LypyL Action Enhancements

LypyL has been working on enhancing the range of supported action objects in dungeons. Not only did he fix bugs in my code, LypyL has discovered previously unknown action records for things like teleporting. Here’s a video of him testing out the new code. Great work, man. Great work.

Roadmap July 2015

Roadmap July 2015

With 1.3 freshly released, it’s time for me to update my roadmap again. While my overall heading remains constant, the beats and rhythm of my short-term goals are always in flux, constantly evolving as I converge on solutions and new priorities emerge. I find that right after a release is the perfect time to step back and consider my next move. So let’s take a look at where I’m up to and what’s on the horizon.

Maintaining 1.3

To start with, I’m wrapping up tutorials and fixing any bugs discovered in 1.3 release version. I plan to support the stable 1.3 branch right up to the next major release. Any minor bugs will be fixed quickly, followed by an incremental release. It’s possible some of the key features in 1.3 will be refined, but no major changes will be made at this point. Anyone waiting for 1.3 to settle down can confidently get started with it.

For those of you waiting on tutorials, Prefab Basics should be up in the next day, and Streaming World sometime next week. I’m hoping all the tutorials earmarked for 1.3 will be completed in next 2-3 weeks.

Work In Progress Branch

In case you missed it, there is now a work-in-progress branch on GitHub. New features will be added first to the work-in-progress branch and only promoted to the master branch once they become complete and stable. At which point, I go into maintenance mode for the new release and start work on the next major release. This is a much better workflow and should result in faster turnaround of fixes in maintenance releases. It’s very likely you will see a few more version points added to 1.3 before it’s done.

Developer Focus

Every major release (two betas, 1.0, 1.1, 1.2, 1.3) involved heavy changes to the underlying code. This often broke your work and was very frustrating for you. I’m sorry about that. Hopefully 1.3 marks the last major paradigm shift for some time (jumping to Unity 5 was a big one), and the new event and prefab systems help you create your projects without needing to modify the core tools.

I plan to continue this developer focus moving forward by adding more events, prefabs, interfaces, editor scripts, etc. to help you work, and make fewer breaking changes to the underlying code base. It’s obviously inevitable this will happen sooner or later, but hopefully not to the extent seen between 1.0-1.3.

Feature Focus

To date, I’ve worked very broadly across the entire spectrum of Daggerfall. This has been essential as much work was involved in setting up content readers, building a bridge into Unity, and seeing the entire game world up and running. Those system are huge, difficult, and take a mountain of work to pull off. And they are so interdependent that it’s necessary to build all of them before any of them really work. This is where most Daggerfall projects stall and fail. And guess what! We’ve pushed past that like a boss and still have plenty of steam out the other side. This is where the really interesting stuff kicks in and all that hard work pays off.

Starting with 1.4, I’m going to focus on a very specific set of features each release, plus some iteration on previous features. In this way, we climb the mountain one challenge at a time. The work becomes easier because the next goal is always in sight, then the next, and so on until the top.

Roadmap 1.4

Work on 1.4 has already begun. The only thing I haven’t shared with you are my plans for this release. Here’s a list of everything I have either already started work on, or plan to include. And the good news? From here on, everything is geared to helping you add real gameplay features.

  • Daggerfall save game import. Yep, the ability to read vanilla Daggerfall save games and import this data.
  • Unity save games. The foundations of a Unity save-game system will be implemented for new demo titles. You can use this, ignore it, or extend as needed.
  • Real player scripts. With stats, special advantages, special disadvantages, reputations, etc.
  • Real monster scripts. With all their properties read straight from the game files.
  • Real combat formulas. For real melee combat. Spellcraft will be added later.
  • Item and loot table support. Play with real Daggerfall items and their properties. Custom loot tables for monsters and dungeons.
  • Starting work on effects system.  At first this will only be complete enough to support player and monster scripts, and melee combat. Spellcraft will be added later.
  • Demo character sheet UI. With inventory and paper doll support.

Some of you may be wondering about the text & translation features I had originally planned for 1.3. Unfortunately the Unity 5 release dropped a huge spanner in the works and a great deal of effort was involved bringing everything up to par with the new paradigm. In that time, I have rethought my approach toward text and translation based on feedback from other developers.

Obviously this remains an important consideration, but I no longer believe it prudent to add a locked-in translation system to core tools. The problem with this approach is it limits options for other developers who may wish to use some other translation system, or a different approach entirely.

The original system as planned would also have added some complexity for developers who just wanted to get stuck in for their own amusement, and are not concerned about targeting multiple languages. Obviously larger projects that want to localize have a plethora of options available in the Unity landscape alone. There is simply no need for me to re-invent the wheel here.

So I’m going to keep doing what I do best, which is provide hooks back to the game files and the means for other developers to easily suck out the data they need. This data obviously includes text, and will have everything a developer needs to pair with their preferred translation system. This set of features are now part of the 1.5 update, which includes quest file support.

Conclusion

That’s everything for now. As usual, Daggerfall Tools for Unity is a constantly evolving beast and this is all subject to refinement. However the journey remains the same. I may not be remaking the game personally, but by Julianos I’m going to create all the moving parts needed to help that become a reality, and with your help it’s practically inevitable.

So if you’re a developer who loves Daggerfall, get started today and help show the world what can be accomplished!

Daggerfall Tools for Unity 1.3.31 Released!

Release1331

Daggerfall Tools for Unity 1.3.31 release version now available!

Key features of this version are:

  • Full Unity 5 compatibility.
  • New material and texture system.
  • Using Standard shader everywhere possible.
  • Simplified custom shaders for tile maps and billboard batches.
  • Significantly improved batching and performance when using new Deferred path.
  • Improved billboard batching using geometry shader, now supporting animation.
  • Event system.
  • Started using prefabs for layout of non-static items such as doors, lights, and enemies.
  • Many bug fixes and improvements.
  • New Feature Gallery.
  • New Tutorials.
Download Daggerfall Tools for Unity - 1.3.31

New Tutorials

As part of the 1.3 release, I will be refreshing all existing tutorials and writing some new ones to round everything out.

The manual has now been retired and rolled into a brand new tutorial called Getting Started which covers the extreme basics of setting up a new project. This will be the new starting point for any Basic and Intermediate tutorials. I was covering a lot of the same ground at the beginning of each tutorial, and from now on I can just reference beginners to the Getting Started tutorial instead of repeating steps. Follow-on tutorials will be a bit leaner as a result.

Following is a list of the tutorials coming online for 1.3 release. Links will be added as each tutorial becomes ready.

Basic Series 1.3

Intermediate Series 1.3

  • Streaming World (expanded)
  • Distributing Builds

Advanced Series 1.3

  • Using Events

The above will keep me busy for a few weeks. If you think a tutorial is missing from the lineup just let me know. Depending on complexity, I can add it to the lineup now or some time after 1.3 launches.

Also please let me know of any problems you find with the documents.

DFTFU Developer Preview 1.3.29

Edit: Release version 1.3.31 now available.

Daggerfall Tools for Unity 1.3.29 is now available to developers for testing. This is a close-to-final version that still has a few kinks to iron out. However it should be close enough for developers to start testing and migrating their work.

The key new features of this version are:

  • Full Unity 5 compatibility.
  • New material and texture system.
  • Using Standard shader everywhere possible.
  • Simplified custom shaders for tile maps and billboard batches.
  • Significantly improved batching and performance when using new Deferred path.
  • Improved billboard batching using geometry shader, now supporting animation.
  • Events at key locations.
  • Started using prefabs for layout of non-static items such as doors, lights, and enemies.
  • Many small bug fixes and improvements.

I did mention there were still a few kinks to iron out, so I value any feedback on this version prior to full release.

I am also working on a full documentation refresh and new tutorials for some of the new features (like events and prefabs). This will take a few weeks to complete. I am planning full online API documentation starting from release 1.3.

Happy testing! If you have any feedback, please use the following forum thread.

http://forums.dfworkshop.net/viewtopic.php?f=7&t=74

Enemy Prefab

The below short video demonstrates using the new enemy prefab inside editor. The same prefab is instantiated when building scenes procedurally. There’s a lot more I want to do with this, including the ability to override classic billboard-based enemies with full 3D models.

Prefab Dubstep

Starting with release 1.3 many dynamic scene objects will be instantiated from prefab. This video shows a bit of fun with extending the city lights prefab.

Physically Based Materials

The new material system is finally humming away. Apart from improved batching and faster scenes, one of the best features has to be support for Physically Based Shading in Unity 5. Obviously Daggerfall can’t take full advantage just yet, those tiny textures are never going to cut it. But with the Standard shader now… well, standard, those of you with an artistic mind can start authoring high quality materials to take advantage of everything Unity has on offer. Combine physically based materials with some higher quality models and there might come a time when Daggerfall could be genuinely breathtaking.

In the meantime, here’s an Ultra HD screenshot taken from on top of Castle Daggerfall. There’s a little SSAO, bloom, and some global fog thrown in for good measure. Click for full size.

Almost There

The 1.3.x preview release will be up in a few more days, over coming weekend at the latest. I’m stomping more bugs than expected and making some last-minute improvements. This is also my busy time of year (end of financial year here in Australia) so I’ve got some heavy work on from all sides.

In the meantime, keep an eye on the Github page as I’ll be checking in code frequently leading up to first 1.3.x preview build.

I also want to offer a partial retraction regarding my earlier statement about the number of batches being 10x improved over previous version. This turns out to be highly variable based on the number of lights affecting objects, shadows, time of day, etc. The real figure is closer to 5-10x less. It’s still a great improvement, but I don’t want to mislead anyone if possible.

1.3 Incoming

Daggerfall Tools for Unity 1.3 is almost done. This version has a whole swag of improvements for developers to play with. Here’s the summary of new features.

Extending

  • Events have been added to key locations throughout the library. It should be easier than ever to write custom code to extend Daggerfall Tools for Unity without changing the core library.
  • I have started moving over to prefabs for non-static elements such as enemies, lights, hinged doors, etc. This will also help developers extend these elements more easily.

Materials

  • Overhauled material loading and caching system. The new material system now uses a pair of structures to request a material and receive results, rather than complex methods with a dozen in/out parameters.
  • Have moved to the Unity 5 Standard shader everywhere possible. A unified shader means simpler processes and better batching. It also means Physically Based Shading and real-time Global Illumination is possible. This could prove a huge boon for texture artists seeking to improve the quality of Daggerfall’s materials.
  • Normal maps can now be auto-generated for materials.
  • Emission maps are auto-generated for windows and bright materials such as fireplaces and lights.
  • Faster loading and processing of all materials.

Billboard System

  • Now using geometry shader with animation support for all surface billboards. Dungeons and interiors are still using standalone billboards for now.

Batching

  • The number of render batches and state changes required to draw a scene is now around 10x less. Therefore scenes which required approx. 5000 batches in 1.2 now require less than 500 in 1.3.

Image Processing

  • The texture loader can perform image processing using a matrix convolution filter.
  • Events allow you to setup your own image filtering when textures are loaded.

Builds

  • Softer coupling to Arena2 folder, allowing developers to release static scenes without contents of Arena2. An example would be a standalone dungeon game. Obviously procedural elements will require Arena2 folder.
  • Arena2 folder can now be stored in _Data folder for standalone Windows, Linux, and Mac builds. WebPlayer builds must continue using Resources method.
  • Added very early support for WebGL builds for static scenes only. Supporting procedural scenes in WebGL is in the pipeline for a future release.

Fixes

  • Fully compatible with Unity 5.1.
  • There are hundreds of small bug fixes and usability improvements. A huge thanks to all the brains at forums.dfworkshop.net for their help and advice.

I am in the process now of squashing bugs and testing. I should have an updated Developer Preview ready over the weekend some time (or mid next week at the latest). My plan is to hold the release version in Developer Preview status for about a month before promoting to full release. There are a few reasons for this (apart from the fact I really like lists (and parentheses)).

  • I need a full documentation and tutorial refresh. Some of the new features also need entirely new tutorials. This is a time-consuming process.
  • I would like time to get feedback from developers on the new features so I can make any last-minute bug fixes or changes.
  • I want to give developers time to finish up any mods/extensions they are working on to work with 1.3. it would be great to release an updated Mod Showcase Demo for 1.3.

That should about cover it for now. If you have any questions, please don’t hesitate to contact me.

Generating Normals

Daggerfall Tools for Unity generates very large, complex procedural scenes entirely at runtime. If you haven’t seen it already, check out the mod showcase video to see just how large these environments are. Every texture, billboard, mesh, town, and dungeon are imported and converted procedurally from native DOS binary data at runtime.

While converting our material system over to the Standard shader for Unity5, I thought how great it would be to add normal maps to the procedurally generated scenes. Unfortunately, Daggerfall is such a classic game (polite way of saying very old) it doesn’t come with any normal maps built-in. And the requirement for this to happen at runtime added several challenges along the way. This journal entry details how I go about it.

The basic strategy of generating normal maps is to create a bump map from the colour image based on light and dark areas, then change the bump map into a normal map. It’s a simple idea in theory, but how does it look in practice? Turns out it looks pretty good, even helping the pixel art to “pop” a little.

Identical scene with and without normal maps

In the scene with normals, textures take on a bit more character and bumpiness, helping them to feel more like real surfaces than just textured polygons. While the effect will never be as great as hand-painted normal maps, the generated approach works surprisingly well with Daggerfall’s painterly, somewhat cartoony textures. So how does it work behind the scenes?

The magic all begins at import time with a matrix convolution filter in our ImageProcessing class. The first step is to run source textures through something called a sobel filter to find edges in the image. After a lot of experimentation, I settled on a two-pass (horizontal then vertical) sobel filter as this produced noticeably better results than one-pass filters. The passes are combined together to produce our final bump map.

BumpMap

Bump map shows texture gradients based on colour value

Thanks to the sobel filter, we now have a reasonable understanding of the gradient at each pixel. Brighter pixels have a steeper gradient than dark pixels. Armed with this information, we can derive our normals for any pixel by sampling the gradient of every pixel around it, then calculating the cross product from the gradiants. Here’s the code.

// Look up the heights to either side of this pixel
float left = GetIntensity(ref colors, x - 1, y, width, height);
float right = GetIntensity(ref colors, x + 1, y, width, height);
float top = GetIntensity(ref colors, x, y - 1, width, height);
float bottom = GetIntensity(ref colors, x, y + 1, width, height);

// Compute gradient vectors, then cross them to get the normal
Vector3 dx = new Vector3(1, 0, (right - left) * strength);
Vector3 dy = new Vector3(0, 1, (bottom - top) * strength);
Vector3 normal = Vector3.Cross(dx, dy);
normal.Normalize();

We also need to write the colours back into the array. The normal is also inverted at this time. Thank you to Huknar on Reddit for the tip.

// This is a standard normal texture without Unity packing
newColors[y * width + x] = new Color32(
    (byte)((normal.x + 1.0f) * 127.5f),
    (byte)(255 - ((normal.y + 1.0f) * 127.5f)),
    (byte)((normal.z + 1.0f) * 127.5f),
    0xff);

And here is the resulting normal map shown as colours. Each pixel now communicates a little bit of 3D information for Unity’s graphics engine work with.

Normal map shown as 2D colours

Now there’s one more wrinkle we need to deal with. If you’re astute, you would have noticed something about “Unity packing” in that bit of code above. When importing textures in to Unity via the editor, you need to check a box so the engine knows to treat it as a normal map. Unfortunately for us, we’re importing textures procedurally and building materials on the fly – there’s no check box here. So how does Unity know it should treat our texture as a normal map?

Besides just sticking the texture into the _BumpMap parameter of the shader, we also need to repack the texture in the same way Unity does when you tick that box to import a texture as a normal map. Internally, Unity actually repacks normal maps from X,Y,Z,1 (or R,G,B,1) to Y,Y,Y,X (or G,G,G,R). We need to do the same thing for our normal map to be understood. This code replaces the last block above.

// Store result packed for Unity
byte r = (byte)((normal.x + 1.0f) * 127.5f);
byte g = (byte)(255 - ((normal.y + 1.0f) * 127.5f));
newColors[y * width + x] = new Color32(g, g, g, r);

The end result looks like below. It’s still a normal map, just one pre-packed for Unity’s shaders. You don’t normally see this in action as it all happens behind the scenes.

Normal map packed for Unity’s shaders

The final step is to create the material and assign the right texture maps to parameters in the shader. You also need to enable the keyword _NORMALMAP or Unity does not process the normals.

material.SetTexture("_BumpMap", normalMap);
material.EnableKeyword("_NORMALMAP");

GeneratedMaterialProcedural material with normal map generated at runtime

With everything put together, we get our scene with extra-bumpy textures thanks to the underlying normal information.

Split shaded and normal views from editor

Putting all of this together was a fun process. I learned a lot about image processing and about how Unity works under the hood. Best of all, you can now add normal maps to your procedural scenes in Daggerfall Tools for Unity by simply ticking a box. Easy.

MaterialReaderOptions

StrengthComparison

Comparing normal strength of 0.1 vs 1.0