The 1.3.x preview release will be up in a few more days, over coming weekend at the latest. I’m stomping more bugs than expected and making some last-minute improvements. This is also my busy time of year (end of financial year here in Australia) so I’ve got some heavy work on from all sides.
In the meantime, keep an eye on the Github page as I’ll be checking in code frequently leading up to first 1.3.x preview build.
I also want to offer a partial retraction regarding my earlier statement about the number of batches being 10x improved over previous version. This turns out to be highly variable based on the number of lights affecting objects, shadows, time of day, etc. The real figure is closer to 5-10x less. It’s still a great improvement, but I don’t want to mislead anyone if possible.
Daggerfall Tools for Unity 1.3 is almost done. This version has a whole swag of improvements for developers to play with. Here’s the summary of new features.
Events have been added to key locations throughout the library. It should be easier than ever to write custom code to extend Daggerfall Tools for Unity without changing the core library.
I have started moving over to prefabs for non-static elements such as enemies, lights, hinged doors, etc. This will also help developers extend these elements more easily.
Overhauled material loading and caching system. The new material system now uses a pair of structures to request a material and receive results, rather than complex methods with a dozen in/out parameters.
Have moved to the Unity 5 Standard shader everywhere possible. A unified shader means simpler processes and better batching. It also means Physically Based Shading and real-time Global Illumination is possible. This could prove a huge boon for texture artists seeking to improve the quality of Daggerfall’s materials.
Normal maps can now be auto-generated for materials.
Emission maps are auto-generated for windows and bright materials such as fireplaces and lights.
Faster loading and processing of all materials.
Now using geometry shader with animation support for all surface billboards. Dungeons and interiors are still using standalone billboards for now.
The number of render batches and state changes required to draw a scene is now around 10x less. Therefore scenes which required approx. 5000 batches in 1.2 now require less than 500 in 1.3.
Events allow you to setup your own image filtering when textures are loaded.
Softer coupling to Arena2 folder, allowing developers to release static scenes without contents of Arena2. An example would be a standalone dungeon game. Obviously procedural elements will require Arena2 folder.
Arena2 folder can now be stored in _Data folder for standalone Windows, Linux, and Mac builds. WebPlayer builds must continue using Resources method.
Added very early support for WebGL builds for static scenes only. Supporting procedural scenes in WebGL is in the pipeline for a future release.
Fully compatible with Unity 5.1.
There are hundreds of small bug fixes and usability improvements. A huge thanks to all the brains at forums.dfworkshop.net for their help and advice.
I am in the process now of squashing bugs and testing. I should have an updated Developer Preview ready over the weekend some time (or mid next week at the latest). My plan is to hold the release version in Developer Preview status for about a month before promoting to full release. There are a few reasons for this (apart from the fact I really like lists (and parentheses)).
I need a full documentation and tutorial refresh. Some of the new features also need entirely new tutorials. This is a time-consuming process.
I would like time to get feedback from developers on the new features so I can make any last-minute bug fixes or changes.
I want to give developers time to finish up any mods/extensions they are working on to work with 1.3. it would be great to release an updated Mod Showcase Demo for 1.3.
That should about cover it for now. If you have any questions, please don’t hesitate to contact me.
Daggerfall Tools for Unity generates very large, complex procedural scenes entirely at runtime. If you haven’t seen it already, check out the mod showcase video to see just how large these environments are. Every texture, billboard, mesh, town, and dungeon are imported and converted procedurally from native DOS binary data at runtime.
While converting our material system over to the Standard shader for Unity5, I thought how great it would be to add normal maps to the procedurally generated scenes. Unfortunately, Daggerfall is such a classic game (polite way of saying very old) it doesn’t come with any normal maps built-in. And the requirement for this to happen at runtime added several challenges along the way. This journal entry details how I go about it.
The basic strategy of generating normal maps is to create a bump map from the colour image based on light and dark areas, then change the bump map into a normal map. It’s a simple idea in theory, but how does it look in practice? Turns out it looks pretty good, even helping the pixel art to “pop” a little.
Identical scene with and without normal maps
In the scene with normals, textures take on a bit more character and bumpiness, helping them to feel more like real surfaces than just textured polygons. While the effect will never be as great as hand-painted normal maps, the generated approach works surprisingly well with Daggerfall’s painterly, somewhat cartoony textures. So how does it work behind the scenes?
The magic all begins at import time with a matrix convolution filter in our ImageProcessing class. The first step is to run source textures through something called a sobel filter to find edges in the image. After a lot of experimentation, I settled on a two-pass (horizontal then vertical) sobel filter as this produced noticeably better results than one-pass filters. The passes are combined together to produce our final bump map.
Bump map shows texture gradients based on colour value
Thanks to the sobel filter, we now have a reasonable understanding of the gradient at each pixel. Brighter pixels have a steeper gradient than dark pixels. Armed with this information, we can derive our normals for any pixel by sampling the gradient of every pixel around it, then calculating the cross product from the gradiants. Here’s the code.
// Look up the heights to either side of this pixel
float left = GetIntensity(ref colors, x - 1, y, width, height);
float right = GetIntensity(ref colors, x + 1, y, width, height);
float top = GetIntensity(ref colors, x, y - 1, width, height);
float bottom = GetIntensity(ref colors, x, y + 1, width, height);
// Compute gradient vectors, then cross them to get the normal
Vector3 dx = new Vector3(1, 0, (right - left) * strength);
Vector3 dy = new Vector3(0, 1, (bottom - top) * strength);
Vector3 normal = Vector3.Cross(dx, dy);
We also need to write the colours back into the array. The normal is also inverted at this time. Thank you to Huknar on Reddit for the tip.
// This is a standard normal texture without Unity packing
newColors[y * width + x] = new Color32(
(byte)((normal.x + 1.0f) * 127.5f),
(byte)(255 - ((normal.y + 1.0f) * 127.5f)),
(byte)((normal.z + 1.0f) * 127.5f),
And here is the resulting normal map shown as colours. Each pixel now communicates a little bit of 3D information for Unity’s graphics engine work with.
Normal map shown as 2D colours
Now there’s one more wrinkle we need to deal with. If you’re astute, you would have noticed something about “Unity packing” in that bit of code above. When importing textures in to Unity via the editor, you need to check a box so the engine knows to treat it as a normal map. Unfortunately for us, we’re importing textures procedurally and building materials on the fly – there’s no check box here. So how does Unity know it should treat our texture as a normal map?
Besides just sticking the texture into the _BumpMap parameter of the shader, we also need to repack the texture in the same way Unity does when you tick that box to import a texture as a normal map. Internally, Unity actually repacks normal maps from X,Y,Z,1 (or R,G,B,1) to Y,Y,Y,X (or G,G,G,R). We need to do the same thing for our normal map to be understood. This code replaces the last block above.
// Store result packed for Unity
byte r = (byte)((normal.x + 1.0f) * 127.5f);
byte g = (byte)(255 - ((normal.y + 1.0f) * 127.5f));
newColors[y * width + x] = new Color32(g, g, g, r);
The end result looks like below. It’s still a normal map, just one pre-packed for Unity’s shaders. You don’t normally see this in action as it all happens behind the scenes.
Normal map packed for Unity’s shaders
The final step is to create the material and assign the right texture maps to parameters in the shader. You also need to enable the keyword _NORMALMAP or Unity does not process the normals.
Procedural material with normal map generated at runtime
With everything put together, we get our scene with extra-bumpy textures thanks to the underlying normal information.
Split shaded and normal views from editor
Putting all of this together was a fun process. I learned a lot about image processing and about how Unity works under the hood. Best of all, you can now add normal maps to your procedural scenes in Daggerfall Tools for Unity by simply ticking a box. Easy.
Just a quick update for you. I am returning to work on Thursday. Once I catch up a little, I’ll get stuck back into the DFTFU 1.3 release. I’m also working on a new technical post for the Workshop Blog that should be up before the weekend. The 1.3 release is shaping up nicely, with lots of great new features for developers to play with. I can’t wait to have it ready for you guys.
Something I mentioned on Twitter a few weeks back is that I would like to talk more about my game projects outside of anything related to Daggerfall. Sometime in the next month or two, I’m going to launch a new site to share my progress and learning experiences creating my own game. I have always enjoyed watching people go through the journey of game development, even way back in the Zzap!64 days when the likes of Martin Walker detailed their process month by month. In turn, I enjoy sharing my experiences with everyone and I value the feedback I receive. I’ve never been fortunate enough to work in the game industry proper, but it has always been a passion for me that I don’t think I could ever abandon.
This may or may not be something you are interested in, so I will keep it separate from the Workshop. I also hope you will understand this will take some time away from my Daggerfall hobby, but hopefully not to its detriment. I promise not to abandon Daggerfall Tools for Unity, but to be an active part of the community and making contributions long into its maturity. Exciting progress is being made by everyone, and I hope to be as much a part of that as possible well into the foreseeable future.
With all the great contributions lately, I can see Daggerfall Tools for Unity becoming self-sustaining. There will come a time when the community here will take things much further than I ever could by myself – in fact I believe that process has already started. Daggerfall has been on my mind for almost 18 years (wow!) and I need to start looking for my next passion. I want to start something original and use everything I have learned to create something of my own, something I can be proud of no matter how feeble it is. If any of you are interested in this new journey, I look forward to seeing you there soon.
I have decided to fast-track 1.3 into release status, as currently Unity 5 compatibility is in Developer Preview 1.3 only. A lot of new Unity 5 developers are downloading the latest 1.2 release version and experiencing problems, which leaves a bad first-impression of the tools.
As a result, the promised text and translation features will be only partially completed in next release. I’m going to bump text and translation to 1.4, and bring some of the material changes planned for 1.4 down into 1.3 which relate to Unity 5 improvements.
There will also be a refresh on the documentation and tutorials for Unity 5, now that features like Deferred Shading are available to all.
Look for the release to be ready sometime within the next week.
I’ve said it many times: Daggerfall Tools for Unity is a platform. Besides helping to recreate the game, DFTFU can be used to remaster and re-imagine Daggerfall, or create whole new games in a similar style. Merging Daggerfall’s content with a powerful engine like Unity provides limitless potential.
Nothing shows off this potential more than mods created over the last several weeks by Nystul, LypyL, and Uncanny_Valley. These range from graphical enhancements and world improvements to silly fun like turning Daggerfall into a flight simulator. I’m so very proud to show off their hard work in the below video and to provide a playable demo for you to experience their work first-hand.