DFTFU Developer Preview 1.3.24

Edit: Updated to build 1.3.24

Daggerfall Tools for Unity 1.3.24 is now available to developers for testing. This is a close-to-final version that still has a few kinks to iron out. However it should be close enough for developers to start testing and migrating their work.

 

The key new features of this version are:

  • Full Unity 5 compatibility.
  • New material and texture system.
  • Using Standard shader everywhere possible.
  • Simplified custom shaders for tile maps and billboard batches.
  • Significantly improved batching and performance when using new Deferred path.
  • Improved billboard batching using geometry shader, now supporting animation.
  • Events at key locations.
  • Started using prefabs for layout of non-static items such as doors, lights, and enemies.
  • Many small bug fixes and improvements.

I did mention there were still a few kinks to iron out, so I value any feedback on this version prior to full release.

I am also working on a full documentation refresh and new tutorials for some of the new features (like events and prefabs). This will take a few weeks to complete. I am planning full online API documentation starting from release 1.3.

Happy testing! If you have any feedback, please use the following forum thread.

http://forums.dfworkshop.net/viewtopic.php?f=7&t=74

Enemy Prefab

The below short video demonstrates using the new enemy prefab inside editor. The same prefab is instantiated when building scenes procedurally. There’s a lot more I want to do with this, including the ability to override classic billboard-based enemies with full 3D models.

Physically Based Materials

The new material system is finally humming away. Apart from improved batching and faster scenes, one of the best features has to be support for Physically Based Shading in Unity 5. Obviously Daggerfall can’t take full advantage just yet, those tiny textures are never going to cut it. But with the Standard shader now… well, standard, those of you with an artistic mind can start authoring high quality materials to take advantage of everything Unity has on offer. Combine physically based materials with some higher quality models and there might come a time when Daggerfall could be genuinely breathtaking.

In the meantime, here’s an Ultra HD screenshot taken from on top of Castle Daggerfall. There’s a little SSAO, bloom, and some global fog thrown in for good measure. Click for full size.

Almost There

The 1.3.x preview release will be up in a few more days, over coming weekend at the latest. I’m stomping more bugs than expected and making some last-minute improvements. This is also my busy time of year (end of financial year here in Australia) so I’ve got some heavy work on from all sides.

In the meantime, keep an eye on the Github page as I’ll be checking in code frequently leading up to first 1.3.x preview build.

I also want to offer a partial retraction regarding my earlier statement about the number of batches being 10x improved over previous version. This turns out to be highly variable based on the number of lights affecting objects, shadows, time of day, etc. The real figure is closer to 5-10x less. It’s still a great improvement, but I don’t want to mislead anyone if possible.

1.3 Incoming

Daggerfall Tools for Unity 1.3 is almost done. This version has a whole swag of improvements for developers to play with. Here’s the summary of new features.

Extending

  • Events have been added to key locations throughout the library. It should be easier than ever to write custom code to extend Daggerfall Tools for Unity without changing the core library.
  • I have started moving over to prefabs for non-static elements such as enemies, lights, hinged doors, etc. This will also help developers extend these elements more easily.

Materials

  • Overhauled material loading and caching system. The new material system now uses a pair of structures to request a material and receive results, rather than complex methods with a dozen in/out parameters.
  • Have moved to the Unity 5 Standard shader everywhere possible. A unified shader means simpler processes and better batching. It also means Physically Based Shading and real-time Global Illumination is possible. This could prove a huge boon for texture artists seeking to improve the quality of Daggerfall’s materials.
  • Normal maps can now be auto-generated for materials.
  • Emission maps are auto-generated for windows and bright materials such as fireplaces and lights.
  • Faster loading and processing of all materials.

Billboard System

  • Now using geometry shader with animation support for all surface billboards. Dungeons and interiors are still using standalone billboards for now.

Batching

  • The number of render batches and state changes required to draw a scene is now around 10x less. Therefore scenes which required approx. 5000 batches in 1.2 now require less than 500 in 1.3.

Image Processing

  • The texture loader can perform image processing using a matrix convolution filter.
  • Events allow you to setup your own image filtering when textures are loaded.

Builds

  • Softer coupling to Arena2 folder, allowing developers to release static scenes without contents of Arena2. An example would be a standalone dungeon game. Obviously procedural elements will require Arena2 folder.
  • Arena2 folder can now be stored in _Data folder for standalone Windows, Linux, and Mac builds. WebPlayer builds must continue using Resources method.
  • Added very early support for WebGL builds for static scenes only. Supporting procedural scenes in WebGL is in the pipeline for a future release.

Fixes

  • Fully compatible with Unity 5.1.
  • There are hundreds of small bug fixes and usability improvements. A huge thanks to all the brains at forums.dfworkshop.net for their help and advice.

I am in the process now of squashing bugs and testing. I should have an updated Developer Preview ready over the weekend some time (or mid next week at the latest). My plan is to hold the release version in Developer Preview status for about a month before promoting to full release. There are a few reasons for this (apart from the fact I really like lists (and parentheses)).

  • I need a full documentation and tutorial refresh. Some of the new features also need entirely new tutorials. This is a time-consuming process.
  • I would like time to get feedback from developers on the new features so I can make any last-minute bug fixes or changes.
  • I want to give developers time to finish up any mods/extensions they are working on to work with 1.3. it would be great to release an updated Mod Showcase Demo for 1.3.

That should about cover it for now. If you have any questions, please don’t hesitate to contact me.

Generating Normals

Daggerfall Tools for Unity generates very large, complex procedural scenes entirely at runtime. If you haven’t seen it already, check out the mod showcase video to see just how large these environments are. Every texture, billboard, mesh, town, and dungeon are imported and converted procedurally from native DOS binary data at runtime.

While converting our material system over to the Standard shader for Unity5, I thought how great it would be to add normal maps to the procedurally generated scenes. Unfortunately, Daggerfall is such a classic game (polite way of saying very old) it doesn’t come with any normal maps built-in. And the requirement for this to happen at runtime added several challenges along the way. This journal entry details how I go about it.

The basic strategy of generating normal maps is to create a bump map from the colour image based on light and dark areas, then change the bump map into a normal map. It’s a simple idea in theory, but how does it look in practice? Turns out it looks pretty good, even helping the pixel art to “pop” a little.

Identical scene with and without normal maps

In the scene with normals, textures take on a bit more character and bumpiness, helping them to feel more like real surfaces than just textured polygons. While the effect will never be as great as hand-painted normal maps, the generated approach works surprisingly well with Daggerfall’s painterly, somewhat cartoony textures. So how does it work behind the scenes?

The magic all begins at import time with a matrix convolution filter in our ImageProcessing class. The first step is to run source textures through something called a sobel filter to find edges in the image. After a lot of experimentation, I settled on a two-pass (horizontal then vertical) sobel filter as this produced noticeably better results than one-pass filters. The passes are combined together to produce our final bump map.

BumpMap

Bump map shows texture gradients based on colour value

Thanks to the sobel filter, we now have a reasonable understanding of the gradient at each pixel. Brighter pixels have a steeper gradient than dark pixels. Armed with this information, we can derive our normals for any pixel by sampling the gradient of every pixel around it, then calculating the cross product from the gradiants. Here’s the code.

// Look up the heights to either side of this pixel
float left = GetIntensity(ref colors, x - 1, y, width, height);
float right = GetIntensity(ref colors, x + 1, y, width, height);
float top = GetIntensity(ref colors, x, y - 1, width, height);
float bottom = GetIntensity(ref colors, x, y + 1, width, height);

// Compute gradient vectors, then cross them to get the normal
Vector3 dx = new Vector3(1, 0, (right - left) * strength);
Vector3 dy = new Vector3(0, 1, (bottom - top) * strength);
Vector3 normal = Vector3.Cross(dx, dy);
normal.Normalize();

We also need to write the colours back into the array. The normal is also inverted at this time. Thank you to Huknar on Reddit for the tip.

// This is a standard normal texture without Unity packing
newColors[y * width + x] = new Color32(
    (byte)((normal.x + 1.0f) * 127.5f),
    (byte)(255 - ((normal.y + 1.0f) * 127.5f)),
    (byte)((normal.z + 1.0f) * 127.5f),
    0xff);

And here is the resulting normal map shown as colours. Each pixel now communicates a little bit of 3D information for Unity’s graphics engine work with.

Normal map shown as 2D colours

Now there’s one more wrinkle we need to deal with. If you’re astute, you would have noticed something about “Unity packing” in that bit of code above. When importing textures in to Unity via the editor, you need to check a box so the engine knows to treat it as a normal map. Unfortunately for us, we’re importing textures procedurally and building materials on the fly – there’s no check box here. So how does Unity know it should treat our texture as a normal map?

Besides just sticking the texture into the _BumpMap parameter of the shader, we also need to repack the texture in the same way Unity does when you tick that box to import a texture as a normal map. Internally, Unity actually repacks normal maps from X,Y,Z,1 (or R,G,B,1) to Y,Y,Y,X (or G,G,G,R). We need to do the same thing for our normal map to be understood. This code replaces the last block above.

// Store result packed for Unity
byte r = (byte)((normal.x + 1.0f) * 127.5f);
byte g = (byte)(255 - ((normal.y + 1.0f) * 127.5f));
newColors[y * width + x] = new Color32(g, g, g, r);

The end result looks like below. It’s still a normal map, just one pre-packed for Unity’s shaders. You don’t normally see this in action as it all happens behind the scenes.

Normal map packed for Unity’s shaders

The final step is to create the material and assign the right texture maps to parameters in the shader. You also need to enable the keyword _NORMALMAP or Unity does not process the normals.

material.SetTexture("_BumpMap", normalMap);
material.EnableKeyword("_NORMALMAP");

GeneratedMaterialProcedural material with normal map generated at runtime

With everything put together, we get our scene with extra-bumpy textures thanks to the underlying normal information.

Split shaded and normal views from editor

Putting all of this together was a fun process. I learned a lot about image processing and about how Unity works under the hood. Best of all, you can now add normal maps to your procedural scenes in Daggerfall Tools for Unity by simply ticking a box. Easy.

MaterialReaderOptions

StrengthComparison

Comparing normal strength of 0.1 vs 1.0

Returning Soon, And More

Just a quick update for you. I am returning to work on Thursday. Once I catch up a little, I’ll get stuck back into the DFTFU 1.3 release. I’m also working on a new technical post for the Workshop Blog that should be up before the weekend. The 1.3 release is shaping up nicely, with lots of great new features for developers to play with. I can’t wait to have it ready for you guys.

Something I mentioned on Twitter a few weeks back is that I would like to talk more about my game projects outside of anything related to Daggerfall. Sometime in the next month or two, I’m going to launch a new site to share my progress and learning experiences creating my own game. I have always enjoyed watching people go through the journey of game development, even way back in the Zzap!64 days when the likes of Martin Walker detailed their process month by month. In turn, I enjoy sharing my experiences with everyone and I value the feedback I receive. I’ve never been fortunate enough to work in the game industry proper, but it has always been a passion for me that I don’t think I could ever abandon.

This may or may not be something you are interested in, so I will keep it separate from the Workshop. I also hope you will understand this will take some time away from my Daggerfall hobby, but hopefully not to its detriment. I promise not to abandon Daggerfall Tools for Unity, but to be an active part of the community and making contributions long into its maturity. Exciting progress is being made by everyone, and I hope to be as much a part of that as possible well into the foreseeable future.

With all the great contributions lately, I can see Daggerfall Tools for Unity becoming self-sustaining. There will come a time when the community here will take things much further than I ever could by myself – in fact I believe that process has already started. Daggerfall has been on my mind for almost 18 years (wow!) and I need to start looking for my next passion. I want to start something original and use everything I have learned to create something of my own, something I can be proud of no matter how feeble it is. If any of you are interested in this new journey, I look forward to seeing you there soon.

Some Personal Time

My father passed away last night. This has been coming for some time so is no surprise, but hits me hard all the same.

I will be taking some personal leave time away from work and my side projects until things settle down.

I will resume work on the 1.3 release as soon as I am able. Until then, please don’t forget you can download the 1.3.8 Developer Preview for Unity 5 compatibility.

Thank you for your patience. I wish you all the best.

Fast-Tracking 1.3 Release

I have decided to fast-track 1.3 into release status, as currently Unity 5 compatibility is in Developer Preview 1.3 only. A lot of new Unity 5 developers are downloading the latest 1.2 release version and experiencing problems, which leaves a bad first-impression of the tools.

As a result, the promised text and translation features will be only partially completed in next release. I’m going to bump text and translation to 1.4, and bring some of the material changes planned for 1.4 down into 1.3 which relate to Unity 5 improvements.

There will also be a refresh on the documentation and tutorials for Unity 5, now that features like Deferred Shading are available to all.

Look for the release to be ready sometime within the next week.

Mod Showcase 1 Demo

I’ve said it many times: Daggerfall Tools for Unity is a platform. Besides helping to recreate the game, DFTFU can be used to remaster and re-imagine Daggerfall, or create whole new games in a similar style. Merging Daggerfall’s content with a powerful engine like Unity provides limitless potential.

Nothing shows off this potential more than mods created over the last several weeks by Nystul, LypyL, and Uncanny_Valley. These range from graphical enhancements and world improvements to silly fun like turning Daggerfall into a flight simulator. I’m so very proud to show off their hard work in the below video and to provide a playable demo for you to experience their work first-hand.

DFTFU Mod Showcase 1 – Playable Demo

What’s Included

  • Daggerfall Tools for Unity 1.3.11 [Interkarma/Gavin Clayton]
  • Increased Terrain Distance 1.4 [Nystul/Michael Rauter]
  • Floating Origin Updates [LypyL]
  • Enhanced Sky 1.6 [LypyL]
  • DaggerJet 1.1a [LypyL]
  • TravelMap2.1 [LypyL]
  • Real Grass 1.01 [Uncanny_Valley]
  • Birds In Daggerfall 1.0 [Uncanny_Valley]

Controls

See below for complete list. There is also a handy key reference created by Daschluba on the forums (click for full size).

Mod Showcase Keys

General

  • Mouse Move Look
  • W, S, A, D Move
  • Shift (hold) Run
  • Space Jump
  • Left Mouse Activate: open doors, enter dungeons, operate switches, etc.
  • Esc Exit game (standalone)

Weapons

  • Z Toggle weapons
  • Right Mouse Drag Right-click and drag mouse to swing weapon

Transport

  • R (outside) Random location, season, time of day, and weather
  • R (dungeon) Recall back to dungeon start
  • 1-7 Preset location (keeps current season, weather, etc.)

Music

  • P Toggle songs on/off
  • [ Play previous song in context playlist
  • ] Play next song in context playlist

Timescale

  • PageUp Increase timescale.
  • PageDown Decrease timescale.

Other

  • I Toggle mouse invert on/off
  • H Toggle high speed running on/off
  • L Toggle debug text on/off
  • O Toggle brighter torch in dungeons and at night
  • V Fast travel menu (see below for instructions)

Sky Control

  • T Toggle Enhanced/Native Sky

Jet Control

  • F6 Toggle laser (can be used without jet)
  • F7 Teleport to where laser is pointed, and exit jet if flying
  • F9 Summon and enter jet
  • F10 Change jet camera position
  • F11 Toggle free mouse look
  • F12 Spawn flying enemy while in jet
  • W, S, A, D Steer jet
  • Mouse0 Fire weapon at laser target
  • Space Air brakes – slow down jet while flying

Using Fast Travel Menu (LypyL)

  1. Hit V key to open/close travel menu.
  2. You can either set your coordinates directly using the sliders, or select a Region & Location using the Location menu and hit the Set Coordinates button.
  3. Use the toggles below the location menu button to choose what type of location to show in the menu (Cities, Dungeons, Taverns, Graveyard etc.)
  4. Once You’re ready to travel, just hit the GO button.

Fast Travel Notes (LypyL)

  • Red marker indicates your current location, the white shows where you will travel to.
  • Click and drag on the location & region menus to scroll through them
  • It’s very rare, but it’s possible to fall through the ground after traveling. If it happens (and you don’t have another solution, like the editor), just fast travel again somewhere to fix it.