DFConnect Library 0.4.5

DFConnect Library 0.4.5 is now available for Download. Please check the Issues page for current known issues, and the System Requirements page for help getting started.

The key feature of this build is RMB and RDB blocks are almost fully supported by the library. This release includes an RMB Block Viewer Demo (binary and source code included with library download). The control scheme of this tool follows the RDB viewer:

RMB (Outdoor) Block Viewer

  • Click and hold the left mouse button while moving the mouse to look around in a first-person manner.
  • Click and hold the right mouse button while moving the mouse forward and backwards to move along the camera’s facing vector.
  • Click and hold the middle mouse button while moving the mouse to go up, down, left, and right relative to the camera’s facing vector.
rmb viewer 1 rmb viewer 2
rmb viewer 5 rmb viewer 3
rmb viewer 7 rmb viewer 4

The upcoming 0.5.0 build will introduce Maps.bsa support, along with climate processing (i.e. using correct textures for swamp, desert, etc.). This build will also include a simple map viewer demo similar to early screenshots of Daggerfall Scout.

The 0.5.5 build will be a bug-fixing and polishing build. If you have any problems with DFConnect not mentioned on the Issues or Troubleshooting pages, please Contact me so I can address them.

Starting with 0.6.0, I will be adding more usability features to the root DFConnect namespace. While the individual classes in the DFConnect.Arena2 namespace greatly simplify access to Daggerfall’s files, the addition of custom readers will make this easier.

I have started work on standalone applications that demonstrate advanced use of DFConnect. The plan is to release these separately to the DFConnect Library as new exploring tools. Details of these tools will be released shortly after the 0.6.0 milestone.

Improving UV Generation

I recently engaged in a fruitful conversation on the DaggerXL forums with another DF hacker called DigitalMonk, whose specialty is Daggerfall’s textures. His interest was piqued when I made an informal comment regarding UV generation. After bouncing a couple of posts off each either, DigitalMonk’s thoughts and observations resulted in improvement to my current method of UV generation. He also made a discovery that will result in yet another improvement down the track. If you want to track the whole conversation, read back from here. I will summarise the outcome below so you don’t have to re-read the whole thread unless you want to.

Anyone who used Daggerfall Explorer would have noticed some 3D objects have skewed textures. At the core of this problem is how Daggerfall stores UV coordinates (UV mapping is a way of describing how textures are painted on a mesh). Rather than storing a value from 0.0 to 1.0 for every point, Daggerfall only describes UV coordinates for the first three points of any face (and optionally on the fourth point, but research has shown DF does not use this value). This probably doesn’t sound so bad, triangles only have three points right? Unfortunately Daggerfall doesn’t use triangles to describe 3D objects, it uses ngons that can exceed a dozen points per face. With only the first three UV coordinates to work from, the problem is how do we calculate UVs for the remaining points? Check out the main article on the UESP for more information on the problem and the solution we’ve been using up until now. You get some feeling from this short article on how much special handling is needed to cover all possible faces.

The problem with this solution is the constructed matrix can generate UVs that look a little weird, even when the determinant for that linear equation believes the face should be solvable. There are also multiple linear equations based on whether the face is coplanar to XZ, XY, YZ, etc. When UV generation failed, branching logic was needed to post-fix the UVs for certain faces, or skip the matrix generator for another technique, or just slap the correct UVs on directly. All of this made UV generation a bloated exercise that was tricky to debug. Fixing one problem would often break something else. All in all, there are three classes of texture problems:

  • Type 1. These are faces for which UVs cannot be generated using the existing linear equations, or are generated incorrectly.
  • Type 2. These are faces with bad source UVs (i.e. the coordinates are incorrect in Daggerfall’s files). Check out the eastern side of Wayrest Castle in-game for an example of this problem. This class of problem is visible inside the game.
  • Type 3. I used to think these were Type 2 problems, but DigitalMonk discovered these source UVs actually have extra bits packed into the coordinate. By removing these bits, the UV coordinate works as expected. This class of problem is not visible in the game, meaning Daggerfall uses these bits for some purpose, or at the very least just strips them out.

After my conversation with DigitalMonk, I’ve made some improvements to my UV generation that removes all Type1 problems with no special handling required. I managed this after realising UV generation is really just a 2D problem in 3D space. All of those extra linear equations were only needed due to the extra degree of freedom. By moving all the points into 2D first, I would only need the single XY linear equation to solve for all possible faces.

This worked perfectly, and allowed me to remove all my special handling. Below are a couple of screenshots from the test block viewer. The screenshots on the left demonstrate some problem faces with all special handling disabled. You don’t see these problems in the RDB Block Viewer demo included with DFConnect because of that special handling. On the right is the same scene using the new UV generator, with absolutely no special handling. All UV coordinates “just work”.

uv fixing 1 uv fixing 2
uv fixing 3 uv fixing 4

The next step will be to strip those extra bits for Type3 problems, and eventually write a simple UV patching function for the worst Type2 problems. Once this is completed, UV generation in DFConnect will be almost perfect.

I can’t thank DigitalMonk enough for taking the time to clarify the various UV problems and help me reach a better solution. Cheers mate.

DFConnect Library 0.4.0

The 0.4.0 library release is now ready for download. The Tutorials pages have been updated due to a few namespace changes. I have also added a System Requirements page.

The demos are very basic examples of how to render content out of Daggerfall’s files. Demos will be added for all major content types as the features are added to DFConnect.

I need to add pages specific to each demo. Until then, the controls for the two initial demos are:

Mesh Viewer

  • Click and hold the left mouse button while moving the mouse left and right to rotate the mesh.
  • Click and hold the right mouse button while moving the mouse forward and backwards to zoom in and out.

RDB (Dungeon) Block Viewer

  • Click and hold the left mouse button while moving the mouse to look around in a first-person manner.
  • Click and hold the right mouse button while moving the mouse forward and backwards to move along the camera’s facing vector.
  • Click and hold the middle mouse button while moving the mouse to go up, down, left, and right relative to the camera’s facing vector.

The FAQ and Issues pages have been posted. A Troubleshooting page has been added for common problems relating to things like build settings and dependencies.

The 0.4.5 release should be ready this weekend. This build will support RMB (Outdoor) blocks. From that point, I will start adding map and world reading for the 0.5.0 build.

Workshop Updates

Lately, I have been spending most of my development time on the DFConnect Library. Besides making Daggerfall’s file formats more accessible, I now have the added benefit of a framework for rapidly discovering new things. I hadn’t realised just how cumbersome my old code had become until I started writing tutorials and demos for DFConnect. I can do things with DFConnect in a few lines of clear code that previously required a lot of jumping through hoops. There’s a lot to be said for a clean rewrite and rethink. Time spent on DFConnect is time saved in the future. Best of all, it’s something the entire community can take advantage of.

I’ll give you an overview of what I’ve been up to.

DFConnect 0.4.0

The next build of DFConnect is almost finished, and will be up in the next few days (this weekend at the latest). This build supports reading from BLOCKS.BSA, with a strong focus on dungeon (RDB) blocks. Outdoor (RMB) blocks are implemented, but still need some testing. So dungeon blocks will be the focus of this release, and outdoor blocks the release after.

DFConnect Demos

To complement the tutorials, I have started work on a series of demos for getting the most of DFConnect. These demos will cover things like proper 3D visualisation of models, blocks, and maps. Future demos will cover action records in dungeons (moving blocks, levers, etc.) and working with world data.

These demos will be available as pre-built explorers for everyone to play with. The source code for these demos will be bundled with the DFConnect Library starting from 0.4.0.

WorkshopDemoEngine

This is a self-contained 3D rendering engine – basically a simple wrapper for Ogre3D 1.6.3. All DFConnect demos will use this engine for 3D visualisation.

The WorkshopDemoEngine does not expose the full functionality of Ogre, just enough to do the job at hand. The goal is to provide a series of simple and direct examples of how to work with DFConnect data without cluttering these examples with engine-specific boilerplate. The implementation should be generic enough for you to learn the important concepts and apply them to your own engine of choice.

The second purpose of the WorkshopDemoEngine is a simplistic foundation for everyone. If you are just getting started with the DFConnect examples, or C# itself, the engine will hopefully be an easy starting point for you to get cool results without struggling with a 3D library on top of everything else.

Dungeon Block Viewer

Below are a couple of screenshots of a test dungeon block viewer. This uses DFConnect and the WorkshopDemoEngine. A cleaned up version of this will be released with DFConnect 0.4.0, along with source code.

Also, a big thanks to Lucius for his tips on cleaning up the edges of my billboards. The 2D flats have lost their black edges, and now blend slightly into the environment instead (not really visible in jpeg below). Enabling anisotropic filtering on supported hardware has cleaned up texture noise on long stretches adjacent to the view direction (see bottom-right screen shot).

blockviewer1 blockviewer2
blockviewer3 blockviewer5

Future of Daggerfall Scout

I’ve been thinking a lot about Daggerfall Scout, and where it belongs in regards to how I spend my time on this hobby. A few things have made me rethink the purpose of this tool:

  1. The best part of Daggerfall Scout has always been exploring cities and dungeons. This will very shortly be possible using the DFConnect demos. While these demos won’t have all the features planned for DF Scout, they will do much the same thing – with source code and faster to boot.
  2. Working with Daggerfall has never been easier for me than it is now with the DFConnect Library. As Daggerfall Scout is based out of a much older code base, it carries a lot of baggage along with it. Every time I look at the DF Scout code these days, I spend all my time wishing I’d written DFConnect first to make life easier on myself.
  3. Last but not least is DaggerXL. What Lucius is doing here is incredible! Much of the “wow” factor of DF Scout has been far exceeded by DaggerXL. If I had to choose between exploring the world in DF Scout, or in DaggerXL, I’d pick DaggerXL every time. If that’s how I feel about my own program, then others probably feel much the same.

When I started on DF Scout earlier in the year, the Daggerfall scene was stagnant. Since then, Daggerfall has been released for free and Lucius has come along to bring our remake dreams back to life. With the community revitalised, DF Scout is just looking really tired. To be brutally honest, I think I’m wasting my time with it.

Moving forwards, everything good from DF Scout will be cannibalised for DFConnect. So rather than releasing one big program at some point in the future, I get to release lots of small purposeful programs almost right away. For people who just want to explore the game files, these programs will be available for download. For people who want to hack away at the Daggerfall files themselves, the source code for these programs will help you get started.

I feel this adds the most community benefit, and is more in line with the spirit of Daggerfall Workshop. Thinking back to the glory days of this site, I can only imagine how much more could have been accomplished if I’d started work on a Daggerfall library then, rather than sinking that time into other projects that ultimately went nowhere.

Community Forums

To start rebuilding our development community, I am reopening the Workshop forums in the next few weeks. This will be a place where Daggerfall hackers can get together and talk about their projects. If you are using the DFConnect Library, this will be the place to share your ideas and get support from the community. Promising projects will be given their own subforum, just like in the past.

In the meantime, I am opening up comments in the blog for new posts starting with this one. I look forward to hearing all your thoughts.