Building From Source and Other Issues

I’ve received a few emails from programmers trying to build Daggerfall Modelling from source and experiencing problems. The reason for their frustrations is that Daggerfall Modelling is built as a VS2008/XNA3.1 project. This means it will only build with Visual Studio (Express) 2008 with XNA Game Studio 3.1 installed. Attempting to build with VS2010/XNA4.0 will fail due to breaking changes in the latest version of XNA.

I also get the occasional email from Windows XP users with an older version of the .NET Framework 2.0 (pre-SP2) that experience a crash when opening a dungeon. This problem is caused by a bug in the .NET Framework 2.0 that was resolved in Service Pack 2, and can be fixed by upgrading/patching .NET.

The reason I stuck to older versions was to ensure my tools remained accessible to more people. On the flip side, not everyone wants to install an older version of Visual Studio and XNA just to compile my code. I’m also running the risk that Windows XP users will come across that crash without having read my getting started page for Daggerfall Modelling.

After serious thought, I have decided to make some changes to my source code:

  • Solution format upgraded to Visual Studio 2010 / Visual C# 2010 Express.
  • Daggerfall Modelling and XNALibrary will be refactored for XNA Game Studio 4.0.

The DaggerfallConnect class library and Daggerfall Imaging 2 will remain based on .NET 2.0 for now, but I might bring these forward at a later date.

The SVN will be updated in the next few days with these changes. The next release of Daggerfall Modelling (Beta 2) will require the latest versions of .NET and XNA Frameworks to run.

Visual Diary: XNALibrary Update

After a hectic mid-year rush things are finally settling back to normal. I will have more time to dedicate to my Daggerfall tools for another few months before the silly season begins. I am now working on getting Beta 1 of Daggerfall Modelling ready for release.

When I started Daggerfall Modelling the goal was to make a quick and easy Collada exporter for Daggerfall’s 3D assets. It didn’t take long before I realised an opportunity to revive Daggerfall Scout in a more useful form. Like all projects, escalating the scope after most of the code has been written is a recipe for trouble. While trying to wedge in action records to my scenes (levers, switches, etc.), I came to the realisation my code had become as messy as hell. To accomplish what I had in mind a total rewrite of the scene manager was needed.

As an adjunct to Daggerfall Connect, I have also been developing an XNA class library that makes using Daggerfall assets with XNA a snap (creatively called XNALibrary). This code base is something I wish reuse for other projects, so it made sense to get all the scene management and rendering out of Daggerfall Modelling and build it properly into XNALibrary. The end result is a small XNA 3D engine designed specifically for rendering Daggerfall scenes. I then had to rip out all the old code from Daggerfall Modelling and port it over to XNALibrary as the primary engine. It was a lot of work just to end up back where I started, but internally the code is much improved. Following is a quick tour of what’s new in XNALibrary, as seen from Daggerfall Modelling Beta 1.

 

Scene Graph

Previously, Daggerfall Modelling used a one-dimensional array of objects and looped through them to draw scenes. This is fine for small environments but doesn’t scale up very well.  It also makes visibility testing and render batching harder than it needs to be. When it came time to link up the levers and switches, I realised it was time to put in a scene graph.

I decided to use a simple Bounding Volume Hierarchy based around spheres. The beauty of bounding spheres is that you don’t have to worry about axis alignment as you do with boxes. Spheres may not wrap objects quite as tightly, but they’re a lot easier to handle when transformed (rotated, translated, scaled, etc.) as the volume is always the same shape and alignment in 3D space.

On the left are a couple of screen-shots of the bounding volumes, rendered as wireframe spheres. The root node is red, general models are white, and sprites are green. Resources are added to the scene from the top down, with renderable objects like models and sprites in the leaves of the tree. This hierarchy makes it easy to cull non-visible groups – if a node isn’t visible, then none of its children can be visible either. It also simplifies mouse picking and collision as these tests are only performed where necessary. Batching visible polygons by texture becomes easier, which greatly improves rendering times. And finally, linking up chained action records becomes simplified thanks to each scene node being a discrete class aware of its own state.

Action Chains

Certain objects in Daggerfall have action records that describe how this object can move, rotate, etc. An example every Daggerfall player will be familiar with is the throne room in Privateer’s Hold (pictured left). When you throw the switch an action record makes the lever rotate, which chains to another action record to make the platform rise, which finally chains to the throne to make it rise also. The end result is you throw the switch and the platform rises (along with the throne and the player).

If you open up the screenshot to the left, you will see a red line going from switch, to platform, to throne. This is the action chain linking the objects together. The next screenshot shows the same linkage in Mantellan Crux – from the axe on the floating island, to the crossbow, to the giant sword. At some point, I will number the nodes so it becomes easier to visualise where action chains begin and end.

In Daggerfall Modelling Beta 1 it will be possible to trigger action chains and watch the animations run. There are other actions, such as teleporting the player or casting spells, but I am only going to support action that cause things to move around the scene.

Once gravity and collision are implemented, you will be able to explore dungeons from a first-person perspective with the ability to operate doors and platforms. You can also just turn off collision and gravity to fly around the scene with total freedom.

Add-In Components

The renderer in XNALibrary can plug in components that add functionality. This is quite basic at the moment, but will be extended down the road as needed.

Currently, the add-ins are a billboard manager for rendering sprites like trees, animals, NPCs, and a sky manager for drawing the background sky in location exteriors. You can see these components in action to the left.

The next release of Daggerfall Modelling should be ready in 3-4 weeks. This will coincide with an update to the Daggerfall Connect library download with XNALibrary updates included. All of this is in the SVN now, if you want to download the code as it’s being written. Just keep in mind not everything may be in a stable state in the latest SVN version.

Improving UV Generation

I recently engaged in a fruitful conversation on the DaggerXL forums with another DF hacker called DigitalMonk, whose specialty is Daggerfall’s textures. His interest was piqued when I made an informal comment regarding UV generation. After bouncing a couple of posts off each either, DigitalMonk’s thoughts and observations resulted in improvement to my current method of UV generation. He also made a discovery that will result in yet another improvement down the track. If you want to track the whole conversation, read back from here. I will summarise the outcome below so you don’t have to re-read the whole thread unless you want to.

Anyone who used Daggerfall Explorer would have noticed some 3D objects have skewed textures. At the core of this problem is how Daggerfall stores UV coordinates (UV mapping is a way of describing how textures are painted on a mesh). Rather than storing a value from 0.0 to 1.0 for every point, Daggerfall only describes UV coordinates for the first three points of any face (and optionally on the fourth point, but research has shown DF does not use this value). This probably doesn’t sound so bad, triangles only have three points right? Unfortunately Daggerfall doesn’t use triangles to describe 3D objects, it uses ngons that can exceed a dozen points per face. With only the first three UV coordinates to work from, the problem is how do we calculate UVs for the remaining points? Check out the main article on the UESP for more information on the problem and the solution we’ve been using up until now. You get some feeling from this short article on how much special handling is needed to cover all possible faces.

The problem with this solution is the constructed matrix can generate UVs that look a little weird, even when the determinant for that linear equation believes the face should be solvable. There are also multiple linear equations based on whether the face is coplanar to XZ, XY, YZ, etc. When UV generation failed, branching logic was needed to post-fix the UVs for certain faces, or skip the matrix generator for another technique, or just slap the correct UVs on directly. All of this made UV generation a bloated exercise that was tricky to debug. Fixing one problem would often break something else. All in all, there are three classes of texture problems:

  • Type 1. These are faces for which UVs cannot be generated using the existing linear equations, or are generated incorrectly.
  • Type 2. These are faces with bad source UVs (i.e. the coordinates are incorrect in Daggerfall’s files). Check out the eastern side of Wayrest Castle in-game for an example of this problem. This class of problem is visible inside the game.
  • Type 3. I used to think these were Type 2 problems, but DigitalMonk discovered these source UVs actually have extra bits packed into the coordinate. By removing these bits, the UV coordinate works as expected. This class of problem is not visible in the game, meaning Daggerfall uses these bits for some purpose, or at the very least just strips them out.

After my conversation with DigitalMonk, I’ve made some improvements to my UV generation that removes all Type1 problems with no special handling required. I managed this after realising UV generation is really just a 2D problem in 3D space. All of those extra linear equations were only needed due to the extra degree of freedom. By moving all the points into 2D first, I would only need the single XY linear equation to solve for all possible faces.

This worked perfectly, and allowed me to remove all my special handling. Below are a couple of screenshots from the test block viewer. The screenshots on the left demonstrate some problem faces with all special handling disabled. You don’t see these problems in the RDB Block Viewer demo included with DFConnect because of that special handling. On the right is the same scene using the new UV generator, with absolutely no special handling. All UV coordinates “just work”.

uv fixing 1 uv fixing 2
uv fixing 3 uv fixing 4

The next step will be to strip those extra bits for Type3 problems, and eventually write a simple UV patching function for the worst Type2 problems. Once this is completed, UV generation in DFConnect will be almost perfect.

I can’t thank DigitalMonk enough for taking the time to clarify the various UV problems and help me reach a better solution. Cheers mate.

Technical Content: The Daggerfall Cache

A companion to the Visual Diary, my Technical Content series will go into more arcane aspects of Daggerfall Scout. Today’s tech-talk is about how XML is used in the Daggerfall Cache to store world files in a format both Humans and computers can read. Before going into more detail, a quick primer for The Daggerfall Cache is required.

After running the Cache Manager to build your personal cache, the following directory structure is built:

-MyDaggerfallCache
	-Default
		+Bitmaps
		+Blocks
		+Maps
		+Meshes
		+Materials
	Default.Provider.xml

In this model, Daggerfall Scout is both a cache provider and a cache consumer. When opening DF Scout, it first locates the Default.Provider.xml file containing a description of subpaths in which content is held. DF Scout is built to understand the layout created by the Default provider. It would be possible to build your own cache provider or consumer. Each distinct cache provider has a unique parent folder and .Provider.xml file describing the layout and builder versions. A consumer program opens its matching provider to locate content.

Each of the content groups has been homogenized into one media type. Where Daggerfall has multiple bitmap formats, the Default builder converts all of these into alpha-enabled PNG files. 3D objects and materials are Ogre .mesh and .material files. Blocks and Maps are saved as pure XML. Starting with a birds-eye view, let’s look deeper at these XML files and how one is used to build a simple location.

When opening a map, DF Scout locates map filenames in this format: RegionName_LocationName.map.xml. So the city of Daggerfall in the region of Daggerfall would be: Daggerfall_Daggerfall.map.xml. A map simply contains layout information for specific blocks. Take a look at Daggerfall_The Ashcroft Graveyard.map.xml below. This map contains only a single location block and a very small dungeon.

[code lang=”xml”]

















[/code]

The <Map> element encapsulates the document. Its attributes describe the map name as seen on the travel map, a native (Daggerfall-specific) type number, and a descriptive string. Of particular note is the HasDungeon attribute, denoting if the map contains dungeon data or not. This can also be derived by the existence of a <Dungeon&#62 element, but the HasDungeon attribute helps in selectively parsing the XML file using a forward-only reader.

Both the <Location&#62 and <Dungeon&#62 tags encapsulate block layout on a 2D grid. Child <Block&#62 elements describe the filename for each block, and its position on the grid. Now that we’ve seen how maps are laid out, let’s drill down into a block to see how it’s composed. In this case, we’re using GRVEAS15.RMB as it’s the only location block used to represent The Ashcroft Graveyard.

A single block XML file is a bit large to fit comfortably in this post. You can view the entire contents of GRVEAS15.RMB if you wish. The general block structure is as follows:

[code lang=”xml”]













[/code]
The <Block&#62 element encapsulates all block data. <Records&#62 describes general 3D objects populating the block, such as houses and temples. <MiscMeshes&#62 are 3D clutter such as wagons, fountains, and fences. <MiscFlats&#62 represent 2D clutter like cows, horses, shrubs, and statues. Finally, the <GroundPlane&#62 element details the 16×16 texture square grid under each block. Every block is 4096×4096 world-space units in size, and the ground plane sits under this perfectly, each texture square being 256×256 world-space units.

So what does all this stuff actually mean? Taking the map XML file above, you could render individual components to visit The Ashcroft Graveyard just as it appears in the game. Map layouts range from 1×1 blocks, all the way up to 8×8 blocks for the largest areas such as Daggerfall City. Rendering large areas is just a matter of snapping all the blocks together and rendering their individual components.

Speaking of components, let’s go into the <Records&#62 element in a little more detail. An individual record looks like this:
[code lang=”xml”]








[/code]

Attributes of the parent element describes how this 3D object should be rotated and translated into world space, relative to the parent block. <Meshes&#62 describes how many individual 3D objects make up this particular record. For outdoor areas there is typically only one mesh per record, although there are certainly exceptions. <InsideMeshes&#62 plots out how the interior of this record should appear. A good example is any house in Daggerfall – the act of going inside the house loads the inside meshes. This is used extensively in cities to keep locations with interiors nicely grouped. For outdoor scenery, such as GRVEAS15.RMB, the <InsideMeshes&#62 element is rarely populated.

Back to the <Block&#62 itself, <MiscMeshes&#62 and <MiscFlats&#62 are simply strewn relative to their parent. Their transforms are detailed as attributes for each child item.

Each block <GroundPlane&#62 is described by 256 <Square&#62 elments. These are a bit unusual as they don’t specify an image archive, just an image index. This is because there are several viable ground texture sets, based on the terrain (temperate, desert, swamp, and mountain) and the climate (snowing or raining). By applying different texture sets to the meshes and ground planes, Daggerfall achieves a wide variety of appearances using the same 3D data. Each <Square&#62 may also be rotated 90 degrees or flipped in both X an Y axes.

Finally, you will notice the record above specifies a mesh with an identifier of “750”. This tells the engine which mesh to load for that record before applying terrain/weather processing (if required) and transforming into position.

There is much more data held inside a block: doors, trees, action records, NPCs, and more. As these features come online in Daggerfall Scout, I will add them to the XML files exported by Cache Manager. Every time I release a new version of Daggerfall Scout, it’s possible to increment the internal cache version. The program will prompt you to rebuild the cache from your Daggerfall CD so you can explore the new feature.

This Technical Content article has been an overview of how Daggerfall Scout represents blocks and maps using XML. The goal is to stay close to how Daggerfall stores data, while representing it in a Human-readable format. The purpose of this design is to allow technically-minded players to explore the game world using nothing more than Notepad, and never need to deal with complex native file formats.

Below, you can see the results of loading The Ashcroft Graveyard in Daggerfall Scout. The engine neatly parses all the XML and loads the appropriate resources.

The Ashcroft Graveyard The Ashcroft Graveyard Dungeon