Technical Content: The Daggerfall Cache

A companion to the Visual Diary, my Technical Content series will go into more arcane aspects of Daggerfall Scout. Today’s tech-talk is about how XML is used in the Daggerfall Cache to store world files in a format both Humans and computers can read. Before going into more detail, a quick primer for The Daggerfall Cache is required.

After running the Cache Manager to build your personal cache, the following directory structure is built:

-MyDaggerfallCache
	-Default
		+Bitmaps
		+Blocks
		+Maps
		+Meshes
		+Materials
	Default.Provider.xml

In this model, Daggerfall Scout is both a cache provider and a cache consumer. When opening DF Scout, it first locates the Default.Provider.xml file containing a description of subpaths in which content is held. DF Scout is built to understand the layout created by the Default provider. It would be possible to build your own cache provider or consumer. Each distinct cache provider has a unique parent folder and .Provider.xml file describing the layout and builder versions. A consumer program opens its matching provider to locate content.

Each of the content groups has been homogenized into one media type. Where Daggerfall has multiple bitmap formats, the Default builder converts all of these into alpha-enabled PNG files. 3D objects and materials are Ogre .mesh and .material files. Blocks and Maps are saved as pure XML. Starting with a birds-eye view, let’s look deeper at these XML files and how one is used to build a simple location.

When opening a map, DF Scout locates map filenames in this format: RegionName_LocationName.map.xml. So the city of Daggerfall in the region of Daggerfall would be: Daggerfall_Daggerfall.map.xml. A map simply contains layout information for specific blocks. Take a look at Daggerfall_The Ashcroft Graveyard.map.xml below. This map contains only a single location block and a very small dungeon.

<?xml version="1.0" encoding="UTF-8"?>
<Map Name="The Ashcroft Graveyard" NativeType="140" TypeDesc="Ancient Graveyard" HasDungeon="1">
  <Location Identifier="20934" BlocksWide="1" BlocksHigh="1">
    <Block GridX="0" GridY="0" Filename="GRVEAS15.RMB">
    </Block>
  </Location>
  <Dungeon Identifier="20935" BlocksWide="2" BlocksHigh="2">
    <Block GridX="-1" GridY="0" Filename="B0000008.RDB" PlayerStartBlock="0">
    </Block>
    <Block GridX="0" GridY="-1" Filename="B0000006.RDB" PlayerStartBlock="0">
    </Block>
    <Block GridX="0" GridY="0" Filename="M0000000.RDB" PlayerStartBlock="1">
    </Block>
    <Block GridX="0" GridY="1" Filename="B0000003.RDB" PlayerStartBlock="0">
    </Block>
    <Block GridX="1" GridY="0" Filename="B0000001.RDB" PlayerStartBlock="0">
    </Block>
  </Dungeon>
</Map>

The <Map> element encapsulates the document. Its attributes describe the map name as seen on the travel map, a native (Daggerfall-specific) type number, and a descriptive string. Of particular note is the HasDungeon attribute, denoting if the map contains dungeon data or not. This can also be derived by the existence of a <Dungeon&#62 element, but the HasDungeon attribute helps in selectively parsing the XML file using a forward-only reader.

Both the <Location&#62 and <Dungeon&#62 tags encapsulate block layout on a 2D grid. Child <Block&#62 elements describe the filename for each block, and its position on the grid. Now that we’ve seen how maps are laid out, let’s drill down into a block to see how it’s composed. In this case, we’re using GRVEAS15.RMB as it’s the only location block used to represent The Ashcroft Graveyard.

A single block XML file is a bit large to fit comfortably in this post. You can view the entire contents of GRVEAS15.RMB if you wish. The general block structure is as follows:

<Block Identifier="GRVEAS15.RMB">
  <Records RecordCount="17">
    ...
  </Records>
  <MiscMeshes Count="34">
    ...
  </MiscMeshes>
  <MiscFlats Count="2">
    ...
  </MiscFlats>
  <GroundPlane>
    ...
  </GroudPlane>
</Block>

The <Block&#62 element encapsulates all block data. <Records&#62 describes general 3D objects populating the block, such as houses and temples. <MiscMeshes&#62 are 3D clutter such as wagons, fountains, and fences. <MiscFlats&#62 represent 2D clutter like cows, horses, shrubs, and statues. Finally, the <GroundPlane&#62 element details the 16×16 texture square grid under each block. Every block is 4096×4096 world-space units in size, and the ground plane sits under this perfectly, each texture square being 256×256 world-space units.

So what does all this stuff actually mean? Taking the map XML file above, you could render individual components to visit The Ashcroft Graveyard just as it appears in the game. Map layouts range from 1×1 blocks, all the way up to 8×8 blocks for the largest areas such as Daggerfall City. Rendering large areas is just a matter of snapping all the blocks together and rendering their individual components.

Speaking of components, let’s go into the <Records&#62 element in a little more detail. An individual record looks like this:

<Record RotateX="0" RotateY="0" RotateZ="0" TranslateX="-1536" TranslateY="0" TranslateZ="512">
  <Meshes Count="1">
    <Mesh Identifier="750" RotateX="0" RotateY="0" RotateZ="0" TranslateX="-24" TranslateY="0" TranslateZ="24">
    </Mesh>
  </Meshes>
  <InsideMeshes Count="0">
  </InsideMeshes>
</Record>

Attributes of the parent element describes how this 3D object should be rotated and translated into world space, relative to the parent block. <Meshes&#62 describes how many individual 3D objects make up this particular record. For outdoor areas there is typically only one mesh per record, although there are certainly exceptions. <InsideMeshes&#62 plots out how the interior of this record should appear. A good example is any house in Daggerfall – the act of going inside the house loads the inside meshes. This is used extensively in cities to keep locations with interiors nicely grouped. For outdoor scenery, such as GRVEAS15.RMB, the <InsideMeshes&#62 element is rarely populated.

Back to the <Block&#62 itself, <MiscMeshes&#62 and <MiscFlats&#62 are simply strewn relative to their parent. Their transforms are detailed as attributes for each child item.

Each block <GroundPlane&#62 is described by 256 <Square&#62 elments. These are a bit unusual as they don’t specify an image archive, just an image index. This is because there are several viable ground texture sets, based on the terrain (temperate, desert, swamp, and mountain) and the climate (snowing or raining). By applying different texture sets to the meshes and ground planes, Daggerfall achieves a wide variety of appearances using the same 3D data. Each <Square&#62 may also be rotated 90 degrees or flipped in both X an Y axes.

Finally, you will notice the record above specifies a mesh with an identifier of “750”. This tells the engine which mesh to load for that record before applying terrain/weather processing (if required) and transforming into position.

There is much more data held inside a block: doors, trees, action records, NPCs, and more. As these features come online in Daggerfall Scout, I will add them to the XML files exported by Cache Manager. Every time I release a new version of Daggerfall Scout, it’s possible to increment the internal cache version. The program will prompt you to rebuild the cache from your Daggerfall CD so you can explore the new feature.

This Technical Content article has been an overview of how Daggerfall Scout represents blocks and maps using XML. The goal is to stay close to how Daggerfall stores data, while representing it in a Human-readable format. The purpose of this design is to allow technically-minded players to explore the game world using nothing more than Notepad, and never need to deal with complex native file formats.

Below, you can see the results of loading The Ashcroft Graveyard in Daggerfall Scout. The engine neatly parses all the XML and loads the appropriate resources.

The Ashcroft Graveyard The Ashcroft Graveyard Dungeon

Visual Diary: Adding Ground Detail

Every now and then, I plan on adding a visual diary showing a feature as it progresses. Today’s diary is about ground details.

 It's A Start

It’s A Start

I’ve added a simple ground plane under the city map. The ground in Daggerfall is composed of squares fitting perfectly under a block. The ground is tiled together in the same way blocks are tiled together to form a complete map.

The next step is texture it.

 Um Not Quite

Um, Not Quite

Even though I’ve written ground builders a few times in the past (the first was in Daggerfall Explorer), adding this feature into a new engine is bound to encounter bugs. I’ve not quite nailed it this attempt.

 Done

Textures Done

The ground textures are now being placed and aligned correctly across all maps.

 Needs Region Textures

Still Needs Region Processing

Daggerfall uses texture swaps for each region/climate type. You can play with these combinations in Daggerfall Explorer. Here is Sentinel using the default “temperate” texture set used while developing the ground plane.

At some point, I need to re-code the ability to swap in the correct texture. But for now I want to do something a little more interesting.

 New Flats

New Flats

In this screenshot, I’ve started adding flats. My first attempt isn’t that successful. They aren’t at the right height, size, or transparent. But they are standing in the right places, so it’s is a good start.

Another issue with these flats you can’t see in the screenshot is they always face the camera. If you fly into the air and look down, they’re lying flat on the ground! A quick change to the axis alignment will sort that out.

Flats1
Flats2

Working Flats

Flats are now correctly positioned, axis aligned, and alpha-enabled. I feel the scaling needs some work. At this time, it is unknown which numbers Daggerfall uses to scale flats. I will refine this as development progresses.

I’ve never been a huge fan of flat billboards in games, but they were a standard technique in days past and part of the Daggerfall landscape. Despite being antique tech, I feel some of the old charm has started to make its way back into the otherwise dreary maps in Daggerfall Scout.

The next step is to do some gardening and add the local plant life, such as trees, etc. These aren’t stored in the flats records along with everything else. I need to find them before I can draw them from the game files.

I hope you enjoyed this first visual diary.

What’s Green And Right-Handed?

I spent the last few days exchanging the graphics engine in Daggerfall Scout from Irrlicht to Ogre. This came with enough surprises and frustrations to last me a few weeks.

To paraphrase one of the best quotes I’ve read while comparing these two engines: “Irrlicht quickly lets you get your project to the point where you need Ogre to complete it.” That certainly was the case DF Scout. Irrlicht required very little investment in time to get up and running, but poor mesh loading times and limited support for dynamic content were starting to be a drag.

Fortunately DF Scout talks to an engine wrapper that handles all the dirty work. I should just be able to swap in a new wrapper and everything will be peachy, right? The initial work was easy, and before long I had a new Ogre wrapper happily chugging coloured pixels into a viewport.

The next step was to build .mesh and .material exporters into the Cache Manager to change all that Daggerfall content into a format the engine could load and render. This required a few changes to DFLib to break meshes down into named submeshes rather than use attribute buffers (which is more common in D3D). Again, not so hard. I’ll even go so far as to say I enjoyed it.

Problems started when I loaded my first city and everything was back to front. All the meshes were in the right place, but they were flipped. Imagine looking a sign and the letters are back to front, and then imagine a whole building flipped that way. The effect was a little unsettling.

The root cause was that my old engines, and more recently Irrlicht, happened to use a left-handed coordinate system. This coordinate handedness is ubiquitous in my code (and my thinking). I use left-handed coordinates everywhere in DFLib, in DF Scout, in the content builders, in the scene files. Now in comes Ogre with its fancy right-handed coordinate system making my buildings look peculiar.

Of course, I didn’t work this out right away and spent a lot of time trying to fix my mesh builders, assuming I did something wrong in the new submesh breakdowns. When the penny finally dropped (with a clang), I just kind of sat there stunned thinking “how the hell am I gonna fix this, I use left-handed coordinates everywhere.”

After a quick cup of coffee and wishing I was a smoker – or that my girlfriend who is a smoker was home so I could at least get a passive hit – things didn’t seem so bad. In the end, all I had to do was change face index ordering in the mesh builder, and teach the engine wrapper how to deal with the conversion (a quick sign inversion of the Z axis). Of course this isn’t ideal. While it’s easy enough to fix handedness in the engine wrapper, the engine itself is still right-handed while the content builders are left-handed. This will probably lead to confusion down the road, and the confused person will probably be me.

And the results? Well, they’re awesome. Ogre loads its native mesh format incredibly quickly – even the largest cities fly into the scene almost instantly. The material system is excellent, far more power than required to render Daggerfall’s old content. This should give me some interesting tools as my experience with Ogre grows. Overall, the work has been worth it.

The downside is that I’ve spent a few days gutting DF Scout and more or less getting back to where I started. Oh well, back to the fun stuff.

Daggerfall Cache

Anyone that has written mods for Daggerfall, or even peeked at the file formats, will notice just how unusual these formats are. There are five different bitmap formats, each with their own specification, three 3D model versions, two major block formats and, thankfully, only a single map format.

Looking deeper things get even stranger. The 3D models aren’t stored as triangles – a single face primitive might have upwards of 12 vertices. Considering modern gaming cards generally feed off triangles, these models have to be converted before they can be rendered properly. Loading a city or dungeon into memory can take several seconds of processing on a fast computer just to load a small map.

My solution was to pre-convert Daggerfall’s files into something manageable, rather than convert everything at runtime as I have in my previous tools. Daggerfall Scout has a feature called the Cache Manager which builds a personal cache from your Daggerfall CD containing pre-converted textures, 3D objects, blocks, and maps. The following table shows the process in a little more detail.

No Cache

Fresh Install

The first time you open Daggerfall Scout, there’s nothing much to look at. I don’t distribute any copyrighted content with my programs, and I don’t plan on starting. Even the world map image is Bethesda’s property. By itself, Daggerfall Scout is useless.

Before you can do anything with Daggerfall Scout, you have to build your personal cache. If you open the screenshot to the left, you will see a message on the status panel guiding you to the Cache Manager.

Cache Manager Start

Cache Manager

Opening the Cache Manager shows that you need an update. This only needs to be done once, but you may also need to update your cache after downloading a new version of Daggerfall Scout.

Clicking the Update button starts the process.

Select CD

Select Your CD

The next step is to insert your Daggerfall CD. You must tick the box confirming this is an original and legally purchased CD before the program will allow you to continue.

After clicking OK, the update process begins. This can take several minutes based on the speed of your PC. Once the cache has been built, you can put away your Daggerfall CD.

Cache Complete

Cache Complete

The cache has been built, and all groups show they are updated and ready. Clicking Close will take you back to the main program.

Scout Ready

Scout Ready

After closing the Cache Manager, Daggerfall Scout detects the cache is now up-to-date and initialises the exploring components. The world map and location explorer come online, and the 3D engine is prepared to render.

From this point you can start using Daggerfall Scout to explore.

The contents of the cache are easy to work with. Bitmaps are saved to standard PNG files, and constructs like blocks and maps are written out as XML documents. This makes exploring Daggerfall a faster, cleaner, and more stable process.

Introducing Daggerfall Scout

One of the key goals for my abandoned Daggerfall Project was to bring the virtual world of Daggerfall back to life. Prior to that, all of my exploring tools were about specific media. If you wanted to listen to Daggerfall’s music, you fired up Daggerfall Jukebox. If you wanted to view and export bitmaps, Daggerfall Imaging was the tool for you. For most other things, Daggerfall Explorer and Cartographer had it covered. But none of these tools gave you the ability to really visit a place.

While The Daggerfall Project was never close to a finished work, I put a lot of effort into visualising the world. It’s a shame to let all that go to waste. My solution is a new exploring tool, something that lets you explore the Illiac Bay for the hell of it. This new tool is called Daggerfall Scout, and you will be able to track its progress at this blog over the coming months.

To get things started, I want to talk about the goals I have for Daggerfall Scout, and what you can expect to see as work progresses. The most important thing to realise is that I am approaching this as a hobby. Some people play golf, or paint, or fish. I work on exploring Daggerfall. It’s my version of a good day out on the water drinking beer and reeling in snapper. My number one goal is to have fun.

Just as important is to keep things legal. Daggerfall is Bethesda’s property, not mine. Like all of my tools, you will need to legally own the full version of Daggerfall before you can use Daggerfall Scout. No content belonging to Bethesda is distributed with my programs. If you don’t have your original Daggerfall CD handy, you won’t be able to play with Daggerfall Scout.

Last but not least is to keep it simple. In the past, I’ve written my own 3D engines. This time around I’m using a free, open-source graphics engine as the back-end to my tools. Daggerfall is not a graphics powerhouse and it doesn’t need a hand-rolled engine to render the world. Time previously spent on developing an engine can now be better put to use on the fun stuff.

Keeping those three goals in mind, my overall objective is to take the best parts of the abandoned Daggerfall Project and put them into an exploring tool that will hopefully recapture some of the magic of Daggerfall and put it at your fingertips.

With all that out of the way, let’s take a look at where I’m up to, and where I’m headed:

Privateer's Hold

Privateer’s Hold

This first image shows the basic layout of Daggerfall Scout. The user interface can be docked to the top or bottom of the window, or collapsed down to a small status bar with just the basics.

In the bottom-left is a map of the Illiac Bay, allowing you to choose a region to explore. Just to the right of that is a hierarchy of locations that can be filtered by type. If you’ve ever used Daggerfall Cartographer, you will be familiar with this concept. The difference here is that Daggerfall Scout supports all world locations, including dungeons and interiors.

The blue square on the right will eventually be used for several purposes, such as displaying a graphical minimap and controlling properties.

Along the top of the user interface is a status bar and commonly used buttons. When collapsed, this is the only visible portion of the user interface.

Currently loaded is the centre portion of Privateer’s Hold.

dDaggerfall City

Daggerfall City

You can see from this screenshot I’m only part-way through porting my world builders from Absinthe (engine written for The Daggerfall Project) to Irrlicht (a simple and easy to use open-source engine). The ground plane and the sky are missing! I will slowly fill in the blanks as development progresses.

If you open the thumbnail and look at the location view, you will notice there are two Daggerfalls in the tree view. The parent Daggerfall is the city. The child Daggerfall is the dungeon contained inside Castle Daggerfall. I am working on a free-roaming mode allowing you to enter dungeons and buildings through the front door, just like you would in the game. For now, all dungeons follow this format, allowing you to swap between world cells and dungeon cells with a single mouse click.

Daggerfall Dungeon

Castle Daggerfall

This screenshot is an overhead view of the twisting labyrinth inside Castle Daggerfall. If you look to the right of frame, you can just see the Treasury. How many hours have we all spent search for the Totem in that twisting mess?

Another limitation of the current build is lighting. Ultimately, I want to render Daggerfall’s light sources as they appear in game – recapturing the atmosphere of those dark places.

While exploring you will be able to toggle gravity and collision, allowing you fly through the map with ease or explore on foot.

Daggerfall Treasury

Daggerfall Treasury

This is a close-up of the Treasury inside Castle Daggerfall. Using the first-person flying camera with no collision, I have positioned myself high in the room.

One of the long-term features of Daggerfall Scout is to allow world interaction. There is a complex puzzle in this room to find your way to the Totem. I would like to enable the explorer to action levers, teleporters, doors, etc.

Mantellan Crux

Mantellan Crux

This famous dungeon marks the end of the main quest in Daggerfall. Each part of the dungeon is a self-contained puzzle room where you are beset by foes. Only the most powerful characters could find their way past these challenges.

This screenshot also shows the user interface collapsed down to just the status panel.

Blind God

Blind God

A close-up of the fire room in the Mantellan Crux, looking down on the Blind God.

I hope to release the first version of Daggerfall Scout in a few months time. The plan is to release a fun world explorer and continue to add features in subsequent builds.