Questing Part 2 – Compiling

In the first part of this series, I discussed how I’ll be using source files output by Tipton’s Template v1.11 for quests in Daggerfall Unity. If you’d like to check these out in full, the quest source files are already on GitHub. Follow this link to view them (ignore the .meta files).

Some interesting quest files to investigate are:

  • _BRISIEN.txt – This is the quest that launches when player starts in Privateer’s Hold. It sets the whole story arc in motion and starts timers for delivering letters that prompt player to visit Lady Brisienna. Failing to visit Lady B in time is ultimately a failure condition of main quest.
  • S0000999.txt – Starts when you leave Privateer’s Hold. Delivers letters from Prince Lhotun and Morgiah and sets key global variables.
  • S0000977.txt – Also starts when leaving Privateer’s Hold. Sets up regular ghost and wraith spawns in Daggerfall and plays the VENGEANCE! sound.
  • TUTOR.txt – Is the tutorial quest started when selecting Yes to the prompt at start of game.

As another interesting tidbit, longtime players may have noticed the initial journal entry that begins with…

"I am on a mission from the emperor to investigate
the shade of King Lysandus. His spirit has been
haunting the city of Daggerfall. The emperor
himself has charged me with the duty of laying
his ghost to rest."

…will stick around in your log forever. Even after you’ve completed the game, this first journal entry will remain. As part of researching the quest system a while back, I looked into why this was happening and found the above journal entry is added twice and removed only once. The below image sums it up visually. You may have already seen this when I posted it to Twitter back in September.



So the quest system in Daggerfall is quite powerful. It’s responsible for adding and removing journal text, starting timers, delivering letters, spawning enemies, playing sounds, setting global variables, moving NPCs, and a slew of other functions. At first blush it can appear a lot like a full programming language, but it’s only superficially so. The structure is actually more like a complex INI file, or even a kind of markup file. Everything is neatly categorized into its own section and execution flow generally only happens in a few different ways. Most commonly are at startup, when a timer ends, and due to a variable or condition changing state.

Thus despite the initial similarity to a programming language, the quest source is really just a collection of defined objects with some very basic scripting functions. Most of these functions revolve around spawning something, starting something, playing something, or changing something. Complex enough to get the job done, but don’t feel like you need to be a programming expert to create new quests in the future. Using Tipton’s Template v1.11, you could even start creating quests in classic Daggerfall now and later port them into Daggerfall Unity.

Likewise, the job of compiling the quest source back into Daggerfall Unity is not that difficult. It’s going to have some challenges, but nothing on the scale of building a real compiler, something I thought I’d be facing at the outset. The problem is largely just splitting source file up into correct parts and handing that source off to classes designed to support that part. For example, messages will go to a Message class, timers will go to a Timer class, etc. When serializing live quests as part of save games, the QuestMachine will save/load JSON state for each live quest along with the global variables for the current game.

To help Daggerfall Unity re-compile Template’s source output, I’m going to make some minor changes to the expected source files. I’m trying to keep these changes to an absolute minimum.

First change is for the quest header to be uncommented in source file. For example, _BRISIEN has the following header. The dash ‘-‘ prefix starts a comment.

-- StartsBy: letter
-- Questee: anyone
-- Repute: 0
-- QuestId: 0

This reason this can be commented out is that Template is designed to re-compile quests back into the original QBN/QRC format. When commented out, Template will just use the information already in the quest QBN data. However, Daggerfall Unity relies entirely on the quest source and will need the above uncommented to:

StartsBy: letter
Questee: anyone
Repute: 0
QuestId: 0

This way the information can be parsed back into the quest data at compile time.

The second change is that I would like an explicit startup task, rather than an implicit startup after the QBN object definitions (like Person, Clock, etc.). Currently the startup task looks like this:

-- Quest start-up:
log 1010 step 0 
pc at PiratesHold set _exitstarter_ 
say 1025 

I would like to change this to:

log 1010 step 0 
pc at PiratesHold set _exitstarter_ 
say 1025 

This makes the startup task an explicit task object to be executed when quest begins. This is already the current behaviour, but I think this change makes the task clearer to identify for both quest creators (even without a comment) and for my compiler. I could definitely get away without this change, but I prefer the explicitness to the current setup.

And of course, the above is subject to change as the quest system matures. If I learned anything from rolling out items, the feature will probably look different again by the time it’s mostly complete. Daggerfall Unity is not just a game remake, it’s an ongoing research project into the guts of Daggerfall. That adds a few twists outside of normal gamedev process, and I just need to roll with it.

One of the upcoming features not shown yet is the quest debugger currently in development. Hopefully, I will have something to show on this by part 3. The quest debugger will be used to manually start new quests, terminate and restart executing quests, and inspect state of global variables and quest objects. It’s a big UI that will continue to grow as development progresses. Should turn out to be quite useful though.


For more frequent updates on Daggerfall Unity, follow me on Twitter @gav_clayton.

Questing Part 1 – Source

With items and loot more or less finished, it’s time to place my sights on the next big feature group. I had originally planned to work on spells and effects in the 0.4 cycle, but also had quests in mind as an equally important feature. So back in July, I used a Twitter poll to sample what the community wanted priority on. The answer came firmly in preference for quests over spells.

Fast forward a couple months: 0.3 stable is now tagged on Live Builds page and I’ve started work on the quest system in earnest. This post is the first in a series documenting the journey from raw bytes to a working feature.

When adding a big new feature, there’s always a technical planning stage before commencement. I need to understand which of Daggerfall’s binary files are involved in the feature, learn how much is already known about those files (sometimes a lot, sometimes almost nothing), map out the additional details I need to know before I can re-implement that feature, and finally start cutting some code.

Sometimes it takes a few iterations, as writing code to pull apart the binaries is itself a journey where new information is learned. I don’t know what I don’t know, and often that new information results in a new iteration with better understanding. There’s basically a research-implement-compare loop going at all times throughout development of Daggerfall Unity. It can be a bit draining.

Items for example needed considerable research to bridge the gap between “enough knowledge to make an item editor” and “enough knowledge to fully implement items in a new engine”. Believe me, there’s a huge amount of work needed between having a file spec with lots of unknowns and building a completely functional equivalent. Most of my time is spent somewhere in that gap.

So when starting quests, I set out to learn everything I could about the binary formats and the work others had done before me. I was pleasantly surprised to learn:

  • Quests are among the most well-understood formats, thanks to the work of several early Daggerfall hackers (notably Donald Tipton and Dave Humphrey).
  • Tools exist to edit live quests, but more importantly decompile the QRC and QBN files to a human-readable source file (thanks to Template v1.11 by Tipton).
  • Early Daggerfall modders already had a good understanding of creating quests for Daggerfall – there’s an established process.
  • Quest files are one of the few cleanly decoupled things in Daggerfall. Unlike items which have template data hard-coded in the executable, quests are nicely self-contained.

With the above in mind, it was no longer necessary for me write a quest decompiler from scratch, design a new file format, decouple quest data, or chase half a dozen other time sinks. I can almost go directly to implementation. Almost. But first I need quest data in a usable form.

For the first step, I’m using Template v1.11, which you can find on UESP. This tool can decompile all the QRC (quest text) and QBN (quest programming) files to a single easy-to-read text file. So the information goes from raw binary data to the following (snippet from starting quest _BRISIEN).

Item _letter1_ letter used 1020
Item _letter2_ letter used 1021
Item stopMainQuest letter used 1022

Person ladyBrisienna face 1 named Lady_Brisienna anyInfo 1012 rumors 1013

Place PiratesHold permanent PirateerHold1
Place _dirtypit_ remote tavern

Clock _invitepc_ 7.00:00 14.00:00
Clock _remindpc_ 30.00:00 0 flag 1 range 0 1
Clock _pcfailed_ 14.00:00 0 flag 1 range 0 1
Clock _oneday_ 1.00:00 0 flag 1 range 0 1
Clock _nowwhat_ 29.09:00 0 flag 1 range 0 1

-- Quest start-up:
 log 1010 step 0 
 pc at PiratesHold set _exitstarter_ 
 say 1025

That can still look a bit daunting if you’re not comfortable with programming concepts, but the source format generated by Template v1.11 is generally very easy to read and understand – more like a complex INI file than a real programming language. It’s so good that I’ve decided to adopt Template’s output as the source data for quests in Daggerfall Unity. You won’t need to recompile quest source back to QBN/QRC, rather you will use the source directly in Daggerfall Unity where it will be compiled at runtime by the quest system.

This means the de facto standard for creating Daggerfall quests will remain essentially the same in Daggerfall Unity (with some minor differences I’ll discuss later). If you’ve ever created a quest for Daggerfall, or even seriously looked into it, then you already have the skills to create quests in Daggerfall Unity. If you’re new to Daggerfall quests then you can learn from existing quests and read Template’s excellent documentation. Just as I don’t need to start from scratch, neither do you.

I will need to change the format slightly for unique needs on my side, but I aim to keep my input format as close as possible to Template’s output format. There’s also the need to emulate support for every single condition and action in Daggerfall Unity to behave similarly to Daggerfall given the same quest source. That will probably be an ongoing process all the way through to 1.0. But we need to start somewhere, and this is a great starting point.

The next article in series will look at parsing quest source files and how quests will be executed inside Daggerfall Unity.


For more frequent updates on Daggerfall Unity, follow me on Twitter @gav_clayton.

Building Names

One of Daggerfall’s long-running puzzles is how to generate the correct building name for any given building in a location. Daggerfall’s binary data exposes this information only as a seed value with no obvious correlation to the final name. From today, I’m happy to say this has been solved and I will be able to generate proper building names in the future. This article is a summary of the technical journey, minus all the dead ends and frustration.

The seed value used to generate building names has been known about for some time. This can be found in the BuildingData structure (link to UESP). The first step along the way was to generate some known values by changing a known seed value in MAPS.BSA. I started at the location Ashfield Hall in the Daggerfall province, which has a single tavern and some residences. Taverns are a great place to start as they have a very obvious PartA + PartB structure. For example The Bat And Skull. In Ashfield Hall, our single tavern is the The Howling Stag with a name seed value of 27748.

The first thing I did was change the name seed value for The Howling Stag in MAPS.BSA then start up Daggerfall to see how the name changes. Here’s a small sample of names generated from seeds 0-3. Keep this list in mind as we’ll return to it later.

0 = The Dancing Chasm
1 = The Knave and Scorpian
2 = The Pig and Ogre
3 = The Thirsty Fairy

Now I have somewhere to begin. I know the building is a tavern and have a sample group of seeds that result in specific names. The next trick is to work out how Daggerfall derives these names from the seed value.

I open up FALL.EXE in a hex viewer and search through for strings like “The Dancing” and “Chasm”. These strings are easy enough to locate, but these are just resources packed into the executable. What I need is the actual PartA and PartB tables Daggerfall is selecting from at runtime.

To get this information, I first have to use the DOSBox debugger to dump out memory from Daggerfall while it’s running. I can then search not just for strings, but for memory offsets pointing to those strings. I write a small bit of code to do the searches for me, and it doesn’t take long to find the correct offset tables for Part A and Part B of tavern names. Just think of this as a pair of arrays. In this case, both arrays are 36 elements long. Here they are as captured from the spreadsheet I dumped them out to.


So how do we go from a seed of 0 to The Dancing Chasm? This is where most of the difficulty started. It was obvious Daggerfall used a random number generator to pick both parts, but the trick was to find the correct random number generator used by Daggerfall’s C compiler circa 1994-1996. Fortunately, I also needed this for correct texture table generation (still an open problem at time of writing) and had previously researched the correct random generator, known as a linear congruential generator, specific to Daggerfall. Here it is for completeness.

static ulong next;
public static void srand(int seed)
    next = (uint)seed;
public static uint rand()
    next = next * 1103515245 + 12345;
    return ((uint)((next >> 16) & 0x7FFF));

There are two methods here, one to set the seed (srand) and another to generate the next random number from that seed (rand). This is pretty much the standard ANSI LCG but specific to Daggerfall’s needs. Implementing this manually ensures that critical random number generation will always work just like Daggerfall, regardless of platform.

Now that I have the right random number generator, let’s feed it our test seeds from earlier and see what comes out. Starting with seed=0 and generating two numbers (indices into Part A and Part B name tables above), I get the following results.

PartA = 0
PartB = 12

First obvious thing is the spreadsheet starts from 1, not from 0. Just need to +1 each number to match the tables above (although zero-based arrays will be used in actual code). Matching these numbers to the above name table we get: Chasm The Dancing. OK, so Daggerfall obviously generates PartB first then PartA. Let’s try that again with the +1 and order swapped.

Seed = 0
  PartA = 13 (The Dancing)
  PartB = 1  (Chasm)
  Result: The Dancing Chasm

Using our handy table we can match row 13 with row 1 and we get The Dancing Chasm. Good! Let’s run some more tests and prove the concept.

Seed = 1
  PartA = 35 (The Knave and)
  PartB = 27 (Scorpion)
  Result: The Knave and Scorpion

Seed = 2
  PartA = 30 (The Pig and)
  PartB = 9  (Ogre)
  Result: The Pig and Ogre

Seed = 3
  PartA = 16 (The Thirsty)
  PartB = 36 (Fairy)
  Result: The Thirsty Fairy

So far, so good! Building names are output just like Daggerfall given the same inputs. Let’s try the original, unmodified seed value of 27748 which should give us The Howling Stag.

Seed = 27748
  PartA = 21 (The Howling)
  PartB = 33 (Stag)
  Result: The Howling Stag

And there we have it! Building name generation from initial seed value resulting in a string exactly matching Daggerfall.

From here, I still need to extract hard-coded name tables for other building types like armorers and weapon-smiths. This isn’t hard though, I just need to find the tables using the same methods as taverns. I also need to assign full building data from MAPS.BSA to the correct models in Unity scene and wire up API methods to query this data when inspecting or entering a building. One challenge at a time though.

For regular small updates on Daggerfall Unity, I can be found on Twitter @gav_clayton.

Items Part 3 – Paper Doll

With item bitmaps and dyes out of the way, it’s finally time to begin work on paper dolls. The concept of layering cutouts of clothing and other accessories over a figure is centuries old, and a perfect solution for early video games where memory was at a premium. Daggerfall’s paper doll system is easily one of the most extensive to be found in video games of the time.

Before equipping anything to the paper doll, a few key pieces had to be researched.

  • Body Morphology. Every bit of armour and clothing has 8 variations to suit the male and female body shapes of Argonians, Elves, Humans, and Khajiit. The correct texture set must be mapped to the correct race and gender.
  • Position. The X, Y coordinates of each item on paper doll is coded into their texture files. This is tightly coupled to morphology.
  • Draw Order. Every item template has a value to determine the correct item rendering order on paper doll.
  • Equip Table. The equipment slots available to player and rules for what is equipped where.

I won’t go into detail about the first three, that information is just managed by the API as part of importing or generating items. The equip table is a little interesting however with a total of 27 slots available. I use the same index setup as Daggerfall itself.

  • 00 Amulet0 (amulets, torcs, etc.)
  • 01 Amulet1
  • 02 Bracelet0
  • 03 Bracelet1
  • 04 Ring0
  • 05 Ring1
  • 06 Bracer0
  • 07 Bracer1
  • 08 Mark0
  • 09 Mark1
  • 10 Crystal0
  • 11 Crystal1
  • 12 Head (helms)
  • 13 RightArm (right pauldron)
  • 14 Cloak1 (casual cloak, formal cloak)
  • 15 LeftArm (left pauldron)
  • 16 Cloak2
  • 17 ChestClothes (shirt, straps, armbands, eodorics, tunics, surcoats, robes, etc.)
  • 18 ChestArmor (cuirass)
  • 19 RightHand (right-hand weapon, two-hand weapon)
  • 20 Gloves (gauntlets)
  • 21 LeftHand (left-hand weapon, shield)
  • 22 Unknown1
  • 23 LegsArmor (greaves)
  • 24 LegsClothes (khajiit suits, loincloths, skirts, etc.)
  • 25 Unknown2
  • 26 Boots (boots, shoes, sandals, etc.)

The two unknowns could just be reserved indices as I was unable to find any equipment Daggerfall mapped to these slots. If there’s more to this, I’m confident it will be found in future testing.

As usual the API handles equipping items for developer, it’s easy as calling EquipItem(item) on the entity’s equip table. If an item of that type is already equipped, it will be dropped in the next compatible slot (if one is free) or swap out an existing item based on swap rules for that item template.

Now that we know which items the player has equipped, the textures to use, and their position and draw order, it’s fairly trivial to start layering down bitmaps onto the paper doll. But as usual, a couple of additional problems must be solved.

First up are cloaks, which have both interior and exterior parts drawn at different stages of the build. The below image shows how the two parts work together.


Cloak Components


The interior is drawn first, then the avatar, then the cloak exteriors. It’s actually possible to wear two formal or casual cloaks in Dagerfall (slots 14 and 16). Note: the above image was taken prior to order being fixed which is why the loincloth is slightly eroded in first and third images.

Our next problem is masking. Daggerfall has a special mask index allocated to hide hair that would otherwise be drawn outside of helmets. During the build process, the mask becomes transparent and overwrites anything else in that position. Masking is used for both helmets and hooded robes/cloaks.


Mask Components


Other items can then be drawn based on their draw order. The below animation shows a step-by-step paper doll build after sorting items by draw order.



Now that items are equipped, we need a way of removing them again. Daggerfall allows you to click directly on paper doll itself to remove an item from your avatar. This is accomplished by creating a special selection mask where each pixel is an index mapping to 0-26 on the equip table above. This isn’t actually visible, it’s just an array sampled when player clicks on paper doll. Following is how the selection mask looks when rendered out to an image using grey values to represent indices. Each grey value maps to an item slot on paper doll.





I’m finally nearing the end of initial item support in Daggerfall Unity. There is still much to do (loot tables, shops, dropping items, repairing, storing items, effects, and so on) but those problems can each be tackled in turn. What I want to do now is clean up some code and begin a new test release cycle. This will allow me to fix any early bugs before moving onto the next stage of item support. I will post more news on this soon.

Items Part 2 – Dyes

Before moving on to equipping items, I thought it would be fun to show off the equipment dye system in Daggerfall Unity.

Classic Daggerfall uses a 320*200 pixel 256-colour display – or more specifically a Mode 13h display. Back in this era, bitmap graphics were typically a width*height byte array of indices into a 256-colour RGB palette. One of the coolest tricks available to graphics programmers at the time was to change index ranges to substitute colours, change brightness, animate textures, and so on. Daggerfall uses index changes and palette swaps to accomplish all of these tricks and then some – it would be possible to write a series on that subject alone. This article is just about changing dyes to re-colour weapons, armour, clothing, and how the old index swaps can be realised in a true RGBA renderer like Unity.

If all of that is difficult to visualise, let’s start with an example. Here’s a pair of basic boots without any changes.


Every pixel above is just a single byte index into a 256-colour palette. For example, index 0x70 points to RGB #DCDCDC in the default texture palette for a very light grey. For weapons and armour, the 16 indices 0x70 to 0x7F are reserved for index swaps (clothing reserves indices 0x60-0x6F). In the case of these boots every pixel falls between 0x70-0x7F, but that isn’t true of all items. Sometimes only a small part of the image will support dyes. If we just substitute every index between 0x70-0x7F to another random index between 0x00-0xFF we get the below.


Quite the mess, but it demonstrates that changing indices can radically change the appearance of an indexed bitmap. The important thing to keep in mind is that every pixel is not by itself a colour. Rather its just an index pointing to a colour.

The first challenge in bringing indexed colours into Unity is that every time we read in a Daggerfall bitmap it must be converted to true 32-bit RGBA values where every pixel actually is a specific colour. Fortunately converting to 32-bit RGBA in Unity isn’t difficult. The general process is:

  1. Allocate a Color32 array with the same number of elements as width*height of source bitmap.
  2. For every pixel index in source bitmap, sample the RGB colour of that index to a Color32 value.
  3. Write colour sampled from palette into correct position in Color32 array.
  4. Create a new Texture2D of same width*height as source bitmap.
  5. Promote Color32 array to our Texture2D using SetPixels32() and Apply().
  6. Use this Texture2D as needed.

When it comes to changing the dyes, all that’s required is to substitute the correct indices in step 2 before sampling palette. So where do these colour swaps come from and how does Daggerfall know which swaps to use for what items? Daggerfall actually has a couple of different methods for generating swaps. Let’s start with weapons and armour.

Buried inside Daggerfall’s executable FALL.EXE at offset 0x1BD1E2 (for DaggerfallSetup version) are the metal swap tables. There is one 16-byte swap table per metal type. For example, when encountering index 0x70 for a Daedric pair of boots, replace 0x70 with swap index found at daedricSwapTable[0]. For index 0x71 replace with index found at daedricSwapTable[1]. And so on. These swaps have been known about for some time and you can find more details on this archived page from the wonderful old Svatopluk site.

Clothing does not appear to use pre-defined tables like metals. Rather, each swap table is just 16x sequential indices. For example, purple is 0x30-0x3F and green is 0xA0-0xAF. Swap tables can be generated on the fly using a dye enum mapped to starting index. Daggerfall appears to do this as these sequences are not found in the executable like metal swap tables.

Armed with the power to create textures and swap indices, we can now generate our final boots image based on metal type. here are some examples.








One benefit of using the same generic process for metals and clothing is that it becomes possible to use clothing dyes on armour, something Daggerfall can probably do but doesn’t make available to players. This could allow for dye station mods down the road for players to further customise their equipment. With Unity using a true 32-bit palette this could extend well beyond Daggerfall’s 256-colours. Anyway, for an example of armour dyed something different:


Blue Chain Boots


The next challenge now that we’re using true 32-bit textures is a red pair of boots becomes a completely different texture to a green pair of boots. Whereas in Daggerfall the same bitmap can be used both times by just changing indices as described above in the software blitting function. Between this and the inherent (but minor) performance impact of converting indexed bitmaps to Texture2D, we need some way of minimising CPU time and garbage creation. Caching to the rescue.

Daggerfall Tools for Unity (the underlying API suite) already uses texture caching for general world materials, but items have their own set of problems to solve. To this end, I created a new item helper class to serve up equipment icons and handle the caching based on properties unique to items.

Every time an equipment icon is requested, a unique 32-bit key is generated by packing that request’s variables into a bitfield. The packing looks like below.


  • Colour enum index refers to the swap table in use. This value matches Daggerfall’s own colour enum stored within base item templates.
  • Variant index is an alternate image for this item.
  • Archive index is the texture file number (e.g. TEXTURE.245) containing the icon.
  • Record index is the icon index within the texture archive.
  • Mask bit is used to enable/disable a special mask used to overwrite pixels like hair around helmets. More on this in a later post.
  • There are a few reserved bits to grow the key system later.

It’s worth pointing out the end programmer doesn’t need to worry about how these values are packed. This all happens automatically under the hood when calling GetItemImage(). What matters is the API has a way of uniquely identifying any individual equipment icon based on its display properties.

When calling GetItemImage() the API will first check cache to see if this exact icon has already been converted from Daggerfall’s native file formats. If not, it is converted and stored in the cache for the next time its needed.

To wrap things up, here’s a new gfy showing a variety of dyed items in the inventory UI.

For regular micro-updates on Daggerfall Unity, I can be found on Twitter @gav_clayton.

Items Part 1 – Bootstrapping

Loot. Kit. Swag. Treasure. Whatever you call it, items are an important part of any RPG game loop. They provide the means for your character to defeat ever more powerful foes and create incentive to keep playing in search of the next big upgrade. While Daggerfall’s items don’t quite tickle the reward centres of the brain like Diablo 3 or World of Warcraft, they’re still a vital part of the play experience. Without decent gear and enchantments, you’re unlikely to survive the grueling ordeal of Mantellan Crux.

In this series, I’ll describe the process of adding items to Daggerfall Unity. I wanted to approach items early on as they will be involved at almost every level of the game. Shops sell them, blacksmiths repair them, monsters drop them, quests reward them. Your character may have a special affinity for bladed weapons, or be forbidden the use of shields. Even the biography questions when building a character can grant you items like the near-essential Ebony Dagger. With items embedded in almost every major game system, the hardest part was working out where to begin.

I decided to start with existing items as part of importing classic Daggerfall saves then bootstrap the whole item back-end from there. That way I could be certain I was dealing with the most real-world data possible. Having built support for classic saves in 0.1, I could already identify item records parented to the main character record and visualise them with a custom Unity Editor script. They looked a bit like this at first:



It’s not much, but at least I could find item records belonging to the character. The “Container” record is just a generic parent record. In this context, think of it as the character’s backpack.

The next step was to break apart the item record format. Fortunately the UESP came to the rescue here with most of the bytes already solved, but far from the whole story as you’ll see once the names are revealed:


A Frosty what of Ice Storms? OK, so there’s more to this than just the save record. How to we go about filling in the blanks? The key here is the “category” 16-bit field in that UESP article. This is actually a pair of 8-bit values. The first byte is the item group, the second byte is a table lookup for the item template within that group. The template indexed by this lookup has all the missing pieces of information we need to complete our item data. Now we have two more problems to solve. Where are the templates, and how to use those category bytes to find them? Let’s start with the templates.

Item templates are actually built into FALL.EXE. The offset is a little different depending on your version, but the easy way to locate them is open a hex editor and search for “ruby”. You will find the following data:


Here are all the item templates laid out one after the other. They even follow a certain kind of logic, with gems, weapons, armor, etc. all more or less grouped together. Fortunately this isn’t exactly unknown data and the UESP came to the rescue again with a good starting point for these templates. I just had to fill in some blanks.

I didn’t want to keep this data in the .exe however, it’s much harder to modify these templates later. That’s why I exported the item templates to JSON format. Once exported the above data looks like this:

Much easier to work with. There are still a few unknowns to work out but those will be solved over time. The next problem was how to link up instantiated items like our Frosty %it of Ice Storms back to their original template. I had to reproduce the lookup table Daggerfall was using internally.

It was here Lypyl provided a helping hand thanks to his research into magical items and artifacts. The file format of MAGIC.DEF is very similar to instantiated items found in save games. Furthermore, the creators of old item editors had solved quite a few of these problems back then. Armed with all this, Lypyl could derive enough information to rebuild the group and item tables which he kindly provided to me in C# enums. All I had to do then was link the enums back to their template index in the above JSON file.

The main group enum looks like below. It corresponds to the first byte of the earlier category short.


For every element in the above enum (such as Armor, Weapons, etc.) there is an enum for every individual item in that group. For example:


For the item enum, the individual item value is an index back into the template table. The order within the enum corresponds to the second byte of the category short. With a helper class to bring all this together, it was now possible to perform lookups from instantiated items back to their template data. This is how our items viewer looks now:


Success! We can now resolve an item’s template by type to discover the full name and other useful information. The next step was to determine which items are equipped on the character. Fortunately the “equipped” table is just another record in your save game, and was already known about thanks to that first UESP article. I just had to work out how that table referenced items and I could isolate which were equipped. Items marked by an asterisk are equipped to character.


There are almost two dozen equipment slots in total that map to specific parts of the character’s body and elsewhere. I will describe this in more detail in a future article.

With all of that research out of the way, my next job was even less visual than above. I had to write support classes such as API helpers and an entity item collection class. I also required a new type of image reader to handle the job of loading and caching item images for the inventory UI, tinting them based on material, cutting out unique alpha indices like the hair mask, and so on. Anyway, boring or not these new classes form the foundation of items in Daggerfall Unity and will continue to grow as needed.

With everything finally in place, I could start building the equipment UI to sort, view, and equip items imported from classic Daggerfall save games. Besides a few UI enhancements and fixes, the following came together fairly quickly.

Some of the enhancements in this gfy include a scrollbar and mouse wheel scrolling. No reason we can’t have a few light modern touches to make our lives easier.

More About Scaling

One of the issues that caught me by surprise was a UI scaling problem. This turned out to be more interesting than expected, so I decided to write up a visual diary post about it.

For those who didn’t experience the problem, this is how it looked (1280×960).


What’s happening here isn’t technically a scaling problem, it’s a positioning problem. See, Daggerfall has a fixed-size UI of 320×200 pixels. My custom UI system was designed from the ground up to scale Daggerfall’s fixed UI to any resolution while maintaining correct aspect ratio. Depending on the width to height ratio of your resolution, the fixed UI may need to be pillarboxed (black bars down the sides) or letterboxed (black bars top and bottom). The goal is to fit the entire UI into the display without any clipped off edges and keeping that pixel-perfect ratio.

So what’s going in the screenshot above? The answer is that I forgot to turn on vertical alignment in the UI for the parent panel. This means the UI is stuck to the top of the screen instead of letterboxing like it’s supposed to. This is how the above display should look (1280×960).


The scaling and aspect ratio are correct in both cases, just the UI wasn’t centering vertically like it should for the letter-box effect at that resolution.

Unfortunately, it really isn’t possible to avoid pillarboxing or letterboxing with a fixed UI, unless you have a resolution that is an exact multiple of 320×200. For example, the screenshot below is a perfect x4 multiple of 320×200 and fits the frame completely at the correct aspect ratio (1280×800).


Now it occurs to me that some people simply don’t want letterboxing or pillarboxing. The best solution I can offer is a new option in the INI called “FreeScaling”. When this is enabled, the GUI will scale width and height independently. Here’s an example with FreeScaling enabled (1280×960).


The result is the UI is stretched as required to fill entire viewport. This obviously means the aspect ratio is no longer correct, but the chunky pixels still don’t look that bad with a little stretching. I’m willing to bet a lot of people actually play this way in DOSBox without noticing. It’s all down to personal preference anyway. If you want perfect aspect ratio, just leave things at default and the UI will scale and position itself properly now. If you definitely want to get rid of the black bars, then enable FreeScale and enjoy.

It’s also worth noting this does not apply to the game view rendering, which always fills the entire viewport. Only the classic 320×200 UI has this quirk.

I’m just happy my retro UI system is robust enough to handle all these different resolutions, scales, and positions while still working as it should. That’s an accomplishment by itself.

Generating Normals

Daggerfall Tools for Unity generates very large, complex procedural scenes entirely at runtime. If you haven’t seen it already, check out the mod showcase video to see just how large these environments are. Every texture, billboard, mesh, town, and dungeon are imported and converted procedurally from native DOS binary data at runtime.

While converting our material system over to the Standard shader for Unity5, I thought how great it would be to add normal maps to the procedurally generated scenes. Unfortunately, Daggerfall is such a classic game (polite way of saying very old) it doesn’t come with any normal maps built-in. And the requirement for this to happen at runtime added several challenges along the way. This journal entry details how I go about it.

The basic strategy of generating normal maps is to create a bump map from the colour image based on light and dark areas, then change the bump map into a normal map. It’s a simple idea in theory, but how does it look in practice? Turns out it looks pretty good, even helping the pixel art to “pop” a little.

Identical scene with and without normal maps

In the scene with normals, textures take on a bit more character and bumpiness, helping them to feel more like real surfaces than just textured polygons. While the effect will never be as great as hand-painted normal maps, the generated approach works surprisingly well with Daggerfall’s painterly, somewhat cartoony textures. So how does it work behind the scenes?

The magic all begins at import time with a matrix convolution filter in our ImageProcessing class. The first step is to run source textures through something called a sobel filter to find edges in the image. After a lot of experimentation, I settled on a two-pass (horizontal then vertical) sobel filter as this produced noticeably better results than one-pass filters. The passes are combined together to produce our final bump map.


Bump map shows texture gradients based on colour value

Thanks to the sobel filter, we now have a reasonable understanding of the gradient at each pixel. Brighter pixels have a steeper gradient than dark pixels. Armed with this information, we can derive our normals for any pixel by sampling the gradient of every pixel around it, then calculating the cross product from the gradiants. Here’s the code.

// Look up the heights to either side of this pixel
float left = GetIntensity(ref colors, x - 1, y, width, height);
float right = GetIntensity(ref colors, x + 1, y, width, height);
float top = GetIntensity(ref colors, x, y - 1, width, height);
float bottom = GetIntensity(ref colors, x, y + 1, width, height);

// Compute gradient vectors, then cross them to get the normal
Vector3 dx = new Vector3(1, 0, (right - left) * strength);
Vector3 dy = new Vector3(0, 1, (bottom - top) * strength);
Vector3 normal = Vector3.Cross(dx, dy);

We also need to write the colours back into the array. The normal is also inverted at this time. Thank you to Huknar on Reddit for the tip.

// This is a standard normal texture without Unity packing
newColors[y * width + x] = new Color32(
    (byte)((normal.x + 1.0f) * 127.5f),
    (byte)(255 - ((normal.y + 1.0f) * 127.5f)),
    (byte)((normal.z + 1.0f) * 127.5f),

And here is the resulting normal map shown as colours. Each pixel now communicates a little bit of 3D information for Unity’s graphics engine work with.

Normal map shown as 2D colours

Now there’s one more wrinkle we need to deal with. If you’re astute, you would have noticed something about “Unity packing” in that bit of code above. When importing textures in to Unity via the editor, you need to check a box so the engine knows to treat it as a normal map. Unfortunately for us, we’re importing textures procedurally and building materials on the fly – there’s no check box here. So how does Unity know it should treat our texture as a normal map?

Besides just sticking the texture into the _BumpMap parameter of the shader, we also need to repack the texture in the same way Unity does when you tick that box to import a texture as a normal map. Internally, Unity actually repacks normal maps from X,Y,Z,1 (or R,G,B,1) to Y,Y,Y,X (or G,G,G,R). We need to do the same thing for our normal map to be understood. This code replaces the last block above.

// Store result packed for Unity
byte r = (byte)((normal.x + 1.0f) * 127.5f);
byte g = (byte)(255 - ((normal.y + 1.0f) * 127.5f));
newColors[y * width + x] = new Color32(g, g, g, r);

The end result looks like below. It’s still a normal map, just one pre-packed for Unity’s shaders. You don’t normally see this in action as it all happens behind the scenes.

Normal map packed for Unity’s shaders

The final step is to create the material and assign the right texture maps to parameters in the shader. You also need to enable the keyword _NORMALMAP or Unity does not process the normals.

material.SetTexture("_BumpMap", normalMap);

GeneratedMaterialProcedural material with normal map generated at runtime

With everything put together, we get our scene with extra-bumpy textures thanks to the underlying normal information.

Split shaded and normal views from editor

Putting all of this together was a fun process. I learned a lot about image processing and about how Unity works under the hood. Best of all, you can now add normal maps to your procedural scenes in Daggerfall Tools for Unity by simply ticking a box. Easy.



Comparing normal strength of 0.1 vs 1.0

Text And Localization – First Look

Text, text, text! There’s so much text in Daggerfall – from “Rat just died” all the way up to multi-page books. Before it’s possible to tackle many gameplay elements, something must be done about all this text. All the better if the text engine provides localization features at the same time.

Rather than start from scratch, I have integrated the excellent free Unity asset Smart Localization into Daggerfall Tools for Unity. On this solid foundation, I’m building import tools to convert Daggerfall’s various text sources into a format that can be easily localized and consumed at runtime.

Let’s take a look at the basic workflow. Everything starts with creating a “Localization Workspace” in your project Assets. By the way, it’s possible to work on translations without Unity installed, more on this towards the end.


What you see above is the Smart Localization workspace. I have already created 3 languages for English – en, French – fr, and German – de. There is also a Root Language File, which needs a bit more explanation.

The Root Language is a database of native key/object pairs that can be morphed into other supported languages. In addition to strings, the Root Language also supports images, sounds, and other game objects.

I am writing one-click importers that transfer text data out of Daggerfall’s various files (even the .exe) into their own text namespace. See below for an example after importing the contents of TEXT.RSC.


There’s a fair bit going on here, so I’ll break down all the parts.

Type is the type of asset being worked with. These are all STRING for now, but GAME_OBJECT, AUDIO, and TEXTURE are also supported.

Key is the unique key for this resource. As we’re importing from Daggerfall, the tool maintains each unique key in its own namespace. RSC means that TEXT.RSC was the source and the following digits are the unique ID for that text record. Each distinct resource will have its own three-letter namespace (BOK, IMG, SND, etc.) and unique ID.

Comment is the actual value, or contents, of that record. Anyone that has worked with Daggerfall’s text data will be familiar with the % codes. What’s a little different are how the control bytes are parsed into text format, for example [0xFD] for end of line. This is done because the control bytes are not valid characters for XML (which is how everything is saved to disk) and they are not generally human readable. The [0x00] format makes it very clear where control bytes are used and simple for a human to edit. It’s also trivial to parse these back into byte format.

A little more on those control bytes for anyone unfamiliar with them. They are basically for TTY-styled formatting. They control things like text position, justification, and font, and are very specific to the native Daggerfall GUI at 320×200 resolution. It’s very likely that future developers will prefer not to use the control bytes for their modern scalable UI. However, I pass through the control bytes exactly as-is and its up to the end developer how they wish to use them. If you would like read more about Daggerfall’s text then this article would be of use.

Once the Root Lanuage is ready, it’s time to create our translations. As the text is already in English – en it’s just a simple matter of opening that language and copying all values from Root.


Now the great thing about Smart Localization is that it supports CSV import/export. This means you can export out your language data to CSV file to edit in Excel or similar tool.


Because everything is uniquely keyed, translation teams can build “master” CSV files that implementors can use to quickly add/update languages in their projects. For example, below is the CSV for German – de from their TEXT.RSC file.


And then imported into Unity ready to use in a game project.


One issue with existing translations is that localization teams have needed to squeeze more complicated character sets down into Daggerfall’s 8-bit format. The good news is this is no longer necessary – you are no longer limited by Daggerfall’s engine. The bad news is that a lot of text will probably need to be rewritten for true character set translations for modern platforms. I will work with international Daggerfall fans to ensure they can use native character sets in the translation tool chain.

So we can create a giant resource database and spin-off various translations. How does this get used in a game project?

Accessing localized resources is very easy. For example, to retrieve a localized text string above, you would use the following code. Other resources are just as easy.

string textValue = LanguageManager.Instance.GetTextValue("RSC.1004");

As the native IDs are preserved, it will be easy for upstream developers to call for the correct translations as linked from quests and elsewhere. You can also add custom values for your own project, just need to give them a unique key.

Fell free to hit me up in the comments or via email if you have any question.

Improved Terrain Tiling

To date, I have used a brute-force approach when tiling textures in cities and terrains. Each texture tile is mapped to a quad of vertices where each corner describes a UV coordinate allowing its texture to be drawn normally, rotated, flipped, or both rotated and flipped. This saves a great deal of texture space as only one version of texture is required to achieve four variations.

While it saves on texture memory (great for a 1996 game), this system has some inherent problems, not least of which is the number of vertices being used. For example, a full 8×8 block terrain has a total of 128×128 quad tiles with 4 vertices each. That’s a total of 65536 vertices just to handle the ground plane of a big city like Daggerfall or Wayrest. This is larger than Unity’s maximum mesh size, so everything needs to be split up into chunks. As cities are completely flat, wouldn’t it be great if we could just render all these tiles onto a single quad, using only 4 vertices for an entire city?

Happily, this is not only possible, we can solve a few other problems with tiled textures in the process. All through the magic of custom shaders.

The first step is to change how the ground atlas is stored. Rather than just reference a single texture and modify using UVs, we’re going to pre-transform each texture in all 4 possible configurations (normal, rotated, flipped, rotated+flipped). This removes the need for UV changes and saves time in the fragment shader, and allows for some extra packing to happen as you’ll see shortly. The new tile set atlas looks like below (note that Unity texture [0,0] is bottom-left).


In order to reference these tiles in the fragment shader, we also create a tile map texture. This texture packs an index into the red channel that is basically a number 0-255 corresponding to a tile in the atlas above. This texture must be point-sampled to ensure no corruption of indices occurs. The tile map for Daggerfall city is below.


Now the shader has an atlas of tiles and indices referencing which tile to use. The following bit of code will read index from tile map then render out correct texture tile from the atlas.

int index = tex2D(_TilemapTex, i.texcoord0).x * _MaxIndex;
int xpos = index % _TilesetDim;
int ypos = index / _TilesetDim;
float2 uv = float2(xpos, ypos) / _TilesetDim;
float xoffset = frac(i.texcoord0.x * _TilemapDim) / _TilesetDim;
float yoffset = frac(i.texcoord0.y * _TilemapDim) / _TilesetDim;
uv += float2(xoffset, yoffset);
return tex2D(_MainTex, uv);

The result is a fully tiled city quad using just 4 vertices.


This works great, and it’s fast. If we were just using point filtering with no mipmaps, our job would be done already. Unfortunately linear filtering adds one set of problems and mipmaps add another. The first problem is texture colours bleeding in from adjacent tiles due to how linear filtering works. Image below demonstrates problem, note the dirt texture being sample at edge of water.


There are a few of ways to fix this, but one of the most robust is to manually wrap textures around to their opposite side in the atlas. This means that when the linear filter samples outside the tile area, texture wrapping will be simulated. The new atlas shows this concept with each tile wrapped 50% on all sides. Note: This solution works for tiling textures, but not for non-tiling textures where texture clamping needs to be simulated instead. The atlas generator is yet to support both wrapped and clamped tiles.


With a new atlas format, our fragment shader needs updating to sample interior part of each texture.

// Get offset to tile in atlas
int index = tex2D(_TilemapTex, i.texcoord0).x * _MaxIndex;
int xpos = index % _TilesetDim;
int ypos = index / _TilesetDim;
float2 uv = float2(xpos, ypos) / _TilesetDim;

// Offset to fragment position inside tile
float xoffset = frac(i.texcoord0.x * _TilemapDim) / _GutterSize;
float yoffset = frac(i.texcoord0.y * _TilemapDim) / _GutterSize;
uv += float2(xoffset, yoffset) + _GutterSize / _AtlasSize;

// Return fragment
return tex2D(_MainTex, uv);

These additions fix linear sampling to all but remove bleeding for tiling textures. Compare this image to the earlier version.


With texture bleeding taken care of, the next challenge is dealing with mipmaps. The graphics card has no understanding of our atlas, so tile wrapping makes it sample from the wrong mip level, which creates a whole new problem with cracks as shown below (you might need to open image to full size to see the cracking discussed).


Fixing this problem requires manually calculating the mip level in fragment shader. Fortunately there’s a lot of references online for dealing with this. Rather than just naively return the texture sample using tex2d, we need to use tex2dlod instead. Below is change to final line of code.

// Manually set mip level and return fragment
float mipLevel = GetMipLevel(i.texcoord0, _AtlasSize);
return tex2Dlod(_MainTex, float4(uv.xy, 0, mipLevel));

And the GetMipLevel() function.

float GetMipLevel(float2 iUV, float2 iTextureSize)
	float2 dx = ddx(iUV * iTextureSize.x);
	float2 dy = ddy(iUV * iTextureSize.y);
	float d = max(dot(dx, dx), dot(dy,dy));
	return 0.5 * log2(d);

With this new code, our mip cracking problem has vanished.


And there we have it. A super-fast method of tile mapping from an atlas which defeats both texture bleeding and mip sampling problems, and only needs a single quad mesh.

There’s still more work to do, mainly around atlas generation. I need to handle wrapped tiles and clamped tiles differently, and manually pack mip levels to improve overall quality. I will get back to this in the new year with my performance review. On the subject of performance, these concepts can be expanded to reduce draw calls for the terrain system and city environments. This is just the first spoke in a big wheel of optimizations.

This last screenshot is of the material inside Unity editor, which shows how everything hangs together.



I could not have pulled this off without the help of several people much smarter than myself. All the links below were used in formulating this solution.

Connor Hollis – Fast Tilemap Shader

IceFall Games – Terrain Engine

nVidia – Improve Batching Using Texture Atlasing

0 FPS – Texture atlases wrapping and mip-mapping

Streaming World – Part 2

In the first part of this series, we looked at how the world in Daggerfall is constructed from multiple 1000×500 pixel maps, and how procedural techniques can be applied to add more fine detail to Daggerfall’s height map.

This article shows how continuous height map data generated by world readers is injected into scene as continuous meshed terrain and how terrain is landscaped into flat areas for cities.

First up, I have chosen not to use Unity’s native terrain system. There are several reasons for this, but chiefly is that Daggerfall terrain textures work very differently from Unity. In Unity (like most modern 3D engines) the terrain is textured using a splat map – a special texture which determines how detail textures are to be blended at every vertex. Daggerfall on the other hand uses a grid of quads textured from a selection of 56 tiles. These tiles are earth, sand, grass, stone, and road with hand-painted transitions between most states.


Above is the temperate terrain set as viewed from Daggerfall Imaging 2. There is one complete set of 56 tiles per climate base (desert, temperate, mountain, swamp) with one variant for snow and another for rain. To save texture space, these tiles also use UV modification to be rotated, flipped, or both rotated and flipped where required.

In order for the terrain to blend seamlessly from wilderness into city tiles, it becomes necessary to build a custom terrain system which understands how Daggerfall works. This unfortunately excludes the default Unity terrain. The code is modular however, so you can always replace part or all of my terrain system with something else.

With the decision made to create a Daggerfall-like terrain system, the next stage is to transform raw height data into streaming mesh samples and format ground for locations.

The first step of this journey was actually implemented a few weeks back in the 1.1 update. See article Time & Space and More for details about PlayerGPS and WorldTime components. Suffice to say, the toolset already has a good understanding of where the player is in the world and what locations are nearby. I will copy an image from the time and space article here as it pertains to world streaming.


Above is a snapshot of the world in pure numerical form. Each of these tiles represents a Map Pixel (discussed in part 1), which is the size of a full-sized city. The player is standing at the origin of Daggerfall city at map pixel 207, 213. In the immediate vicinity are Ripwych Commons to the north-west (206, 212) and Copperfield Manor to the south (207, 214). Keep an eye on them, as we’ll be seeing more of them soon.

At a high level, the StreamingWorld component stores data much like the above. It keeps track of a small bubble of world space around player in a 2D array. As the player moves, tiles are shifted up, down, left, or right in the array (based on direction player is moving). New map pixels are loaded as required and everything is occasionally snapped back to near origin, which avoids precision errors at extreme limits of the map. From the player’s perspective, they are walking endlessly into the distance. In reality, they are on a terrain treadmill.

Inside each of these map pixels a DaggerfallTerrain object is created for the entire area. This is generated procedurally from Daggerfall’s height map combined with smooth noise to add small details. This terrain is then broken up into several DaggerfallTerrainChunk objects, which represent the actual mesh objects and colliders for the player to see and walk on.

The edge vertices and normals of each adjacent terrain are stitched together to create a nice continuous terrain. The next screenshot shows a 3×3 terrain area, once again centred on Daggerfall, with the tiling texture system discussed above. Right now it is just set to grass. I also have turned up the noise scale to make the deformations a little more interesting.


The locations are there in memory, but nothing is being drawn yet. Before we can insert locations, some foundations must be laid. First come the appropriate texture tiles. Unlike the earlier screenshot, the locations are centred inside their map pixel. Daggerfall city is in the middle, Copperfield Manor is to the south, and Ripwych Commons is just barely visible to the north-west.



Now location textures are in the right place, but there’s a big problem – cities in Daggerfall need a flat plane to sit on. This raises two questions. What level should the terrain be flattened to and how can everything be smoothed out cleanly?

I evaluated several different methods for selecting city level. The below screenshot shows a few of them without any smoothing.


I didn’t like the highest point as it made everything feel too raised. The average is pretty good as the city is leveled evenly in between all deformations, but I rejected it also as another pass was required to determine average elevation after applying noise. World height was very cheap to obtain but sometimes felt a little too high or too low. I ended up settling on median bilinear height (from the map reader), which is also very cheap and consistently landed somewhere between the world height and true average. So that’s the level I went with.

The next stage was to blend the flat areas into the random terrain. Interestingly, even full-sized cities like Daggerfall do not really fill their entire map pixel. They have a band of 14 tiles around the outside to use as blend space. These even have special index (>55) to indicate not to use RMB-defined tiles and instead blend with terrain.

The blending process gave me serious headaches for a few days. I tried several different methods of blending heights but none of them gave me a perfect transition from the rectangular city area all the way to edge of terrain. In the end, I created a system of building scale maps with linear rolloff from edge of city to edge of terrain. This uses linear interpolation for the straight rects and bilinear in the corner rects to keep everything smooth.

What is important is the scale is 0 all the way around terrain bounds and smoothly reaches 1 at the location bounds. Drawing the scale map for Copperfield Manor looks like the following. Each of the points below represents the blend weight for random terrain vs level terrain. The further out from the centre rect, the more influence random terrain will have. It looks similar to a pure radial falloff, but if you look carefully you will notice everything converges on a rectangular area, not a single point as would be the case with radial.


Applying the scale map gives us a nice smooth transition from flat area into random terrain. The terrain features can still express themselves, but the transition feels natural and remains completely walkable even with lots of noise.


From here, it’s a simple matter of plopping down locations where needed and let the player go exploring.

The next part of this series will be about enhancing details of terrain. I will discuss adding textures outside of cities, improving elevation noise, and adding trees and other wilderness flats.