Items Part 2 – Dyes

Before moving on to equipping items, I thought it would be fun to show off the equipment dye system in Daggerfall Unity.

Classic Daggerfall uses a 320*200 pixel 256-colour display – or more specifically a Mode 13h display. Back in this era, bitmap graphics were typically a width*height byte array of indices into a 256-colour RGB palette. One of the coolest tricks available to graphics programmers at the time was to change index ranges to substitute colours, change brightness, animate textures, and so on. Daggerfall uses index changes and palette swaps to accomplish all of these tricks and then some – it would be possible to write a series on that subject alone. This article is just about changing dyes to re-colour weapons, armour, clothing, and how the old index swaps can be realised in a true RGBA renderer like Unity.

If all of that is difficult to visualise, let’s start with an example. Here’s a pair of basic boots without any changes.

dyes1

Every pixel above is just a single byte index into a 256-colour palette. For example, index 0x70 points to RGB #DCDCDC in the default texture palette for a very light grey. For weapons and armour, the 16 indices 0x70 to 0x7F are reserved for index swaps (clothing reserves indices 0x60-0x6F). In the case of these boots every pixel falls between 0x70-0x7F, but that isn’t true of all items. Sometimes only a small part of the image will support dyes. If we just substitute every index between 0x70-0x7F to another random index between 0x00-0xFF we get the below.

dyes2

Quite the mess, but it demonstrates that changing indices can radically change the appearance of an indexed bitmap. The important thing to keep in mind is that every pixel is not by itself a colour. Rather its just an index pointing to a colour.

The first challenge in bringing indexed colours into Unity is that every time we read in a Daggerfall bitmap it must be converted to true 32-bit RGBA values where every pixel actually is a specific colour. Fortunately converting to 32-bit RGBA in Unity isn’t difficult. The general process is:

  1. Allocate a Color32 array with the same number of elements as width*height of source bitmap.
  2. For every pixel index in source bitmap, sample the RGB colour of that index to a Color32 value.
  3. Write colour sampled from palette into correct position in Color32 array.
  4. Create a new Texture2D of same width*height as source bitmap.
  5. Promote Color32 array to our Texture2D using SetPixels32() and Apply().
  6. Use this Texture2D as needed.

When it comes to changing the dyes, all that’s required is to substitute the correct indices in step 2 before sampling palette. So where do these colour swaps come from and how does Daggerfall know which swaps to use for what items? Daggerfall actually has a couple of different methods for generating swaps. Let’s start with weapons and armour.

Buried inside Daggerfall’s executable FALL.EXE at offset 0x1BD1E2 (for DaggerfallSetup version) are the metal swap tables. There is one 16-byte swap table per metal type. For example, when encountering index 0x70 for a Daedric pair of boots, replace 0x70 with swap index found at daedricSwapTable[0]. For index 0x71 replace with index found at daedricSwapTable[1]. And so on. These swaps have been known about for some time and you can find more details on this archived page from the wonderful old Svatopluk site.

Clothing does not appear to use pre-defined tables like metals. Rather, each swap table is just 16x sequential indices. For example, purple is 0x30-0x3F and green is 0xA0-0xAF. Swap tables can be generated on the fly using a dye enum mapped to starting index. Daggerfall appears to do this as these sequences are not found in the executable like metal swap tables.

Armed with the power to create textures and swap indices, we can now generate our final boots image based on metal type. here are some examples.

dyes3

Orcish

dyes4

Dwarven

dyes5

Daedric

 

One benefit of using the same generic process for metals and clothing is that it becomes possible to use clothing dyes on armour, something Daggerfall can probably do but doesn’t make available to players. This could allow for dye station mods down the road for players to further customise their equipment. With Unity using a true 32-bit palette this could extend well beyond Daggerfall’s 256-colours. Anyway, for an example of armour dyed something different:

dyes6

Blue Chain Boots

 

The next challenge now that we’re using true 32-bit textures is a red pair of boots becomes a completely different texture to a green pair of boots. Whereas in Daggerfall the same bitmap can be used both times by just changing indices as described above in the software blitting function. Between this and the inherent (but minor) performance impact of converting indexed bitmaps to Texture2D, we need some way of minimising CPU time and garbage creation. Caching to the rescue.

Daggerfall Tools for Unity (the underlying API suite) already uses texture caching for general world materials, but items have their own set of problems to solve. To this end, I created a new item helper class to serve up equipment icons and handle the caching based on properties unique to items.

Every time an equipment icon is requested, a unique 32-bit key is generated by packing that request’s variables into a bitfield. The packing looks like below.

dye-key-bitfield

  • Colour enum index refers to the swap table in use. This value matches Daggerfall’s own colour enum stored within base item templates.
  • Variant index is an alternate image for this item.
  • Archive index is the texture file number (e.g. TEXTURE.245) containing the icon.
  • Record index is the icon index within the texture archive.
  • Mask bit is used to enable/disable a special mask used to overwrite pixels like hair around helmets. More on this in a later post.
  • There are a few reserved bits to grow the key system later.

It’s worth pointing out the end programmer doesn’t need to worry about how these values are packed. This all happens automatically under the hood when calling GetItemImage(). What matters is the API has a way of uniquely identifying any individual equipment icon based on its display properties.

When calling GetItemImage() the API will first check cache to see if this exact icon has already been converted from Daggerfall’s native file formats. If not, it is converted and stored in the cache for the next time its needed.

To wrap things up, here’s a new gfy showing a variety of dyed items in the inventory UI.

[gfycat data_id=”WetUncomfortableBetafish”]

For regular micro-updates on Daggerfall Unity, I can be found on Twitter @gav_clayton.

Items Part 1 – Bootstrapping

Loot. Kit. Swag. Treasure. Whatever you call it, items are an important part of any RPG game loop. They provide the means for your character to defeat ever more powerful foes and create incentive to keep playing in search of the next big upgrade. While Daggerfall’s items don’t quite tickle the reward centres of the brain like Diablo 3 or World of Warcraft, they’re still a vital part of the play experience. Without decent gear and enchantments, you’re unlikely to survive the grueling ordeal of Mantellan Crux.

In this series, I’ll describe the process of adding items to Daggerfall Unity. I wanted to approach items early on as they will be involved at almost every level of the game. Shops sell them, blacksmiths repair them, monsters drop them, quests reward them. Your character may have a special affinity for bladed weapons, or be forbidden the use of shields. Even the biography questions when building a character can grant you items like the near-essential Ebony Dagger. With items embedded in almost every major game system, the hardest part was working out where to begin.

I decided to start with existing items as part of importing classic Daggerfall saves then bootstrap the whole item back-end from there. That way I could be certain I was dealing with the most real-world data possible. Having built support for classic saves in 0.1, I could already identify item records parented to the main character record and visualise them with a custom Unity Editor script. They looked a bit like this at first:

 

Items1

It’s not much, but at least I could find item records belonging to the character. The “Container” record is just a generic parent record. In this context, think of it as the character’s backpack.

The next step was to break apart the item record format. Fortunately the UESP came to the rescue here with most of the bytes already solved, but far from the whole story as you’ll see once the names are revealed:

Items2

A Frosty what of Ice Storms? OK, so there’s more to this than just the save record. How to we go about filling in the blanks? The key here is the “category” 16-bit field in that UESP article. This is actually a pair of 8-bit values. The first byte is the item group, the second byte is a table lookup for the item template within that group. The template indexed by this lookup has all the missing pieces of information we need to complete our item data. Now we have two more problems to solve. Where are the templates, and how to use those category bytes to find them? Let’s start with the templates.

Item templates are actually built into FALL.EXE. The offset is a little different depending on your version, but the easy way to locate them is open a hex editor and search for “ruby”. You will find the following data:

Items3

Here are all the item templates laid out one after the other. They even follow a certain kind of logic, with gems, weapons, armor, etc. all more or less grouped together. Fortunately this isn’t exactly unknown data and the UESP came to the rescue again with a good starting point for these templates. I just had to fill in some blanks.

I didn’t want to keep this data in the .exe however, it’s much harder to modify these templates later. That’s why I exported the item templates to JSON format. Once exported the above data looks like this:

Items4
Much easier to work with. There are still a few unknowns to work out but those will be solved over time. The next problem was how to link up instantiated items like our Frosty %it of Ice Storms back to their original template. I had to reproduce the lookup table Daggerfall was using internally.

It was here Lypyl provided a helping hand thanks to his research into magical items and artifacts. The file format of MAGIC.DEF is very similar to instantiated items found in save games. Furthermore, the creators of old item editors had solved quite a few of these problems back then. Armed with all this, Lypyl could derive enough information to rebuild the group and item tables which he kindly provided to me in C# enums. All I had to do then was link the enums back to their template index in the above JSON file.

The main group enum looks like below. It corresponds to the first byte of the earlier category short.

Items5

For every element in the above enum (such as Armor, Weapons, etc.) there is an enum for every individual item in that group. For example:

Items6

For the item enum, the individual item value is an index back into the template table. The order within the enum corresponds to the second byte of the category short. With a helper class to bring all this together, it was now possible to perform lookups from instantiated items back to their template data. This is how our items viewer looks now:

Items7

Success! We can now resolve an item’s template by type to discover the full name and other useful information. The next step was to determine which items are equipped on the character. Fortunately the “equipped” table is just another record in your save game, and was already known about thanks to that first UESP article. I just had to work out how that table referenced items and I could isolate which were equipped. Items marked by an asterisk are equipped to character.

Items8

There are almost two dozen equipment slots in total that map to specific parts of the character’s body and elsewhere. I will describe this in more detail in a future article.

With all of that research out of the way, my next job was even less visual than above. I had to write support classes such as API helpers and an entity item collection class. I also required a new type of image reader to handle the job of loading and caching item images for the inventory UI, tinting them based on material, cutting out unique alpha indices like the hair mask, and so on. Anyway, boring or not these new classes form the foundation of items in Daggerfall Unity and will continue to grow as needed.

With everything finally in place, I could start building the equipment UI to sort, view, and equip items imported from classic Daggerfall save games. Besides a few UI enhancements and fixes, the following came together fairly quickly.

[gfycat data_id=”PeacefulReflectingBarebirdbat”]

Some of the enhancements in this gfy include a scrollbar and mouse wheel scrolling. No reason we can’t have a few light modern touches to make our lives easier.

Back On Deck For 2016

Happy New Year everyone! I’m back from holidays and almost on top of my RL workload again. That means a whole new round of updates to Daggerfall Unity and DFTFU are about to begin. I’ve picked up where I left off last year with the item and inventory system, and will be posting more on this shortly.

Sometime in the next few weeks, I’ll start adding new test builds leading up to the 0.2 release. Key features of 0.2 will be:

  • Basic inventory system and loot tables. Import items from classic saves.
  • Travel map interface (by LypyL).
  • Dungeon and interior automap interface (by Nystul).
  • More bug fixes and incremental updates.
  • Some more community resources for contributors.

I’ve also made a small new year’s resolution to post more technically-minded articles in 2016, as I let this slip with all the rapid-fire updates leading to 0.1. It was quite a shift for me going from pure tool development to building a game, and I rather miss just talking about what I’m working on.

Thanks for all your patience during the holiday season. I look forward to reading your feedback with the next round of updates.

Daggerfall Unity 0.1 Release

DaggerfallUnity0.1Release

 

After several test iterations, Daggerfall Unity is feeling reasonably solid across the two main platforms Windows and Linux. A huge thanks to all testers who helped discover early bugs and quirks. There’s obviously still more to fix, and a lot of game features to implement, but we have a pretty good starting point with this build. At some point, I need to push a stick into the ground and say “it starts here.”

 

It Starts Here

Daggerfall Unity 0.1 is now available for general download. Key features of this build are:

  • Create a new character or load an existing character from a classic Daggerfall save.
  • The entire world of the Illiac Bay is ready to explore at 1:1 scale to Daggerfall itself. Go anywhere, enter any dungeon, and explore any town.
  • Quick-save and quick-load your progress.
  • Dungeons are populated with fixed and random enemies just like in Daggerfall.
  • Basic combat mechanics with the ebony dagger.
  • Game console for enabling god mode, setting run speed, teleporting around world, etc.
  • Lypyl’s enhanced sky mod with dynamic skybox, procedural clouds, and even phases of the moon!
  • Nystul’s beautiful far terrain mod and improved terrain generation.

 

Download

Updated to 0.1.1.

[ddownload id=”2298″ text=”Download Daggerfall Unity 0.1.1 (Windows)”]

[ddownload id=”2299″ text=”Download Daggerfall Unity 0.1.1 (Linux)”]

 

Manual

A PDF manual is included with the download, but you can also download a standalone copy.

 

Feedback

If you would like to offer feedback and bug reports, please use this thread on the forums or contact me directly.

 

What’s Next

I will be a little quiet the next few weeks due to current work commitments, but rest assured the next release is already being worked on. I will soon repeat the process of dropping test builds leading up to the next release milestone. Here’s a list of features scheduled for the next release.

  • Early item support! Inventory will be imported from classic saves.
  • Ability to open inventory window and equip items.
  • Loot enemy corpses and treasure piles.
  • Full save/load UI allowing for multiple saves.
  • In-game options UI for changing game settings.
  • The ability to rest to recover health.
  • More contributor mods.
  • More bug fixes.

There’s a lot of open source activity happening at the moment from contributors, so it’s possible we might end up with more features in the second release than listed above. I will reveal these on Twitter as they become ready to show off.

Daggerfall Unity Test Build 3

DaggerfallUnityTest3Splash

Download

[ddownload id=”2287″ text=”Download Daggerfall Unity 0.0.6 (Windows)”]

[ddownload id=”2288″ text=”Download Daggerfall Unity 0.0.6 (Linux)”]

Please also download the ReadMe!

 

Patch Notes

  • Changed how genders are imported from classic saves based on new idea that gender byte is actually a bitfield and only first bit controls gender (0 for male, 1 for female). This should allow more classic saves to open. Requires more testing, please report any further issues with gender-swapped characters. Vampire/werewolf characters are still not considering working.
  • Fixed issue where player would appear in wrong exterior cell after loading game to an interior cell.
  • Fixed issue where saved player position would be incorrect after changing terrain sampler (e.g. enabling/disabling Nystul’s improved terrain then loading a saved game). When a change to terrain sampler is detected, player will be relocated to outside current location, or to origin of map cell if no location present. This logic may trigger when loading a quick-save created prior to 0.0.6.
  • Fixed another casing issue when lazy-loading saves in Linux for save importer.
  • Some buildings do not have an interior. These will now display the old chestnut “This house has nothing of value.” rather than throw an exception.
  • Class-based enemies (e.g. Spellsword, Thief) now have a health pool and can hurt you. Look out!

 

Controls

General

  • Mouse to look.
  • W, S, A, D to move.
  • SHIFT (hold) to run.
  • SPACE to jump.
  • LEFT-CLICK mouse to open doors, enter dungeons, operate switches, etc.
  • ESC to pause game or go back to previous window.
  • F5 to open Character Sheet.

Weapons

  • Z to toggle weapon.
  • RIGHT-CLICK and drag mouse to swing weapon.

Save/Load

  • F9 to quick-save.
  • F12 to quick-load.

 

Feedback

Preferred method of feedback is the following thread on the forums.

http://forums.dfworkshop.net/viewtopic.php?f=18&t=161

 

When reporting bugs, please include the following in your report.

  • A clear description of what went wrong and process to reproduce problem (if possible).
  • Main system specs (OS, CPU, RAM, GPU).
  • Any output logs or save files as requested. See ReadMe for more information.

 

Thankyou!

And once again, a big thankyou to all contributors, testers, and supporters of this project. You are each helping to create something wonderful.

Daggerfall Unity Test Build 2

DaggerfallUnityTest2Splash

Download

Updated to 0.0.5.

[ddownload id=”2276″ text=”Download Daggerfall Unity 0.0.5 (Windows)”]

[ddownload id=”2277″ text=”Download Daggerfall Unity 0.0.5 (Linux)”]

Please also download the ReadMe!

 

Controls

General

  • Mouse to look.
  • W, S, A, D to move.
  • SHIFT (hold) to run.
  • SPACE to jump.
  • LEFT-CLICK mouse to open doors, enter dungeons, operate switches, etc.
  • ESC to pause game or go back to previous window.
  • F5 to open Character Sheet.

Weapons

  • Z to toggle weapon.
  • RIGHT-CLICK and drag mouse to swing weapon.

Save/Load

  • F9 to quick-save.
  • F12 to quick-load.

 

Feedback

Preferred method of feedback is the following thread on the forums.

http://forums.dfworkshop.net/viewtopic.php?f=18&t=154

 

When reporting bugs, please include the following in your report.

  • A clear description of what went wrong and process to reproduce problem (if possible).
  • Main system specs (OS, CPU, RAM, GPU).
  • Any output logs or save files as requested. See ReadMe for more information.

 

Thankyou!

And once again, a big thankyou to all contributors, testers, and supporters of this project. You are each helping to create something wonderful.