December 2016 Test Build

Hello everyone! This will be my last post for 2016. It’s been a great year for Daggerfall Unity with solid updates across the board. I was hoping to have basic quest support in by now, but sadly not everything goes to plan. This will return as a priority early in 2017 so watch this space!

One thing that always amazes me is the quality of work this community is willing to put back into Daggerfall Unity. It’s always a pleasure to find a new pull request on git from someone willing to contribute their personal time to make this project even better. So this post is going to focus almost exclusively on contributions from community members. It’s time for kudos and credits all around, and you’ll see very little of me this post.

 

Texture & Mesh Replacement

This feature began with Uncanny_Valley and has lately been updated and maintained by TheLacus. It allows for runtime injection of new textures and meshes into Daggerfall Unity’s scene builders, setting the stage for updated models, higher resolution materials, and improvements to Daggerfall’s vanilla UI. It’s still early days but the potential is incredible. Here’s a few screenshots of new assets by community members.

 

At time of writing, mesh and texture replacements aren’t quite ready for download. But now support for this is baked into the core, you should start seeing community-created packs in the near future. You can read more about mesh and texture replacement in this thread on the forums.

 

Early Bow Combat

New contributor electrorobobody added basic bow combat to the lineup of supported weapon animation. No counting arrows yet, and you’ll need to roll a new character for your free silver bow, but it’s awesome to finally burn down enemies with ranged kiting. Looking forward to bows becoming a strong part of the game in future.

 

Save & Load Weather

Daggerfall Unity added basic weather events a while back, but they would not be saved and loaded with your games. Thanks to midopa, the current state of weather will be saved and loaded. This will only get better once correct weather events are wired up based on climate and season.

 

Enemy Steering

Another epic update by midopa. He added a little steering to enemy AI to prevent enemies from stacking on top of small creatures like rats. As hilarious as this problem could be, it’s good to see a workable solution for this bug. It’s still possible for enemies to slightly stack in edge cases, but the problem is much improved and they will no longer ride around on each other (imagine skeletons surfing rats and rat-rat-rat stacks).

 

Potion Recipes

The perennial InconsolableCellist returned with some amazing updates for us. Credit goes completely to him for working out potion recipe format and integrating with Daggerfall Unity. This also means potion recipes will display properly in inventory, and they’re even usable to see the individual ingredients. This is really important ground-work for a bunch of other things down the line.

 

Books

I wrote the initial book reader UI ages ago, but InconsolableCellist wrapped it up along with random book drops in loot, correct tooltips, and all-round awesomeness. Books currently exhibit the same formatting problems as classic (because it’s the same book data). That’s something yet to be fixed.

 

Exterior Automap

Nystul has done it again with the perfect companion UI to his dungeon and interior automap. Yep, exterior automaps are now a thing! It even supports proper tagging of buildings, zoom, and rotation. As always, I’m completely blown away by how complete this is right from the start. It’s still waiting on full building name integration and building identification in scene, but that will come. For now, all the buildings are tagged by type. Go explore!

 

Spotlight: Allofich

I can’t give Allofich high enough praise. He has worked incredibly hard tuning up different areas of Daggerfall Unity to make it more true to the original. He fixed a wide range of UI problems, identified sound effects, linked sounds to their correct actions, fixed clothing and item problems, and so on. Check out his full list of commits here. It’s hard to show these off properly in screenshot form because the changes are either subtle improvements or related to audio, but below is one of the UIs he has cleaned up. Note the poor texture joins in the “before” image (circled). Huge props to Allofich for his work!

 

Thanks To: AnKor

OK this is embarrassing. I was looking everywhere for the animations used when player was riding horse or cart. These turned out to be in the overlooked CFA image file format. Somehow, I had completely disregarded these files which are a format holdover from Arena. Yep, I’m only human. Fortunately, AnKor pointed this out on the forums and I was able to implement CFA support in no time. Now we have this:


It’s only a short jump from here to having these transport options in the game.

 

Thanks To: Testers

I also want to send out a huge thanks to all the amazing people who tested Daggerfall Unity in 2016 and reported the bugs and problems you found. There are simply too many people to list, but you know who you are. You’re on the forums, and Twitter, and Reddit, and sending me emails. You guys rock!

 

Bugs and Problems

Yep, we got ’em! Any large update like this will bring its fair share of new bugs. If you come across a bug during tests and would like to report it, please lodge this in the Bug Reports forum. Don’t forget to read the Guidelines to help you provide the best information to developers.

Contributors, please keep an eye on the Bug Reports forum for anything that might fall into your wheelhouse.

 

Where To Download?

You can always download the latest version of Daggerfall Unity from the Live Builds page. If this is your first time downloading Daggerfall Unity, welcome! Other information on Live Builds page should also help you get started. If you have any troubles, or just want to discuss updates, please go to the December 2016 Test Builds Updated thread on forums.

 

That’s it for 2016! Thank you everyone for visiting and all your kind words of support. Here’s wishing you all a very Merry Christmas and Happy New Year, and all the best for 2017!

 

For more frequent updates on Daggerfall Unity, follow me on Twitter @gav_clayton.

Tutorial – Getting Started

The latest version of Daggerfall Tools for Unity is now available. You can either clone the full project from GitHub or download the standalone asset package from the Daggerfall Tools for Unity project page.

This version of DFTFU might need to be updated a few times as tutorials are rolled out. The current version is 1.6.1, please grab later version if available.

I have also posted the updated Getting Started tutorial to the DFTFU Tutorials page. Image link below will take you straight to tutorial.

GettingStartedTutorialImage

Click image to open tutorial

Upcoming Release 0.3

The 0.3 point release is coming together and should be available sometime late in July. Here’s a quick summary of upcoming features.

 

Modding Support

Lypyl’s mod framework is undoubtedly the star of 0.3. You can read his post about it and check out a few work-in-progress tutorials on the forums. While still early days, I couldn’t be happier with this feature and the potential it brings to Daggerfall Unity.

  • Mods can be created using Unity Personal (free version) and Daggerfall Tools for Unity.
  • Completed mods are packaged to a standalone .dfmod file (asset bundle) for distribution.
  • Integrated mod loader at startup with ability to change load order.
  • Fully integrated run-time C# compiler.
  • Total access to inner workings of Daggerfall Unity.
  • Catch events, display UI windows, spawn world objects, drive game logic.

The mod system is already powerful enough to handle the current round of mods, which will eventually be migrated into .dfmod format. As mod creators grow in experience and the underlying code is expanded to provide more options almost anything will be possible down the road.

 

Treasure & Loot

Random treasure piles and corpse markers will now be lootable, providing gold and new items to you during testing.

  • Player will now find gold and items in random treasure piles and on the bodies of slain foes.
  • Generated items will not have magical powers until the spells & effects features are live.
  • It will be possible to drop items to the ground, but like Daggerfall dropped items will disappear when you leave the area. Non-volatile player storage will be implemented much later as part of housing.

As shops are not implemented yet, I’m going to ignore weight limits on the player and the wagon so you can carry as much as you want. Proper encumbrance tracking will be added in the future after shops come online.

 

New UI Windows

A few UI windows are in the works for 0.3, although not all of them will be ready for initial release. They will come online over the 0.3 cycle.

  • Control mapping UI.
  • Rest UI.
  • Save/Load UI.

As part of updating save/load UI, the way games are saved will be expanded during 0.3 to accommodate the growing amount of data needed to support saving game state.

 

Standalone Builds

From 0.3, I’m going to provide a standalone build of Daggerfall Unity with game files bundled. This will be in addition to the smaller builds where you must provide your own game files. This change is to help users who just wish to quickly try out Daggerfall Unity or have trouble installing a compatible version for any reason.

  • Standalone builds will be substantially larger and updated less often than the trimmed-down test builds.
  • Test builds will be smaller and require you to provide your own set of compatible game files. This is still the best download for dedicated testers.

 

Bug Fixes & Small Improvements

Last but by no means least will be the usual round of small fixes and improvements to features already added. There’s also a bit going on behind the scenes to support future systems like spells & effects, NPCs, factions, shops, and questing. These additions will be slowly rolled out as more features come online post 0.3.

For more frequent micro updates and news, follow me on Twitter @gav_clayton.

Modding Support

Interkarma has asked me to write a quick overview about the upcoming modding feature that will be in version 0.3 that I’ve been working on.

Currently mods need to be built into a Daggerfall Unity project to work, which is an obvious limitation.  Interkarma has gone out of his way to support a bunch of different mods, even including them into Daggerfall Unity releases, but this takes up a lot of his time and has always been a temporary solution.

The new modding system takes advantage of Unity’s asset bundles, and allows mod creators to export the assets that make up their mods (textures, models, c# scripts, prefabs and so on) from Unity to a single file which is then loaded at runtime.

Exporting to a mod file is easy and done with a simple window inside the Unity editor:

editorwindow

 

The mod files are detected and loaded during the Startup Scene:

modlist

This means that going forward, modders will no longer have to rely on their mods being part of Daggerfall Unity to work.  Modders will be able to create, share, and release updates for their mods all on their own.  And using mods will require little more than the player placing the mod file in a directory.

We’ll be providing more details on how it works, including examples and tutorials in the near future.  Stay tuned!

Faction Support

Just a quick post today. I have implemented the faction back-end for Daggerfall Unity, which is a key pillar of quests and NPC dialog. Here’s the data shown as a flat list in the Unity editor. All parent-child relationships are actually in place, list is just drawn flat for debugging.

Factions

Starting faction data is parsed from FACTION.TXT and your save games are supported too! Importing a classic save will now also import your standing with all factions.

While there isn’t much happening with factions yet in Daggerfall Unity, it’s impossible to implement many gameplay systems without them. I look forward to doing much more with this data in a future build.

Building Names

One of Daggerfall’s long-running puzzles is how to generate the correct building name for any given building in a location. Daggerfall’s binary data exposes this information only as a seed value with no obvious correlation to the final name. From today, I’m happy to say this has been solved and I will be able to generate proper building names in the future. This article is a summary of the technical journey, minus all the dead ends and frustration.

The seed value used to generate building names has been known about for some time. This can be found in the BuildingData structure (link to UESP). The first step along the way was to generate some known values by changing a known seed value in MAPS.BSA. I started at the location Ashfield Hall in the Daggerfall province, which has a single tavern and some residences. Taverns are a great place to start as they have a very obvious PartA + PartB structure. For example The Bat And Skull. In Ashfield Hall, our single tavern is the The Howling Stag with a name seed value of 27748.

The first thing I did was change the name seed value for The Howling Stag in MAPS.BSA then start up Daggerfall to see how the name changes. Here’s a small sample of names generated from seeds 0-3. Keep this list in mind as we’ll return to it later.

0 = The Dancing Chasm
1 = The Knave and Scorpian
2 = The Pig and Ogre
3 = The Thirsty Fairy

Now I have somewhere to begin. I know the building is a tavern and have a sample group of seeds that result in specific names. The next trick is to work out how Daggerfall derives these names from the seed value.

I open up FALL.EXE in a hex viewer and search through for strings like “The Dancing” and “Chasm”. These strings are easy enough to locate, but these are just resources packed into the executable. What I need is the actual PartA and PartB tables Daggerfall is selecting from at runtime.

To get this information, I first have to use the DOSBox debugger to dump out memory from Daggerfall while it’s running. I can then search not just for strings, but for memory offsets pointing to those strings. I write a small bit of code to do the searches for me, and it doesn’t take long to find the correct offset tables for Part A and Part B of tavern names. Just think of this as a pair of arrays. In this case, both arrays are 36 elements long. Here they are as captured from the spreadsheet I dumped them out to.

TavernNamePartsAB

So how do we go from a seed of 0 to The Dancing Chasm? This is where most of the difficulty started. It was obvious Daggerfall used a random number generator to pick both parts, but the trick was to find the correct random number generator used by Daggerfall’s C compiler circa 1994-1996. Fortunately, I also needed this for correct texture table generation (still an open problem at time of writing) and had previously researched the correct random generator, known as a linear congruential generator, specific to Daggerfall. Here it is for completeness.

static ulong next;
public static void srand(int seed)
{
    next = (uint)seed;
}
public static uint rand()
{
    next = next * 1103515245 + 12345;
    return ((uint)((next >> 16) & 0x7FFF));
}

There are two methods here, one to set the seed (srand) and another to generate the next random number from that seed (rand). This is pretty much the standard ANSI LCG but specific to Daggerfall’s needs. Implementing this manually ensures that critical random number generation will always work just like Daggerfall, regardless of platform.

Now that I have the right random number generator, let’s feed it our test seeds from earlier and see what comes out. Starting with seed=0 and generating two numbers (indices into Part A and Part B name tables above), I get the following results.

PartA = 0
PartB = 12

First obvious thing is the spreadsheet starts from 1, not from 0. Just need to +1 each number to match the tables above (although zero-based arrays will be used in actual code). Matching these numbers to the above name table we get: Chasm The Dancing. OK, so Daggerfall obviously generates PartB first then PartA. Let’s try that again with the +1 and order swapped.

Seed = 0
  PartA = 13 (The Dancing)
  PartB = 1  (Chasm)
  Result: The Dancing Chasm

Using our handy table we can match row 13 with row 1 and we get The Dancing Chasm. Good! Let’s run some more tests and prove the concept.

Seed = 1
  PartA = 35 (The Knave and)
  PartB = 27 (Scorpion)
  Result: The Knave and Scorpion

Seed = 2
  PartA = 30 (The Pig and)
  PartB = 9  (Ogre)
  Result: The Pig and Ogre

Seed = 3
  PartA = 16 (The Thirsty)
  PartB = 36 (Fairy)
  Result: The Thirsty Fairy

So far, so good! Building names are output just like Daggerfall given the same inputs. Let’s try the original, unmodified seed value of 27748 which should give us The Howling Stag.

Seed = 27748
  PartA = 21 (The Howling)
  PartB = 33 (Stag)
  Result: The Howling Stag

And there we have it! Building name generation from initial seed value resulting in a string exactly matching Daggerfall.

From here, I still need to extract hard-coded name tables for other building types like armorers and weapon-smiths. This isn’t hard though, I just need to find the tables using the same methods as taverns. I also need to assign full building data from MAPS.BSA to the correct models in Unity scene and wire up API methods to query this data when inspecting or entering a building. One challenge at a time though.

For regular small updates on Daggerfall Unity, I can be found on Twitter @gav_clayton.

Items Part 1 – Bootstrapping

Loot. Kit. Swag. Treasure. Whatever you call it, items are an important part of any RPG game loop. They provide the means for your character to defeat ever more powerful foes and create incentive to keep playing in search of the next big upgrade. While Daggerfall’s items don’t quite tickle the reward centres of the brain like Diablo 3 or World of Warcraft, they’re still a vital part of the play experience. Without decent gear and enchantments, you’re unlikely to survive the grueling ordeal of Mantellan Crux.

In this series, I’ll describe the process of adding items to Daggerfall Unity. I wanted to approach items early on as they will be involved at almost every level of the game. Shops sell them, blacksmiths repair them, monsters drop them, quests reward them. Your character may have a special affinity for bladed weapons, or be forbidden the use of shields. Even the biography questions when building a character can grant you items like the near-essential Ebony Dagger. With items embedded in almost every major game system, the hardest part was working out where to begin.

I decided to start with existing items as part of importing classic Daggerfall saves then bootstrap the whole item back-end from there. That way I could be certain I was dealing with the most real-world data possible. Having built support for classic saves in 0.1, I could already identify item records parented to the main character record and visualise them with a custom Unity Editor script. They looked a bit like this at first:

 

Items1

It’s not much, but at least I could find item records belonging to the character. The “Container” record is just a generic parent record. In this context, think of it as the character’s backpack.

The next step was to break apart the item record format. Fortunately the UESP came to the rescue here with most of the bytes already solved, but far from the whole story as you’ll see once the names are revealed:

Items2

A Frosty what of Ice Storms? OK, so there’s more to this than just the save record. How to we go about filling in the blanks? The key here is the “category” 16-bit field in that UESP article. This is actually a pair of 8-bit values. The first byte is the item group, the second byte is a table lookup for the item template within that group. The template indexed by this lookup has all the missing pieces of information we need to complete our item data. Now we have two more problems to solve. Where are the templates, and how to use those category bytes to find them? Let’s start with the templates.

Item templates are actually built into FALL.EXE. The offset is a little different depending on your version, but the easy way to locate them is open a hex editor and search for “ruby”. You will find the following data:

Items3

Here are all the item templates laid out one after the other. They even follow a certain kind of logic, with gems, weapons, armor, etc. all more or less grouped together. Fortunately this isn’t exactly unknown data and the UESP came to the rescue again with a good starting point for these templates. I just had to fill in some blanks.

I didn’t want to keep this data in the .exe however, it’s much harder to modify these templates later. That’s why I exported the item templates to JSON format. Once exported the above data looks like this:

Items4
Much easier to work with. There are still a few unknowns to work out but those will be solved over time. The next problem was how to link up instantiated items like our Frosty %it of Ice Storms back to their original template. I had to reproduce the lookup table Daggerfall was using internally.

It was here Lypyl provided a helping hand thanks to his research into magical items and artifacts. The file format of MAGIC.DEF is very similar to instantiated items found in save games. Furthermore, the creators of old item editors had solved quite a few of these problems back then. Armed with all this, Lypyl could derive enough information to rebuild the group and item tables which he kindly provided to me in C# enums. All I had to do then was link the enums back to their template index in the above JSON file.

The main group enum looks like below. It corresponds to the first byte of the earlier category short.

Items5

For every element in the above enum (such as Armor, Weapons, etc.) there is an enum for every individual item in that group. For example:

Items6

For the item enum, the individual item value is an index back into the template table. The order within the enum corresponds to the second byte of the category short. With a helper class to bring all this together, it was now possible to perform lookups from instantiated items back to their template data. This is how our items viewer looks now:

Items7

Success! We can now resolve an item’s template by type to discover the full name and other useful information. The next step was to determine which items are equipped on the character. Fortunately the “equipped” table is just another record in your save game, and was already known about thanks to that first UESP article. I just had to work out how that table referenced items and I could isolate which were equipped. Items marked by an asterisk are equipped to character.

Items8

There are almost two dozen equipment slots in total that map to specific parts of the character’s body and elsewhere. I will describe this in more detail in a future article.

With all of that research out of the way, my next job was even less visual than above. I had to write support classes such as API helpers and an entity item collection class. I also required a new type of image reader to handle the job of loading and caching item images for the inventory UI, tinting them based on material, cutting out unique alpha indices like the hair mask, and so on. Anyway, boring or not these new classes form the foundation of items in Daggerfall Unity and will continue to grow as needed.

With everything finally in place, I could start building the equipment UI to sort, view, and equip items imported from classic Daggerfall save games. Besides a few UI enhancements and fixes, the following came together fairly quickly.

Some of the enhancements in this gfy include a scrollbar and mouse wheel scrolling. No reason we can’t have a few light modern touches to make our lives easier.

Back On Deck For 2016

Happy New Year everyone! I’m back from holidays and almost on top of my RL workload again. That means a whole new round of updates to Daggerfall Unity and DFTFU are about to begin. I’ve picked up where I left off last year with the item and inventory system, and will be posting more on this shortly.

Sometime in the next few weeks, I’ll start adding new test builds leading up to the 0.2 release. Key features of 0.2 will be:

  • Basic inventory system and loot tables. Import items from classic saves.
  • Travel map interface (by LypyL).
  • Dungeon and interior automap interface (by Nystul).
  • More bug fixes and incremental updates.
  • Some more community resources for contributors.

I’ve also made a small new year’s resolution to post more technically-minded articles in 2016, as I let this slip with all the rapid-fire updates leading to 0.1. It was quite a shift for me going from pure tool development to building a game, and I rather miss just talking about what I’m working on.

Thanks for all your patience during the holiday season. I look forward to reading your feedback with the next round of updates.

Remastering Daggerfall

Less than 12 months have passed since I began work on Daggerfall Tools for Unity. In that time, my little project has grown substantially and attracted a lot of attention from around the world, including articles on Kotaku AU and Rock Paper Shotgun.

As word about Daggerfall Tools for Unity has spread, a lot of hopeful Daggerfall players have visited looking for word on a remake – only to find a set of development tools not aimed towards players. While everyone seems excited about the potential Daggerfall Tools for Unity offers, there’s a lot of latency between new developers coming on board and getting up to speed before they can offer their contributions. So as a developer community, we may be climbing a steady ladder towards remaking Daggerfall, but there’s very little for non-developers to experience and no clear direction from which a remake will come. I’m contacted by passionate Daggerfall fans almost every week and it breaks my heart that I don’t have more for them.

I want that situation to change. I want the average Daggerfall player to be able to experience progress and have a clear sense of direction that Daggerfall Tools for Unity can be used for remaking Daggerfall, and then some.

What this means is that I can’t just create a toolset. Raw building blocks aren’t enough. I also need to create a scaffold for remastering Daggerfall that includes everyone, developers and players alike.

So let’s get this out of the way. I will shortly release a burgeoning Daggerfall remake, proudly using the Unity engine and Daggerfall Tools for Unity.

For players, this means you will be able to download regular builds to experience the project as it grows. You will become part of the process, able to offer your feedback and constructive criticism to the betterment of the project. Developers win out as well, because you get a functioning Daggerfall remake – completely open source – to build on and change as you desire. For the first time we can all, developers and players alike, come together and contribute.

So where does this leave Daggerfall Tools for Unity? Nothing changes at all. My remake project will be built on the same Daggerfall Tools for Unity that everyone has access to. If anything, this will only improve Daggerfall Tools for Unity by creating a proving-ground for new features and increasing the number of testers.

I will soon release a new web site, solely for distributing the latest playable build of my Daggerfall remake. Daggerfall Workshop will remain as the hub of social activity and development news.

And the best news for players is there’s a swag of new features coming in Daggerfall Tools for Unity 1.4 and 1.5, all aimed at creating playable systems and filling in those blanks between a toolset and a game.

Feel free to leave your comments on this post, or head over the forum post where I’ve kicked off the conversation.

Daggerfall VID Playback

I’m going all-out in Daggerfall Tools for Unity 1.4. You will also be able to play Daggerfall’s VID movies right inside Unity to any texture you like. I’ve even provided a VID player panel for the native UI discussed in my previous post.

This is just a prototype, currently without audio or correct timings. I’ll show this off in a YouTube clip sometime in the future once its properly integrated.

Native User Interface Intro

The portable UI being developed for Daggerfall Tools for Unity 1.4 is coming together. Rather than use a uGUI simulacrum of the Daggerfall interface, I decided to re-implement Daggerfall’s native UI screen by screen, making it as true to Daggerfall as possible.

StartScreenUI

 

 

This has involved a fair amount of work to ensure everything is rendered at true native resolutions and scales proportionately to any screen size. Here’s an early test clip where I’m playing with the scaling, which runs in real-time.

Another challenge was pixel-perfect text rendering. The internal UI in Daggerfall Tools for Unity uses Daggerfall’s pixel fonts exactly, which required a custom text renderer. Below is the book reader UI flicking through The Real Barenziah. The formatting codes from Daggerfall are followed so precisely that badly formatted text even overflows in the same places it does in game.

Behind the scenes, the GUI runs on a state machine stack where windows can be pushed and popped as required. Each window is a hierarchy of components (panels, buttons, labels, etc.) that all follow the same consistent layout behaviour and handle fundamental UI tasks like firing messages when clicked. Everything is rendered and hit-tested internally at Daggerfall’s native 320×200 resolution.

Communication between windows uses a custom messaging system that runs independently of platform. The messages themselves can even work a bit like command strings in that they’re able to pass parameters directly inside the message.

And the best news is that you aren’t bound to using this GUI in your projects. It exists inside the Demo namespace, which means that nothing in the core library or API has any dependency on it. You’re free to use this GUI or ignore it completely to implement your own interface using whatever UI platform you’re most comfortable with.

There’s still a lot of work to do before it’s ready for Daggerfall Tools for Unity 1.4, but everything is taking shape very quickly.