Guild Systems

Hi everyone, Hazelnut here. Interkarma has asked me to write a blog post about some of the work I have been doing recently, so for this post Daggerfall Workshop has been taken over by me! Muhahahaha… etc. I’m sure your regularly scheduled Interkarma posts will return soon, so don’t worry.

Anyway, allow me to introduce myself, you may know me from such features as horse riding, inventory upgrades, shopping, persistent state, icons, tavern rooms, ships and houses, as well as various other bits and bobs. So why did I spring up from nowhere and start contributing to DFU?

I bought Daggerfall on xmas after its release and played like crazy for 7 days until the end of the christmas break, then stopped. Because it’s such a huge game, never really got back into it for years due to lack of time & small children – until shortly before Morrowind came out. By then the game was simply far too dated for me to persevere, especially because I forgot about the WSAD option and I was using the default control scheme of stupid mouse pointer arrows to move. So I’ve actually never played beyond the first couple of character levels. Always wanted to, and do intend to once DF Unity is done. This is why I am absent for any discussion or work around the main quest! Spoilers! Yep even after all this time.

While the progress has been fantastic (Interkarma has Orc level stamina apparently) it’s a big project and takes a long time, and since I’m a software engineer for the past 2+ decades who now does software design so don’t get to write much code at work… I figured I’d see if I could contribute. Also my kids are all teenagers now, so I have more free time than I’ve had for 2 decades. I had no knowledge of Daggerfall and its data structures, hardly ever touched C# and never used Unity before, so it’s been quite a learning curve. Anyway, after I finished my work on ships, houses, shops and taverns I asked Interkarma what he thought I should tackle next and between us we decided it was time to implement guilds. I’d already done some work with the guild services menus, but now I needed to create the guild membership systems.

Guilds

Guilds in Daggerfall follow some common rules for promotion etc but offer different services and quests. The first thing to focus on was ranks and building some foundations that would make each guild easy to implement, allowing the common behavior to be shared. I also wanted to ensure that new guilds could be added by mods, as I was sure that several people in the community were keen to add new guilds having seen discussions on the forums, and I had some ideas of my own that would be best integrated into the base game by adding a new guild. I decided early on that the guild code should be designed to support modding from the start, and the only way to ensure this was to actually implement a guild mod early on to prove the concept.

So, quite a lot to take on. Fortunately I had 3 solid days free coming up to get started on this. The fighters guild is the simplest of them all so I started with that. First step was to test classic and see how guild membership worked with rank changes etc. To support modding, the guild classes are designed so that they know about the players membership status, rank, and what benefits and services the guild provides at that rank. Service and guild management code asks the guild class by calling the appropriate method when they need to know. They also supply guild specific messages to the ranking system which is shared. This means that a new guild can simply be added by implementing a new guild class and registering it with the guild manager.

Now by this time you could join the Fighters guild.

Join Fighters

Joining the Fighters

Continue reading

Town Populations

I’ve had my head in the quest system for several months and really needed a short break. I also happened to need a solution for spawning enemies outdoors in cities (for example, the ghosts and wraiths in Daggerfall at night) and noted there was a good amount of overlap between spawning enemies and NPCs in town environments because they all need to avoid placement inside building geometry. And whatever solution I use for placement could probably be used for navigation as well. I had scheduled wandering NPCs for 0.5 cycle, but decided to make an early start on this while solving town placement and navigation. And what better way to test this solution than to actually watch NPCs walk around?

The first problem I had was how to find an appropriate placement position. My initial idea was to use the foliage placement array in exterior data. This formed a nice grid over each block, but it also marched over water and under buildings. That would not be suitable. I considered just dropping in mobiles and using a combination of rays and colliders to refine their position until they found open space, but that approach seemed way too messy and inefficient.

That’s when I had a eureka moment thinking about how perfectly automap image data lined up with the game world. Take the below screenshot as an example. In game, I’m standing outside the Odd Blades looking at the entrance door. On the automap, once unit conversions are done, I’m in exactly the same place.

 

So Daggerfall’s automap perfectly skins building footprints. This should mean I can take the inverse of automap data to work out which parts of the environment are open. I quickly prototype by placing white cubes on open environment and avoiding the building footprints. Take a look at the results.

 

This is beyond perfect. The automap doesn’t just contain data for building footprints, but for flat placement and decorative geometry as well. I have a strong suspicion Daggerfall also uses the automap data in this manner, it’s just too precise and detailed to be a coincidence.

With a solution in mind, I now have to execute the idea. I create a new class called CityNavigation which is added to the location GameObject at scene layout time by StreamingWorld. This constructs a navigation grid at the same time location and automap data is read so only a small amount of additional processing is done per location. With the inverse of automap blocked out, we get the following:

This is good – the white areas can be used for placement and navigation, but it’s not perfect. It also needs to account for tilemap under location. We can’t place NPCs on water tiles, and they should try to avoid those tiles when walking around. Rather than just block out unwalkable tiles, I take this one step further and allocate each tile a specific weight, where a higher weight means the tile is more favourable. Here’s the final navgrid where black areas are “no-go” and brighter areas are preferred over darker areas. You can probably see right away this creates a strong preference for the road network:

 

What you can’t see in the above image is that each weight occupies the upper 4 bits of a single byte. The lower 4 bits are reserved for a flag system, giving me up to 4 bits to control NPC behaviours. This will be important later in this article.

Now that I have a nice procedurally generated map of any exterior location, the next problem is converting between all the different coordinate systems. If you’ve ever tried to make a big world in Unity, you’ll know that precision problems kick in after a few thousand scene units or so from origin (position 0,0,0 in world). This manifests itself through jittery movement and shadows, imprecise feeling of control, and issues with physics system. The game map in Daggerfall rocks in over 819,000 x 409,000 scene units, way beyond what Unity can handle with fine floating-point precision. I overcame this challenge very early on by using a fixed point coordinate system for the world (Daggerfall units) and normal floating point units for the scene (Unity units). The world is built around the player in chunks close to origin, and when player runs too far in one direction, the whole world is brought back towards origin. To the player it feels like they are running continuously through a huge open world, when in fact the world is being constructed around them one chunk at a time. The player never moves more than 1000 units from origin in the X-Z plane.

What does all of this have to do with the navgrid above? Well, now I have yet another coordinate system to glue together. I have not only the Daggerfall units and Unity units, but the X,Y position inside the navgrid array where any virtual mobile objects have to move around. So the next thing I do is write some helpers in CityNavigation to convert from navgrids to world space, world space to scene space, and back again, and so on. This chewed up a solid chunk of Sunday to get working properly, and there’s still a few precision issues due to the large differences in scale. Something to refine down the track.

With all the math out of the way, I can now start placing mobile NPCs into the world. One problem though, I hadn’t written any code to render wandering NPCs yet. So I started with these guys just to confirm the navgrid through scene conversions were working. Sometimes in game development, you have to bust out some programmer art to get the job done.

 

With placement working, next came the process of building the mobile NPC billboard properly – including that stop-and-stare thing they do when you get too close to them.

Town NPC Billboards

[gfycat data_id=”YawningGrotesqueBluegill”]

With rendering done, I can start moving them around the navgrid using a simple motor. They will generally follow roads when encountered (because roads have a higher weight), but there’s enough randomness to let them change directions and wander around elsewhere on the grid.

And do you remember me mentioning the navgrid can store flags in the lower 4 bits? The first flag I created is an “occupied” bit that lets a mobile claim a navgrid tile before walking into it. This prevents two or more NPCs trying to occupy the same tile at a time. The next clip shows the mobile movement, path following, and dynamic avoidance of each other. I’ve cranked up the spawn count and movement speeds because it helps me observe the behaviour (and it’s kind of fun to watch).

Mobile Pathing and Dynamic Avoidance

[gfycat data_id=”ActualFondGrouper”]

Despite everything accomplished, I still have more to do. The next step is working out which races are placed in which towns. I put my travelling boots on and tracked around classic Daggerfall’s world until I found which races appeared in which climate zones. I built this into the helper which returns climate data for texture swaps, etc. and now the correct NPC races (either Redguard, Nord, or Breton) will appear across the game world.

 

The final step was to build a PopulationManager class. This code handles spawning/despawning of NPCs around player as you move through town environments so the location feels populated. After a bit of experimentation, I used a population index per 16 RMB blocks so that small towns feel like they have a smaller overall population than large cities. One of my challenges here is the draw distances in Daggerfall Unity are huge compared to classic Daggerfall. While classic can place NPCs safely in the fog out of sight, in Daggerfall Unity you can see two full city sizes distant across the map. This means that hiding pop-in and pop-out of NPCs is a little trickier. For now, I mitigate this by trying to only show or hide NPCs when player is not looking directly at them (as Daggerfall does) and only allow them to pop-in when a certain distance from player.

There’s still a few bugs to iron out. You can still catch them pop-in nearby if you happen to look in the right direction at the right time, and they sometimes glide slightly in the wrong direction on spawn due to precision issues as they align to the grid. And of course you can’t talk to them yet, because the “talk” system won’t be introduced until 0.5 sometime. But overall, the feeling of crowds is quite satisfactory and Daggerfall-ish, and it’s wonderful to finally see these sprite people bustling around cities.

I hope you enjoyed reading about some of the work that goes into creating even a small feature like this one. If you’d like to read more, I try to post regular micro-updates to my Twitter feed @gav_clayton.

 

Questing: Place Resource

I’ve decided to go in a slightly different direction regarding dev articles about quest system from now on. Instead of the traditional numbered sequence split into a few parts, I will post more about what I’m working on in the moment. There are two reasons for this. First, questing is a huge subject that resists being neatly split into a progressive narrative like items or streaming terrain. I often have to cross back and forth between related systems. Second, it gets a bit daunting trying to construct that narrative from such fragmented work. The end result is that I’m posting less and feeling more weight on my shoulders every time I try to construct a post. Just how do I describe to everyone I spent weeks working on research without a satisfying answer to a particular problem?

My solution is to tear the band-aid off and just post something. I’m slowly learning that many people enjoy the development process itself, including the potential for frustration and dead ends. Those outcomes aren’t particularly enjoyable for me when they happen, but maybe I can turn them into something positive and entertain my readers. This should result in more posts but they will often be shorter (which in itself might be a good thing). This post probably won’t be shorter, but it is my first in a while. 🙂

So here’s what I’ve been working through lately…

In a QBN section, Daggerfall quests defines resources that kind of glue the whole shebang together. They define places in the world, people to visit, timers, items, and enemies. Basically all the world-stuff player will contact to unroll that quest’s execution. A high-priority resource is Place, which defines locations the player must use in the quest, like a dungeon, shop, or residence. Here are some examples.

Place _mondung_ remote dungeon9
Place _house_ local house2
Place _inn_ local tavern
Place _palace_ remote palace
Place PiratesHold permanent PirateerHold1

To deconstruct the above: Place tells the compiler this is a place resource; _symbol_ is the name of that place for the compiler to reference; remote/local/permanent is where to select that place from; and the final bit is site which tells quest engine what type of place to select. The site code is actually a reference to a places data table which defines the parameters for these site types. I won’t go into these in much detail, just keep in mind they define what kind of building or dungeon to select.

Let’s use “local tavern” for now. This means find a random tavern within the same town as where player received quest. If this was “remote tavern”, we’d select a random tavern from another town in the same region. The site code “tavern” is then looked up in the places data table and the parameters tell us this is building type 15 (0x0F).

Now the quest engine knows it needs a tavern (building type 15) and it should select from the map player is currently in. Fortunately, Daggerfall keeps a handy list of buildings for every map definition in MAPS.BSA. This gives us information like the building name seed, faction, quality (“rusty relics” through “incense burning”), and best of all building type. Great! We already know we need a tavern, and it’s trivial to select taverns from this list of buildings and grab one at random. Now we have our quest location… well, almost.

A Daggerfall city is made up of geomorph-like blocks – up to 8×8 blocks for really big cities. They are like puzzle pieces laid out on a grid. Here’s the block layout for the City of Daggerfall, you might recognise this from Daggerfall Modelling.

 

Every block also carries a list of buildings it contains, sharing the same data structure as MAPS.BSA – with a few differences. One key difference is that a few of the values present at map level (e.g. name seed) are not present at block level. The reason for this is obvious when you consider how often blocks are reused across locations. If the name seed was stored in block data, our tavern would have the same name in every location that block is used. Thus the name seed is carried at map level and merged down to block level as needed. Same goes for the block texturing, which changes based on climate. Put together, this all makes blocks somewhat polymorphic in that they can take different forms depending on the location they belong to.

Back to our tavern which we found in building list at map level, how then do we map this to the actual physical building at block level? Besides other jobs like setting the correct names on automap, this is needed to determine correct physical location of quest buildings in the world. This is where some of my troubles started.

At time of writing, it is unknown exactly how Daggerfall links building data at map level down to building data at block level. To compound my problems, only a subset of buildings are represented at map level. A large city like Daggerfall will have many more physical buildings than listed for that location in MAPS.BSA. Even in small locations where the totals match, the individual building type counts don’t always match. For example, map data might define 6x House1 and 4x House2, while block data will contain 5x House1 and 5x House2. There doesn’t seem to be any consistent way to link these two sets of data together. If this were a relational database, I would expect a stable primary and foreign key matching on both sides. But not only does a key not seem to exist between them, the totals cannot be trusted to line up between map and block buildings.

Despite this, I have been able to make some good progress. I discovered that named buildings (taverns, stores, guilds, etc.) are always in the same order in the map building data as the block layout data. Couple this with the building name seed discussed a while back, it’s possible to merge down named buildings without much trouble. This is also needed for the automap like below. Credit goes to Nystul for the excellent automap implementation.

 

When it comes to generic premises like House1/House2, things become more difficult. I can’t depend on an exact match between building type distribution, and can’t find a way these could possibly be linked in a stable manner. Both Lypyl and myself have been through the binary data in great detail. Despite finding a few other interesting leads, we have not found a satisfying answer to the question of mapping non-named buildings from map level to block level.

After many frustrating weeks of searching, I’ve more or less come to the conclusion that it really doesn’t matter. The only buildings that need to be mapped 1:1 are named buildings, and I already have a working solution for this. When it comes to generic premises (which are selected randomly anyway) it should not matter which House1 or House2, etc. Daggerfall Unity selects for a quest. As long as the end behaviour closely matches that of Daggerfall by selecting a building of the correct type, I’m basically free to implement this in a manner suitable for Daggerfall Unity.

In hindsight, the apparent lack of stable linking between map and block data could account for some quirks in Daggerfall itself. For example, it’s not uncommon to find a tavern which Daggerfall thinks is a residence. And only about half the buildings in most towns can be entered anyway (“this house has nothing of value”). My feeling now is that when a location is loaded, Daggerfall simply distributes map building data amongst block buildings using some kind of algorithm where named buildings are populated first and generic buildings are distributed to best effort. As long as named buildings are handled (they are), and other buildings are selected randomly (they seem to be), then I can at least reproduce this behaviour in a way that feels exactly the same to the player. My solution might not be exactly what Daggerfall is doing, but it’s likely very close.

So that’s where I’m at right now – working through implementation of Place resource, selecting locations in the world, and ensuring this maps to physical environment correctly. During this process, I’ve added some fun bits along the way such as the flavour text when entering a dungeon area outdoors. This text is based on dungeon type and reads from strings in Daggerfall’s text database using the same variations.

The other side of a Place resource is the “pc at” condition which tests if player is at a Place or not. This is highly intertwined with Place resource handling so I’m likely to bounce back and forth between these two for a while until I’m happy.

That’s all for now. Rather than go through a long delay between posts again, I’ll try to post as regularly as possible with updates on what I’m working on at the time. Even if the post is short, I’ll try to put in some details of the development process for you to read.

If you enjoy this sort of post, please let me know! Feedback helps me work out what people enjoy reading and hopefully write better posts in the future. Thank you for reading, everyone. 🙂

Questing Part 3 – Anatomy of a Task

Example custom quest action created for this post

The quest system back-end is coming along. I’m sad it’s not further along by now, but life has a way of disrupting plans. The important thing is I’m still making progress and have some good stuff to share today.

In this post, I will dissect tasks along with their conditions and actions, which together form the meat of a quest. There’s a great deal of technical content ahead, I want this post to be a kind of primer for contributors to quest system. My apologies to those of you who don’t enjoy code-heavy updates.

Before getting started, please have a quick skim through Donald Tipton’s excellent documentation for his Template v1.11 quest compiler/decompiler. As discussed in Questing Part 1: Source, these source files are also used by Daggerfall Unity’s quest parser so we have common ground with classic. This even allows quest source to be shared back and forth with classic for testing. I’ve actually rolled back some of my ideas from that first post and will use Template source files directly as-is without any changes. This might have to change in time but right now I’m aiming for total parity.

If you’d like to see some real quest source, you’ll find all of Daggerfall’s decompiled quests in the StreamingAssets/Quests directory (link to GitHub, ignore .meta files). But today we’re just zooming in on tasks and actions to see how these are handled by Daggerfall Unity.

 

Tasks

Daggerfall quests have four distinct forms of tasks (so far). All of the examples below are from the quest _BRISIEN.

Standard – This is a basic task which does not start unless explicitly triggered somehow. The task name (e.g. _invitepc_) is also a boolean symbol which can be queried to see if task has been triggered (i.e. is active). The lines under the task header are the conditions and actions making up that task.

_invitepc_ task:
	start timer _remindpc_ 
	give pc _letter1_ notify 1026 
	create npc at _dirtypit_ 
	place npc ladyBrisienna at _dirtypit_ 

Repeating – These tasks execute continuously until the symbol they reference (the boolean state of another task or variable name) is triggered. In below case, the task will persist until _exitstarter_ is triggered. Repeating tasks appear to be triggered automatically at startup.

until _exitstarter_ performed:
	start quest 999 999 
	start quest 977 977 
	start timer _invitepc_ 
	remove log step 0

Variable – A variable is really a kind of task with trigger state only. Trigger state may be set/unset by other tasks.

variable _exitstarter_

Headless – Every quest must have a single headless task. This is the entry point to be executed automatically at quest start-up. Unlike other tasks, a headless task does not have a symbol to query trigger state. It just executes to bootstrap the rest of quest. This is the entry point of _BRISIEN:

--	Quest start-up:
	log 1010 step 0 
	pc at PiratesHold set _exitstarter_ 
	say 1025 

At time of writing, Daggerfall Unity will parse through quest source to instantiate tasks and try to match component actions to a registered template (more on this below).

 

Conditions

Other than being triggered at startup or by other tasks and clock time-outs, a task can have one or more conditions that might cause it to be triggered. For example, if player is in a specific place at a certain time (e.g. Daggerfall at night) then some action can be performed (e.g. play the “vengeance” effect). This makes it possible to chain together tasks which trigger on and off based on the trigger state of other tasks.

I won’t go much into conditions right now as they have not been implemented yet. I’ve just barely stubbed out a bit of starting code that will be replaced later. If you like, you can read more about quest conditions here.

 

Actions

A quest action is a bit of text that does something. This is usually a single thing like playing a sound, displaying a message, or starting another task. Don’t think of actions like a normal programming command though. They aren’t necessarily run and done (although they might be). Try to think of actions as components attached to a task in a similar way that Unity components are attached to a GameObject. This isn’t a perfect analogy, but its a start. Like GameObjects in Unity, tasks can switch on and off and their component actions perform bits of work over time.

Actions are a great way for contributors to help build out the quest system. There are many different kinds of actions, some will be very simple others very complex.

 

Building Actions

So how does an action go from a line of text to actually doing something in the game? The rest of this post will cover the fundamentals and show a real working example of a custom action… in action.

The bones of every action begins with the ActionTemplate class, an abstract implementation of IQuestAction interface. All actions must inherit from ActionTemplate and implement the required parts of IQuestAction. This ensures that every action template has a few key features:

  • Pattern – A regex string used to pair a line of source text with this action. Two actions cannot have the same match pattern.
  • Test – Checks if provided source string matches the regex pattern expected for this action.
  • Create – An action template is special in that it can also factory (i.e. generate) a new instance of itself with default settings. This allows the QuestMachine which hosts active quests to hold a list of self-replicating action templates that can be instantiated as required.
  • GetSaveData – Gets a data packet from action live state. This will be passed on to JSON serialization system when saving a game.
  • ResoreSaveData – Sends a data packet to action from serialized state. This will be used to restore action state when loading a game.
  • Update – Called by the task owning this action. Allows the action to do work every frame as needed.

To show all of this working, I wrote an example called JuggleAction which simulates the player juggling some number of objects with a percent chance to drop one. Click here for the full source code and I’ll break it down below. Let’s start with the pattern:

public override string Pattern
{
    get { return @"juggle (?<numberOfThings>\d+) (?<thingName>\w+) every (?<interval>\d+) seconds drop (?<dropPercent>\d+)%"; }
}

This is just a basic regex match string that looks for a pattern like “juggle 5 apples every 2 seconds drop 40%”. Everything the action needs to execute is contained in the pattern. Sometimes an action might take different forms and the pattern string must cover these variants also.

The parser uses Test to find a registered action template with pattern matching source. When a match is found, the JuggleAction template will factory a new instance of itself with default settings by way of Create.

public override IQuestAction Create(string source, Quest parentQuest)
{
    // Source must match pattern
    Match match = Test(source);
    if (!match.Success)
        return null;

    // Factory new action and set default data as needed
    JuggleAction action = new JuggleAction(parentQuest);
    action.thingName = match.Groups["thingName"].Value;
    action.thingsRemaining = Parser.ParseInt(match.Groups["numberOfThings"].Value);
    action.interval = Parser.ParseInt(match.Groups["interval"].Value);
    action.dropPercent = Parser.ParseInt(match.Groups["dropPercent"].Value);

    return action;
}

You’ll notice the action parameters are exposed directly by the Match class returned by Test. This makes it easy to read out the values involved. At this time, our new action is ready and is added to a collection stored in the Task object. During quest runtime, the task will call Update on each action to do the work required. Here it just counts off time and if still holding any objects, calls the Juggle() method. Note that we’re using Time.realtimeSinceStartup instead of Time.deltaTime. The reason for this is that QuestMachine ticks at a slower rate than Unity (currently 10 times per second). So we need to measure time without using something that only changes frame-to-frame.

public override void Update(Task caller)
{
    // Increment timer
    if (Time.realtimeSinceStartup < nextTick)
        return;

    // Juggle 'em if you got 'em
    if (thingsRemaining > 0)
    {
        Juggle();
    }

    // Update timer
    nextTick = Time.realtimeSinceStartup + interval;
}

Below is the Juggle() method for completeness. It just spits out some notification text to HUD and randomly decrements object count until none are remaining.

private void Juggle()
{
    // Juggle current things
    DaggerfallUI.AddHUDText(string.Format("Juggling {0} {1}...", thingsRemaining, thingName));

    // We might drop something!
    int roll = Random.Range(1, 101);
    if (roll < dropPercent)
    {
        thingsRemaining--;
        DaggerfallUI.AddHUDText("Oops, I dropped one!");
    }

    // Give up if we've dropped everything
    if (thingsRemaining == 0)
    {
        DaggerfallUI.AddHUDText(string.Format("Dropped all the {0}. I give up!", thingName));
        return;
    }
}

I won’t touch on GetSaveData and RestoreSaveData yet as quest state serialization has a ways to go. You can check the full source of JuggleAction linked above for an example implementation.

You might recall I said something about registering new actions with QuestMachine. This might change later, but right now our action class JuggleAction is registered in QuestMachine from RegisterActionTemplates() like below. The template is only being used as a factory so it doesn’t need to pass in an owning quest at construction.

RegisterAction(new JuggleAction(null));

Registering the action template allows the quest machine to find it (using Test) and factory a new instance from the template.

Now that we have an action and registered it to quest machine, we actually need a quest that uses this action for real. I created a cut-down quest just for this example called __DEMO01.

- Minimal example quest used to demonstrate how to script custom actions

Quest: _BASIC01

QRC:

- No text resources

QBN:

- Headless entry point with custom action
juggle 5 apples every 2 seconds drop 40%

All that remains is to instantiate the quest itself. I will add a console command soon for this, but in the meantime I’m calling the following bit of code from StartGameBehaviour.

GameManager.Instance.QuestMachine.InstantiateQuest("__DEMO01");

This loads our custom quest into the quest machine and starts executing supported actions, which right now is just the demo quest and juggle action. When starting a game, this will be the output:

[gfycat data_id=”ConsiderateZanyFlyingsquirrel”]

 

Next Steps

For now, I will continue to work on the quest machine, parser, and related frameworks. My immediate next step will be to get the full tutorial quest working along with some foundation conditions and actions, and a few supporting user interfaces (quest log, quest debugger UI).

I would like to invite the more experienced contributors to review the quest source documentation in more detail and see if any actions might fall into their range of interest. I would also love some help with quest resources other than tasks (e.g. Place, Item, Foe, Person, etc.). I’ve stubbed out the Clock resource as a starting point. If there is something you would like to work with, please start a conversation on the forums and let’s see where it takes us.

If you have any questions or would like to dicuss this post in more detail, please don’t hesitate to find me on the forums!

 

For more frequent updates on Daggerfall Unity, follow me on Twitter @gav_clayton.

Questing Part 2 – Compiling

In the first part of this series, I discussed how I’ll be using source files output by Tipton’s Template v1.11 for quests in Daggerfall Unity. If you’d like to check these out in full, the quest source files are already on GitHub. Follow this link to view them (ignore the .meta files).

Some interesting quest files to investigate are:

  • _BRISIEN.txt – This is the quest that launches when player starts in Privateer’s Hold. It sets the whole story arc in motion and starts timers for delivering letters that prompt player to visit Lady Brisienna. Failing to visit Lady B in time is ultimately a failure condition of main quest.
  • S0000999.txt – Starts when you leave Privateer’s Hold. Delivers letters from Prince Lhotun and Morgiah and sets key global variables.
  • S0000977.txt – Also starts when leaving Privateer’s Hold. Sets up regular ghost and wraith spawns in Daggerfall and plays the VENGEANCE! sound.
  • TUTOR.txt – Is the tutorial quest started when selecting Yes to the prompt at start of game.

As another interesting tidbit, longtime players may have noticed the initial journal entry that begins with…

"I am on a mission from the emperor to investigate
the shade of King Lysandus. His spirit has been
haunting the city of Daggerfall. The emperor
himself has charged me with the duty of laying
his ghost to rest."

…will stick around in your log forever. Even after you’ve completed the game, this first journal entry will remain. As part of researching the quest system a while back, I looked into why this was happening and found the above journal entry is added twice and removed only once. The below image sums it up visually. You may have already seen this when I posted it to Twitter back in September.

 

questlogmessagestuck

So the quest system in Daggerfall is quite powerful. It’s responsible for adding and removing journal text, starting timers, delivering letters, spawning enemies, playing sounds, setting global variables, moving NPCs, and a slew of other functions. At first blush it can appear a lot like a full programming language, but it’s only superficially so. The structure is actually more like a complex INI file, or even a kind of markup file. Everything is neatly categorized into its own section and execution flow generally only happens in a few different ways. Most commonly are at startup, when a timer ends, and due to a variable or condition changing state.

Thus despite the initial similarity to a programming language, the quest source is really just a collection of defined objects with some very basic scripting functions. Most of these functions revolve around spawning something, starting something, playing something, or changing something. Complex enough to get the job done, but don’t feel like you need to be a programming expert to create new quests in the future. Using Tipton’s Template v1.11, you could even start creating quests in classic Daggerfall now and later port them into Daggerfall Unity.

Likewise, the job of compiling the quest source back into Daggerfall Unity is not that difficult. It’s going to have some challenges, but nothing on the scale of building a real compiler, something I thought I’d be facing at the outset. The problem is largely just splitting source file up into correct parts and handing that source off to classes designed to support that part. For example, messages will go to a Message class, timers will go to a Timer class, etc. When serializing live quests as part of save games, the QuestMachine will save/load JSON state for each live quest along with the global variables for the current game.

To help Daggerfall Unity re-compile Template’s source output, I’m going to make some minor changes to the expected source files. I’m trying to keep these changes to an absolute minimum.

First change is for the quest header to be uncommented in source file. For example, _BRISIEN has the following header. The dash ‘-‘ prefix starts a comment.

-- StartsBy: letter
-- Questee: anyone
-- Repute: 0
-- QuestId: 0

This reason this can be commented out is that Template is designed to re-compile quests back into the original QBN/QRC format. When commented out, Template will just use the information already in the quest QBN data. However, Daggerfall Unity relies entirely on the quest source and will need the above uncommented to:

StartsBy: letter
Questee: anyone
Repute: 0
QuestId: 0

This way the information can be parsed back into the quest data at compile time.

The second change is that I would like an explicit startup task, rather than an implicit startup after the QBN object definitions (like Person, Clock, etc.). Currently the startup task looks like this:

-- Quest start-up:
log 1010 step 0 
pc at PiratesHold set _exitstarter_ 
say 1025 

I would like to change this to:

startup:
log 1010 step 0 
pc at PiratesHold set _exitstarter_ 
say 1025 

This makes the startup task an explicit task object to be executed when quest begins. This is already the current behaviour, but I think this change makes the task clearer to identify for both quest creators (even without a comment) and for my compiler. I could definitely get away without this change, but I prefer the explicitness to the current setup.

And of course, the above is subject to change as the quest system matures. If I learned anything from rolling out items, the feature will probably look different again by the time it’s mostly complete. Daggerfall Unity is not just a game remake, it’s an ongoing research project into the guts of Daggerfall. That adds a few twists outside of normal gamedev process, and I just need to roll with it.

One of the upcoming features not shown yet is the quest debugger currently in development. Hopefully, I will have something to show on this by part 3. The quest debugger will be used to manually start new quests, terminate and restart executing quests, and inspect state of global variables and quest objects. It’s a big UI that will continue to grow as development progresses. Should turn out to be quite useful though.

 

For more frequent updates on Daggerfall Unity, follow me on Twitter @gav_clayton.

Questing Part 1 – Source

With items and loot more or less finished, it’s time to place my sights on the next big feature group. I had originally planned to work on spells and effects in the 0.4 cycle, but also had quests in mind as an equally important feature. So back in July, I used a Twitter poll to sample what the community wanted priority on. The answer came firmly in preference for quests over spells.

Fast forward a couple months: 0.3 stable is now tagged on Live Builds page and I’ve started work on the quest system in earnest. This post is the first in a series documenting the journey from raw bytes to a working feature.

When adding a big new feature, there’s always a technical planning stage before commencement. I need to understand which of Daggerfall’s binary files are involved in the feature, learn how much is already known about those files (sometimes a lot, sometimes almost nothing), map out the additional details I need to know before I can re-implement that feature, and finally start cutting some code.

Sometimes it takes a few iterations, as writing code to pull apart the binaries is itself a journey where new information is learned. I don’t know what I don’t know, and often that new information results in a new iteration with better understanding. There’s basically a research-implement-compare loop going at all times throughout development of Daggerfall Unity. It can be a bit draining.

Items for example needed considerable research to bridge the gap between “enough knowledge to make an item editor” and “enough knowledge to fully implement items in a new engine”. Believe me, there’s a huge amount of work needed between having a file spec with lots of unknowns and building a completely functional equivalent. Most of my time is spent somewhere in that gap.

So when starting quests, I set out to learn everything I could about the binary formats and the work others had done before me. I was pleasantly surprised to learn:

  • Quests are among the most well-understood formats, thanks to the work of several early Daggerfall hackers (notably Donald Tipton and Dave Humphrey).
  • Tools exist to edit live quests, but more importantly decompile the QRC and QBN files to a human-readable source file (thanks to Template v1.11 by Tipton).
  • Early Daggerfall modders already had a good understanding of creating quests for Daggerfall – there’s an established process.
  • Quest files are one of the few cleanly decoupled things in Daggerfall. Unlike items which have template data hard-coded in the executable, quests are nicely self-contained.

With the above in mind, it was no longer necessary for me write a quest decompiler from scratch, design a new file format, decouple quest data, or chase half a dozen other time sinks. I can almost go directly to implementation. Almost. But first I need quest data in a usable form.

For the first step, I’m using Template v1.11, which you can find on UESP. This tool can decompile all the QRC (quest text) and QBN (quest programming) files to a single easy-to-read text file. So the information goes from raw binary data to the following (snippet from starting quest _BRISIEN).

QBN:
Item _letter1_ letter used 1020
Item _letter2_ letter used 1021
Item stopMainQuest letter used 1022

Person ladyBrisienna face 1 named Lady_Brisienna anyInfo 1012 rumors 1013

Place PiratesHold permanent PirateerHold1
Place _dirtypit_ remote tavern

Clock _invitepc_ 7.00:00 14.00:00
Clock _remindpc_ 30.00:00 0 flag 1 range 0 1
Clock _pcfailed_ 14.00:00 0 flag 1 range 0 1
Clock _oneday_ 1.00:00 0 flag 1 range 0 1
Clock _nowwhat_ 29.09:00 0 flag 1 range 0 1

-- Quest start-up:
 log 1010 step 0 
 pc at PiratesHold set _exitstarter_ 
 say 1025

That can still look a bit daunting if you’re not comfortable with programming concepts, but the source format generated by Template v1.11 is generally very easy to read and understand – more like a complex INI file than a real programming language. It’s so good that I’ve decided to adopt Template’s output as the source data for quests in Daggerfall Unity. You won’t need to recompile quest source back to QBN/QRC, rather you will use the source directly in Daggerfall Unity where it will be compiled at runtime by the quest system.

This means the de facto standard for creating Daggerfall quests will remain essentially the same in Daggerfall Unity (with some minor differences I’ll discuss later). If you’ve ever created a quest for Daggerfall, or even seriously looked into it, then you already have the skills to create quests in Daggerfall Unity. If you’re new to Daggerfall quests then you can learn from existing quests and read Template’s excellent documentation. Just as I don’t need to start from scratch, neither do you.

I will need to change the format slightly for unique needs on my side, but I aim to keep my input format as close as possible to Template’s output format. There’s also the need to emulate support for every single condition and action in Daggerfall Unity to behave similarly to Daggerfall given the same quest source. That will probably be an ongoing process all the way through to 1.0. But we need to start somewhere, and this is a great starting point.

The next article in series will look at parsing quest source files and how quests will be executed inside Daggerfall Unity.

 

For more frequent updates on Daggerfall Unity, follow me on Twitter @gav_clayton.