Visual Diary: Building Maps – Part 1

The Cartographer view in Daggerfall Scout is a continuous map interface, allowing you to zoom all the way out to see the whole Illiac Bay at once, or zoom right in to street level to see individual ground detail and buildings. The location browser on the left is for quickly finding a location.

 CartographerDev2

Prototype

The first step was to quickly prototype a dynamic scrolling map. The screenshot to the left renders each world cell using a single texture (from TEXTURE.002, .102, .302, and .402 depending on climate type). Locations are just plotted as red squares.

This fairly basic setup solves most of the early problems, and helps me visually confirm my new ultra-fast world queries are working. Now to throw out the experimental code and write it again properly.

 CartographerDev4

Top-Level Map

There will be several levels of fidelity to the map. To start with we have the whole Illiac Bay on-screen at once. On the left is the first version of this map. Water is blue, land is grey, and locations are just yellow dots. So many locations!

I tried several different approaches here before settling on dynamic textures mapped to quads. The next step is to add some details.

 

 CartographerDev6

Region Coding

This picture colours each region band appropriately. The colours have been averaged from the actual region ground textures. Location dots have been disabled so I can see details.

 CartographerDev7

Height Shading

Next I have modified climate colours based on elevation of the landscape. As you zoom in, this will be replaced by actual deformed terrain and textured ground.

Nothing here is pre-generated, this is all built dynamically by reading data from MAPS.BSA (locations), CLIMATE.PAK (climate), POLITIC.PAK (region) and WOODS.WLD (elevation data). Every “world pixel” is built from a combination of data from these sources.

 CartographerDev8

Zooming In

This screenshot is a slightly zoomed-in version of the above. This is about the limit of the top-level map, once you zoom in much further, 3D elevation maps will start to be paged in. My next post will show the process of this being added.

Visual Diary: Odds & Ends

During the busy period at work, I’ve been tinkering on various odds and ends in Daggerfall Scout. This Visual Diary is a wrap-up of the things I’ve been playing with.

 SkyDev1

Skies

The skies in Daggerfall are not constructed by a sky box, sphere, or infinite plane. Rather they are a backdrop composed by two tiling images, one for the west part of the sky and one for the east part. As the player looks left and right, the images are scrolled left and right. As the player looks up and down (or levitates) the images are scrolled up and down slightly to match the horizon line. This approach made for some dazzling skies back in the day, but has a few technical limitations for modern interpretations.

  • While the sky looks quite decent at 1024×768 (screenshot), it does not scale well to high resolutions. On a widescreen monitor running at 1920×1200, the visible portion of our sky backdrop is being enlarged by over 30 times its original size. No amount of filtering can save it from looking stretched, and this spoils the effect greatly.
  • By using backdrops, the sky does not actually go above the player’s head (or very far below the horizon), which limits the maximum pitch (angle looking up or down) to around 60-70 degrees.. If you’ve played Daggerfall, you might remember not being able to look straight up or down as you can in modern games. This is because there’s nothing up there to see!
  • The sky backdrops are designed to give the illusion of sky to a player standing at ground level, or levitating slightly above it. My goal in Daggerfall Scout is to allow the explorer to walk around towns, or zoom all the way out to view a whole city at once. In the former case, skies would look fine, in the latter any pairing between horizon lines is lost and the illusion fails: the sky is visibly just an image painted behind a tiny floating city.

With the above in mind, I need to make a choice on how best to represent the sky in Daggerfall Scout. Do I turn the sky on when walking around at street level, then off again when the explorer zooms out too far? I would also prefer to be able to look 90 degrees up and down, but Daggerfall’s sky backdrops do not allow for this.

Unfortunately, there is no perfect solution. My options are to either limit mobility and view angle, turn skies on and off as required, or to implement another type of sky (which means losing Daggerfall flavour). I will return to this later.

 

 ShadowDev1

Shadows

This one is easy to turn on in Ogre (screenshot made using basic stencil shadows), and the effect is quite dramatic even with traditional per-face lighting. Shadows can be adjusted for time of day so they are long in the early morning or late afternoon, and short during the middle of the day.

I am not using shadows permanently at this stage, as the final decision on how to handle shadows has not been made. I will probably end up using a unified light and shadow model before the final version, so this particular shadow technique is not final.

Also, If you look to the bottom-right of this screenshot, you will notice a working compass. This should help explorers keep oriented in cities.

 

 LightDev10LightDev9LightDev5LightDev6

LightDev7

Lights

Before illuminating my scenes properly, I wanted to get an idea of where lights are positioned, and how their radius extends to the buildings surrounding them. As a mock-up, I dropped in a red semi-transparent sphere over every light source in Daggerfall city. You might remember from Visual Diary: Bright Ideas that street lights are added wherever a flat using a texture from TEXTURE.210 (Lights) is present. In the first screenshot, you can see the lamp-post at the middle of the red sphere.

In the second screenshot is the entire city of Daggerfall, with all “lights” visible. There are several dozen potential light sources in this scene. This is also a great insight into how the developers positioned lights. They line walkways, surround gardens and flood the wealthy market district. Not all cities are as well lit as Daggerfall. Wayrest only has several in key points, and Sentinel is almost lacking street lights entirely.

In the next three screenshots, I have enabled a simple per-pixel shader that creates a light radius against buildings based on their distance from a light source. I’ve turned off textures so I can get a better idea of how the values look moving from light to shade. If anything, it looks a bit too smooth. I quite liked the rougher bands of the old Daggerfall lights, but this is something that can be implemented after some time playing with shaders.

Keeping in mind that I wish to view cities from street level, or all at once, one of the frustrations I have with lighting is the performance hit on lower-end hardware. To show the whole scene from above with all lights visible, a forward renderer requires a tremendous number of passes. There are several ways to optimise this (LuciusDXL has given me some great pointers), but the absolute best results come from using a different pipeline (deferred rendering, or light indexed deferred rendering).

I have chosen not to write my own engine this time around so I can focus on exploring Daggerfall, and not sink valuable time into engine building. The downside to this is that I’m limited to the features my chosen engine (Ogre) offers. Now, I’m aware Ogre’s compositor framework allows you to build a deferred renderer, but this involves more time than I am willing to invest. So for now at least, I’m limiting myself to forward rendering.

What this means is that I need to optimise lights when at ground level, and turn them off when the explorer zooms out to view the whole city at once. Alternatively when zoomed out, I can just switch off per-pixel materials and add simple point lights to the scene. It’s not like you can see much detail from all the way up there anyway.

Visual Diary: Climate Processing

The various regions/provinces in Daggerfall can take on a broad range of appearances. From the dry deserts of Alik’r, to the fecund swamps of Totambu, Daggerfall applies local flavour to almost every corner of the map. Different parts of the game world not only have distinct textures for most buildings, but varied plant life as well.

A primary goal for Daggerfall Scout is to faithfully display world maps as they would appear in the game. This is where climate processing comes in. When a city is correctly passed through the climate processer, it will take on the appearance of a desert, swamp, mountainous, or temperate location. Before I could do this, I had to find where Daggerfall itself stored information about a location’s climate, including native plant life.  Today’s Visual Diary steps through the process from research to implementation.

texturearchivemap

A Red Herring

The first place I looked was the map data in WOODS.WLD. Every pixel on the small map data has a corresponding large map chunk. One of the data values in the header for this chunk always corresponds to a ground texture. This seemed like a promising place to start.

After a bit of experimentation, I found the texture value only seemed valid some of the time. In many cases, it was flat out wrong. The city of Sentinel is a good example. The large map chunk for this world pixel reported the ground texture as 302 (temperate), not 2 (desert).

In frustration, I saved out the data into a visual format with intriguing results. Only the far eastern part of the world map, and some of the south, had any definition. The rest was a flat sea of “temperate”. This obviously wasn’t right, so I had to keep looking.

I’m guessing the data visualised to the left is either from an old version of the game world, or used in some as-yet not understood way.

climate226

CLIMATE.PAK

The next place to look was CLIMATE.PAK, which also defines a single pixel that can be overlaid on the standard 1000×500 world map. This data has been known about for some time, but I’ve not looked deeply into how it is used.

The first step was to see what kinds of numbers we’re dealing with. Each climate pixel is a value from 223 to 232. To see this visually, I saved out a series of climate maps, each with a single climate highlighted where it’s used in the world. The screenshot to the left shows climate 226, which covers the mountainous regions of Wrothgaria, Dragontail, and interestingly the Isle of Balfiera. This seemed far more promising. Each climate zone neatly overlaid visually distinct areas of the world.

Once I knew which areas were affected by a particular climate, I fired up Daggerfall and started taking notes. After a few hours of testing, I found Daggerfall always uses the same texture base and plant archive for any given climate value. This allowed me to build a table of which climate pixels uses which texture series for buildings and ground scenery.

If you’ve used Daggerfall Explorer, then you have probably played with my climate texture swaps already. Knowing how to make the swap is one thing. Understanding where the game draws this data from is quite another. With this in place, I was armed to build climate processing into Daggerfall Scout.

regiondev1

Pretty!

New code doesn’t always go to plan. The trees and other ground scenery are processed correctly, but the ground and buildings textures aren’t quite right. Castle Sentinel is particularly attractive.

After quickly fixing a few bugs, things start looking better.

regiondev2-swampregiondev3-desertregiondev4-mountain

Working Climate Processing

Finally, Daggerfall Scout will build a map using the correct building textures and ground scenery for that climate. This has been confirmed by travelling through Daggerfall with Scout running alongside to many locations in each climate band.

The screenshots to the left show maps using swamp, desert, and mountain climates, with distinct scenery in place.

The next step will be to drop in skies. Once again I need to do some research to find out exactly which skies Daggerfall uses where. I’ll put up another visual diary once I’ve made more progress.

Visual Diary: Bright Ideas

Note: This diary is about research into the file formats. Screenshots are from Daggerfall running in DOSBox.

So I’ve been trying to find where Daggerfall stores extra information about flat billboard objects strewn throughout the world. Cows moo, cats meow, and lights… well they light things. For some time I just assumed this information was somewhere unknown in the file formats, and lately I’ve been going mad trying to find it. After not making any progress, I decided that sound effects and lighting properties must be hard-coded based on the texture used.

To test this out, I quickly modified my BSA readers to write back different textures for miscellaneous block flats in BLOCKS.BSA. In the following screenshots the only thing I have modified is the texture index.

lighttest1

Default Scene

This screenshot is of a General Store facade at night. Note the street light is illuminating the store nicely.

Before taking this screenshot, I have prepared a few modified BLOCKS.BSA files with different flats in place of the defaults. The purpose was to see how Daggerfall handled a simple texture index change. If the light kept flickering, then I knew lights were stored elsewhere and had to keep looking.

lighttest2

Lights Out!

After changing the texture index from a street light to a spell effect, the lighting effect is no longer applied. This is a good sign that I’m on the right track.

lighttest3

Testing A Theory

After putting the lights out, it seemed likely that Daggerfall was injecting a standard light source whenever the engine encountered a flat billboard using any texture from TEXTURE.210 (Lights). The billboard is also set to be emissive so that it appears to be self-lit.

To test this out, I replaced the street lights with braziers, and the lights returned. This was enough to prove how lights work, but what about cows and cats? Are billboard sound effects played based on texture alone?

lighttest4

Daisy

For the next test, I found a nice dark spot with a cow standing near a building. This cow is mooing happily – and frequently!

lighttest5

Flame On

I then replaced Daisy with a brazier. Not surprisingly, the mooing stopped completely and the billboard started illuminating the building behind it.

The lesson learned here is never make assumptions when dealing with Daggerfall’s file formats. Just because it makes sense for something to work a certain way doesn’t mean Daggerfall does it like that. Hard-coded lights and sounds, based on texture index, is a good example of how Daggerfall manages not to conform.

The good news is this behaviour is very easy to reproduce in Daggerfall Scout, and I can stop looking for light and sound definitions in RMB blocks. I can start working on my own lighting model to reproduce those wonderfully atmospheric flickering lights.

I also have a suspicion that Daggerfall scales trees and other objects in a hard-coded manner as well. To test this, I’ll use the same texture modification technique on trees and find out. If the results are interesting, I’ll add to this visual diary.

Visual Diary: Adding Ground Detail

Every now and then, I plan on adding a visual diary showing a feature as it progresses. Today’s diary is about ground details.

 It's A Start

It’s A Start

I’ve added a simple ground plane under the city map. The ground in Daggerfall is composed of squares fitting perfectly under a block. The ground is tiled together in the same way blocks are tiled together to form a complete map.

The next step is texture it.

 Um Not Quite

Um, Not Quite

Even though I’ve written ground builders a few times in the past (the first was in Daggerfall Explorer), adding this feature into a new engine is bound to encounter bugs. I’ve not quite nailed it this attempt.

 Done

Textures Done

The ground textures are now being placed and aligned correctly across all maps.

 Needs Region Textures

Still Needs Region Processing

Daggerfall uses texture swaps for each region/climate type. You can play with these combinations in Daggerfall Explorer. Here is Sentinel using the default “temperate” texture set used while developing the ground plane.

At some point, I need to re-code the ability to swap in the correct texture. But for now I want to do something a little more interesting.

 New Flats

New Flats

In this screenshot, I’ve started adding flats. My first attempt isn’t that successful. They aren’t at the right height, size, or transparent. But they are standing in the right places, so it’s is a good start.

Another issue with these flats you can’t see in the screenshot is they always face the camera. If you fly into the air and look down, they’re lying flat on the ground! A quick change to the axis alignment will sort that out.

Flats1
Flats2

Working Flats

Flats are now correctly positioned, axis aligned, and alpha-enabled. I feel the scaling needs some work. At this time, it is unknown which numbers Daggerfall uses to scale flats. I will refine this as development progresses.

I’ve never been a huge fan of flat billboards in games, but they were a standard technique in days past and part of the Daggerfall landscape. Despite being antique tech, I feel some of the old charm has started to make its way back into the otherwise dreary maps in Daggerfall Scout.

The next step is to do some gardening and add the local plant life, such as trees, etc. These aren’t stored in the flats records along with everything else. I need to find them before I can draw them from the game files.

I hope you enjoyed this first visual diary.