Visual Diary: XNALibrary Update

After a hectic mid-year rush things are finally settling back to normal. I will have more time to dedicate to my Daggerfall tools for another few months before the silly season begins. I am now working on getting Beta 1 of Daggerfall Modelling ready for release.

When I started Daggerfall Modelling the goal was to make a quick and easy Collada exporter for Daggerfall’s 3D assets. It didn’t take long before I realised an opportunity to revive Daggerfall Scout in a more useful form. Like all projects, escalating the scope after most of the code has been written is a recipe for trouble. While trying to wedge in action records to my scenes (levers, switches, etc.), I came to the realisation my code had become as messy as hell. To accomplish what I had in mind a total rewrite of the scene manager was needed.

As an adjunct to Daggerfall Connect, I have also been developing an XNA class library that makes using Daggerfall assets with XNA a snap (creatively called XNALibrary). This code base is something I wish reuse for other projects, so it made sense to get all the scene management and rendering out of Daggerfall Modelling and build it properly into XNALibrary. The end result is a small XNA 3D engine designed specifically for rendering Daggerfall scenes. I then had to rip out all the old code from Daggerfall Modelling and port it over to XNALibrary as the primary engine. It was a lot of work just to end up back where I started, but internally the code is much improved. Following is a quick tour of what’s new in XNALibrary, as seen from Daggerfall Modelling Beta 1.

 

Scene Graph

Previously, Daggerfall Modelling used a one-dimensional array of objects and looped through them to draw scenes. This is fine for small environments but doesn’t scale up very well.  It also makes visibility testing and render batching harder than it needs to be. When it came time to link up the levers and switches, I realised it was time to put in a scene graph.

I decided to use a simple Bounding Volume Hierarchy based around spheres. The beauty of bounding spheres is that you don’t have to worry about axis alignment as you do with boxes. Spheres may not wrap objects quite as tightly, but they’re a lot easier to handle when transformed (rotated, translated, scaled, etc.) as the volume is always the same shape and alignment in 3D space.

On the left are a couple of screen-shots of the bounding volumes, rendered as wireframe spheres. The root node is red, general models are white, and sprites are green. Resources are added to the scene from the top down, with renderable objects like models and sprites in the leaves of the tree. This hierarchy makes it easy to cull non-visible groups – if a node isn’t visible, then none of its children can be visible either. It also simplifies mouse picking and collision as these tests are only performed where necessary. Batching visible polygons by texture becomes easier, which greatly improves rendering times. And finally, linking up chained action records becomes simplified thanks to each scene node being a discrete class aware of its own state.

Action Chains

Certain objects in Daggerfall have action records that describe how this object can move, rotate, etc. An example every Daggerfall player will be familiar with is the throne room in Privateer’s Hold (pictured left). When you throw the switch an action record makes the lever rotate, which chains to another action record to make the platform rise, which finally chains to the throne to make it rise also. The end result is you throw the switch and the platform rises (along with the throne and the player).

If you open up the screenshot to the left, you will see a red line going from switch, to platform, to throne. This is the action chain linking the objects together. The next screenshot shows the same linkage in Mantellan Crux – from the axe on the floating island, to the crossbow, to the giant sword. At some point, I will number the nodes so it becomes easier to visualise where action chains begin and end.

In Daggerfall Modelling Beta 1 it will be possible to trigger action chains and watch the animations run. There are other actions, such as teleporting the player or casting spells, but I am only going to support action that cause things to move around the scene.

Once gravity and collision are implemented, you will be able to explore dungeons from a first-person perspective with the ability to operate doors and platforms. You can also just turn off collision and gravity to fly around the scene with total freedom.

Add-In Components

The renderer in XNALibrary can plug in components that add functionality. This is quite basic at the moment, but will be extended down the road as needed.

Currently, the add-ins are a billboard manager for rendering sprites like trees, animals, NPCs, and a sky manager for drawing the background sky in location exteriors. You can see these components in action to the left.

The next release of Daggerfall Modelling should be ready in 3-4 weeks. This will coincide with an update to the Daggerfall Connect library download with XNALibrary updates included. All of this is in the SVN now, if you want to download the code as it’s being written. Just keep in mind not everything may be in a stable state in the latest SVN version.

Visual Diary: Collada Export

Daggerfall Modelling will be capable of exporting individual models to Collada format. This is a universal format for describing digital content such as 3D models and scenes. When this feature goes live in the next release, you will be able to export Daggerfall’s 3D data and load it back into modelling packages such as Blender and 3DS Max. You could also use this data in another 3D engine. One potential such use will be replacing Daggerfall’s standard models with high-poly variants in DaggerXL.

I’ve been working on the Collada export over the last couple of days and would like to document progress so far.


Wiring Up

The first step is to create a C# wrapper for the Collada DOM.

I started by creating a static build of the Collada DOM DLL, which is then imported into a CLI/C++ project. This wrapper project is able to handle communication between native code (the Collada DOM in this case) and managed code. Fortunately it was not necessary to wrap the entire Collada DOM, just a few methods of the DAE and daeElement classes with some string marshalling thrown in. The end result is a self-contained .NET assembly exposing the Collada DOM to any .NET language.

All I need to do now is code the exporter. I started with a simple model with a distinctive shape (pictured left). This arrow turned out to be extra useful for tackling axis-alignment in the next step.


First Import

My first pass at the exporter creates a tidy .dae file which Blender opens without any complaints. I love it when code works first run.

I did notice however the arrow was no longer pointing along the correct axis. The reason for this is that Blender uses +Z for up and XNA uses +Y for up. A quick scan of the Collada spec and forums shows this is a common issue with an easy solution. My exporter class is updated so that any axis can be considered up. This is achieved with a rotation on the model node and setting the <up_axis> in <asset> to reflect the appropriate setting.

With the model vertices in place, it’s now time to add textures.

Textured Models

This was more work than I expected. Each set of triangles references a material, each material instantiates an effect, every effect points to an image, which in turn points to a file on disk. Oh, and the materials have to be bound to the model in the scene setup.

Once my head stopped spinning from all the redirection, I could see the versatility of this method. I was also impressed how detailed scenes can be constructed using Collada, right down to cameras and lights. I want to look into exporting blocks, cities, and dungeons to Collada files at a later date.

On the left are a few screenshots of some distinctive Daggerfall models as viewed in Blender.

The exporter is more or less ready and just needs a little tidy-up. I have confirmed that my exported .dae files pass the Collada Coherency Test without any errors, and so far all the models I’ve loaded into Blender look perfect.

The Collada export feature will be available in the next release of Daggerfall Modelling. I’m looking forward to seeing what you all do with it in the future.

Visual Diary: Daggerfall Modelling Views

In my last post, I talked about the thumbnails view in Daggerfall Modelling. This is just one of the possible ways of exploring 3D data from Daggerfall. Read on to see all supported view modes and get an idea of how they work together.



Thumbnails

This has been tweaked since my last post. I’ve selected a different viewing angle to give more of a 3/4 view of each model rather than a frontal view. I have disabled back-face culling to ensure a thumbnail is rendered even when facing away from the camera. The overall effect is that almost all thumbnails show a decent representation of the model that’s easy for the eye to track.

Thumbnails are generated in real-time from model data. I plan to add a few embelishments like watching the model spin when the mouse passes over a thumbnail.

Scrolling the thumbnail view can be performed from your mouse wheel or by dragging the page as you would the touch screen of an iPhone. You can impart velocity to the scroll with a flick of the mouse. The view will keep scrolling until you grab it again or roll the mouse wheel.

The thumbnail view isn’t just for viewing thousands of models either. When you have a city or dungeon loaded, you can switch over to thumbnails to view just the models used for that location. This filtered list of thumbnails will be helpful for finding the exact model to export in large areas where models can be duplicated dozens of times.

If you double-click on a thumbnail, it takes you straight to our next view, the Single Model View.


Single Model View

After finding a model in the thumbnails view, you can explore that model in greater detail. This view is similar to the model view in Daggerfall Explorer.

It is possible to search for individual models by ID (check the left-hand pane of top screenshot).

When viewing a location, you can also jump to single model view by clicking on any model within the location itself. Daggerfall Modelling uses bounding volumes so you can pick any model right out of the scene.

Location View – Top-Down Camera

The first way of viewing locations is the default top-down camera. This camera is always facing down on the scene and you scroll around as you would a 2D map. Clicking on a specific block in the map (screenshot) will take you directly to that block.

If you’re only interested in one block, you will be able to filter out everything but the block you’ve clicked on.

When you see a model you’re interested in, you can switch to inspecting that model in the Single Model View without losing your place in the Location View (unless you change locations).


Location View – Free Camera

The free camera allows you to move around in first-person mode through city streets or narrow dungeon corridors. Pictured left is the city of Sentinel (top) and Mantellan Crux (bottom).

The Mantellan Crux screenshot shows how the dungeon layout map is displayed. Each square represents a dungeon block. Currently the starting block (dungeon entrance) is blue-green, and other blocks are gold. The RGB values match those used in Daggerfall to display dungeon layouts.

Before this tool is completed, I plan to implement an “interior view” so you can crack open building interiors to select the furniture and other models within.

You can see there will be several ways of getting around Daggerfall’s 3D data. As a pure exploring tool, Daggerfall Modelling will replace Daggerfall Explorer, Daggerfall Cartographer, and Daggerfall Scout. The goal is to create a rich visual path for exploring models individually, or as part of the locations and dungeons they build. Every view will offer some technique for identifying, isolating, and exporting a model.

I hope to have a beta of the exploring side up for download in the next few weeks. Once any problems with the GUI have been ironed out, I will begin work on export queues and saving models to Collada format.

Visual Diary: Building Maps – Part 1

The Cartographer view in Daggerfall Scout is a continuous map interface, allowing you to zoom all the way out to see the whole Illiac Bay at once, or zoom right in to street level to see individual ground detail and buildings. The location browser on the left is for quickly finding a location.

 CartographerDev2

Prototype

The first step was to quickly prototype a dynamic scrolling map. The screenshot to the left renders each world cell using a single texture (from TEXTURE.002, .102, .302, and .402 depending on climate type). Locations are just plotted as red squares.

This fairly basic setup solves most of the early problems, and helps me visually confirm my new ultra-fast world queries are working. Now to throw out the experimental code and write it again properly.

 CartographerDev4

Top-Level Map

There will be several levels of fidelity to the map. To start with we have the whole Illiac Bay on-screen at once. On the left is the first version of this map. Water is blue, land is grey, and locations are just yellow dots. So many locations!

I tried several different approaches here before settling on dynamic textures mapped to quads. The next step is to add some details.

 

 CartographerDev6

Region Coding

This picture colours each region band appropriately. The colours have been averaged from the actual region ground textures. Location dots have been disabled so I can see details.

 CartographerDev7

Height Shading

Next I have modified climate colours based on elevation of the landscape. As you zoom in, this will be replaced by actual deformed terrain and textured ground.

Nothing here is pre-generated, this is all built dynamically by reading data from MAPS.BSA (locations), CLIMATE.PAK (climate), POLITIC.PAK (region) and WOODS.WLD (elevation data). Every “world pixel” is built from a combination of data from these sources.

 CartographerDev8

Zooming In

This screenshot is a slightly zoomed-in version of the above. This is about the limit of the top-level map, once you zoom in much further, 3D elevation maps will start to be paged in. My next post will show the process of this being added.

Visual Diary: Odds & Ends

During the busy period at work, I’ve been tinkering on various odds and ends in Daggerfall Scout. This Visual Diary is a wrap-up of the things I’ve been playing with.

 SkyDev1

Skies

The skies in Daggerfall are not constructed by a sky box, sphere, or infinite plane. Rather they are a backdrop composed by two tiling images, one for the west part of the sky and one for the east part. As the player looks left and right, the images are scrolled left and right. As the player looks up and down (or levitates) the images are scrolled up and down slightly to match the horizon line. This approach made for some dazzling skies back in the day, but has a few technical limitations for modern interpretations.

  • While the sky looks quite decent at 1024×768 (screenshot), it does not scale well to high resolutions. On a widescreen monitor running at 1920×1200, the visible portion of our sky backdrop is being enlarged by over 30 times its original size. No amount of filtering can save it from looking stretched, and this spoils the effect greatly.
  • By using backdrops, the sky does not actually go above the player’s head (or very far below the horizon), which limits the maximum pitch (angle looking up or down) to around 60-70 degrees.. If you’ve played Daggerfall, you might remember not being able to look straight up or down as you can in modern games. This is because there’s nothing up there to see!
  • The sky backdrops are designed to give the illusion of sky to a player standing at ground level, or levitating slightly above it. My goal in Daggerfall Scout is to allow the explorer to walk around towns, or zoom all the way out to view a whole city at once. In the former case, skies would look fine, in the latter any pairing between horizon lines is lost and the illusion fails: the sky is visibly just an image painted behind a tiny floating city.

With the above in mind, I need to make a choice on how best to represent the sky in Daggerfall Scout. Do I turn the sky on when walking around at street level, then off again when the explorer zooms out too far? I would also prefer to be able to look 90 degrees up and down, but Daggerfall’s sky backdrops do not allow for this.

Unfortunately, there is no perfect solution. My options are to either limit mobility and view angle, turn skies on and off as required, or to implement another type of sky (which means losing Daggerfall flavour). I will return to this later.

 

 ShadowDev1

Shadows

This one is easy to turn on in Ogre (screenshot made using basic stencil shadows), and the effect is quite dramatic even with traditional per-face lighting. Shadows can be adjusted for time of day so they are long in the early morning or late afternoon, and short during the middle of the day.

I am not using shadows permanently at this stage, as the final decision on how to handle shadows has not been made. I will probably end up using a unified light and shadow model before the final version, so this particular shadow technique is not final.

Also, If you look to the bottom-right of this screenshot, you will notice a working compass. This should help explorers keep oriented in cities.

 

 LightDev10LightDev9LightDev5LightDev6

LightDev7

Lights

Before illuminating my scenes properly, I wanted to get an idea of where lights are positioned, and how their radius extends to the buildings surrounding them. As a mock-up, I dropped in a red semi-transparent sphere over every light source in Daggerfall city. You might remember from Visual Diary: Bright Ideas that street lights are added wherever a flat using a texture from TEXTURE.210 (Lights) is present. In the first screenshot, you can see the lamp-post at the middle of the red sphere.

In the second screenshot is the entire city of Daggerfall, with all “lights” visible. There are several dozen potential light sources in this scene. This is also a great insight into how the developers positioned lights. They line walkways, surround gardens and flood the wealthy market district. Not all cities are as well lit as Daggerfall. Wayrest only has several in key points, and Sentinel is almost lacking street lights entirely.

In the next three screenshots, I have enabled a simple per-pixel shader that creates a light radius against buildings based on their distance from a light source. I’ve turned off textures so I can get a better idea of how the values look moving from light to shade. If anything, it looks a bit too smooth. I quite liked the rougher bands of the old Daggerfall lights, but this is something that can be implemented after some time playing with shaders.

Keeping in mind that I wish to view cities from street level, or all at once, one of the frustrations I have with lighting is the performance hit on lower-end hardware. To show the whole scene from above with all lights visible, a forward renderer requires a tremendous number of passes. There are several ways to optimise this (LuciusDXL has given me some great pointers), but the absolute best results come from using a different pipeline (deferred rendering, or light indexed deferred rendering).

I have chosen not to write my own engine this time around so I can focus on exploring Daggerfall, and not sink valuable time into engine building. The downside to this is that I’m limited to the features my chosen engine (Ogre) offers. Now, I’m aware Ogre’s compositor framework allows you to build a deferred renderer, but this involves more time than I am willing to invest. So for now at least, I’m limiting myself to forward rendering.

What this means is that I need to optimise lights when at ground level, and turn them off when the explorer zooms out to view the whole city at once. Alternatively when zoomed out, I can just switch off per-pixel materials and add simple point lights to the scene. It’s not like you can see much detail from all the way up there anyway.

Visual Diary: Climate Processing

The various regions/provinces in Daggerfall can take on a broad range of appearances. From the dry deserts of Alik’r, to the fecund swamps of Totambu, Daggerfall applies local flavour to almost every corner of the map. Different parts of the game world not only have distinct textures for most buildings, but varied plant life as well.

A primary goal for Daggerfall Scout is to faithfully display world maps as they would appear in the game. This is where climate processing comes in. When a city is correctly passed through the climate processer, it will take on the appearance of a desert, swamp, mountainous, or temperate location. Before I could do this, I had to find where Daggerfall itself stored information about a location’s climate, including native plant life.  Today’s Visual Diary steps through the process from research to implementation.

texturearchivemap

A Red Herring

The first place I looked was the map data in WOODS.WLD. Every pixel on the small map data has a corresponding large map chunk. One of the data values in the header for this chunk always corresponds to a ground texture. This seemed like a promising place to start.

After a bit of experimentation, I found the texture value only seemed valid some of the time. In many cases, it was flat out wrong. The city of Sentinel is a good example. The large map chunk for this world pixel reported the ground texture as 302 (temperate), not 2 (desert).

In frustration, I saved out the data into a visual format with intriguing results. Only the far eastern part of the world map, and some of the south, had any definition. The rest was a flat sea of “temperate”. This obviously wasn’t right, so I had to keep looking.

I’m guessing the data visualised to the left is either from an old version of the game world, or used in some as-yet not understood way.

climate226

CLIMATE.PAK

The next place to look was CLIMATE.PAK, which also defines a single pixel that can be overlaid on the standard 1000×500 world map. This data has been known about for some time, but I’ve not looked deeply into how it is used.

The first step was to see what kinds of numbers we’re dealing with. Each climate pixel is a value from 223 to 232. To see this visually, I saved out a series of climate maps, each with a single climate highlighted where it’s used in the world. The screenshot to the left shows climate 226, which covers the mountainous regions of Wrothgaria, Dragontail, and interestingly the Isle of Balfiera. This seemed far more promising. Each climate zone neatly overlaid visually distinct areas of the world.

Once I knew which areas were affected by a particular climate, I fired up Daggerfall and started taking notes. After a few hours of testing, I found Daggerfall always uses the same texture base and plant archive for any given climate value. This allowed me to build a table of which climate pixels uses which texture series for buildings and ground scenery.

If you’ve used Daggerfall Explorer, then you have probably played with my climate texture swaps already. Knowing how to make the swap is one thing. Understanding where the game draws this data from is quite another. With this in place, I was armed to build climate processing into Daggerfall Scout.

regiondev1

Pretty!

New code doesn’t always go to plan. The trees and other ground scenery are processed correctly, but the ground and buildings textures aren’t quite right. Castle Sentinel is particularly attractive.

After quickly fixing a few bugs, things start looking better.

regiondev2-swampregiondev3-desertregiondev4-mountain

Working Climate Processing

Finally, Daggerfall Scout will build a map using the correct building textures and ground scenery for that climate. This has been confirmed by travelling through Daggerfall with Scout running alongside to many locations in each climate band.

The screenshots to the left show maps using swamp, desert, and mountain climates, with distinct scenery in place.

The next step will be to drop in skies. Once again I need to do some research to find out exactly which skies Daggerfall uses where. I’ll put up another visual diary once I’ve made more progress.