Workshop Updates

Lately, I have been spending most of my development time on the DFConnect Library. Besides making Daggerfall’s file formats more accessible, I now have the added benefit of a framework for rapidly discovering new things. I hadn’t realised just how cumbersome my old code had become until I started writing tutorials and demos for DFConnect. I can do things with DFConnect in a few lines of clear code that previously required a lot of jumping through hoops. There’s a lot to be said for a clean rewrite and rethink. Time spent on DFConnect is time saved in the future. Best of all, it’s something the entire community can take advantage of.

I’ll give you an overview of what I’ve been up to.

DFConnect 0.4.0

The next build of DFConnect is almost finished, and will be up in the next few days (this weekend at the latest). This build supports reading from BLOCKS.BSA, with a strong focus on dungeon (RDB) blocks. Outdoor (RMB) blocks are implemented, but still need some testing. So dungeon blocks will be the focus of this release, and outdoor blocks the release after.

DFConnect Demos

To complement the tutorials, I have started work on a series of demos for getting the most of DFConnect. These demos will cover things like proper 3D visualisation of models, blocks, and maps. Future demos will cover action records in dungeons (moving blocks, levers, etc.) and working with world data.

These demos will be available as pre-built explorers for everyone to play with. The source code for these demos will be bundled with the DFConnect Library starting from 0.4.0.

WorkshopDemoEngine

This is a self-contained 3D rendering engine – basically a simple wrapper for Ogre3D 1.6.3. All DFConnect demos will use this engine for 3D visualisation.

The WorkshopDemoEngine does not expose the full functionality of Ogre, just enough to do the job at hand. The goal is to provide a series of simple and direct examples of how to work with DFConnect data without cluttering these examples with engine-specific boilerplate. The implementation should be generic enough for you to learn the important concepts and apply them to your own engine of choice.

The second purpose of the WorkshopDemoEngine is a simplistic foundation for everyone. If you are just getting started with the DFConnect examples, or C# itself, the engine will hopefully be an easy starting point for you to get cool results without struggling with a 3D library on top of everything else.

Dungeon Block Viewer

Below are a couple of screenshots of a test dungeon block viewer. This uses DFConnect and the WorkshopDemoEngine. A cleaned up version of this will be released with DFConnect 0.4.0, along with source code.

Also, a big thanks to Lucius for his tips on cleaning up the edges of my billboards. The 2D flats have lost their black edges, and now blend slightly into the environment instead (not really visible in jpeg below). Enabling anisotropic filtering on supported hardware has cleaned up texture noise on long stretches adjacent to the view direction (see bottom-right screen shot).

blockviewer1 blockviewer2
blockviewer3 blockviewer5

Future of Daggerfall Scout

I’ve been thinking a lot about Daggerfall Scout, and where it belongs in regards to how I spend my time on this hobby. A few things have made me rethink the purpose of this tool:

  1. The best part of Daggerfall Scout has always been exploring cities and dungeons. This will very shortly be possible using the DFConnect demos. While these demos won’t have all the features planned for DF Scout, they will do much the same thing – with source code and faster to boot.
  2. Working with Daggerfall has never been easier for me than it is now with the DFConnect Library. As Daggerfall Scout is based out of a much older code base, it carries a lot of baggage along with it. Every time I look at the DF Scout code these days, I spend all my time wishing I’d written DFConnect first to make life easier on myself.
  3. Last but not least is DaggerXL. What Lucius is doing here is incredible! Much of the “wow” factor of DF Scout has been far exceeded by DaggerXL. If I had to choose between exploring the world in DF Scout, or in DaggerXL, I’d pick DaggerXL every time. If that’s how I feel about my own program, then others probably feel much the same.

When I started on DF Scout earlier in the year, the Daggerfall scene was stagnant. Since then, Daggerfall has been released for free and Lucius has come along to bring our remake dreams back to life. With the community revitalised, DF Scout is just looking really tired. To be brutally honest, I think I’m wasting my time with it.

Moving forwards, everything good from DF Scout will be cannibalised for DFConnect. So rather than releasing one big program at some point in the future, I get to release lots of small purposeful programs almost right away. For people who just want to explore the game files, these programs will be available for download. For people who want to hack away at the Daggerfall files themselves, the source code for these programs will help you get started.

I feel this adds the most community benefit, and is more in line with the spirit of Daggerfall Workshop. Thinking back to the glory days of this site, I can only imagine how much more could have been accomplished if I’d started work on a Daggerfall library then, rather than sinking that time into other projects that ultimately went nowhere.

Community Forums

To start rebuilding our development community, I am reopening the Workshop forums in the next few weeks. This will be a place where Daggerfall hackers can get together and talk about their projects. If you are using the DFConnect Library, this will be the place to share your ideas and get support from the community. Promising projects will be given their own subforum, just like in the past.

In the meantime, I am opening up comments in the blog for new posts starting with this one. I look forward to hearing all your thoughts.

Introducing DFConnect

DFConnect is a class library that handles the nitty-gritty of reading Daggerfall’s known file formats. Written completely in C#, this is a ground-up rewrite of the code I have developed over several years, and will allow other developers to create their own Daggerfall tools without facing complicated format specifications.

The key feature of DFConnect is simplicity. Supported file formats can be opened and enumerated with only a few lines of code. Data is fetched easily from native files (no need to allocate or convert anything, just fetch away), and emitted back to your application in a format that can be effortlessly used. Data is emitted as standard managed types (e.g. strings, images, and structures) but can also be emitted as byte arrays for some data types (such as A8R8G8B8 texture data for marshalling into your favourite 3D engine).

Built on the Microsoft .NET Framework 2.0, DFConnect is compatible with modern Microsoft operating systems (XP, Vista, and Windows 7). The DFConnect library can be used in any .NET language, such as C# and VB.NET. This also grants you a wide variety of options for 3D visualisation with graphics libraries like MOGRE, Irrlicht.NET, and XNA that also support these languages.

While DFConnect is still in the early stages of development, I am going to make it available to everyone for testing right from the start. A series of basic tutorials will accompany the initial release, and more will be added over time. Please keep an eye on the Tutorials and Download pages for more information.

Project Work

I’ve had some project work come up that will keep me busy for 2-3 weeks. I probably won’t have much free time, but will update if I can.

Visual Diary: Building Maps – Part 1

The Cartographer view in Daggerfall Scout is a continuous map interface, allowing you to zoom all the way out to see the whole Illiac Bay at once, or zoom right in to street level to see individual ground detail and buildings. The location browser on the left is for quickly finding a location.

 CartographerDev2

Prototype

The first step was to quickly prototype a dynamic scrolling map. The screenshot to the left renders each world cell using a single texture (from TEXTURE.002, .102, .302, and .402 depending on climate type). Locations are just plotted as red squares.

This fairly basic setup solves most of the early problems, and helps me visually confirm my new ultra-fast world queries are working. Now to throw out the experimental code and write it again properly.

 CartographerDev4

Top-Level Map

There will be several levels of fidelity to the map. To start with we have the whole Illiac Bay on-screen at once. On the left is the first version of this map. Water is blue, land is grey, and locations are just yellow dots. So many locations!

I tried several different approaches here before settling on dynamic textures mapped to quads. The next step is to add some details.

 

 CartographerDev6

Region Coding

This picture colours each region band appropriately. The colours have been averaged from the actual region ground textures. Location dots have been disabled so I can see details.

 CartographerDev7

Height Shading

Next I have modified climate colours based on elevation of the landscape. As you zoom in, this will be replaced by actual deformed terrain and textured ground.

Nothing here is pre-generated, this is all built dynamically by reading data from MAPS.BSA (locations), CLIMATE.PAK (climate), POLITIC.PAK (region) and WOODS.WLD (elevation data). Every “world pixel” is built from a combination of data from these sources.

 CartographerDev8

Zooming In

This screenshot is a slightly zoomed-in version of the above. This is about the limit of the top-level map, once you zoom in much further, 3D elevation maps will start to be paged in. My next post will show the process of this being added.

Updated User Interface

I have been working on a new user interface for Daggerfall Scout. I was often frustrated by the previous UI, due to conflicting requirements between the navigation and exploration components. I wanted to put more detail into the navigation pane (lots more detail), but I also wanted the 3D exploring pane to have as much real estate as possible. After trialling many solutions, a simple tabbed view proved the most elegant.

The free release of Daggerfall also prompted a complete rewrite of how I cache content. My initial design only supported an original Daggerfall CD (i.e. optical media). The new cache is more streamlined, in addition to supporting the free version of Daggerfall.

Note: I am a bit further along than these screenshots show, but future posts will be confusing unless I talk about the new UI first.

 DFScoutUI1

First Run

The first time you open Daggerfall Scout, it prompts you to set your Arena2 path. This is more in line my previous tools. The Arena2 path can be a Daggerfall CD, full installation, or unzip of the free download.

Once set, Daggerfall Scout will automatically attach to your Arena2 folder next time you open it. If you move/delete your Arena2 folder, DF Scout will simply reprompt until it is provided a valid Arena2 folder.

You will notice the two tabs at the top allow you to switch between the Cartographer and Explorer views. More on this below.

 DFScoutUI2

The New Cache

Once your Arena2 path is set, Daggerfall Scout will initialise the cache. Unlike the old cache that required everything to be built ahead of time (which could take several minutes), the new cache simply performs a few essential startup tasks. The whole process takes a few seconds the first time, and less than a second on subsequent opens. Content is then cached as needed.

This means DF Scout is ready to go in a matter of seconds, without burdening the user with a cache builder dialog. Not only is this a huge plus for everyone, it allows me to quickly trial changes to my converters without waiting for a complete build or navigating through several windows.

 DFScoutUI3

Cartographer & Explorer

It doesn’t look like it yet, but the Cartographer view is turning into something special. On the left is the familiar region-name browser you saw in the old UI. On the right, I am developing a complete 3D map of the game world you can zoom and pan around (think Google Earth). This 3D map is constructed from random landscape tiles and populated with all cities and locations. When zoomed out these appear as coloured dots, as you zoom in these progress from a simple 2D map to the actual location itself. This view will be the subject of my next several posts.

The Explorer view is more or less what you’ve already seen in previous screenshots (fly and walk around a town/dungeon), but with one exception. You are placed properly in the world itself, and all random landscape is generated around you. You will be able to fly from one end of the map to the other.

To make the two views as seamless as possible, I will be creating several methods of switching between them based on context. For example, if you are hovering over a location you’d like to explore in the Cartographer view, a quick shortcut will take you directly to that place in the Explorer view where you can explore at ground level.

This has just been an introduction to the new user interface. I will be posting more in the near future and showing off early stages of the Cartographer view.

Visual Diary: Odds & Ends

During the busy period at work, I’ve been tinkering on various odds and ends in Daggerfall Scout. This Visual Diary is a wrap-up of the things I’ve been playing with.

 SkyDev1

Skies

The skies in Daggerfall are not constructed by a sky box, sphere, or infinite plane. Rather they are a backdrop composed by two tiling images, one for the west part of the sky and one for the east part. As the player looks left and right, the images are scrolled left and right. As the player looks up and down (or levitates) the images are scrolled up and down slightly to match the horizon line. This approach made for some dazzling skies back in the day, but has a few technical limitations for modern interpretations.

  • While the sky looks quite decent at 1024×768 (screenshot), it does not scale well to high resolutions. On a widescreen monitor running at 1920×1200, the visible portion of our sky backdrop is being enlarged by over 30 times its original size. No amount of filtering can save it from looking stretched, and this spoils the effect greatly.
  • By using backdrops, the sky does not actually go above the player’s head (or very far below the horizon), which limits the maximum pitch (angle looking up or down) to around 60-70 degrees.. If you’ve played Daggerfall, you might remember not being able to look straight up or down as you can in modern games. This is because there’s nothing up there to see!
  • The sky backdrops are designed to give the illusion of sky to a player standing at ground level, or levitating slightly above it. My goal in Daggerfall Scout is to allow the explorer to walk around towns, or zoom all the way out to view a whole city at once. In the former case, skies would look fine, in the latter any pairing between horizon lines is lost and the illusion fails: the sky is visibly just an image painted behind a tiny floating city.

With the above in mind, I need to make a choice on how best to represent the sky in Daggerfall Scout. Do I turn the sky on when walking around at street level, then off again when the explorer zooms out too far? I would also prefer to be able to look 90 degrees up and down, but Daggerfall’s sky backdrops do not allow for this.

Unfortunately, there is no perfect solution. My options are to either limit mobility and view angle, turn skies on and off as required, or to implement another type of sky (which means losing Daggerfall flavour). I will return to this later.

 

 ShadowDev1

Shadows

This one is easy to turn on in Ogre (screenshot made using basic stencil shadows), and the effect is quite dramatic even with traditional per-face lighting. Shadows can be adjusted for time of day so they are long in the early morning or late afternoon, and short during the middle of the day.

I am not using shadows permanently at this stage, as the final decision on how to handle shadows has not been made. I will probably end up using a unified light and shadow model before the final version, so this particular shadow technique is not final.

Also, If you look to the bottom-right of this screenshot, you will notice a working compass. This should help explorers keep oriented in cities.

 

 LightDev10LightDev9LightDev5LightDev6

LightDev7

Lights

Before illuminating my scenes properly, I wanted to get an idea of where lights are positioned, and how their radius extends to the buildings surrounding them. As a mock-up, I dropped in a red semi-transparent sphere over every light source in Daggerfall city. You might remember from Visual Diary: Bright Ideas that street lights are added wherever a flat using a texture from TEXTURE.210 (Lights) is present. In the first screenshot, you can see the lamp-post at the middle of the red sphere.

In the second screenshot is the entire city of Daggerfall, with all “lights” visible. There are several dozen potential light sources in this scene. This is also a great insight into how the developers positioned lights. They line walkways, surround gardens and flood the wealthy market district. Not all cities are as well lit as Daggerfall. Wayrest only has several in key points, and Sentinel is almost lacking street lights entirely.

In the next three screenshots, I have enabled a simple per-pixel shader that creates a light radius against buildings based on their distance from a light source. I’ve turned off textures so I can get a better idea of how the values look moving from light to shade. If anything, it looks a bit too smooth. I quite liked the rougher bands of the old Daggerfall lights, but this is something that can be implemented after some time playing with shaders.

Keeping in mind that I wish to view cities from street level, or all at once, one of the frustrations I have with lighting is the performance hit on lower-end hardware. To show the whole scene from above with all lights visible, a forward renderer requires a tremendous number of passes. There are several ways to optimise this (LuciusDXL has given me some great pointers), but the absolute best results come from using a different pipeline (deferred rendering, or light indexed deferred rendering).

I have chosen not to write my own engine this time around so I can focus on exploring Daggerfall, and not sink valuable time into engine building. The downside to this is that I’m limited to the features my chosen engine (Ogre) offers. Now, I’m aware Ogre’s compositor framework allows you to build a deferred renderer, but this involves more time than I am willing to invest. So for now at least, I’m limiting myself to forward rendering.

What this means is that I need to optimise lights when at ground level, and turn them off when the explorer zooms out to view the whole city at once. Alternatively when zoomed out, I can just switch off per-pixel materials and add simple point lights to the scene. It’s not like you can see much detail from all the way up there anyway.

Daggerfall For Free!

The wonderfully huggable Bethesda have released Daggerfall as a free download. Announcement on the Bethesda Blog is here. A huge thanks to Bethesda for taking this step and releasing Daggerfall to the community.

This also means my exploring tools are now available to everyone – not just people who already own Daggerfall. I’m assuming Daggerfall Explorer, Jukebox, etc. will work with the free download of Daggerfall. I will test this out over the weekend. I will also update Daggerfall Scout so it no longer requires the original CD, and instead works with the Zip file released by Bethesda.

Happy days!

Back On Board

End of financial is finally over for another year (happy!). I’m almost on top of things, and will be back into regular updates starting this weekend.

DaggerXL

In case you’ve missed it, a chap called LuciusDXL has started work on a Daggerfall engine remake called DaggerXL. What he has shown so far is incredibly promising. You will find his site, along with screenshots and movies here.

I’m very excited by this ambitious project, and look forward to watching his progress unfold.

EOFY

Sorry for the decrease in updates lately. Australia is heading into end of financial year, and I have plenty of work coming in to keep me busy (not to mention preparing my tax return – blegh).

I’m still putting in a little time for the Workshop every night, and I should have something more interesting post in another couple of weeks. Once July 1 rolls around the worst of it will be over.

Interface Overhaul

Just a quick update today. After realising I spend nearly all my time in Daggerfall Scout with the interface collapsed, I started getting annoyed every time I had to expand the interface to pick another location to explore. If something annoys me, it’s probably going to really annoy someone else.

I’ve started re-working the user interface to provide a nicer experience, and make the location browser more entertaining in itself. I’ll post more about this once the new interface has progressed to a point where it’s screenshot-worthy.

Visual Diary: Climate Processing

The various regions/provinces in Daggerfall can take on a broad range of appearances. From the dry deserts of Alik’r, to the fecund swamps of Totambu, Daggerfall applies local flavour to almost every corner of the map. Different parts of the game world not only have distinct textures for most buildings, but varied plant life as well.

A primary goal for Daggerfall Scout is to faithfully display world maps as they would appear in the game. This is where climate processing comes in. When a city is correctly passed through the climate processer, it will take on the appearance of a desert, swamp, mountainous, or temperate location. Before I could do this, I had to find where Daggerfall itself stored information about a location’s climate, including native plant life.  Today’s Visual Diary steps through the process from research to implementation.

texturearchivemap

A Red Herring

The first place I looked was the map data in WOODS.WLD. Every pixel on the small map data has a corresponding large map chunk. One of the data values in the header for this chunk always corresponds to a ground texture. This seemed like a promising place to start.

After a bit of experimentation, I found the texture value only seemed valid some of the time. In many cases, it was flat out wrong. The city of Sentinel is a good example. The large map chunk for this world pixel reported the ground texture as 302 (temperate), not 2 (desert).

In frustration, I saved out the data into a visual format with intriguing results. Only the far eastern part of the world map, and some of the south, had any definition. The rest was a flat sea of “temperate”. This obviously wasn’t right, so I had to keep looking.

I’m guessing the data visualised to the left is either from an old version of the game world, or used in some as-yet not understood way.

climate226

CLIMATE.PAK

The next place to look was CLIMATE.PAK, which also defines a single pixel that can be overlaid on the standard 1000×500 world map. This data has been known about for some time, but I’ve not looked deeply into how it is used.

The first step was to see what kinds of numbers we’re dealing with. Each climate pixel is a value from 223 to 232. To see this visually, I saved out a series of climate maps, each with a single climate highlighted where it’s used in the world. The screenshot to the left shows climate 226, which covers the mountainous regions of Wrothgaria, Dragontail, and interestingly the Isle of Balfiera. This seemed far more promising. Each climate zone neatly overlaid visually distinct areas of the world.

Once I knew which areas were affected by a particular climate, I fired up Daggerfall and started taking notes. After a few hours of testing, I found Daggerfall always uses the same texture base and plant archive for any given climate value. This allowed me to build a table of which climate pixels uses which texture series for buildings and ground scenery.

If you’ve used Daggerfall Explorer, then you have probably played with my climate texture swaps already. Knowing how to make the swap is one thing. Understanding where the game draws this data from is quite another. With this in place, I was armed to build climate processing into Daggerfall Scout.

regiondev1

Pretty!

New code doesn’t always go to plan. The trees and other ground scenery are processed correctly, but the ground and buildings textures aren’t quite right. Castle Sentinel is particularly attractive.

After quickly fixing a few bugs, things start looking better.

regiondev2-swampregiondev3-desertregiondev4-mountain

Working Climate Processing

Finally, Daggerfall Scout will build a map using the correct building textures and ground scenery for that climate. This has been confirmed by travelling through Daggerfall with Scout running alongside to many locations in each climate band.

The screenshots to the left show maps using swamp, desert, and mountain climates, with distinct scenery in place.

The next step will be to drop in skies. Once again I need to do some research to find out exactly which skies Daggerfall uses where. I’ll put up another visual diary once I’ve made more progress.