Streaming World – Part 1

For the last week or so, I’ve been knuckling down on creating a fully streaming overworld in Daggerfall Tools for Unity. If you’re not sure what I aim to accomplish, here’s a brief overview of how I want this to work from the Unity Editor.

  1. Setup DaggerfallUnity singleton as normal.
  2. Add prefab StreamingWorld into scene hierarchy.
  3. Add prefab PlayerAdvanced into scene hierarchy.
  4. Set player virtual position in PlayerGPS and virtual time in WorldTime.
  5. Hit Play and explore entire Illiac Bay, entering any building, any dungeon, and experiencing full day/night cycle with climate and seasons.

In many ways, I’m already very close to this goal. The tools have procedural loading of cities and dungeons, virtual time and space, climates and seasons, day and night, interiors and exteriors, and more. What’s missing is a full terrain setup and intelligent loading/unloading of blocks as player traverses the world.

You can probably imagine what a huge job this is. While I am working very quickly towards this goal, it will be another 3-4 weeks before it all comes together in a state that I’m happy with. To maintain regular updates over that time, I will split news on my progress over multiple articles.

In this first article, let’s take a look at Daggerfall’s height map and how it translates into useful terrain.

Similar to the 1000×500 maps found in POLITIC.PAK and CLIMATE.PAK, Daggerfall also stores a 1000×500 elevation map in WOODS.WLD. Here’s a grayscale dump of that data (click for full size).

HeightmapFull

You can also see a nice false-colour version of this map here.

Like most height maps, dark areas are low elevations (values of 2 or less are water) and bright areas are high elevations. Daggerfall also stores a noise map in WOODS.WLD, which I’m not going to talk about as I plan to use a better method of noise generation.

To better understand the scale of this data, here is a zoomed-in 10×8 sample grabbed from along the northern coastline (I have brightened image so individual points are easier to make out). Each of the below squares represents a single “map pixel”, equivalent to one full-sized city like Daggerfall or Wayrest. If you measure the time it takes to cross from one side of Wayrest to the other, it would take 10x that amount of time to cross the below sample west-to-east.

HeightmapZoomOriginal

Once the scale is understood, it becomes apparent there’s not much height data here considering the actual size of terrain represented. This is why Daggerfall’s terrain is mostly flat. It’s stretching a single height sample over a huge area then modulating that with a little noise. Sometimes you can fluke a nice bit of terrain, but it’s very rare. On the whole, the overworld in Daggerfall is very flat and bland.

To improve this situation, there needs to exist more data in between height samples above. This new data must be quick to sample, use very little memory, and create a continuous grade between samples with plenty of interesting variety. To accomplish this, I am combining a few basic techniques.

The first problem to solve is the rate of elevation change between map pixels. It’s incredibly boring to have a huge flat area abruptly stepping up or down into yet another huge flat area. Our baseline needs to be a nice continuous elevation change from sample-to-sample.

I have added a new API method called GetHeightBilinear(). This quickly samples any point in the 1000×500 map with any number of bilinear interpolations between samples. The result is a much smoother overworld full of curves as each height sample blends into the next.

Below is the same zoomed-in terrain sample using bilinear interpolation to create additional sample points across the surface.

HeightmapZoomWithBilinear

If written out to a full-size image, this would be 128000×64000 pixels, enough to create a reasonably detailed overworld at the same grid resolution found in RMB blocks. New sample points are created on the fly from existing data, and it doesn’t require any additional memory.

There’s still a problem however. Despite having nice continuous samples, the terrain is still quite boring. The next technique is to add some noise and break things up a little. I’m using a fast simplex noise generator to create small variations at ground level. Below is an example of noise overlayed with interpolated heightmap.

HeightmapZoomWithBilinearNoise

You won’t see it in the thumbnail, but if you click through to full-size image you will see small whorls of coherent noise added to the height data. Compare this with the blocky first image, and you can see just how much fine data has been mathematically inserted. This can be scaled to any grid resolution and tweaked as required.

Of course, this is just the beginning. The next step is to create real terrain chunks from this data and tune noise generation towards interesting-looking terrain based on climate data.

Before wrapping up, I will leave you with the following image of a small patch of terrain (equivalent to 2×2 RMB blocks, or 1/4 the size of a full city. I have noise turned up to show the kind of deformations possible.

TerrainChunks

 

Over the course of the next few weeks, I will continue building on this foundation, adding better texturing, cities, and climates into the mix.

First Android Build

This is cool. One of our community members, Eric, successfully created a standalone city build on Android using Daggerfall Tools for Unity.

daggerfallTCC27

There are some issues with the sky and controls (these are bugs on my side), but on the whole everything seems to be working OK. Interior transitions and sounds are all working. Great work Eric!

What’s great about the above screenshot is that it shows Daggerfall could run on tablets without emulation. That’s not a DOSBox build up there, that’s a native Android build. The textures, models, and sounds are all created dynamically from raw Daggerfall binary files at runtime.

I’ve created a new blog category called “Community Spotlight”. Let me know if you make anything with the toolset and I’d be more than happy to feature it here.

Release 1.1 and City Basics Part Two

Daggerfall Tools for Unity 1.1 is now available for Download. This version introduces PlayerGPS and WorldTime components, along with several bug fixes and other enhancements.

There is also a new tutorial following on from City Basics, called City Basics Part Two. You will find this both inside the latest distribution and on the Tutorials & Docs page. This tutorial will help you create a fully automated day/night/season cycle using world time and a variable timescale.

I hope you enjoy this release and new tutorial. As usual, please let me know if you find any bugs or have trouble following the documentation.

Full 1.1 release notes are below.

  • Tutorials are now included with zip distribution.
  • Fixed inconsistent origins between blocks and world space. Imported blocks and locations are now all laid out in a positive X-Z direction with origin at south-west corner.
  • Added WorldTime and PlayerGPS components to core DaggerfallUnity singleton. These manage flow of time and player’s virtual position in world.
  • Added Time & Space options to DaggerfallUnity options. These options automate many time-based changes such as day/night cycles and seasonal changes.
  • Fixed RMB and RMB static assignments not working as they should.
  • Locations now store additional metadata about their world position.
  • Static door enumeration is now performed by API when initially processing model data. Door finding now has minimal overhead compared to previous post-layout method. Doors are also cached along with model data, so zero impact on future calls to GetModelData().
  • Removed individual door triggers for buildings and interiors. Doors are now tracked by a single component attached to building mesh. It stores an array of door positions evaluated used only when user clicks on building.
  • Added support for animated texture on imported Daggerfall models.
  • Added a SunlightManager to control intensity of sunlight rigs (including any number of secondary lights) and simulate time of day using a single key light for the sun.
  • Improved player position and facing when exiting buildings. The player is now positioned above ground using a ray hit and facing is determined by door normal.

Time & Space and More

Daggerfall Tools for Unity has good fundamentals. It can load any texture, model, sound effect, city, dungeon, season, and climate. It has weapons, skies, enemies, combat, building interiors, tons of editor options, and a solid API for reading anything else you need from Daggerfall into Unity.

The big missing piece right now is loading up the right city based on world position and automatically applying things like time of day and seasonal effects. Starting from 1.1, many of these additional extras will be included in the tools. This post talks a little bit about the new features with some technical bits thrown in.

If you’ve already used Daggerfall Tools for Unity, then you know it uses a singleton object called DaggerfallUnity as an interface and broker between Unity and the Arena2 binary data. There are several components attached to DaggerfallUnity for various tasks. In 1.1 you will find a couple of new components in the mix that bring proper time and space handling. These are PlayerGPS and WorldTime. Let’s take a look at PlayerGPS first.

PlayerGPS

 

PlayerGPS is fairly basic on the surface. It exposes two variables to manage a virtual player position anywhere in the Illiac Bay. The world map in Daggerfall is 1000×500 “world pixels”, with each “pixel” being 32768×32768 native units in size. This means the total world size ranges from 0,0 (bottom-left, or south-west corner of map) through to 32768000, 16384000 (top-right, or north-west corner of map) The numbers you see above (6782976, 9371648) are world coordinates for the city of Daggerfall.

At code level, PlayerGPS exposes more functionality. It can provide information about climate and politics, check to see if a location is present, etc. New static methods in the MapsFile class also provide the ability to move between different coordinate systems used by Daggerfall – such as longitude and latitude, world pixel, world coordinate, and location ID. The location ID is actually a special number derived from the longitude and latitude. It allows for any location to be looked up quickly via hash. PlayerGPS exposes this via lookup functions into a Dictionary collection.

To help support all this from editor and code, the internal DaggerfallLocation component (a self-assembling map layout for cities and dungeons) now hold additional metadata with locations.

Location Metadata

Most of what you need to know about a location is either in this metadata, or can be read from the API using this information as a starting point. The good news is that you don’t need to worry about any of this if you don’t want to. The toolset automates all of this for you in different ways. More on this shortly.

Next comes WorldTime. This component is an actual Daggerfall-styled Tamrielic calendar.

WorldTime

When you start a game, WorldTime starts counting the seconds using whatever timescale you have set. You can set timescale at 0.5 to make world time run at half speed, or at 100,000,000 to watch months flip by in moments. To make it easy to use, the individual time units are broken down in a very human-readable format (year, month, day, etc.).

This is far more than just a timer. WorldTime exposes real information about seasons, birth signs, month names, and so on. When coupled with PlayerGPS, it becomes possible to pinpoint the exact climate, season, and time of day for player position and make the world react as expected.

This means everything from rotating the sun through an animated sky, to making it snow, to setting the correct sky by climate and season, to lighting up windows at night, to turning street lights on and off. You’ve probably seen this video already, but here is WorldTime in full effect. You don’t see the seasons change, but that is in and working. If you let the simulation run long enough, seasons will roll by as expected.

You don’t need to do anything special to benefit from any of this. Just tick the new boxes below and the tools do the rest for you.

Time & Space Options

More automation will be added as the feature set expands.

Time and space aren’t the only new feature coming in 1.1. There are huge improvements to import speed and the beginnings of true world streaming, a feature scheduled for version 1.2. My goal for 1.2 is to allow you to walk around a fully streaming, fully time-automated world. For now though, world streaming is still in early testing stages, and looks a bit like this.

WorldStreaming

Each of these tiles represent a single “world pixel”. In this test, locations are streamed in around the player’s world position. As the player moves, new world pixels are read in or disposed as required.

In case you’re wondering, yes that actually is the entire city of Daggerfall on a single tile. All the models are rendered as normal, I just have the camera pointing straight down from a high position. The gray tiles will eventually be replaced by a simple terrain system, also scheduled for version 1.2.

You won’t have long to wait for all this stuff. The 1.1 release should be ready within the next week, and 1.2 (which I’m working on now) will only be a few weeks away. When 1.2 is ready you’ll be able to just drop in a few prefabs (or load a premade scene) and hit Play to explore the entire of Illiac Bay.

I’m very excited to get all this through to you as soon as possible.

City Tutorial and 1.0.2 Update

Daggerfall Tools for Unity 1.0.2 is now available. This version completely overhauls the DaggerfallSky component. Skies are now more flexible, working in both Forward and Deferred paths. Skies will thread-load data and allow for dynamic changes at runtime.

The City Basics tutorial is also available from the Tutorials menu page. Please update to 1.0.2 before running this tutorial.

One more quick item. Below is a preview of the the world time feature scheduled for release 1.1 in a few weeks. I will post more about this soon.

Dungeon Tutorial and 1.0.1 Update

Daggerfall Tools for Unity 1.0.1 is now available for download. This version contains minor bug fixes and improvements to the weapon system.

The first tutorial is now also available. This introductory tutorial steps you through creating a new scene, importing a dungeon, setting up weapons, and more. You will find this and all future tutorials on the Tutorials menu page.

Please be sure to update to 1.0.1 before following the tutorial as it fixes a bug you will encounter otherwise.

Let me know what you think of the tutorial, especially if any parts are too hard to follow. I want the “Basics” series to be generally accessible and will revise if necessary to improve information presented.

Daggerfall Tools for Unity 1.0

Daggerfall Tools for Unity 1.0 is now available for general download. Click here for download page. Change notes are below.

I’ve been working on this like a man possessed the last week or so. I definitely need a short break to clear my head and get some perspective on what comes next. I’ll talk more about this when I can.

I hope you have fun with Daggerfall Tools for Unity! If you have any feedback, bug reports, or feature requests, please feel free to contact me.

v1.0.0

  • Exterior door triggers are now sized appropriately to doors.
  • Added support for exterior dungeon doors.
  • Added enum for static door types.
  • Added DaggerfallInterior component to layout interiors.
  • Hinged action doors are now loaded inside building interiors and dungeons. They can be locked, magically held, bashed open, opened by monsters, and play appropriate sounds.
  • Added example class to transition in and out of interiors.
  • Player controller can click into and out of building interiors using example script.
  • Player controller now has moving platform support and ceiling hit detection.
  • Fixed broken sky hemisphere swap, refined sky scrolling.
  • Sky component can now be attached anywhere, such as to an exterior parent object. It will search for MainCamera if one is not specified.
  • Added default unlit billboard shader to MaterialReader. This will be used by light billboards.
  • Added default unlit texture material to MaterialReader. This will be used by fireplaces and similar textures.
  • Added dungeon action records and examples of activation by player.
  • Add iTween to __ExternalAssets namespace for animating hinged doors and action records. iTween was chosen as it will be familiar to the most Unity developers.
  • Removed QuickSize() and QuickScale() from API as they were incompatible with standalone builds.
  • Added size and scale properties to new method GetCachedMaterial(). The cached method slightly reduces scene building time and is compatible with standalone builds.
  • Palette files can now be loaded in standalone builds.
  • Bug fixes to standalone build startup and loading from Resources.
  • Added exception handling when loading invalid enemy indices.
  • Added exception handling for missing Run and Jump inputs in player controller prefab.
  • Changed Scripts/Other folder to Scripts/Demo. Contents are now in DaggerfallWorkshop.Demo namespace. These classes are not part of core tools but show examples of specific tasks such as loading interiors.
  • Added example weapon loading with variable tinting based on metal type.
  • Added example attack manager for gesture-based attacks.
  • Fixed API bug when getting DFSize for animated weapon records.
  • Enemy mobiles now read in “not hostile” flag where set in game data, and will not attack unless provoked. Examples are guard in castles and liches/vampire the King of Worms court.
  • Added DaggerfallAudioSource component for dynamically loading and playing sound effects from DAGGER.SND.
  • Created sound enumeration to name and group effects.
  • Linked sound definitions to enemy mobiles and action triggers.
  • Added ambient effect player.
  • New flags to import sounds automatically.
  • Added flag to MaterialReader controlling mipmap creation.
  • Lots of fixes and extensions to Demo scripts.

Direnni Tower – Full Demo

I’m very close to Daggerfall Tools for Unity 1.0. To celebrate this upcoming milestone, I’ve created a new demo showing a bit of everything the toolset is capable of right now.

Direnni Demo

  • Explore remixed city environment during a late afternoon storm.
  • Enter any building or brave the depths of a transplanted Direnni Tower.
  • Fight dungeon inhabitants while dual-wielding heavy Daedric weapons.
  • Bash open randomly locked doors.
  • Full sound effects, played directly from Daggerfall’s effects file.

WebPlayer URL
http://www.dfworkshop.net/web_demos/direnni-demo/direnni-demo.html

Standalone Windows
www.dfworkshop.net/static_files/demos/DirenniDemoStandalone.zip

Controls

  • Alt+F4 to exit (standalone).
  • Click to capture mouse (webplayer).
  • ESC to uncapture mouse (webplayer).
  • Mouse move to look.
  • W, S, A, D to move.
  • SHIFT (hold) to run.
  • SPACE to jump.
  • LEFT-CLICK mouse to open doors, operate switches, etc.
  • Z to sheathe and draw your weapons.
  • RIGHT-CLICK and HOLD while moving mouse in different directions to attack enemies (Daggerfall style).
  • Attack locked doors with your weapons to bash them open (25% chance per hit to open).
  • P while in dungeon to recall back to entrance.

Sound Effects

After playing through the combat demo a few times, I felt a little sound would really liven things up. This turned into a bigger job than expected (doesn’t it always?) but I’m very happy with the result. This post talks about the new audio features in brief, and may be slightly technical if you’re not familiar with Unity.

So how does this work? To start with, let’s take a look at adding a sound manually. What you need to do is add the component DaggerfallAudioSource to any of your GameObjects. This scripts buddies up with Unity’s AudioSource component and feeds it sounds loaded dynamically from Daggerfall’s sound effects library.

Once the component is added, you set a preset (OnDemand, Looping, or None for no changes) and a Sound Index then click Apply(). This will load in the sound from Daggerfall as a regular AudioClip and apply it, along with your preset settings, to the Unity AudioSource. From there you just treat it as a normal audio source like any other and don’t have to worry about it any more. You can even preview sounds directly from the editor by index, id, or name.

Behind the scenes, things are a bit more complicated. As these audio clips are created dynamically they don’t have a matching file in the project hierarchy, and for some reason Unity doesn’t like to serialize procedural sound clips into the scene in the way it serializes stuff like mesh data. The other complication is that settings like “Play On Awake” don’t work as expected as the clip is not present in memory until after the component is running.

To handle all this, the DaggerfallAudioSource component will check to see if your AudioClip has been lost (from starting/stopping play, serialization/deserialization, recompile, etc.) then loads it back into memory again. It also handles booting the sound manually to simulate Play On Awake.

This is very lightweight and under normal play sessions the clip will only be loaded once at start. It’s only inside the editor where the clip may need to be loaded multiple times a session due to various reasons the editor dumps volatile data. It all just works as expected, even when using the little speaker to preview scene audio.

The other thing which caused some grief was actually converting the 8-bit PCM audio from Daggerfall into floats for AudioClip.SetData(). This needed special wrangling of the float values to convert bit depths properly, and the Unity docs weren’t much help here. A bit of Googling and face-desking saw me through to the end. Mostly, this was only hard because I can be an idiot sometimes.

On the support front, there’s a few other goodies coming with the sound update. I’ve created a SoundClips enum with about 80% of Daggerfall’s effects given a plain text, descriptive name. Where possible, I have grouped these by usage as I understand them. I also managed to work out how Daggerfall assigns sounds to enemies and action triggers, and put this into the toolset and API where appropriate. See below for updated DaggerfallAction editor window.

New DaggerfallAction Editor Window

The new number above is “Action Sound”. This is the sound ID to play when this action is triggered. I had to break out my hex editor again, as this action-sound mapping was previously unknown.

The next bit of support is you can import sound effects with dungeons, enemies, etc. Daggerfall Tools for Unity wires up all the audio sources for you to automatically bring your scenes to life. You can also create, change, and play sounds completely at run-time.

Last but not least, I’ve created a “spooky sound player” example component to randomly play those wonderful dungeon sounds (e.g wind blowing, doors opening, water dripping, etc.). This will later be joined by other example scripts as part of the tutorial series I’m working on.

I should have a new demo ready for you over the weekend with a full complement of sound effects to enjoy.

Privateer’s Hold Combat Demo

Below is a new standalone web build showing off many of the new features soon to be available in Daggerfall Tools for Unity 1.0.

Click the link below to explore Privateer’s Hold and vanquish foes with your trusty Ebony Dagger.

http://www.dfworkshop.net/web_demos/privateers-test2/privateers-test2.html

  • W, S, A, D to move.
  • Mouse move to look.
  • SHIFT to toggle run mode.
  • SPACE to jump.
  • LEFT-CLICK mouse to open doors, operate switches, etc.
  • RIGHT-CLICK and HOLD while moving mouse in different directions to attack enemies.
  • ESC to uncapture mouse.

Roadmap Update

I’m running behind on Release 3, however I’ve also managed to squeeze in features originally planned for 1.0 or later (such as weapon loading). My original plan was for Release 4 to be considered 1.0, but I’m so close to the finish line now that I’ve decided to push through and name the next release 1.0. This should be ready in another week or two, depending how much time I can allocate.

One feature that will be missing from 1.0 is improved content browsing directly from Unity’s editor. After reviewing what I want to achieve here and some limitations of editor scripts, I’ve decided to delay this feature to the post-1.0 pile. Creating a good user interface is always challenging, and there’s nothing functionally wrong with just typing in the name of the model/block/location you want to load. I will improve this, but I need to step back and clear my head before taking it on.

Once the 1.0 release is out, I’ll start work on a series of short YouTube tutorials to help newcomers with Daggerfall Tools for Unity. These will cover the basics of setting up a project all the way to creating a simple dungeon crawler with infinite levels. This will be a nice diversion and help me review the tools by actually putting them to work.

On a personal note, the development of these tools has been a lot of fun and I’ve not had to make the time sacrifices I made two years ago. I can get a lot done in an hour or two per day, plus a little extra on weekends, and I don’t need to miss out on any time with my family. Thanks to Unity doing most of the heavy lifting, I can take a new feature from concept to reality in very little time. <3 Unity.