Streaming World – Improving Terrain Noise

I decided to take a step back and have quick look at my base terrain generation. I wasn’t very happy with the overall appearance and contours of noise, and smooth noise functions alone weren’t enough to achieve the desired result.

To fix this, I changed my mind about using the the large height map data (sometimes called the noise map) which is also in WOODS.WLD. If you’re not familiar with this additional map, it is a 5000×2500 version of the 1000×500 height map I showed in part 1. Not all of this map is usable as there is a grid-like pattern overlayed with an unknown purpose.

Below are 3×3 terrains with city of Daggerfall in the middle. I have increased height scale so any deformations are more visible. This is basically the terrain from my previous article with all noise removed. Bilinear interpolation is used to smooth transition between height samples.

SmallHeightBilinearThis really isn’t very interesting. Next I’m going to use the large height map, which has 3×3 samples per terrain instead of 1×1. This looks like the following when using nearest-neighbour interpolation.

LargeHeightPointSamples

There’s definitely more sample points, but it resembles Minecraft more than Daggerfall. Let’s smooth things out a bit using trusty bilinear interpolation.

LargeHeightBilinearSamples

Ouch, that’s still really ugly. With more sample points, bilinear isn’t really an appropriate choice anymore. The next best thing is bicubic interpolation.

LargeHeightBicubicSamples

Much better! Bicubic ensures a nice continuous curve through all sample points and provides a more interesting base terrain compared to the very first screenshot. I am much happier with this as a starting point.

So far, all this data is being read from Daggerfall’s files. I haven’t added anything besides scaling up the height. As before, I will use a bit of smooth noise to add fine details at ground level and flatten out the city areas.

Now that’s out of the way, I can go back to improving other aspects of terrain quality.

Streaming World – Part 2

In the first part of this series, we looked at how the world in Daggerfall is constructed from multiple 1000×500 pixel maps, and how procedural techniques can be applied to add more fine detail to Daggerfall’s height map.

This article shows how continuous height map data generated by world readers is injected into scene as continuous meshed terrain and how terrain is landscaped into flat areas for cities.

First up, I have chosen not to use Unity’s native terrain system. There are several reasons for this, but chiefly is that Daggerfall terrain textures work very differently from Unity. In Unity (like most modern 3D engines) the terrain is textured using a splat map – a special texture which determines how detail textures are to be blended at every vertex. Daggerfall on the other hand uses a grid of quads textured from a selection of 56 tiles. These tiles are earth, sand, grass, stone, and road with hand-painted transitions between most states.

TerrainTextureTiles

Above is the temperate terrain set as viewed from Daggerfall Imaging 2. There is one complete set of 56 tiles per climate base (desert, temperate, mountain, swamp) with one variant for snow and another for rain. To save texture space, these tiles also use UV modification to be rotated, flipped, or both rotated and flipped where required.

In order for the terrain to blend seamlessly from wilderness into city tiles, it becomes necessary to build a custom terrain system which understands how Daggerfall works. This unfortunately excludes the default Unity terrain. The code is modular however, so you can always replace part or all of my terrain system with something else.

With the decision made to create a Daggerfall-like terrain system, the next stage is to transform raw height data into streaming mesh samples and format ground for locations.

The first step of this journey was actually implemented a few weeks back in the 1.1 update. See article Time & Space and More for details about PlayerGPS and WorldTime components. Suffice to say, the toolset already has a good understanding of where the player is in the world and what locations are nearby. I will copy an image from the time and space article here as it pertains to world streaming.

WorldStreaming

Above is a snapshot of the world in pure numerical form. Each of these tiles represents a Map Pixel (discussed in part 1), which is the size of a full-sized city. The player is standing at the origin of Daggerfall city at map pixel 207, 213. In the immediate vicinity are Ripwych Commons to the north-west (206, 212) and Copperfield Manor to the south (207, 214). Keep an eye on them, as we’ll be seeing more of them soon.

At a high level, the StreamingWorld component stores data much like the above. It keeps track of a small bubble of world space around player in a 2D array. As the player moves, tiles are shifted up, down, left, or right in the array (based on direction player is moving). New map pixels are loaded as required and everything is occasionally snapped back to near origin, which avoids precision errors at extreme limits of the map. From the player’s perspective, they are walking endlessly into the distance. In reality, they are on a terrain treadmill.

Inside each of these map pixels a DaggerfallTerrain object is created for the entire area. This is generated procedurally from Daggerfall’s height map combined with smooth noise to add small details. This terrain is then broken up into several DaggerfallTerrainChunk objects, which represent the actual mesh objects and colliders for the player to see and walk on.

The edge vertices and normals of each adjacent terrain are stitched together to create a nice continuous terrain. The next screenshot shows a 3×3 terrain area, once again centred on Daggerfall, with the tiling texture system discussed above. Right now it is just set to grass. I also have turned up the noise scale to make the deformations a little more interesting.

StartingTerrain

The locations are there in memory, but nothing is being drawn yet. Before we can insert locations, some foundations must be laid. First come the appropriate texture tiles. Unlike the earlier screenshot, the locations are centred inside their map pixel. Daggerfall city is in the middle, Copperfield Manor is to the south, and Ripwych Commons is just barely visible to the north-west.

LocationGroundTextures

 

Now location textures are in the right place, but there’s a big problem – cities in Daggerfall need a flat plane to sit on. This raises two questions. What level should the terrain be flattened to and how can everything be smoothed out cleanly?

I evaluated several different methods for selecting city level. The below screenshot shows a few of them without any smoothing.

FlattenVariants

I didn’t like the highest point as it made everything feel too raised. The average is pretty good as the city is leveled evenly in between all deformations, but I rejected it also as another pass was required to determine average elevation after applying noise. World height was very cheap to obtain but sometimes felt a little too high or too low. I ended up settling on median bilinear height (from the map reader), which is also very cheap and consistently landed somewhere between the world height and true average. So that’s the level I went with.

The next stage was to blend the flat areas into the random terrain. Interestingly, even full-sized cities like Daggerfall do not really fill their entire map pixel. They have a band of 14 tiles around the outside to use as blend space. These even have special index (>55) to indicate not to use RMB-defined tiles and instead blend with terrain.

The blending process gave me serious headaches for a few days. I tried several different methods of blending heights but none of them gave me a perfect transition from the rectangular city area all the way to edge of terrain. In the end, I created a system of building scale maps with linear rolloff from edge of city to edge of terrain. This uses linear interpolation for the straight rects and bilinear in the corner rects to keep everything smooth.

What is important is the scale is 0 all the way around terrain bounds and smoothly reaches 1 at the location bounds. Drawing the scale map for Copperfield Manor looks like the following. Each of the points below represents the blend weight for random terrain vs level terrain. The further out from the centre rect, the more influence random terrain will have. It looks similar to a pure radial falloff, but if you look carefully you will notice everything converges on a rectangular area, not a single point as would be the case with radial.

ScaleMap

Applying the scale map gives us a nice smooth transition from flat area into random terrain. The terrain features can still express themselves, but the transition feels natural and remains completely walkable even with lots of noise.

SmoothedTerrain

From here, it’s a simple matter of plopping down locations where needed and let the player go exploring.

The next part of this series will be about enhancing details of terrain. I will discuss adding textures outside of cities, improving elevation noise, and adding trees and other wilderness flats.

Streaming World – Part 1

For the last week or so, I’ve been knuckling down on creating a fully streaming overworld in Daggerfall Tools for Unity. If you’re not sure what I aim to accomplish, here’s a brief overview of how I want this to work from the Unity Editor.

  1. Setup DaggerfallUnity singleton as normal.
  2. Add prefab StreamingWorld into scene hierarchy.
  3. Add prefab PlayerAdvanced into scene hierarchy.
  4. Set player virtual position in PlayerGPS and virtual time in WorldTime.
  5. Hit Play and explore entire Illiac Bay, entering any building, any dungeon, and experiencing full day/night cycle with climate and seasons.

In many ways, I’m already very close to this goal. The tools have procedural loading of cities and dungeons, virtual time and space, climates and seasons, day and night, interiors and exteriors, and more. What’s missing is a full terrain setup and intelligent loading/unloading of blocks as player traverses the world.

You can probably imagine what a huge job this is. While I am working very quickly towards this goal, it will be another 3-4 weeks before it all comes together in a state that I’m happy with. To maintain regular updates over that time, I will split news on my progress over multiple articles.

In this first article, let’s take a look at Daggerfall’s height map and how it translates into useful terrain.

Similar to the 1000×500 maps found in POLITIC.PAK and CLIMATE.PAK, Daggerfall also stores a 1000×500 elevation map in WOODS.WLD. Here’s a grayscale dump of that data (click for full size).

HeightmapFull

You can also see a nice false-colour version of this map here.

Like most height maps, dark areas are low elevations (values of 2 or less are water) and bright areas are high elevations. Daggerfall also stores a noise map in WOODS.WLD, which I’m not going to talk about as I plan to use a better method of noise generation.

To better understand the scale of this data, here is a zoomed-in 10×8 sample grabbed from along the northern coastline (I have brightened image so individual points are easier to make out). Each of the below squares represents a single “map pixel”, equivalent to one full-sized city like Daggerfall or Wayrest. If you measure the time it takes to cross from one side of Wayrest to the other, it would take 10x that amount of time to cross the below sample west-to-east.

HeightmapZoomOriginal

Once the scale is understood, it becomes apparent there’s not much height data here considering the actual size of terrain represented. This is why Daggerfall’s terrain is mostly flat. It’s stretching a single height sample over a huge area then modulating that with a little noise. Sometimes you can fluke a nice bit of terrain, but it’s very rare. On the whole, the overworld in Daggerfall is very flat and bland.

To improve this situation, there needs to exist more data in between height samples above. This new data must be quick to sample, use very little memory, and create a continuous grade between samples with plenty of interesting variety. To accomplish this, I am combining a few basic techniques.

The first problem to solve is the rate of elevation change between map pixels. It’s incredibly boring to have a huge flat area abruptly stepping up or down into yet another huge flat area. Our baseline needs to be a nice continuous elevation change from sample-to-sample.

I have added a new API method called GetHeightBilinear(). This quickly samples any point in the 1000×500 map with any number of bilinear interpolations between samples. The result is a much smoother overworld full of curves as each height sample blends into the next.

Below is the same zoomed-in terrain sample using bilinear interpolation to create additional sample points across the surface.

HeightmapZoomWithBilinear

If written out to a full-size image, this would be 128000×64000 pixels, enough to create a reasonably detailed overworld at the same grid resolution found in RMB blocks. New sample points are created on the fly from existing data, and it doesn’t require any additional memory.

There’s still a problem however. Despite having nice continuous samples, the terrain is still quite boring. The next technique is to add some noise and break things up a little. I’m using a fast simplex noise generator to create small variations at ground level. Below is an example of noise overlayed with interpolated heightmap.

HeightmapZoomWithBilinearNoise

You won’t see it in the thumbnail, but if you click through to full-size image you will see small whorls of coherent noise added to the height data. Compare this with the blocky first image, and you can see just how much fine data has been mathematically inserted. This can be scaled to any grid resolution and tweaked as required.

Of course, this is just the beginning. The next step is to create real terrain chunks from this data and tune noise generation towards interesting-looking terrain based on climate data.

Before wrapping up, I will leave you with the following image of a small patch of terrain (equivalent to 2×2 RMB blocks, or 1/4 the size of a full city. I have noise turned up to show the kind of deformations possible.

TerrainChunks

 

Over the course of the next few weeks, I will continue building on this foundation, adding better texturing, cities, and climates into the mix.

First Android Build

This is cool. One of our community members, Eric, successfully created a standalone city build on Android using Daggerfall Tools for Unity.

daggerfallTCC27

There are some issues with the sky and controls (these are bugs on my side), but on the whole everything seems to be working OK. Interior transitions and sounds are all working. Great work Eric!

What’s great about the above screenshot is that it shows Daggerfall could run on tablets without emulation. That’s not a DOSBox build up there, that’s a native Android build. The textures, models, and sounds are all created dynamically from raw Daggerfall binary files at runtime.

I’ve created a new blog category called “Community Spotlight”. Let me know if you make anything with the toolset and I’d be more than happy to feature it here.

Release 1.1 and City Basics Part Two

Daggerfall Tools for Unity 1.1 is now available for Download. This version introduces PlayerGPS and WorldTime components, along with several bug fixes and other enhancements.

There is also a new tutorial following on from City Basics, called City Basics Part Two. You will find this both inside the latest distribution and on the Tutorials & Docs page. This tutorial will help you create a fully automated day/night/season cycle using world time and a variable timescale.

I hope you enjoy this release and new tutorial. As usual, please let me know if you find any bugs or have trouble following the documentation.

Full 1.1 release notes are below.

  • Tutorials are now included with zip distribution.
  • Fixed inconsistent origins between blocks and world space. Imported blocks and locations are now all laid out in a positive X-Z direction with origin at south-west corner.
  • Added WorldTime and PlayerGPS components to core DaggerfallUnity singleton. These manage flow of time and player’s virtual position in world.
  • Added Time & Space options to DaggerfallUnity options. These options automate many time-based changes such as day/night cycles and seasonal changes.
  • Fixed RMB and RMB static assignments not working as they should.
  • Locations now store additional metadata about their world position.
  • Static door enumeration is now performed by API when initially processing model data. Door finding now has minimal overhead compared to previous post-layout method. Doors are also cached along with model data, so zero impact on future calls to GetModelData().
  • Removed individual door triggers for buildings and interiors. Doors are now tracked by a single component attached to building mesh. It stores an array of door positions evaluated used only when user clicks on building.
  • Added support for animated texture on imported Daggerfall models.
  • Added a SunlightManager to control intensity of sunlight rigs (including any number of secondary lights) and simulate time of day using a single key light for the sun.
  • Improved player position and facing when exiting buildings. The player is now positioned above ground using a ray hit and facing is determined by door normal.

Time & Space and More

Daggerfall Tools for Unity has good fundamentals. It can load any texture, model, sound effect, city, dungeon, season, and climate. It has weapons, skies, enemies, combat, building interiors, tons of editor options, and a solid API for reading anything else you need from Daggerfall into Unity.

The big missing piece right now is loading up the right city based on world position and automatically applying things like time of day and seasonal effects. Starting from 1.1, many of these additional extras will be included in the tools. This post talks a little bit about the new features with some technical bits thrown in.

If you’ve already used Daggerfall Tools for Unity, then you know it uses a singleton object called DaggerfallUnity as an interface and broker between Unity and the Arena2 binary data. There are several components attached to DaggerfallUnity for various tasks. In 1.1 you will find a couple of new components in the mix that bring proper time and space handling. These are PlayerGPS and WorldTime. Let’s take a look at PlayerGPS first.

PlayerGPS

 

PlayerGPS is fairly basic on the surface. It exposes two variables to manage a virtual player position anywhere in the Illiac Bay. The world map in Daggerfall is 1000×500 “world pixels”, with each “pixel” being 32768×32768 native units in size. This means the total world size ranges from 0,0 (bottom-left, or south-west corner of map) through to 32768000, 16384000 (top-right, or north-west corner of map) The numbers you see above (6782976, 9371648) are world coordinates for the city of Daggerfall.

At code level, PlayerGPS exposes more functionality. It can provide information about climate and politics, check to see if a location is present, etc. New static methods in the MapsFile class also provide the ability to move between different coordinate systems used by Daggerfall – such as longitude and latitude, world pixel, world coordinate, and location ID. The location ID is actually a special number derived from the longitude and latitude. It allows for any location to be looked up quickly via hash. PlayerGPS exposes this via lookup functions into a Dictionary collection.

To help support all this from editor and code, the internal DaggerfallLocation component (a self-assembling map layout for cities and dungeons) now hold additional metadata with locations.

Location Metadata

Most of what you need to know about a location is either in this metadata, or can be read from the API using this information as a starting point. The good news is that you don’t need to worry about any of this if you don’t want to. The toolset automates all of this for you in different ways. More on this shortly.

Next comes WorldTime. This component is an actual Daggerfall-styled Tamrielic calendar.

WorldTime

When you start a game, WorldTime starts counting the seconds using whatever timescale you have set. You can set timescale at 0.5 to make world time run at half speed, or at 100,000,000 to watch months flip by in moments. To make it easy to use, the individual time units are broken down in a very human-readable format (year, month, day, etc.).

This is far more than just a timer. WorldTime exposes real information about seasons, birth signs, month names, and so on. When coupled with PlayerGPS, it becomes possible to pinpoint the exact climate, season, and time of day for player position and make the world react as expected.

This means everything from rotating the sun through an animated sky, to making it snow, to setting the correct sky by climate and season, to lighting up windows at night, to turning street lights on and off. You’ve probably seen this video already, but here is WorldTime in full effect. You don’t see the seasons change, but that is in and working. If you let the simulation run long enough, seasons will roll by as expected.

You don’t need to do anything special to benefit from any of this. Just tick the new boxes below and the tools do the rest for you.

Time & Space Options

More automation will be added as the feature set expands.

Time and space aren’t the only new feature coming in 1.1. There are huge improvements to import speed and the beginnings of true world streaming, a feature scheduled for version 1.2. My goal for 1.2 is to allow you to walk around a fully streaming, fully time-automated world. For now though, world streaming is still in early testing stages, and looks a bit like this.

WorldStreaming

Each of these tiles represent a single “world pixel”. In this test, locations are streamed in around the player’s world position. As the player moves, new world pixels are read in or disposed as required.

In case you’re wondering, yes that actually is the entire city of Daggerfall on a single tile. All the models are rendered as normal, I just have the camera pointing straight down from a high position. The gray tiles will eventually be replaced by a simple terrain system, also scheduled for version 1.2.

You won’t have long to wait for all this stuff. The 1.1 release should be ready within the next week, and 1.2 (which I’m working on now) will only be a few weeks away. When 1.2 is ready you’ll be able to just drop in a few prefabs (or load a premade scene) and hit Play to explore the entire of Illiac Bay.

I’m very excited to get all this through to you as soon as possible.

City Tutorial and 1.0.2 Update

Daggerfall Tools for Unity 1.0.2 is now available. This version completely overhauls the DaggerfallSky component. Skies are now more flexible, working in both Forward and Deferred paths. Skies will thread-load data and allow for dynamic changes at runtime.

The City Basics tutorial is also available from the Tutorials menu page. Please update to 1.0.2 before running this tutorial.

One more quick item. Below is a preview of the the world time feature scheduled for release 1.1 in a few weeks. I will post more about this soon.