AI-written description: A game render of a thin crescent moon in a clear blue sky, its pale, cratered sliver lit along the lower-right edge. cora.earth

tiny neural networks in my space game

read ↓
cora.earth

Β· 9 min

My goal is to make a space exploration sandbox game, with ~20%-scale planets, real physics, and a small colony/factory layer.

This teaches real orbital mechanics skills, like:

  • How do you efficiently turn a rocket sideways as it gets into orbit?
  • How do you reach the moon?
  • How do you safely get a huge gravity assist off a binary black hole pair (without being ripped apart by extreme tidal forces)? spoiler: very flat spacecraft

The factory layer can also teach people about the promise of space industry:

  • What does it take to build a Kevlar space elevator on the moon?
  • How many orbital solar panels does it take to meet all of humanity’s energy needs?
  • Why are huge rockets like Sea Dragon or Starship economically exciting?

This is an outrageously ambitious technical project. Kerbal Space Program 2 spent six years in development with an informally estimated cost of 40 to 60 million USD, never left early access, and ended when its publisher shut down the studio that made it.

To do this, one thing I need is an earth worth returning to.

That earth begins as a few-hundred-million-year whole-planet sim, written in Rust, which simulates plate tectonics, orogeny, volcanoes and lava flows, meteor impacts, glacial erosion (Cordonnier et al. 2023), post-glacial rebound as the earth’s crust regains isostasy, ocean evaporation, rain shadows, rivers, lakes, erosion, and movement of sand in the ocean. (This might be the subject of a future blog post 🌍, but for now we’ll skip over it to some other interesting bits about neural networks. The moon, pictured at the beginning of this post and rendered in Unreal Engine, comes from the same terrain Rust sim, with more emphasis on craters, ejecta, and lava flows.)

That sim takes a few hours to run on my RTX 3080 Ti, and gets us down to ~500-meter resolution for my earth stand-in. But that’s not enough detail, and using first-principles physics sims to get to a finer resolution was too difficult. Rock breaks and erodes in complicated ways, rivers and soil have incredibly complex dynamics; capturing all of that detail was hard.

So, let’s use neural networks to just learn all those complicated dynamics!

Terrain detail with diffusion models πŸ”οΈ

First, we teach a diffusion model to understand fine erosion detail, by training it (from scratch on my gaming PC!) on real earth terrain data. (Thanks ESA!)

Terrain produced using Copernicus WorldDEM-30 Β© DLR e.V. 2010-2014 and Β© Airbus Defence and Space GmbH 2014-2018 provided under COPERNICUS by the European Union and ESA; all rights reserved. The organisations in charge of the Copernicus programme by law or by delegation do not incur any liability for any use of the Copernicus WorldDEM-30. USGS 3D Elevation Program (3DEP) data courtesy of the U.S. Geological Survey.
AI-written description: Four square shaded-relief terrain maps in green and cream sit in a row on a dark background: a blocky 32 Γ— 32 grid of average heights, a noisy state, the model's estimate, and the real ground truth. As a progress bar fills, the noise level falls from Οƒ = 20 to almost zero: the noisy panel clears from static into ridged terrain, and the model's estimate sharpens from soft, blurry ridges into crisp branching valleys that closely match the ground truth. A readout below shows the model's mean slope rising from 21.9Β° to 25.0Β°, against 25.4Β° for the truth.

Performance on a held-out region: Grand Canyon, Arizona, 14.6 km square. Input data is 32x32 averages at 456 meter resolution.

AI-written description: A grid of twelve square shaded-relief terrain maps in four columns, for the Grand Canyon (Arizona), the Southern Alps (New Zealand), the Central Range (Taiwan) and the Cordillera Oriental (Colombia), and three rows labelled smooth base, the model and ground truth. In each column the smooth base looks soft and blurry, while the model's map has sharp ridges and branching valleys much like the real terrain beneath it. The model's mean slopes (25.0Β° to 31.3Β°) are close to the truth's (25.4Β° to 32.0Β°).

Performance on other held-out regions.

Terrain produced using Copernicus WorldDEM-30 Β© DLR e.V. 2010-2014 and Β© Airbus Defence and Space GmbH 2014-2018 provided under COPERNICUS by the European Union and ESA; all rights reserved. The organisations in charge of the Copernicus programme by law or by delegation do not incur any liability for any use of the Copernicus WorldDEM-30. USGS 3D Elevation Program (3DEP) data courtesy of the U.S. Geological Survey. These figures aren’t covered by this site’s CC0 dedication.

Inspired by recent work like Elucidating the Design Space of Diffusion-Based Generative Models (Karras et al. 2022). This takes nine hours to add detail to a tiled cube-sphere heightmap representation of the earth, which is the output from the first erosion sim. It runs over the tiles in raster order, using RePaint (Lugmayr et al. 2022) and a few other tricks to avoid seams.

Sounds good in theory, but how does it look in practice?

AI-written description: A game render from high altitude over a vast range of snowy mountains, white snow on grey rock, stretching to the curved horizon, where a thin band of blue atmosphere fades into a black sky. Fluffy white clouds hang over green lowlands and a turquoise coastline in the lower right.

Thank goodness for Unreal Engine’s Nanite renderer - between an ~8 GB mesh and further tessellation heightmaps for residual detail, the earth’s geometry data alone is over 9 GB. (These are still just flat-color renders - grey for rock, white for snow, green spheres for trees, be forgiving πŸ˜…). Unlike a procedural quad-sphere (the usual solution here, as well as my since-replaced v1), there’s no inefficient subdivision of flat planes or expensive geometry smoothing over heightmaps; there’s just a giant efficient mesh, baked offline. This also lets us do caves, tunnels, and overhangs! Collision might deserve its own little write-up - basically, Nanite renders the planet on the GPU (with tessellation), but we also have an efficiently queryable CPU representation, which matches what Nanite draws exactly, so we can dynamically generate surface colliders around physics objects.

You might also notice these odd fluffy white things hanging volumetrically in the sky. Another perfect candidate for tiny neural networks!

Radiance-predicting neural networks for volumetric cloud rendering ☁️

This work is based on MRPNN (Hu et al. 2023) and tiled-MRPNN (multi-feature radiance-predicting neural networks), but is trained from scratch.

Generating training data is usually quite difficult, but we’re saved once again by Unreal Engine - we generate training data for the neural network in-engine by running Unreal’s path tracer through the same volumetric clouds, in the same game world, and then we train a tiny (~10k weights) neural network to approximate the path-traced reference by more efficiently representing visually important features using sparse, cheap input data.

Costs 0.3–1.9 ms a frame at 1440p on an RTX 3080 Ti. On views it never trained on: typical error about 15% against the path tracer, with no bias from dark to bright.

Here’s a held-out example of what the network is using as input:

AI-written description: A three-panel diagram on a dark background: a target of nine concentric rings around a star (the pivot) with a dotted ray to the sun, a false-color heat map of what the network is fed, and its output, puffy grey-white clouds over a dark ground. The animation steps through the inputs: a highlighted ring grows outward through the nine shells from 50 m to 12.8 km, a marker moves out along the sun ray from 0.4 to 25.6 km, and then sun-map transmittance and sky light each change the heat map. The last frame sets the learned clouds beside a grainy path-traced reference that looks almost identical.

Putting it all together (terrain, atmosphere, and clouds) in-game:

AI-written description: A game render looking down on a thick deck of fluffy white clouds that fills most of the view, piled over a mountain range. Snowy grey peaks poke out along the top edge with small separate clouds floating above them, and slopes dotted with green trees show through on the left.

(Again, please be forgiving, this is an early prototype - this version of the network was trained on just a few hundred clouds!)

Beauty shots πŸŒ„

After several months of work, (now) three total re-implementations of volumetric cloud rendering, and ~four total re-architectures of the planet rendering (I started with a procedural quad-sphere before pivoting to Nanite) and generation sims, the game is finally starting to look… kinda good? Let me show this off for a moment. Screenshots are running in real-time on my gaming PC:

AI-written description: A game render from low altitude over snowy rolling hills in warm, low evening sunlight, with long shadows, bare grey rock on the slopes, patches of dark ground and scattered clusters of small dark conifers. Fluffy, faintly pink-tinted clouds line the horizon under a pale sky, and one large cloud hangs across the middle of the view.
AI-written description: A game render over rolling hills densely covered in small round green trees, lit by a low sun that leaves the valleys in long blue shadow. Fluffy white clouds hang at mid-height in front of a hazy, snow-capped mountain range under a pale blue sky.
AI-written description: A game render from near the edge of space, looking along a long, snowy mountain chain that runs toward the curved horizon under a black sky and a glowing blue band of atmosphere. The low sun tints the scene pink and lilac; small puffy clouds are scattered over the plains on the left, and long ribbons of cloud stretch down the right side.
AI-written description: A game render from low altitude in bright afternoon light, looking across snowy plains patched with dark green forest toward a hazy horizon under a pale blue sky. Puffy white clouds float at eye level, some close to the camera and casting shadows on the snow, and snowy mountains rise on the right.
AI-written description: A game render looking down from above a layer of soft, puffy clouds in evening light, the clouds tinted pale pink and lilac against blue shadow. Rounded cloud tops rise out of the layer, dark forested ground with a few snowy ridges shows through a gap on the left, and jagged snowy peaks line the hazy horizon at the top.

…so pretty. 🀩