Out-of-core precompute:
Enlighten 3.10 supports development for massive world

Introduction

Last November, we released the Enlighten SDK version 3.10, our first major SDK release at Silicon Studio. This release includes a variety of useful new features and fixes. For details, please refer to our release notes for Enlighten SDK and Enlighten UE4. In this article, we will cover in detail one of the key features, “Out-of-core precompute” within Enlighten UE4.

Background

Today, we enjoy a wide variety of gaming experiences thanks to technological advancements. Triple-A titles provide a richer and truly immersive experience, with ever larger open worlds that the player can explore freely. Game developers face many challenges to create such massive worlds. Managing time for the creation of assets is one of those headaches. A massive world requires more content to be created, and more time spent building game ready data.

It is essential that your tools and pipelines can scale to handle this additional content efficiently. . Enlighten precomputes radiosity data to efficiently calculate global illumination at runtime. Larger scenes require more time to run the Enlighten Precompute, which is no surprise.

The Enlighten precompute generally scales well for larger scenes, but for massive worlds that don’t fit within memory of a single system, the computation must be split up into smaller chunks. Using Incredibuild to distribute the computation is a great way to speed up the precompute, but when the world does not fit within memory each task will slow to a crawl.

As game worlds become ever larger Enlighten needed to be modified to help designers to create even massive scenes more comfortably and efficiently. We decided to add Out-of-core precompute to Enlighten 3.10 to address this..

Technical Details

Managing massive scenes

In this section, we briefly discuss general level streaming. If you think you are familiar with this topic, please skip here.

When you treat massive scenes in your games, you know that not all objects can be there just to be operable all the time. Because large amount of memory and enormous processing are expected, you easily eat up all hardware resources. In such a case, those scenes are usually broken up into smaller chunks, so that only the relevant parts of the scenes can be loaded or unloaded as needed during gameplay in order to avoid this issue. This approach is generically called level streaming.

picture: World Composition User Guide.

The basic ideas of this approach are that levels are loaded only when they fall within a certain range of a player’s coordinates. For example, if a player appears on coordinate A as shown in the picture (1) below, levels around it are loaded. If the player starts moving toward Coordinate B from A, level loading starts if those levels fall in the direction of forward movement, while level unloading starts if they lie in the opposite direction. When the player stops moving, its situation becomes like the picture (3) shown below. As a player moves, levels are loaded/unloaded accordingly like this and only minimum data is processed.

(1) State on Coordinate A	(2) State of a player moving from Coordinate A to B	(3) State on Coordinate B

If levels are loaded asynchronously to game updating cycles, a player could be seen as if it appears in one continuous huge scene although only part of the entire scene actually exists. This is a quite simple idea, but is very effective.

Most game engines integrate features like this to support level streaming management. For details about this feature, please refer to Official document (UE4).

Issues in Enlighten 3.09

Enlighten treats complex light interaction called “Global Illumination”. This effect appears when objects are placed in a scene and they influence each other, so you need to extract necessary information from the entire scene for accurate calculation. Up to Enlighten 3.09, the entire scene was loaded during Precompute to narrow it down to the area where influence could be found. This approach works fine with rather smaller scenes, while a huge amount of memory is required if it becomes a certain size or larger, which could cause memory swapping very frequently and should take more time for precompute.

Enlighten internally breaks up scenes into smaller chunks called System to manage radiosity data. When data is generated to support each system, you need to consider the systems in the surrounding area where a system could affect as previously mentioned. Their visibilities between systems need to be checked in order to judge if systems affect each other or not. If you simply try to follow this step, all of the systems are required to be loaded for ray tracing, of which cost should be extremely high without doubt in terms of both memory and processing loads.

If you find any clues about relationships between those systems, you can decide which to exclude from computation before checking all systems. If such a filtering process is done as preparation, whether influence can be found among systems or not will be efficiently determined.

Approach with Enlighten 3.10

Enlighten 3.10 addresses this issue by providing level’s streaming information as a filtering clue. This approach is based on the supposition: If one level has a chance to affect certain levels, you can limit to those levels reasonably if they are loaded simultaneously when that one level is loaded. We won’t dig into details about relationship between systems and levels here, but two instances won’t be included in the same system if one instance belongs to a different level from the other. Therefore, you can simply apply level chunk units to systems.

In order to determine if several levels are sometimes actually loaded simultaneously or not, you need to obtain border information and streaming distance about each level. The following images indicate how they work.

Levels A and B which are loaded simultaneously	Levels A and B which are NOT loaded simultaneously

By doing this, you can narrow down to see which level could affect the other and minimize the area to load for detailed testing. This could substantially reduce the amount of memory to use.

As an aside, these calculated possible influencing factors over systems are outputted as intermediate data and passed to Precompute. If you want to use SDK version to integrate Enlighten into your own engine, you can utilize the Out-of-core precompute like UE4 version by implementing the process similarly. For details about how to configure, refer to Radiosity dependencies in the document.

Configuring with editors

Settings for dependencies

Streaming distances are set to respective levels. If you use world compositions to manage streaming, that distance information is considered automatically. Without any special setting to configure, dependencies are calculated only for the levels which are simultaneously loaded during in-games performances.

For further optimization, use the “Enlighten Bounce Distance” in the World Settings. If a valid value is set to this parameter, dependencies between levels can be judged more strictly.

This setting is valuable if you don’t use world compositions but manage streaming manually (by writing codes, etc.). In such a case, all levels use this distance as their streaming one without exceptions.

Settings for Precompute commandlet

The modifications which have been made for the Out-of-core precompute include optimization for commandlets as well as ones for the above dependencies. If Enlighten Precompute is automated by nightly build or any, Precompute should be executed by commandlets instead of editors. In Enlighten 3.10, the out-of-core mode has been added to load only minimum levels at execution to support this.

This feature is not compatible with certain existing features, so disabled by default. In order to activate this, uncheck the “Share Lightmaps Across Levels” in the World Settings and disable lightmap sharing between levels when Detail lighting mode is used. (*For details about this feature, refer to Document.）

Summary

By using the Out-of-core precompute, you can configure Enlighten efficiently even if you have to develop games which must use massive scenes and make the best of Enlighten’s real time global illumination. Please try our latest Enlighten 3.10 to enjoy its powerful features!