Lightmap Compression in The Witness

In my initial implementation of our lightmapping technology I simply stored lightmap textures in RGBA16F format. This produced excellent results, but at a very high memory cost. I later switched to the R10G10B10A2 fixed point format to reduce the memory footprint of our lightmaps, but that introduced some quantization artifacts. At first glance it seemed that we would need more than 10 bits per component in order to have smooth gradients!

At the time the RGBM color transform seemed to be a popular way to encode lightmaps. I gave that a try and the results weren’t perfect, but it was a clear improvement and I could already think of several ways of improving the encoder. Over time I tested some of these ideas and managed to improve the quality significantly and also reduce the size of the lightmap data. In this post I’ll describe some of these ideas and support them with examples showing my results.

I believe the RGBM transform was first proposed by Capcom in these CEDEC 2006 slides. While Capcom employs it for diffuse textures, it has become a popular way to encode lightmaps. RGBM or some of its variations are used in Unity, Bioshock Infinite, and the Unreal Engine, among others. Its use for standard color textures is not as widespread, but Shane Calimlim found it to be a good fit for the stylized artwork of Duck Tales and suggests it could be a good format in general. However, with so many precedents, I was surprised it had not been analyzed in more detail.

The main challenge of compressing lightmaps is that often they have a wider range than regular diffuse textures. This range is not as large as in typical HDR textures, but it’s large enough that using regular LDR formats results in obvious quantization artifacts. Lightmaps don’t usually have high frequency details, they are often close to greyscale, and only have smooth variations in the chrominance.

In our case, most our lightmap values are within the [0, 16] range, and in the rare occasions when they are outside of that range, we constrain them clamping the colors while preserving the hue to avoid saturation artifacts. Brian Karis also suggests tone mapping the upper section of the range to avoid sharp discontinuities, but I only found this to be a problem when light sources had unreasonably high intensity values.

The shape of the lightmap color distribution varies considerably. Interior lightmaps are predominantly dark with a long tail of brighter highlights:

while outdoor lightmaps have a more Gaussian distribution with a bell-like shape. This particular lightmap is under the shade of some colored fall trees, which give it an orange tone:

Not all lightmaps use all the available range, so after tone mapping the next thing we do is to scale the range to [0, 1].

So, why is RGBM a good choice for data like this? The distribution of distinct values that can be represented with RGBM looks as follows:

It provides much more precision toward 0 than toward 1. This is beneficial for images that are intended to be visualized at multiple exposures. We want to obtain smooth lightmaps without quantization artifacts independently of the camera exposure. However, as we will see later, this provides much more precision around 0 than is actually necessary.

Naive RGBM Encoding

In my initial implementation I simply used RGBA8 textures, squaring the colors to perform gamma correction in the shader. The standard rgb -> RGBM transform is as follows:

m = max(r,g,b)
R = r/m
G = g/m
B = b/m
M = m

A simple improvement I did early on is to divide the quantization interval in two. This is a variation of the idea presented in Microsoft’s LUVW HDR texture paper, but instead of using an extra texture, I simply rely on the RGB and alpha (M) channels.

A similar observation is done by Shane Calimlim:

Gray is encoded as pure white in the color map, which may not always be optimal. Gray is an edge case most of the time, but a smarter encoding algorithm could make vast improvements in its handling. In the simple version of the algorithm the entire burden of representing gray lies with the multiply map; this could be split between both maps, improving precision greatly in scenarios where the color map can accommodate extra data without loss.

But in our case grey is not really an edge case! Lightmaps are mostly grey with slight smooth color variations.

The way I implemented this is by choosing a certain threshold t. For values of m that are lower than t the color is fully encoded using only the RGB components as follows:

R = r/t
G = g/t
B = b/t
M = 0

and for values of m greater than the threshold t, the normalized color is encoded in the RGB components, and the normalization factor m is biased and scaled to store it at a higher precision:

R = r/m
G = g/m
B = b/m
M = (m-t) / (1-t)

That’s equivalent to just doing:

m = max(r,g,b,t)
R = r/m
G = g/m
B = b/m
M = (m-t) / (1-t)

This is useful for several reasons. As Shane notes, splitting the burden of representing the luminance between the RGB and M maps we can obtain more precision and reduce the size of the quantization interval.

It’s important to note that this actually reduces precision around zero, where we don’t actually need so much, because the game camera never has long enough exposures. If we look at the distribution of grey levels that biased RGBM can represent it now looks as follows:

Picking different values of t allows us to use different quantization intervals for different parts of the color range. The optimal choice of t depends on the distribution of colors in the lightmap and the number of bits used to represent each of the components. We chose this value experimentally. For our lightmaps values around 0.3 seemed to work best when encoding them in RGBA8 format.

Optimized RGBM Encoding

With these improvements RGBM was already producing very good results. Visually I could not see any difference between the RGBM lightmaps and the raw half floating point lightmaps. However, I had not reduced the size of the lightmaps by much and ideally I wanted to compress them further.

The next thing that I tried to do was to choose M in a way that minimizes the quantization error. I did that by brute force, trying all possible values of M, computing the corresponding RGB values for that choice of M, and selecting the one that minimized the MSE:

for (int m = 0; m < 256; m++) {
    // Decode M
    float M = float(m) / 255.0f * (1 - threshold) + threshold;

    // Encode RGB.
    int R = ftoi_round(255.0f * saturate(r/ M));
    int G = ftoi_round(255.0f * saturate(g / M));
    int B = ftoi_round(255.0f * saturate(b / M));

    // Decode RGB.
    float dr = (float(R) / 255.0f) * M;
    float dg = (float(G) / 255.0f) * M;
    float db = (float(B) / 255.0f) * M;

    // Measure error.
    float error = square(r-dr) + square(g-dg) + square(b-db);

    if (error < bestError) {
        bestError = error;
        bestM = M;
    }
}

This improved the error substantially, but it introduced interpolation artifacts. The RGBM encoding is not linear, so interpolation of RGBM colors is not correct. With the naive method this was not a big deal, because adjacent texels usually had similar values of M, but the M values resulting from this optimization procedure were not necessarily similar anymore.

However, it was easy to solve this problem by constraining the search to a small range around the M value selected with the naive method:

float M = max(max(R, G), max(B, threshold));
int iM = ftoi_ceil((M - threshold) / (1 - threshold) * 255.0f);

for (int m = max(iM-16, 0); m < min(iM+16, 256); m++) {
    ...
}

This constrain did not reduce the quality noticeably, but eliminated the interpolation artifacts entirely.

While this idea showed that there's a significant optimization potential over the naive approach, it did not get us any closer to our stated goal: to reduce the size of the lightmaps. I tried to use a packed pixel format such as RGBA4, but even with the optimized encoding, it did not produce sufficiently high quality results. To reduce the size further we would have to use DXT block compression.

RGBM-DXT5

Simply compressing the RGBM data produced poor results and compressing the optimized RGBM data did not help, but instead only degraded the results even more.

A brute force compressor is not practical in this case, because when processing blocks of 4x4 colors simultaneously the search space is much larger.

A better approach is to first compress the RGB values obtained through the naive procedure using a standard DXT1 compressor and then choosing the M values to compensate for the quantization and compression errors of the DXT1 component.

That is, we want to compute M so that:

M * (R, G, B) == (r, g, b)

This gives us three equations that we can minimize in the least squares sense. The M that minimizes the error is:

M = dot(rgb, RGB) / dot(RGB, RGB)

In my tests, the resulting M's compress very well in the alpha map and reduced the error significantly.

I also tried to encode RGB again with the newly obtained M, and compress them afterward, but in most cases that did not improve the error. Something that worked well was to simply weight the RGB error by M in the initial compression step.

The number of bits allocated for the RGB and M components is very different than in our initial RGBA8 texture, so the choice of t had to be reviewed. In this case values of t around 0.15 produced best results. I attribute this to the reduced number of bits per pixel used to encode the RGB channels.

Results

In addition to the described formats I also compared the proposed method against BC6. BC6 is specifically designed to encode HDR textures, but it's not available in all hardware. Our optimized RGBM-DXT5 scheme provides nearly the same quality as BC6:

The above chart is displaying RMSE values of the final images after color space conversion and range rescaling.

To study the effectiveness of the encoders it's more useful to look at the errors before rescaling. These look a lot more uniform, but cannot be compared against BC6 anymore, since in that case adjusting the range of the input values does not usually reduce the compression error.

Finally, I thought it would be interesting to use RGBM-DXT5 to compress standard images and compare it against YCoCg-DXT5. The following chart shows the results for the first 8 images of the kodim image set:

YCoCg-DXT5 is clearly a much better choice for LDR color textures.

Conclusions and Future Work

Our proposed RGBM encoder was good enough for our lightmaps, but I'm convinced there's more room for improvement.

One idea would be to pick a different threshold t for each texture. Finding the best t for a given texture to be encoded using the plain RGBM linear format would be easy, but it's not so obvious when using block compression.

The RGB components are encoded with a standard weighted DXT1 compressor. It would be interesting to use a specialized compressor that favored RGB values with errors that the M component could correct. For example, the M values resulting from the least squares minimization are sometimes above 1, but need to be clamped to the [0, 1] range, it should be possible to constrain the RGB endpoints to prevent that. It may also be possible to choose RGB endpoints such that the error of the least squares fitted M are as small as possible.

Finally, DXT5 is not available on most mobile GPUs. I haven't tried this yet, but it seems the ETC2 EAC_RGBA8 format is widely available and would be a good fit for the techniques presented here. It would also be interesting to compare our method against packed floating point formats such as (R11G11B10_FLOAT
R9G9B9E5_SHAREDEXP) and ASTC's HDR mode.

Tables

In all cases I measured the error using the RMSE metric, which is the same metric used to guide the block compressors. It may make more sense to use a metric that takes into account how the lightmaps are visualized in the game. I did exactly that, tone map the lightmaps at different exposures and compute the error in post-tone-mapping space. The tables below show the resulting values and they roughly correlate with the plain RMSE metric.

               Tone mapped error
            e=2.2   e=1.0   e=0.22     average   rmse

RGBM8 naive:

hallway:    0.00026 0.00045 0.00089 -> 0.00053 | 0.00007
hut:        0.00100 0.00102 0.00082 -> 0.00095 | 0.00609
archway:    0.00114 0.00141 0.00190 -> 0.00148 | 0.00818
windmill:   0.00102 0.00133 0.00185 -> 0.00140 | 0.00083
shaft:      0.00201 0.00228 0.00214 -> 0.00214 | 0.00798
hub:        0.00151 0.00182 0.00191 -> 0.00175 | 0.00267
tower:      0.00153 0.00200 0.00299 -> 0.00217 | 0.00160
tunnel:     0.00094 0.00123 0.00171 -> 0.00129 | 0.00093
mine:       0.00105 0.00120 0.00141 -> 0.00122 | 0.00640
theater:    0.00099 0.00126 0.00160 -> 0.00128 | 0.00129

RGBM8 optimized:

hallway     0.00010 0.00015 0.00030 -> 0.00018 | 0.00004
hut         0.00049 0.00043 0.00031 -> 0.00041 | 0.00543
archway     0.00044 0.00060 0.00122 -> 0.00075 | 0.00595
windmill    0.00020 0.00026 0.00036 -> 0.00027 | 0.00024
shaft       0.00059 0.00066 0.00102 -> 0.00076 | 0.00501
hub         0.00038 0.00051 0.00085 -> 0.00058 | 0.00099
tower       0.00060 0.00072 0.00082 -> 0.00072 | 0.00112
tunnel      0.00025 0.00031 0.00042 -> 0.00033 | 0.00048
mine:       0.00044 0.00049 0.00083 -> 0.00058 | 0.00467
theater:    0.00061 0.00076 0.00087 -> 0.00075 | 0.00095

RGBM4 optimized:

hallway:    0.00169 0.00259 0.00562 -> 0.00330 | 0.00063
hut:        0.00932 0.00899 0.00773 -> 0.00868 | 0.08317
archway:    0.00906 0.01287 0.02616 -> 0.01603 | 0.09614
windmill:   0.00424 0.00562 0.00830 -> 0.00606 | 0.00402
shaft:      0.01103 0.01314 0.01978 -> 0.01465 | 0.08204
hub:        0.00868 0.01160 0.01848 -> 0.01292 | 0.01722
tower:      0.01004 0.01217 0.01466 -> 0.01229 | 0.01835
tunnel:     0.00516 0.00687 0.01066 -> 0.00757 | 0.00764
mine:       0.00871 0.01044 0.01742 -> 0.01219 | 0.07510
theater:    0.00683 0.00840 0.00963 -> 0.00829 | 0.01057

DXT5 naive:

hallway:    0.00155 0.00249 0.00570 -> 0.00325 | 0.00048
hut:        0.00487 0.00536 0.00564 -> 0.00529 | 0.02119
archway:    0.00500 0.00656 0.01039 -> 0.00731 | 0.01949
windmill:   0.00214 0.00287 0.00444 -> 0.00315 | 0.00177
shaft:      0.01062 0.01339 0.01977 -> 0.01459 | 0.03412
hub:        0.00616 0.00796 0.01130 -> 0.00848 | 0.01481
tower:      0.00551 0.00712 0.01019 -> 0.00761 | 0.00735
tunnel:     0.00235 0.00308 0.00451 -> 0.00331 | 0.00285
mine:       0.00471 0.00589 0.00877 -> 0.00646 | 0.01809
theater:    0.00332 0.00412 0.00496 -> 0.00413 | 0.00498

DXT5 optimized:

hallway:    0.00125 0.00199 0.00456 -> 0.00260 | 0.00041
hut:        0.00336 0.00373 0.00408 -> 0.00372 | 0.01529
archway:    0.00353 0.00460 0.00719 -> 0.00511 | 0.01285
windmill:   0.00134 0.00180 0.00280 -> 0.00198 | 0.00116
shaft:      0.00801 0.01016 0.01507 -> 0.01108 | 0.02437
hub:        0.00469 0.00602 0.00846 -> 0.00639 | 0.01241
tower:      0.00421 0.00544 0.00781 -> 0.00582 | 0.00599
tunnel:     0.00157 0.00206 0.00306 -> 0.00223 | 0.00193
mine:       0.00338 0.00428 0.00646 -> 0.00471 | 0.01178
theater:    0.00245 0.00302 0.00357 -> 0.00301 | 0.00382

DXT5 optimized with M-weighted RGB:

hallway:    0.00114 0.00184 0.00430 -> 0.00243 | 0.00038
hut:        0.00338 0.00382 0.00443 -> 0.00388 | 0.01478
archway:    0.00356 0.00464 0.00725 -> 0.00515 | 0.01271
windmill:   0.00134 0.00180 0.00281 -> 0.00198 | 0.00113
shaft:      0.00804 0.01023 0.01522 -> 0.01116 | 0.02382
hub:        0.00472 0.00611 0.00868 -> 0.00650 | 0.01088
tower:      0.00421 0.00544 0.00787 -> 0.00584 | 0.00597
tunnel:     0.00157 0.00206 0.00306 -> 0.00223 | 0.00193
mine:       0.00337 0.00428 0.00648 -> 0.00471 | 0.01170
theater:    0.00245 0.00302 0.00356 -> 0.00301 | 0.00382