The Moon's Many Faces.

One surface, four signals

Remote sensing looks at the Moon through many instruments, each recording a different physical property. But the instruments rarely agree on where they have coverage — a region imaged by a camera may have no elevation model, and vice versa. This work treats four properties as four modalities of the same scene and learns to translate between them, so a missing modality can be synthesised from whatever is available.

Grayscale

The raw NAC photograph — a composite of shading and reflectivity that mixes geometry and material together.

Elevation (DEM)

A Digital Elevation Model — the height of the surface at every pixel, the target of 3D reconstruction.

Surface normals

The direction each point faces, with the x, y, z of the normal vector mapped to red, green, blue.

Albedo

Intrinsic reflectivity — how bright the material is once lighting and shape are removed. Tied to composition, not topography.

Six Apollo landing sites

Training uses six narrow-angle-camera (NAC) images from the Lunar Reconnaissance Orbiter covering the Apollo 12, 14, 15, 16 and 17 sites — a deliberately diverse cross-section of lunar geology. Apollo 11 is held out entirely for testing. Only grayscale is measured directly; the other three modalities are derived from each image with a Shape-from-Shading framework (Grumpe & Wöhler, using the Hapke reflectance model), which recovers elevation, normals and albedo from the photograph and a coarse low-resolution DEM.

Each stitched image is sliced into 224×224 patches with a stride of 32, producing 1,720,540 training samples. The same Shape-from-Shading pipeline underpins my earlier work on detecting anomalies at the Apollo sites →

From images to tokens

A transformer does not read pixels — it reads tokens. Each modality gets its own tokenizer: a Vision Transformer encodes the image, a vector quantizer snaps every patch onto the nearest entry in a discrete codebook, and a small CNN decoder can turn those codes back into an image. This is what lets one architecture treat a DEM and a photograph as the same kind of object — sequences drawn from a shared vocabulary.

Learning by filling in the blanks

The model is a masked autoencoder: at each step some tokens are shown as context and the rest are hidden targets it must predict. How they split is drawn from a Dirichlet distribution controlled by a single knob, α. Low α concentrates the context in one modality — teaching pure translation; high α spreads it evenly across all four. Move the knob and resample to see the masking change.

Training is staged so knowledge is never lost: first grayscale ↔ DEM alone (470 h), then surface normals are folded in (81 h), and finally albedo (100 h), each stage inheriting the previous weights.

Any-to-any: you pick the inputs

This is the whole point of the model. Choose any one, two or three modalities as input and it generates the rest — a single network, every direction. The images below are the model's real predictions for one held-out scene. Press and hold any prediction to reveal the ground truth underneath.

Where the model looks

Summing the attention across all layers shows which source modalities the model leans on to predict each target — and the pattern is physically meaningful, not arbitrary. Pick a target modality to see its attention map and what it reveals.

Predicting Grayscale

To rebuild a photograph the model attends most to albedo and normals — combining what the material is with how it is oriented to the light. This is inverse rendering: shading reconstructed from reflectivity and geometry.

How well does it work?

Every arrow in the any-to-any matrix has a score. Each cell below is one translation — a row modality predicting a column modality — coloured by how good it is. Switch the metric to see the picture change: geometry-to-geometry translations are excellent, while anything involving albedo stays hard.

Scaling up to real terrain

Tiled back together, the model reconstructs elevation over full landing-site areas. Drag the divider to wipe between the model's prediction and the ground-truth DEM. Giving it all three companion modalities is near-perfect; from a single grayscale photo it still recovers the shape, but absolute height drifts — the classic bas-relief ambiguity of Shape from Shading.

Ground truth

Predicted

Apollo 11 area — elevation predicted from a single grayscale image vs. the ground-truth DEM. The morphology is right; a slow vertical offset is the main error.

Paper, data & foundations