AI Agents Can Reconstruct 3D Scenes from Photos but Can’t Tell Whether They’re Accurate

A research effort called LEGO-Anything has demonstrated an "Image-to-Code" pipeline in which a coding agent looks at one photograph and writes an executable Blender program that rebuilds the scene. Because the output is code rather than a static mesh, the result can be run, inspected, edited, and queried for downstream image-analysis tasks.

The workflow is iterative rather than one-shot. The agent drafts code, executes it in Blender, examines the rendered result, and revises until the reconstruction approaches the source image. That loop matters: the program explicitly encodes objects, geometry, layout, and camera position, so a user can modify any of those elements the way they would edit any other script.

Background: Why a Benchmark Was Needed

Evaluating this kind of system is awkward. Real photographs carry no precise 3D ground truth to score against, while simple synthetic scenes tend to look artificial. The team's answer is LEGO-Bench, built from professionally authored simulator scenes. It contains 208 images drawn from 104 indoor and outdoor scenes and relies on 443 registered assets.

Rendering from simulator scenes keeps the inputs natural-looking while the exact geometry, depth, and object assignments stay hidden as an answer key for automated scoring. Complexity can be scaled up without altering lighting or camera settings. Each scene is graded on three axes: validity (was a usable artifact delivered at all), reconstruction (how accurate the visible geometry is), and appearance (pixel-by-pixel comparison after re-rendering against the reference).

Development: Strong Validity, Weak Geometry

All six tested GPT configurations produced a working scene almost every time, but accuracy diverged sharply. GPT-6 Astra, the strongest configuration, reached 53.4 percent on indoor scenes and 39.6 percent on outdoor scenes; weaker setups landed near 15 percent. Difficulty scales with scene complexity, and exteriors proved harder than interiors.

Increasing the reasoning budget helped the GPT-6 variants substantially. Astra's score on an office test subset climbed from 32.3 to 61.8 percent. Step-level analysis identified the recurring failure modes: weak first attempts, revisions that rolled back earlier gains, and unreliable self-assessment.

That last problem is the most consequential. Asked to choose which of two versions better matched the original, models scored near or below chance on geometric judgment—an agent effectively cannot tell whether its own scene improved. Even GPT-6 Astra degraded its own output late in a run, falling from 33.9 to 4.4 percent. The authors conclude refinement must lean on concrete measurements rather than the agent's own opinion.

Latest Progress: A Training-Free Plugin

That finding produced LEGO-Plugin, an extension requiring no additional training. It anchors the starting scene to the reference image, replaces unreliable self-judgment with measurable signals, and protects correct progress from regressive edits. All six models improved. Weaker agents gained the most, with boosts up to 62.7 percent, while the already-strong top model added roughly two percentage points.

The team also tested whether reconstructed scenes could support standard vision tasks, since object detection, segmentation, and depth estimation can be read directly from an executable program. Without extra training, results were usable but unremarkable. Detection came closest, reaching about half the performance of the specialized model DINO; the gaps to SAM 3 and Depth Anything 3 were wider for segmentation and depth. The authors describe current coding agents as promising but not yet accurate enough, with a large distance between a working result and a faithful reconstruction.

What to Watch

Astra's commanding LEGO-Bench lead fits a broader pattern. AI researcher Yoav Artzi reads the model as a major step in spatial understanding and suspects training on large volumes of 3D data such as Blender scenes. Tooling is moving in parallel: Unity has shipped official plugins for Claude Code and Codex. Alternative routes avoid code altogether—World Labs reconstructs scenes inside its Atlas world model, while Google DeepMind's GenCeption uses a video model for depth estimation and segmentation that matches specialized models. The open question is whether measurement-driven refinement closes the geometry gap faster than end-to-end world models render code-based reconstruction unnecessary.