LLM2R marks a shift from large language models to three-dimensional reassemblers.

Meta description: LLM2R casts 3D part assembly as sequence modeling with a decoder-only LLM, improving accuracy and inference speed on PartNet, PartNeXt and Breaking Bad.

Tags: LLM2R, 3D part assembly, PartNet, Breaking Bad, PartNeXt

Featured image: Abstract 3D assembly of geometric tokens forming a complete object, in a technology style without people, logos, or text.

LLM2R recasts 3D part assembly as a sequence modeling task for a pretrained decoder-only large language model, or LLM, and produces a rigid SE(3) pose for every part in a single forward pass. The shift matters because assembly is global: a part's correct placement is rarely explained by its own geometry alone, and methods based on pairwise matching or iterative verification struggle as objects contain more parts.

Background: The Task Is Not Local

3D part assembly starts with unassembled parts and asks for the poses that reconstruct a complete shape. The pose is a rigid SE(3) transformation, meaning a 3D rotation combined with a 3D translation. The difficulty comes from dependence among parts. One component may have many locally plausible positions, yet only one arrangement can be compatible with the whole set.

Earlier methods often build the assembly from pairwise relations or from repeated cycles of generation and verification. Those designs can capture local constraints, but they scale poorly when the part count grows. More parts mean more relations to compare and more verification steps, so accuracy and speed become harder to keep together.

Development: From Geometric Tokens to One-Forward-Pass Assembly

LLM2R converts parts into a sequence that a language model can read. A Point-BERT encoder and a Q-Former compress each part into a block of geometric tokens. The blocks for all parts are then concatenated, so the model receives one ordered sequence containing the whole part set.

A causal LLM adapted with Low-Rank Adaptation processes that sequence in one forward pass. Low-Rank Adaptation keeps the pretrained model's sequence capacity while adding a compact trainable update for the assembly task.

The causal structure creates a limitation: a part's hidden states can see only earlier parts. To close that gap, LLM2R appends learnable summary tokens after the last part. These tokens follow every part, so their states aggregate the full assembly at a constant token cost. A part-aware pose head then combines each part's states with the summary tokens before predicting rotation and translation.

Latest Progress: Global Context Improves Accuracy and Speed

A single model can therefore cover both semantic and geometric assembly and output all poses in one non-autoregressive pass. Because it does not generate poses one by one, the method can be faster while still producing a complete set of part transformations.

The benchmark picture is consistent across datasets. LLM2R records the highest part accuracy and success rate on all three PartNet categories. On PartNeXt, it has the best rotation error, translation error, and part accuracy. On Breaking Bad, it delivers the lowest translation error and fastest inference, with only a fracture-specialized method more accurate.

The gain becomes sharper at larger part counts. At 100 parts, LLM2R is 33.5 points ahead of GARF in part accuracy on Breaking Bad and 29.1 points ahead of Assembler on PartNet. It also transfers zero-shot to unseen categories, meaning it is evaluated without explicit training on those categories, with part accuracy more than 16 points above the strongest of three baselines. Zero-shot performance suggests the sequence-level reasoning is not tied only to the categories used for training.

The current stage is a move from local relation matching or iterative verification toward global, single-pass sequence modeling. The next signal is whether summary tokens and part-aware pose heads can continue to close gaps with fracture-specialized methods while preserving their speed advantage as part counts increase.