Pumpire: A Unified Benchmark for Estimating Metric Distances

Meta description: Pumpire is a unified benchmark for metric point-pair distance estimation in 3D foundation models, using 6,400 frames, 100 scenes and 29 baselines.

Tags: Pumpire benchmark, 3D foundation models, point-pair distance, metric depth evaluation

Featured image: An abstract 3D sensor grid formed by translucent cubes and dotted lines, with no real people, no company logos and no text.

The Pumpire benchmark is a unified test for metric point-pair distance estimation in image- and video-level 3D foundation models. It matters because distance is a basic property of 3D understanding, yet a model can look accurate in one view and unstable in another. Pumpire covers systems that use depth priors and systems that do not, giving researchers a shared frame for comparison.

Why Metric Distance Needed a Shared Test

A point-pair distance estimate asks how far apart two points are in real-world units. In a single image, a model may rely on size cues, perspective and learned priors. In a video, the same relationship should remain stable as the camera moves, but small errors can accumulate across frames. Without a common benchmark, researchers compare models using different scenes, different distance definitions and different assumptions about depth. The result is fragmented evaluation. Pumpire addresses that gap by defining a shared task: estimate metric distances between points in real-world scenes, then test whether the estimates hold across views.

How Pumpire Built a 6,400-Frame Measurement Base

The benchmark was built from manually collected material. Its core dataset contains 6,400 frames drawn from 100 real-world scenes. Those frames are annotated with physically measured point-pair distances. That detail is important. Many evaluation sets depend on synthetic geometry or estimated ground truth, which can leave ambiguity about whether a model is wrong or the annotation is imprecise. Physical measurement gives the dataset a clearer claim to real-world reliability. The manual collection also means the scenes are not merely rendered environments; they are spaces where distance can be checked by direct measurement.

The data structure moves from real-world capture to distance annotation: 100 scenes provide the material, 6,400 frames carry the observations, and physically measured point-pair distances define the target for evaluation.

What 29 Baselines Show About Accuracy and Consistency

Pumpire does not stop at data. The benchmark evaluates 29 baseline configurations. Its scope includes image- and video-level 3D foundation models, and it separates models that operate with depth priors from models that operate without depth priors. This creates a two-axis comparison: model form and depth-prior usage. The analysis examines metric accuracy, meaning how close a prediction is to the measured distance, and cross-view consistency, meaning whether the same distance remains stable when observed from different frames.

A unified benchmark like Pumpire can affect research practice in several ways. First, it gives model developers a clearer target. If a method improves average distance error but loses consistency across views, the benchmark exposes that trade-off. Second, it lets users compare models that are not directly related. Some systems may be designed for single images, while others use temporal information from video. A shared annotation format may reduce the gap between these approaches. Third, it clarifies the role of depth priors. If models with depth priors perform differently from models without them, the benchmark can show where that assumption helps or hurts.

Where Pumpire Leaves 3D Foundation Model Evaluation

At present, Pumpire sits at the stage of evaluation infrastructure rather than a single winning model. Its value comes from the combination of physically measured distances, 100 real-world scenes, 6,400 frames and 29 baseline configurations. Future work will likely test whether this framework can support broader comparisons across more 3D foundation models, especially as video understanding systems move from perception tasks toward spatial reasoning. The key question is not only which model has the lowest error, but which model can maintain metric distance estimates consistently across views.