SpatialScout.
Can VLM agents actively
explore and map 3D scenes?
Move the camera. Connect the views. Locate objects in 3D.
An interactive benchmark where observation is part of the reasoning.
A scene to explore.
A map to build.
A VLM controls a virtual camera, chooses new viewpoints, and predicts the geometric centers of target objects in the initial camera frame.
Diverse spaces. One protocol.
80 scenesFrom views to coordinates
Capture RGB views and read camera poses.
Schematic illustrationSeeing more is only
part of the challenge.
Compare eight model configurations on localization and coverage. Switch the evaluation scope to explore all targets or only the submitted objects.
| Model | Tasks |
|---|
Task-average (macro) metrics. Missing targets count as zero for “All objects.” Thresholds are strict. Only trajectories ending with successful finish_mapping are included; task counts differ by configuration.
Three ways a reconstruction can turn out
Ground truth and reconstruction from three viewpoints reveal geometric accuracy, missing objects, and localization failures.
Mapping accuracy.
Measured against cost.
Price and performance span orders of magnitude. Explore the configurations that improve reconstruction quality for a given average task cost.
Hover or focus on a model to see its exact cost and measurement count.
How the cost comparison is measured +
Costs use the saved OpenRouter pricing snapshot (19 October 2026), including input, output and recorded cache usage. The mean uses only tasks with complete token records; coverage differs across models. Astra medium combines its previous IoU score with the newer run’s recorded task costs.
This chart uses object-average (micro) IoU, matching the existing cost figure. The result table above uses task-average (macro) metrics. Dollar amounts are estimates under the saved pricing, not billing receipts.
Open the original cost figure ↗ Download the data snapshot ↓Your model.
The same scenes.
Source code, fixed task definitions, and scene resources are available. Use your own vision-and-tool-capable model endpoint to rerun the 80-task protocol.
# From the repository root, after installation
export VLM_BASE_URL='https://YOUR_API_HOST/v1'
export VLM_API_KEY='YOUR_MODEL_API_KEY'
export VLM_MODEL='EXACT_MODEL_ID'
export EVAL_OUTPUT_DIR=outputs/room_benchmark/my_model
bash scripts/run_evaluation.sh
# Compute the eight IoU metrics
.venv/bin/python -m spatialscout.evaluation.score \
"$EVAL_OUTPUT_DIR"Build on SpatialScout.
If you use this benchmark in your research, please cite our paper.
Download BibTeX ↓