SAGE-Room

Sparse-view Anchor Graph Enhanced Room Reconstruction

SAGE-Room reconstructs an editable semantic indoor CAD scene from sparse, unposed RGB photographs. It separates room-structure recovery from object recovery: a high-quality five-view Plane-DUSt3R window freezes the room coordinate system, while all fifteen views provide depth, semantic masks, object identities, 3D boxes, and spatial relations. Captured depth, LiDAR, ARKit poses, and Gaussian Splatting are not used.

中文简介：本项目从 15 张无位姿普通 RGB 照片恢复可编辑的室内语义 CAD 场景。5 张高质量连续视图负责房间结构，全部 15 张照片负责实体识别、跨视图匹配与三维定位；输入不包含实测深度、LiDAR 或 ARKit 位姿。

Every thumbnail in the pipeline above comes from the included input images or a saved intermediate result of the released experiment.

Highlights

• Sparse RGB input without known camera poses or captured depth.
• RGB-derived relative depth with indoor semantic and instance segmentation.
• One-to-one cross-view identity association instead of class-only merging.
• Five-view room branch plus all-view object geometry fusion.
• Confidence-gated entity relation graph for position and orientation correction.
• Wall, floor, ceiling, opening, support, containment, and collision constraints.
• Editable semantic scene output in JSON, OBJ, GLB, and publication-ready figures.
• No training or fine-tuning is required for the released baseline.

Released configuration

Component	Released setting
Input	15 ordered RGB photographs
Room branch	Frames 09-13, one joint Plane-DUSt3R window
Entity branch	All 15 photographs
Relative depth	Depth Anything V2 Small
Panoptic semantics	OneFormer / ADE20K
Open-vocabulary detection	Grounding DINO Tiny
Instance masks	SAM2 Hiera Tiny
Appearance descriptors	DINOv2 Small
Geometry representation	Non-convex room footprint and regular 3D boxes
Scene output	Semantic CAD assets and relation metadata

The default parameters are defined in configs/default.yaml. They are configuration values rather than scene-specific hard-coded corrections.

Quick start

只想检查代码和可视化是否正常，可运行缓存演示（约数十秒，不下载模型）：

conda env create -f environment.yml
conda activate sage-room
powershell -ExecutionPolicy Bypass -File run_cached_demo.ps1

输出位于 outputs/cached_scene_topview.png。需要从 15 张 RGB 完整重建时，再运行：

powershell -ExecutionPolicy Bypass -File setup.ps1
conda activate sage-room
powershell -ExecutionPolicy Bypass -File run_full_pipeline.ps1

Repository contents

SAGE-Room/
├── assets/                    # Redistributable CAD assets and their license
├── configs/default.yaml       # Frozen baseline configuration
├── data/example_rgb/          # Fifteen normalized example photographs
├── resources/cached/          # Compact cached scene and verified previews
├── scripts/                   # Preprocessing, inference, fusion, and evaluation
├── environment.yml
├── requirements.txt
├── setup.ps1
├── run_cached_demo.ps1
└── run_full_pipeline.ps1

Generated model weights, third-party source trees, run caches, and full experiment outputs are intentionally excluded from version control.

Requirements

• Windows 10 or 11 with PowerShell 5.1 or newer.
• Miniconda or Anaconda.
• Git.
• Git LFS (used for the fifteen full-resolution example photographs).
• NVIDIA GPU with at least 8 GB VRAM for full inference.
• CUDA-capable NVIDIA driver compatible with PyTorch CUDA 12.1.
• Approximately 12 GB of free disk space for environments, checkpoints, and intermediate results.

The cached demonstration only needs the Python environment and does not download model checkpoints.

Cached reconstruction details

git lfs install
git lfs pull
conda env create -f environment.yml
conda activate sage-room
powershell -ExecutionPolicy Bypass -File run_cached_demo.ps1

Expected output:

outputs/cached_scene_topview.png

The cached path verifies the environment, scene schema, coordinate normalization, and renderer without running the deep models.

Full reproduction details

Run the setup script once. It creates the sage-room environment, checks out the pinned Plane-DUSt3R revision, installs its requirements, downloads official pretrained weights, and validates the installation.

powershell -ExecutionPolicy Bypass -File setup.ps1
conda activate sage-room
powershell -ExecutionPolicy Bypass -File run_full_pipeline.ps1

The full pipeline executes the following stages:

1. EXIF-aware orientation normalization and 1024-pixel semantic working copies.
2. RGB-derived relative depth, panoptic semantics, open-vocabulary detections, and instance masks.
3. A single five-view Plane-DUSt3R room reconstruction from frames 09-13, plus overlapping geometry windows used only to localize entities observed across all fifteen views.
4. One-to-one cross-view identity association and all-view entity geometry fusion.
5. Confidence-gated global relation optimization.
6. Wall, floor, ceiling, opening, support, room-boundary, and collision constraints.
7. Semantic CAD replacement, result visualization, comparison, and ablation export.

The first run downloads several model checkpoints and is therefore substantially slower than later runs.

Using another scene

Replace the images in data/example_rgb/ with fifteen ordered JPEG images:

frame_000001.jpg
frame_000002.jpg
...
frame_000015.jpg

Recommended capture protocol:

• Keep one camera lens and focal-length mode throughout the sequence.
• Rotate images to a consistent gravity direction or preserve valid EXIF orientation.
• Move the camera between viewpoints instead of only rotating in place.
• Maintain overlap between adjacent views.
• Ensure key doors, windows, furniture, and both sides of the corridor transition are visible.

For a different number of images or a different structural window, update scene.expected_images, scene.room_view_indices, and scene.entity_windows in configs/default.yaml, then adapt the corresponding command arguments in run_full_pipeline.ps1.

Main outputs

outputs/final/data/full_scene.json
outputs/final/data/ablation_results.csv
outputs/final/figures/
outputs/final/scene_topview.png

full_scene.json records each entity's semantic class, global identity, center, dimensions, orientation, support surface, wall assignment, confidence values, and high-confidence spatial relations. The geometric scene and CAD assets are also exported by the intermediate CAD stage.

Reproducibility boundary

• The algorithm receives only RGB photographs.
• Relative depth is predicted independently from RGB and is not metric sensor depth.
• Room scale is recovered from the configured room-height prior; it is not obtained from LiDAR or ARKit.
• The room branch is frozen before all-view entity fusion so that object observations cannot deform the footprint.
• Low-confidence spatial relations are rejected before global optimization.
• Reported relation residuals, containment, collision, and class-count errors are internal-consistency measures unless an external reference is explicitly supplied.
• The cached example verifies code and data flow; full model outputs are regenerated by run_full_pipeline.ps1.

Third-party components

• Plane-DUSt3R, pinned to commit 9a1ae50650ec6d706bf329352aaaf49efded90a0.
• Depth Anything V2.
• OneFormer.
• Grounding DINO.
• Segment Anything 2.
• DINOv2.
• CAD assets are selected from the Kenney Furniture Kit; its license is included at assets/kenney_furniture_kit/License.txt.

Model checkpoints and third-party repositories are downloaded from their official sources and are not committed to this repository.