IGGT4D: Streaming 4D Instance-Grounded
Geometry Transformer

Zhengyu Zou^1,†, Hao Li², Kuixuan Jiao^1,†, Liu Liu^1,‡, Tingyang Xiao¹,
Xiaolin Zhou¹, Fangzhou Hong², Zhizhong Su¹, Dingwen Zhang^3,✉, Ziwei Liu²

¹Horizon Robotics     ²S-Lab, Nanyang Technological University     ³Institute of Artificial Intelligence, Hefei Comprehensive National Science Center

^†Intern at Horizon Robotics     ^‡Project Leader     ^✉Corresponding Author

---

This is the official repository for the paper IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer.

Overview

IGGT4D is a streaming instance-grounded geometry Transformer for online 4D scene understanding. It processes video frames sequentially, reuses historical context through causal spatiotemporal modeling, and incrementally updates a unified representation of camera motion, scene geometry, and object identity. We also introduce InsScene4D-147K, a large-scale dataset with RGB images, depth, camera poses, and temporally consistent instance masks for unified geometry-instance learning.

Installation

Clone HorizonRobotics/IGGT4D, create a Conda environment with Python 3.10, and install PyTorch. The commands below use PyTorch 2.8.0 with CUDA 12.8 as an example:

git clone https://github.com/HorizonRobotics/IGGT4D.git
cd IGGT4D

conda create -n iggt4d python=3.10 -y
conda activate iggt4d

pip install torch==2.8.0 torchvision==0.23.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128
pip install -e .

Checkpoint

Download the model checkpoint from Hugging Face and save it as checkpoints/model.safetensors.

Inference

Run streaming inference on the demo sequence:

python streaming_infer.py

Options

• --mode full: use full-attention (offline) inference.
• --camera: enable camera pose input.
• --save-npy: save prediction results as NPY files.
• --save-ply: save RGB and instance-feature PCA point clouds.

Run on a custom image sequence:

python streaming_infer.py \
  --image-dir /path/to/images \
  --output-dir /path/to/output

Input

image       # RGB image sequence (B, N, 3, H, W)
extrinsics  # Optional world-to-camera extrinsics (B, N, 4, 4)
intrinsics  # Optional camera intrinsics (B, N, 3, 3)

Output

pred["depth"]        # Predicted depth maps (B, N, H, W)
pred["depth_conf"]   # Per-pixel depth confidence scores (B, N, H, W)
pred["ins_feature"]  # Per-pixel instance embeddings (B, N, 8, H, W)
pred["ins_mask"]     # Instance masks produced by clustering (B, N, H, W)
pred["extrinsics"]   # World-to-camera extrinsics (B, N, 3, 4)
pred["intrinsics"]   # Camera intrinsics (B, N, 3, 3)

To-Do List

• Release the paper and project page
• Release the inference code
• Release the model checkpoints
• Release the dataset
• Release the training code
• Release the benchmark
• Release the evaluation code

Acknowledgements

IGGT4D is built upon the following excellent open-source projects:

• Depth Anything 3
• IGGT

We sincerely thank the authors for making their work publicly available and encourage readers to explore and cite the corresponding works.

License

Licensed under the Apache License 2.0. See LICENSE.

Citation

@article{zou2026iggt4d,
  title={IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer},
  author={Zou, Zhengyu and Li, Hao and Jiao, Kuixuan and Liu, Liu and Xiao, Tingyang and Zhou, Xiaolin and Hong, Fangzhou and Su, Zhizhong and Zhang, Dingwen and Liu, Ziwei},
  journal={arXiv preprint arXiv:2607.19228},
  year={2026}
}