Learning Transferable Dynamics Priors from Action to World Modeling

Ze Huang^{1 *}, Jiahui Zhang^{1 *}, Hairuo Liu^{2 3 *}, Chenxi Zhang², Ran Cheng⁴, Li Zhang^{1 2 †}

¹Fudan University  ²Shanghai Innovation Institute  ³Shanghai Jiao Tong University  ⁴McGill University

^*Equal contribution  ^†Corresponding author

We view action-conditioned world modeling as a transferable dynamics prior for robot learning. A2World is pretrained on 2.1M+ robot manipulation trajectories spanning 20+ embodiments to predict future multi-view manipulation videos from an initial observation and future action chunks. The same pretrained dynamics prior can be adapted into A2World-sim, a long-horizon autoregressive simulator for policy evaluation, and A2World-policy, a video-action joint prediction model for instruction-conditioned robot control.

Abstract

Project page: https://logosroboticsgroup.github.io/A2World/

We study action-conditioned world modeling as a scalable way to learn transferable dynamics priors for robot learning. By pretraining a model to predict how actions drive visual scene evolution, the resulting world model captures reusable interaction dynamics beyond appearance-level video generation.

Concretely, we pretrain a multi-view interactive base diffusion world model, A2World, on large-scale robot manipulation data with real action annotations. We validate the learned dynamics priors from two complementary perspectives. First, we adapt A2World into a task- or scene-specialized real-world simulator, A2World-sim, whose long-horizon rollouts support simulator-based policy evaluation and scalable what-if analysis by replacing real-robot rollouts with world model rollouts. Second, starting from the same pretrained weights, we adapt A2World into a video-action joint prediction model, A2World-policy, that predicts actions under visual and instruction conditioning.

Experiments across simulation benchmarks and real-robot settings demonstrate that action-conditioned world model pretraining yields transferable dynamics priors that benefit both simulator-centric and policy-centric robot learning.

Method

A2World learns action-to-video dynamics from robot data, then transfers the pretrained prior into two downstream variants:

• A2World-sim injects pose-guided history and rolls out future observations autoregressively for long-horizon simulator-based evaluation.
• A2World-policy performs joint video-action diffusion with shared attention and action-specific denoising branches for instruction-conditioned control.

Code Release

The current release focuses on the A2World world-model component:

• multi-view action-conditioned A2World inference;
• history-aware A2World-sim autoregressive rollout generation;
• LIBERO HDF5 conversion, full-parameter fine-tuning, and inference;
• checkpoint validation, model-card metadata, and NVIDIA weight licensing.

cd world_model
python -m pip install -e . --no-deps
python scripts/download_base_models.py

a2world-demo \
  --variant libero \
  --checkpoint /path/to/a2world-libero.pt \
  --input /path/to/agentview.mp4 /path/to/eye_in_hand.mp4 \
  --actions /path/to/actions.npz \
  --base-checkpoints checkpoints \
  --output outputs/libero_rollout.mp4 \
  --autoregressive

See world_model/README.md for environment setup, data conversion, training, and rollout options.

The released checkpoints are hosted at Fleurrr/A2World-World-Model:

• a2world-pretrained.pt
• a2world-libero.pt

Interactive Model Demo

The project page provides button-driven demos. Select actions and press Rollout.

• AgiBot: 216 (6 × 6 × 6) garment rollouts from the pretrained checkpoint.
• RoboCoin: 80 (4 × 4 × 5) keyboard-control cube rollouts from the pretrained checkpoint.
• Galaxea: three bottle-interaction rollouts from the pretrained checkpoint.
• LIBERO: three book-placement rollouts from the dedicated LIBERO checkpoint.

World Model Rollouts

These videos show A2World world model rollouts on real-robot manipulation scenarios. These are predictions of future interaction dynamics, rather than direct camera recordings of policy execution. Click a preview to watch the full rollout.

Flip small box Reorientation under contact-rich manipulation.

Insert RAM module Precision alignment and insertion dynamics.

Lift box high Longer-horizon object lifting and transport.

Put chain in the box Deformable-object handling with container interaction.

Toggle power switch Interactive switch manipulation with small contact changes.

Real-Robot Execution Videos

A2World-policy is evaluated on a Flexiv dual-arm real-robot suite covering precision insertion, reorientation, switch interaction, lifting, and deformable-object handling. Click a preview to watch the full video.

Put chain in the box
Toggle power switch	Flip small box
Insert RAM module	Lift box high

BibTeX

If you find this project helpful, please consider citing our paper, in Proceedings of ECCV 2026:

@inproceedings{huang2026a2world,
    title={Learning Transferable Dynamics Priors from Action to World Modeling},
    author={Huang, Ze and Zhang, Jiahui and Liu, Hairuo and Zhang, Chenxi and Cheng, Ran and Zhang, Li},
    booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
    year={2026},
}