Skip to content

(draft) feat: add LeWM, an action-conditioned latent world model trained with SIGReg - #2032

Draft
gabrielfruet wants to merge 2 commits into
masterfrom
feat/lewm
Draft

(draft) feat: add LeWM, an action-conditioned latent world model trained with SIGReg#2032
gabrielfruet wants to merge 2 commits into
masterfrom
feat/lewm

Conversation

@gabrielfruet

@gabrielfruet gabrielfruet commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

LeWM is a latent world model: it predicts the next frame's embedding from past embeddings and the action taken, never reconstructing pixels. SIGReg, already here for LeJEPA, prevents collapse, so there is no teacher, no EMA and no stop-gradient, and the encoder trains from pixels alongside the predictor. Paper: https://arxiv.org/abs/2603.19312

  • lightly/models/modules/world_model/: LatentDynamicsPredictor, a causal transformer with AdaLN-Zero action conditioning and a rollout() that feeds predictions back over a sliding window, plus an ActionEncoder MLP.
  • lightly/loss/: LeWMLoss (prediction MSE plus lambda_param * SIGReg, default 0.1) and latent_distance, a new l1/l2 helper with optional layer norm for later methods.
  • A PyTorch example, docs, and ~500 lines of tests.
  • scaled_dot_product_attention goes through getattr, with a manual fallback, so type checking passes on torch 1.10.

The API covers only what LeWM needs, the first of a planned sequence; later methods add keyword arguments defaulting to this behavior. Missing: the Lightning example variants and the README table row.

The example synthesizes its own trajectories, a square pushed by the action, so it needs no simulator. Loss falls from 1.05 to 0.33 over 10 epochs. That shows the loop trains, not that it reproduces the paper. 12 CI checks green.

gabrielfruet and others added 2 commits August 15, 2026 09:56
Predictor, action encoder, loss and a PyTorch example, scoped to what LeWM
alone needs. Later world models add arguments that default to this behavior,
so nothing here changes meaning when they land.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Type checking runs against the oldest supported torch, where
scaled_dot_product_attention does not exist yet.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@gabrielfruet gabrielfruet changed the title (draft) feat: add LeWM latent world model (draft) feat: add LeWM, an action-conditioned latent world model trained with SIGReg Aug 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant