(draft) feat: add LeWM, an action-conditioned latent world model trained with SIGReg - #2032
Draft
gabrielfruet wants to merge 2 commits into
Draft
(draft) feat: add LeWM, an action-conditioned latent world model trained with SIGReg#2032gabrielfruet wants to merge 2 commits into
gabrielfruet wants to merge 2 commits into
Conversation
Predictor, action encoder, loss and a PyTorch example, scoped to what LeWM alone needs. Later world models add arguments that default to this behavior, so nothing here changes meaning when they land. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Type checking runs against the oldest supported torch, where scaled_dot_product_attention does not exist yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LeWM is a latent world model: it predicts the next frame's embedding from past embeddings and the action taken, never reconstructing pixels. SIGReg, already here for LeJEPA, prevents collapse, so there is no teacher, no EMA and no stop-gradient, and the encoder trains from pixels alongside the predictor. Paper: https://arxiv.org/abs/2603.19312
lightly/models/modules/world_model/: LatentDynamicsPredictor, a causal transformer with AdaLN-Zero action conditioning and arollout()that feeds predictions back over a sliding window, plus an ActionEncoder MLP.lightly/loss/: LeWMLoss (prediction MSE pluslambda_param* SIGReg, default 0.1) andlatent_distance, a new l1/l2 helper with optional layer norm for later methods.scaled_dot_product_attentiongoes throughgetattr, with a manual fallback, so type checking passes on torch 1.10.The API covers only what LeWM needs, the first of a planned sequence; later methods add keyword arguments defaulting to this behavior. Missing: the Lightning example variants and the README table row.
The example synthesizes its own trajectories, a square pushed by the action, so it needs no simulator. Loss falls from 1.05 to 0.33 over 10 epochs. That shows the loop trains, not that it reproduces the paper. 12 CI checks green.