Skip to content

Re-run the benchmarks on main, with Muon for every task - #73

Merged
AmitMY merged 3 commits into
mainfrom
benchmarks/muon-rerun
Oct 11, 2026
Merged

AmitMY merged 3 commits into
mainfrom
benchmarks/muon-rerun

Conversation

@AmitMY

@AmitMY AmitMY commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Re-runs every benchmark training on main after the migration stack (#63–#71), with Muon for every task. The task configs now set Muon themselves, so the results reproduce from them.

Task Steps ms / step Val. bits / byte Val. word acc. Gen. exact match Gen. chrF
string-repetition 1500 66 (50) 0.014 (0.017) 99.3% (99.2%) 93.8% (93.4%) 98.7 (98.5)
ocr 3000 74 (62) 0.014 (0.038) 99.1% (97.7%) 88.3% (80.5%) 97.1 (95.4)
letter-count 3000 64 (67, Muon) 0.0001 100% 100% (99.2%) 100 (99.7)
machine-translation 10000 192 (172) 0.542 (0.626) 67.5% (64.8%) 8.6% (5.9%) 46.2 (41.0)
signed-to-spoken (SignWriting → text), bytes, new 5940 (10 epochs) 142 0.953 67.8% 0.0% 5.1
signed-to-spoken, bytes + image (rendered SignWriting), new 5940 (10 epochs) 404 1.296 63.0% 1.6% 10.4

Numbers in parentheses are the previous runs, with Adam and before the review fixes.

  • Muon vs. Adam: Muon improves every task, at 10–30% more time per step.
  • Baseline: the causal LM baseline now scores exactly the same bytes as WeLT (Causal LM baseline with Megatron-Bridge #66). On machine translation it reaches 0.599 bits per byte at 633 ms per step, vs. 0.542 for WeLT at 192 ms per step.
  • Speed: the bench config still takes 129 ms per step (127 ms in the chart), vs. 1227 ms for the HF Trainer.
  • Multi-GPU parity: from the post-review run on 2 H100s, 1 GPU / DP=2 / TP=2 reach validation losses of 0.935 / 0.929 / 0.938.

SignWriting → text (signbank-plus, 10 epochs, Muon):

  • Segmentation fix: signs are now whole words, since words-segmentation 0.0.6. Before, they were split into fragments and partly dropped.
  • Results: both models generate fluent words of the target language, but rarely the right one. vLLM matches the trained Megatron models on these prompts, so this is the model, not the generation.
  • Image encoder: it did not help. Over pixel-renderer's SignWriting images, trained from scratch, it scores 1.296 bits per byte vs. 0.953 with bytes only.
  • Fragmented signs: an earlier bytes-only run on the fragmented signs reached chrF 19.5. That hints symbol-level inputs may suit a small model better.

Memory: WeLT datasets no longer pin memory. Batches vary in shape, so PyTorch's pinned-memory cache kept a buffer for every size and grew until the GB10's earlyoom killed training. Also, run_task.sh no longer pipes training through tee, which hung after a failure.

Examples

The first 5 validation examples per task. ␎ and ␏ stand for the shift-block controls \x0E and \x0F.

string-repetition

Input Expected Generated
<text>␎Wouldn't it be more cruel for society to let people die... - ... when with some effort it could save them?␏<repeat> Wouldn't it be more cruel for society to let people die... - ... when with some effort it could save them? Wouldn't it be more cruel for society to let people die... - ... when with some effort it could save them?
<text>␎YOu dOn't know the half Of it.␏<repeat> YOu dOn't know the half Of it. You don't know the half Of it.
<text>␎Superman's exact opposite who lives in the backwards Bizarro World.␏<repeat> Superman's exact opposite who lives in the backwards Bizarro World. Superman's exact opposite who lives in the backwards Bizarro World.
<text>␎- The apology, so- - We're keeping the robot.␏<repeat> - The apology, so- - We're keeping the robot. - The apology, so- - We're keeping the robot.
<text>␎Of course, there were always mama's boyfriends, but as soon as I'd learn their names, mama would kick them out, and there'd be a new toothbrush in the bathroom.␏<repeat> Of course, there were always mama's boyfriends, but as soon as I'd learn their names, mama would kick them out, and there'd be a new toothbrush in the bathroom. Of course, there were always mama's boyfriends, but as soon as I'd learn their names, mama would kick them out, and there'd be a new toothbrush in the bathroom.

ocr (the model sees only the rendered words)

Input Expected Generated
<text>␎Wouldn't it be more cruel for society to let people die... - ... when with some effort it could save them?␏<repeat> Wouldn't it be more cruel for society to let people die... - ... when with some effort it could save them? Wouldn't it be more cruel for society to let people die... - ... when with some effort it could save them?
<text>␎YOu dOn't know the half Of it.␏<repeat> YOu dOn't know the half Of it. YOu daNt! know the half Of it.
<text>␎Superman's exact opposite who lives in the backwards Bizarro World.␏<repeat> Superman's exact opposite who lives in the backwards Bizarro World. Superman's exact opposite who lives in the backwards Bizarro World.
<text>␎- The apology, so- - We're keeping the robot.␏<repeat> - The apology, so- - We're keeping the robot. - The apology, so- - We're keeping the robot.
<text>␎Of course, there were always mama's boyfriends, but as soon as I'd learn their names, mama would kick them out, and there'd be a new toothbrush in the bathroom.␏<repeat> Of course, there were always mama's boyfriends, but as soon as I'd learn their names, mama would kick them out, and there'd be a new toothbrush in the bathroom. Of course, there were always mama's boyfriends, but as soon as I'd learn their names, mama would kick them out, and there'd be a new toothbrush in the bathroom.

letter-count

Input Expected Generated
<text>␎Morel␏<count> M1 O1 R1 E1 L1 M1 O1 R1 E1 L1
<text>␎neurosis␏<count> N1 E1 U1 R1 O1 S2 I1 N1 E1 U1 R1 O1 S2 I1
<text>␎mydaleine␏<count> M1 Y1 D1 A1 L1 E2 I1 N1 M1 Y1 D1 A1 L1 E2 I1 N1
<text>␎forgetter␏<count> F1 O1 R2 G1 E2 T2 F1 O1 R2 G1 E2 T2
<text>␎fiddley␏<count> F1 I1 D2 L1 E1 Y1 F1 I1 D2 L1 E1 Y1

machine-translation

Input Expected Generated
<en>␎Wouldn't it be more cruel for society to let people die... - ... when with some effort it could save them?␏<he> האם לא יהיה זה אכזרי יותר מצד החברה, לתת לאנשים למות... כאשר עם מאמץ מה ניתן להצילם? לא יהיה אכזרי יותר לחברה לתת לאנשים למות... כשמאית שתוכל להציל אותם?
<en>␎YOu dOn't know the half Of it.␏<he> אתה לא יודע חצי מזה. אתה לא יודע חצי מזה.
<en>␎Superman's exact opposite who lives in the backwards Bizarro World.␏<he> ההפך הגמור של סופרמן מי שחי בעולם Bizarro אחורה. הפוסימין הוא המופע היחיד שחי בעולם הביזרק האחורי.
<en>␎- The apology, so- - We're keeping the robot.␏<he> -ההתנצלות, אז... אנחנו שומרים את הרובוט. ההתנצלות, אז... אנחנו שומרים את הרובוט.
<en>␎Of course, there were always mama's boyfriends, but as soon as I'd learn their names, mama would kick them out, and there'd be a new toothbrush in the bathroom.␏<he> כמובן שתמיד היו החברים של אמא שלי, אבל כשלמדתי את שמותיהם, אמא הייתה מעיפה אותם, והייתה מברשת שיניים חדשה באמבטיה. כמובן, היו תמיד החברים של אמא, אבל ברגע שאני אלמד את השמות שלהם, אמא תבעט בהם, ויהיו שיניים חדשות בשירותים.

signed-to-spoken, bytes

Input Expected Generated
<ncs>␎𝠀񌀅񆊱񂌳𝠃𝤠𝥇񌀅𝣴𝣵񆊱𝣶𝤜񂌳𝤅𝤴␏<es> difficult llegar de la cabeza
<ncs>␎𝠀񌀅񆊱񂌳𝠃𝤠𝥇񌀅𝣴𝣵񆊱𝣶𝤜񂌳𝤅𝤴␏<es> dificil llegar de la cabeza
<bzs>␎𝠀񍝁񆇡񄼱񉸒𝠃𝥊𝤳񍝁𝣴𝣵񆇡𝤋𝤅񄼱𝤙𝣼񉸒𝤱𝤚␏<pt> Flavia Para frente para baixo - pare
<ssp>␎𝠀񅯱񅯵񈪇񋾡𝠃𝤨𝥅񅯱𝤙𝤗񅯵𝤘𝤪񈪇𝣽𝤞񋾡𝣴𝣴␏<es> abril cl
<bzs>␎𝠀񆀡𝠃𝤎𝤏񆀡𝣿𝣽␏<pt> M Para baixo do circular

signed-to-spoken, bytes + image

Input Expected Generated
<ncs>␎𝠀񌀅񆊱񂌳𝠃𝤠𝥇񌀅𝣴𝣵񆊱𝣶𝤜񂌳𝤅𝤴␏<es> difficult cara
<ncs>␎𝠀񌀅񆊱񂌳𝠃𝤠𝥇񌀅𝣴𝣵񆊱𝣶𝤜񂌳𝤅𝤴␏<es> dificil cara
<bzs>␎𝠀񍝁񆇡񄼱񉸒𝠃𝥊𝤳񍝁𝣴𝣵񆇡𝤋𝤅񄼱𝤙𝣼񉸒𝤱𝤚␏<pt> Flavia Carlos Carla de Almeida
<ssp>␎𝠀񅯱񅯵񈪇񋾡𝠃𝤨𝥅񅯱𝤙𝤗񅯵𝤘𝤪񈪇𝣽𝤞񋾡𝣴𝣴␏<es> abril contradicción
<bzs>␎𝠀񆀡𝠃𝤎𝤏񆀡𝣿𝣽␏<pt> M CM14_MD_Dorso_VF

🤖 Generated with Claude Code

https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f


Note

Low Risk
Mostly benchmark artifacts and docs; small defaults (pin_memory, dependency pin) and config tweaks affect reproducibility and dataloader memory, not core model logic.

Overview
Re-runs and documents benchmarks on main after the migration, with Muon set explicitly in task YAMLs (string-repetition, machine-translation, baseline, signed-to-spoken) and removed from letter-count only in docs. tasks.csv and benchmarks/README.md are refreshed with new timings/metrics, SignWriting→text rows (bytes-only and bytes+image), Muon-vs-Adam notes, updated multi-GPU parity numbers, causal-LM baseline comparison, and per-task generation example tables.

Signed-to-spoken config now enables a tiny image encoder, 5940 steps (10 epochs), 4 dataloader workers, and Muon; words-segmentation is bumped to ≥0.0.6 in pyproject.toml.

Training hygiene: WeLTDatasetProvider defaults pin_memory=False to avoid pinned-buffer growth on variable batch shapes; run_task.sh redirects training logs without a pipe so failures do not hang the script.

Reviewed by Cursor Bugbot for commit cd87abd. Bugbot is set up for automated code reviews on this repo. Configure here.

autoresearch and others added 3 commits October 8, 2026 19:31
- Task configs (string-repetition and the tasks extending it, machine translation, the baseline) use Muon
- All tasks re-trained, exported, served and evaluated on main; Muon improves every task over Adam
  (e.g. ocr 80.5% -> 88.3% exact match, machine translation 0.626 -> 0.542 bits per byte, chrF 41.0 -> 46.2)
- The causal LM baseline, now scoring the same bytes as WeLT: 0.599 bits per byte at 633 ms / step,
  vs. 0.542 for WeLT at 192 ms / step
- Multi-GPU parity from the post-review run on 2 H100s; bench config still 129 ms / step (127 in the chart)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f
…or every task

- words-segmentation>=0.0.6: SignWriting signs are whole words (they were split, and partly dropped)
- signed-to-spoken trains 10 epochs; with bytes only and with an image encoder over rendered SignWriting
  (bytes 0.953 vs. bytes + image 1.296 bits per byte, chrF 5.1 vs. 10.4)
- WeLT datasets do not pin memory: batches vary in shape, and PyTorch's pinned memory cache grew without bound
  (killed training on the GB10); run_task.sh does not pipe training to tee, which hung after a failure
- The benchmarks README shows the first 5 validation examples of each task: input, expected, generated

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit cd87abd. Configure here.

@AmitMY
AmitMY merged commit f7a54fb into main Oct 11, 2026
4 checks passed
@AmitMY
AmitMY deleted the benchmarks/muon-rerun branch October 13, 2026 06:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant