Repository navigation
Re-run the benchmarks on main, with Muon for every task - #73
Merged
Merged
Conversation
- Task configs (string-repetition and the tasks extending it, machine translation, the baseline) use Muon - All tasks re-trained, exported, served and evaluated on main; Muon improves every task over Adam (e.g. ocr 80.5% -> 88.3% exact match, machine translation 0.626 -> 0.542 bits per byte, chrF 41.0 -> 46.2) - The causal LM baseline, now scoring the same bytes as WeLT: 0.599 bits per byte at 633 ms / step, vs. 0.542 for WeLT at 192 ms / step - Multi-GPU parity from the post-review run on 2 H100s; bench config still 129 ms / step (127 in the chart) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f
…or every task - words-segmentation>=0.0.6: SignWriting signs are whole words (they were split, and partly dropped) - signed-to-spoken trains 10 epochs; with bytes only and with an image encoder over rendered SignWriting (bytes 0.953 vs. bytes + image 1.296 bits per byte, chrF 5.1 vs. 10.4) - WeLT datasets do not pin memory: batches vary in shape, and PyTorch's pinned memory cache grew without bound (killed training on the GB10); run_task.sh does not pipe training to tee, which hung after a failure - The benchmarks README shows the first 5 validation examples of each task: input, expected, generated Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit cd87abd. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Re-runs every benchmark training on
mainafter the migration stack (#63–#71), with Muon for every task. The task configs now set Muon themselves, so the results reproduce from them.Numbers in parentheses are the previous runs, with Adam and before the review fixes.
SignWriting → text (signbank-plus, 10 epochs, Muon):
Memory: WeLT datasets no longer pin memory. Batches vary in shape, so PyTorch's pinned-memory cache kept a buffer for every size and grew until the GB10's earlyoom killed training. Also,
run_task.shno longer pipes training throughtee, which hung after a failure.Examples
The first 5 validation examples per task.
␎and␏stand for the shift-block controls\x0Eand\x0F.string-repetition
<text>␎Wouldn't it be more cruel for society to let people die... - ... when with some effort it could save them?␏<repeat>Wouldn't it be more cruel for society to let people die... - ... when with some effort it could save them?Wouldn't it be more cruel for society to let people die... - ... when with some effort it could save them?<text>␎YOu dOn't know the half Of it.␏<repeat>YOu dOn't know the half Of it.You don't know the half Of it.<text>␎Superman's exact opposite who lives in the backwards Bizarro World.␏<repeat>Superman's exact opposite who lives in the backwards Bizarro World.Superman's exact opposite who lives in the backwards Bizarro World.<text>␎- The apology, so- - We're keeping the robot.␏<repeat>- The apology, so- - We're keeping the robot.- The apology, so- - We're keeping the robot.<text>␎Of course, there were always mama's boyfriends, but as soon as I'd learn their names, mama would kick them out, and there'd be a new toothbrush in the bathroom.␏<repeat>Of course, there were always mama's boyfriends, but as soon as I'd learn their names, mama would kick them out, and there'd be a new toothbrush in the bathroom.Of course, there were always mama's boyfriends, but as soon as I'd learn their names, mama would kick them out, and there'd be a new toothbrush in the bathroom.ocr (the model sees only the rendered words)
<text>␎Wouldn't it be more cruel for society to let people die... - ... when with some effort it could save them?␏<repeat>Wouldn't it be more cruel for society to let people die... - ... when with some effort it could save them?Wouldn't it be more cruel for society to let people die... - ... when with some effort it could save them?<text>␎YOu dOn't know the half Of it.␏<repeat>YOu dOn't know the half Of it.YOu daNt! know the half Of it.<text>␎Superman's exact opposite who lives in the backwards Bizarro World.␏<repeat>Superman's exact opposite who lives in the backwards Bizarro World.Superman's exact opposite who lives in the backwards Bizarro World.<text>␎- The apology, so- - We're keeping the robot.␏<repeat>- The apology, so- - We're keeping the robot.- The apology, so- - We're keeping the robot.<text>␎Of course, there were always mama's boyfriends, but as soon as I'd learn their names, mama would kick them out, and there'd be a new toothbrush in the bathroom.␏<repeat>Of course, there were always mama's boyfriends, but as soon as I'd learn their names, mama would kick them out, and there'd be a new toothbrush in the bathroom.Of course, there were always mama's boyfriends, but as soon as I'd learn their names, mama would kick them out, and there'd be a new toothbrush in the bathroom.letter-count
<text>␎Morel␏<count>M1 O1 R1 E1 L1M1 O1 R1 E1 L1<text>␎neurosis␏<count>N1 E1 U1 R1 O1 S2 I1N1 E1 U1 R1 O1 S2 I1<text>␎mydaleine␏<count>M1 Y1 D1 A1 L1 E2 I1 N1M1 Y1 D1 A1 L1 E2 I1 N1<text>␎forgetter␏<count>F1 O1 R2 G1 E2 T2F1 O1 R2 G1 E2 T2<text>␎fiddley␏<count>F1 I1 D2 L1 E1 Y1F1 I1 D2 L1 E1 Y1machine-translation
<en>␎Wouldn't it be more cruel for society to let people die... - ... when with some effort it could save them?␏<he>האם לא יהיה זה אכזרי יותר מצד החברה, לתת לאנשים למות... כאשר עם מאמץ מה ניתן להצילם?לא יהיה אכזרי יותר לחברה לתת לאנשים למות... כשמאית שתוכל להציל אותם?<en>␎YOu dOn't know the half Of it.␏<he>אתה לא יודע חצי מזה.אתה לא יודע חצי מזה.<en>␎Superman's exact opposite who lives in the backwards Bizarro World.␏<he>ההפך הגמור של סופרמן מי שחי בעולם Bizarro אחורה.הפוסימין הוא המופע היחיד שחי בעולם הביזרק האחורי.<en>␎- The apology, so- - We're keeping the robot.␏<he>-ההתנצלות, אז... אנחנו שומרים את הרובוט.ההתנצלות, אז... אנחנו שומרים את הרובוט.<en>␎Of course, there were always mama's boyfriends, but as soon as I'd learn their names, mama would kick them out, and there'd be a new toothbrush in the bathroom.␏<he>כמובן שתמיד היו החברים של אמא שלי, אבל כשלמדתי את שמותיהם, אמא הייתה מעיפה אותם, והייתה מברשת שיניים חדשה באמבטיה.כמובן, היו תמיד החברים של אמא, אבל ברגע שאני אלמד את השמות שלהם, אמא תבעט בהם, ויהיו שיניים חדשות בשירותים.signed-to-spoken, bytes
<ncs>␎𝠀𝠃𝤠𝥇𝣴𝣵𝣶𝤜𝤅𝤴␏<es>difficultllegar de la cabeza<ncs>␎𝠀𝠃𝤠𝥇𝣴𝣵𝣶𝤜𝤅𝤴␏<es>dificilllegar de la cabeza<bzs>␎𝠀𝠃𝥊𝤳𝣴𝣵𝤋𝤅𝤙𝣼𝤱𝤚␏<pt>FlaviaPara frente para baixo - pare<ssp>␎𝠀𝠃𝤨𝥅𝤙𝤗𝤘𝤪𝣽𝤞𝣴𝣴␏<es>abrilcl<bzs>␎𝠀𝠃𝤎𝤏𝣿𝣽␏<pt>MPara baixo do circularsigned-to-spoken, bytes + image
<ncs>␎𝠀𝠃𝤠𝥇𝣴𝣵𝣶𝤜𝤅𝤴␏<es>difficultcara<ncs>␎𝠀𝠃𝤠𝥇𝣴𝣵𝣶𝤜𝤅𝤴␏<es>dificilcara<bzs>␎𝠀𝠃𝥊𝤳𝣴𝣵𝤋𝤅𝤙𝣼𝤱𝤚␏<pt>FlaviaCarlos Carla de Almeida<ssp>␎𝠀𝠃𝤨𝥅𝤙𝤗𝤘𝤪𝣽𝤞𝣴𝣴␏<es>abrilcontradicción<bzs>␎𝠀𝠃𝤎𝤏𝣿𝣽␏<pt>MCM14_MD_Dorso_VF🤖 Generated with Claude Code
https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f
Note
Low Risk
Mostly benchmark artifacts and docs; small defaults (pin_memory, dependency pin) and config tweaks affect reproducibility and dataloader memory, not core model logic.
Overview
Re-runs and documents benchmarks on main after the migration, with Muon set explicitly in task YAMLs (string-repetition, machine-translation, baseline, signed-to-spoken) and removed from letter-count only in docs.
tasks.csvandbenchmarks/README.mdare refreshed with new timings/metrics, SignWriting→text rows (bytes-only and bytes+image), Muon-vs-Adam notes, updated multi-GPU parity numbers, causal-LM baseline comparison, and per-task generation example tables.Signed-to-spoken config now enables a tiny image encoder, 5940 steps (10 epochs), 4 dataloader workers, and Muon;
words-segmentationis bumped to ≥0.0.6 inpyproject.toml.Training hygiene:
WeLTDatasetProviderdefaultspin_memory=Falseto avoid pinned-buffer growth on variable batch shapes;run_task.shredirects training logs without a pipe so failures do not hang the script.Reviewed by Cursor Bugbot for commit cd87abd. Bugbot is set up for automated code reviews on this repo. Configure here.