Repository navigation
train stories_base at the vocab the pipeline actually builds - #7
Merged
Merged
Conversation
train.py remaps gpt-2 ids into VOCAB_SIZE slots by frequency, but the config asked for 50257 of them. TinyStories only contains ~12k distinct gpt-2 tokens, so the map filled 12354 slots and left three quarters of the embedding rows and head columns without a gradient, at 30.49M parameters instead of 6.85M. The top 4096 tokens cover 98.8% of occurrences. This also makes the config agree with the export command in the readme, which has always passed --vocab_size 4096. The vocab cache is not keyed by VOCAB_SIZE, so a map built for a larger vocab would hand the model ids past its embedding table; it is now rebuilt when it does not fit.
Every published number so far was measured on randomly initialised weights. Training the quick start config for 15k steps gives 101.3 held-out perplexity against a uniform baseline of 4096, and the sample is generated by the C++ engine from a 5.6 MB packed binary so the whole pipeline is exercised. The sample is not cherry-picked and the text does not hold together; saying so is the point of publishing it.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Auto-approved: pull request authored by repository owner @guirguispierre.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Starting a training run surfaced this immediately. train.py loads configs/stories_base.yaml, which asked for a 50257-token vocabulary, so it began training a 30.49M-parameter model rather than the 6.85M one the readme's export command builds. The pipeline does not use gpt-2 ids directly: _init_vocab remaps them into VOCAB_SIZE slots by frequency. TinyStories contains only about 12k distinct gpt-2 tokens, so the map filled 12354 slots and left roughly three quarters of the embedding rows and head columns without a gradient. The top 4096 tokens cover 98.8% of occurrences, and 4096 is what the readme has always passed, so that is what the config now says.
Changing vocab_size exposed a second problem worth fixing in the same breath: the vocab cache is written to weights/ and reloaded without being keyed by VOCAB_SIZE, so after this change a stale map would have handed the model token ids well past the end of its embedding table. It is now rebuilt when it does not fit.
The readme change is the reason I went looking. Every number published in this repo, including the throughput and size figures I added earlier, was measured on randomly initialised weights, so nothing demonstrated that a model trained by this stack says anything at all. 15k steps takes about 25 minutes on an M-series laptop and gives 101.3 held-out perplexity against a uniform baseline of 4096. The sample in the readme comes out of the C++ engine reading a 5.6 MB packed binary, so it exercises training, export, packing and the bare-metal runtime end to end.
Be clear about what that sample is: it has the register and names of TinyStories and local grammar, and it does not hold a sentence together. I have written that in the readme rather than picking a better seed, because a baseline is only useful if it is honest about where it starts. Greedy decoding on the trained model agreed with pytorch for 11 to 20 tokens out of 20 depending on the prompt before forking, which is the tie-break behaviour the parity section already documents and is longer than random weights manage.
The checkpoint itself is not committed, since weights/ is gitignored and it is 82 MB. Publishing it as a release asset is the obvious follow-up.