[waynehacking8]

Overview

With tencent/Hy3 or tencent/Hy3-preview, fastokens 0.3.5 fails on any text with a carriage return ("byte 0x0d has no token in vocabulary"), where Hugging Face tokenizers drops the byte. fastokens 0.3.6 (crusoecloud/fastokens#89, fixing crusoecloud/fastokens#88) drops it too and gives the same ids (both models fail in crusoecloud/fastokens#88 and match in the checks on crusoecloud/fastokens#89).

vLLM uses fastokens in two places: the Rust frontend's tokenizer, and the optional Python backend behind VLLM_USE_FASTOKENS=1. This PR moves both to 0.3.6.

This change was prepared with AI assistance.

Claims

• The Rust frontend encodes Hy3 text with a carriage return the same way as Hugging Face, instead of returning an error.
• With VLLM_USE_FASTOKENS=1, vLLM now requires fastokens 0.3.6 and says so in the import error, the env var help, the docs and the Nemotron 3.5 Lightning recipe that installs it.
• A new Rust test fails on 0.3.5 with the error above and passes on 0.3.6.
• Breaking for opt-in Python users: VLLM_USE_FASTOKENS=1 with fastokens 0.3.5 now fails at tokenizer load with an ImportError that names 0.3.6, for every model. 0.3.6 is on PyPI.

Validation

Served: vllm-rs serve tencent/Hy3 (Hub revision a960ebc3) with vllm-mock-engine, then /tokenize requests, built once with fastokens 0.3.5 locked and once with 0.3.6:

request	0.3.5	0.3.6
prompt "a\rb"	HTTP 500, "byte 0x0d has no token in vocabulary"	200, same ids as Hugging Face
prompt "line one.\r\nline two"	HTTP 500, same error	200, same ids as Hugging Face
chat message with \r\n	HTTP 500, same error	200
prompt without \r	200, same ids as Hugging Face	200, same ids as Hugging Face

The Python-parity test from #65757 (head dc36074) compares the Rust frontend's ids with the Python frontend's on a fixed corpus of 31 texts, 6 of them with \r.

With Hy3 it passes on 0.3.5 because it expects exactly this error for those 6 texts. On 0.3.6 its only failures are "matches Python now" for those 6, in both of its variants (the tokenizer as published, and with truncation and padding set), so all 31 texts give the Python frontend's ids.

New test, fastokens_matches_hf_when_a_byte_has_no_token in rust/src/tokenizer/src/hf.rs: it removes the \r token from the byte-level test vocab, adds a merge across the dropped byte (ĠĠ with Ċ), and compares encode and encode_ordinary with Hugging Face on six texts, with and without the fixture's fused pre-tokenizer setting. crusoecloud/fastokens#89 has its own test, but vLLM's encode_fastokens_ordinary calls the fused tokenize_batch_fused directly, which that test does not reach. It runs in the existing Rust frontend cargo CI step.

# in rust/; for 0.3.5, first set the Cargo.toml requirement to "0.3.5" and run:
cargo update -p fastokens --precise 0.3.5
cargo test -p vllm-tokenizer --lib hf::tests
fastokens 0.3.5: 14 passed, 1 failed: "encoding failed: model error: byte 0x0d has no token in vocabulary"
fastokens 0.3.6 (this PR): 15 passed

cargo fmt --check and cargo clippy -p vllm-tokenizer --all-targets --all-features -- -D warnings pass.

Python path, with transformers 5.22.2 (in vLLM's supported range): AutoTokenizer.from_pretrained on the Hy3 snapshot after fastokens.patch_transformers(), which is what VLLM_USE_FASTOKENS=1 calls, fails on 0.3.5 with the same error for three texts with \r and gives the same ids as Hugging Face on 0.3.6. The Python version gate has no test, and I did not serve through the Python frontend. The mock engine emits random tokens, so these runs say nothing about generation.

Details

• No open PR bumps fastokens past 0.3.5 (searched open PRs for fastokens on 2026-11-08; [Rust Frontend] Use fastokens for tiktoken models #51027 still pins 0.3.1).
• No model evaluation: only text with a carriage return changes, and with Hy3 its ids now equal the Python frontend's.
• Hy3's vocab has no token for \r (which byte-level BPE writes as č), and its BPE has no unk token and no byte fallback. fastokens' merge routines for raw bytes assumed every byte has a token in the vocab. Hugging Face drops a byte it cannot map, and the bytes on either side can still merge into one token across the gap; 0.3.5 returned an error instead.
• The only other change between 0.3.5 and 0.3.6 is perf(tokenizer): share one segment cache between a tokenizer and its deep copies crusoecloud/fastokens#87, which shares the segment cache between a tokenizer and its deep copies. The cache is off unless FASTOKENS_SEGMENT_CACHE is set, and vLLM does not set it.
• The raised Python minimum also stops models whose vocab covers every byte from starting with 0.3.5. The gate was already a hard minimum at 0.3.5; this only raises its version.
• This PR's test covers only the drop case that Hy3 hits; fix(bpe): merge like HF when a byte has no ByteLevel token crusoecloud/fastokens#89 tests the unk-token and byte-fallback cases upstream.
• [Rust Frontend] Check tokenizer ids against the Python frontend #65757 (open) lists this Hy3 failure as an expected divergence of the Rust frontend. This PR does not depend on it; whichever of the two lands second removes that entry.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

Documentation preview: https://vllm--66327.org.readthedocs.build/en/66327/

[mergify]

Recipe preview: https://vllm-recipes-git-preview-vllm-pr-66327-inferact-inc.vercel.app