[AlonKejzman]

Fixes #88.

Problem

tencent/Hy3's byte-level vocab has no token for \r (its ByteLevel char č), and the BPE has no unk_token and no byte fallback. Every raw-byte merge routine in bpe.rs assumes each byte has a token, so any text with a carriage return failed with byte 0x0d has no token in vocabulary. HF's merge_word drops the char instead, or emits its byte-fallback tokens or the unk token when the model has them, and the bytes on either side may then merge with each other.

Change

• Bpe records at load whether a byte that can occur in UTF-8 has no initial token. Hy3 lacks 14 bytes, but only 0x0D can occur; the others (0xC0, 0xC1, 0xF5–0xFF) are ignored.
• merge_all_raw_into, the one entry every byte-level path goes through, merges a pretoken holding such a byte from its ByteLevel text via merge_all_encoded_into. That path already follows merge_word (drop, byte fallback, unk, fuse_unk).
• Models with all 256 byte tokens only check the flag. For Hy3, only pretokens containing \r take the char-based path.

Testing

• New byte_level_bpe_without_a_byte_token_matches_huggingface, against the tokenizers crate. It uses a byte-level BPE without a \r token, in four variants (dropped, unk, fused unk, byte fallback), each with and without regex splitting, and its merges join the bytes around the gap. It fails on main and passes here.
• cargo test: 401 passed, 9 ignored. cargo clippy -- -D warnings and cargo fmt --check are clean.
• Against HF tokenizers 0.23.3 through the Python binding:
  • The issue's examples give HF's ids: [64, 65], [2020, 926, 185, 2020, 1230] and [].
  • No mismatches or errors on ~25k random texts with \r (plus one > 256 KiB) each for tencent/Hy3 (revision a960ebc), Hy3-preview and Hy4-preview, Hy3 with a single ByteLevel pre-tokenizer (forcing merges across the gap), and six synthetic variants. encode_batch matches too.
• Hy3 encodes \r\n text as fast as \n text: 2.1 ms median each on 3.3M chars, where HF takes ~1.1 s.

🤖 Generated with Claude Code