[ly989264]

Summary

A serving stack keeps a pool of copy.deepcopy copies of the tokenizer and
hands each request to the next free copy in turn (vLLM 0.23's
maybe_make_thread_pool keeps renderer_num_workers + 1 copies and deep-copies
another when the pool is empty). With a segment cache per copy, a segment is
missed and stored once per copy, each copy sees only a fraction of the traffic,
and the budget is paid per copy (with a 128 MiB budget, eight copies may hold
up to 1 GiB).

The cache now lives in an Arc. Tokenizer::share_segment_cache_from(&other)
makes a tokenizer use another's cache (or none, if other has none).
_TokenizerShim.__deepcopy__ calls it, so the original and its copies warm
one cache within one budget. add_tokens gives the tokenizer it is called on
a fresh cache of the same budget instead of clearing the shared one: copies
that keep the old vocabulary still rely on its entries.

Measured on an M2 Pro with GLM-5.2 coding-agent traffic: main, where each
copy gets a cache of its own from FASTOKENS_SEGMENT_CACHE=128, against this
branch with the same setting; medians of 3 runs. Through direct tokenizer
calls, on an in-process replica of that pool with 8 copies (handed out in
turn, one client, 480 requests, 35.7M tokens), the text tokenized on misses
drops from 21.0 MiB to 2.8 MiB (hit rate 0.945 → 0.992), the cache holds
6.0 MiB instead of 45.6 MiB and the replay takes 1.10 s instead of 1.23 s.
With 2 copies the hit rate goes 0.985 → 0.992 and the replay time barely
moves (0.80 → 0.79 s). A copy the pool adds under load shares the cache from
its first request.

Through the vLLM 0.23 frontend (renderer_num_workers=3, so up to 4 copies;
16 clients for 15 s; frontend only, no model engine), throughput and latency
are within noise (133.7 → 133.3 requests/s, p50 116.7 → 115.9 ms). Each
copy's cache holds 26–27 MiB and misses about 9.2 MiB of text; the shared
cache holds 28 MiB and misses 9.6 MiB in all (hit rate 0.984 → 0.996), and
the API server's peak RSS goes from 2195 to 2096 MiB. The gain is less
tokenizing and less memory, not frontend latency.

The counters: four copies of one tokenizer, one growing chat

import copy, os
os.environ["FASTOKENS_SEGMENT_CACHE"] = "64"  # MiB; set before loading
import fastokens
from transformers import AutoTokenizer

fastokens.patch_transformers()
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
# the pool vLLM keeps with renderer_num_workers=3
pool = [tok] + [copy.deepcopy(tok) for _ in range(3)]

chat = ""
for i in range(40):  # a chat that grows by one turn per request
    chat += f"<|im_start|>user\nstep {i}: list the files in src/<|im_end|>\n"
    chat += "<|im_start|>assistant\n" + "lib.rs segment_cache.rs " * 50
    chat += "<|im_end|>\n"
    pool[i % 4](chat)  # the next free copy in turn

for i, t in enumerate(pool):
    s = t.backend_tokenizer.segment_cache_stats()
    print(i, {k: s[k] for k in ("hits", "misses", "entries", "bytes")})

On main each copy reports its own cache (hits 721, 760, 799, 838; misses
39, 40, 41, 42; 162 misses and four caches in all). On this branch all four
report the one cache: hits 3238, misses 42, entries 42, bytes 9729.

Notes

Without copies. One tokenizer, one cache, as before; enable_segment_cache
on a copy gives it a cache of its own. With the cache off, the encode paths
are unchanged.

add_tokens. It moves the tokenizer to a fresh cache instead of clearing
the shared one, so its counters now restart (clear kept them); a copy made
afterwards shares the fresh cache. SegmentCache::clear and its unit test are
removed: it lost its only caller and would be dead code under -D warnings;
the add_tokens tests still check that the cache is empty afterwards.

What stays per copy. Truncation and padding are applied after the cached
core ids, so copies with different truncation share a cache and still truncate
their own output (tested). The whole-input cache (enable_input_cache, off by
default) stays per copy, and a tokenizer rebuilt by pickling or constructed
from another shim does not share the cache, as before.

API. Tokenizer::share_segment_cache_from is public, for a Rust stack with
its own pool; a hit is checked against the segment text, not the vocabulary,
so its doc says other must tokenize identically. Python gets
_share_segment_cache_from, for __deepcopy__: it takes other's read lock,
then self's write lock, with the GIL held; sharing with itself is a no-op.

Testing

• cargo fmt --check, cargo clippy -- -D warnings and cargo test (debug,
  as CI): 400 passed on macOS and in a Linux container (Debian 12, glibc 2.36,
  rustc 1.99); cargo test --release on macOS: 400 passed. pytest python/tests with transformers and HF tokenizers: 138 passed on both;
  the patch-transformers job's files also pass with transformers 4.57.1 and
  5.3.0, as does test_segment_cache.py (12 tests), which is outside that
  job's file list.
• Rust: segment_cache_is_shared_with_copies_until_their_vocabularies_differ
  (hits both ways with one set of counters; sharing replaces a cache the copy
  had; add_tokens moves the copy to its own cache and keeps the original's
  entries; sharing from an uncached tokenizer turns the copy's cache off),
  segment_cache_is_shared_safely_between_copies_on_threads (four copies on
  four threads) and segment_cache_stays_off_with_bpe_dropout (sharing into
  a dropout model does not turn the cache on).
• Python: test_a_copy_shares_the_cache_and_applies_its_own_truncation,
  test_a_copy_with_a_cache_of_its_own_joins_the_original (from the env var),
  test_add_tokens_moves_a_copy_to_its_own_cache,
  test_a_copy_of_an_uncached_tokenizer_has_no_cache, and
  test_copies_on_threads_share_one_cache (original plus seven deep copies).
• The pool replica above, built from this branch: the ids of all 480 requests
  equal HF tokenizers' and an uncached tokenizer's in five configurations:
  2 copies with 1 client; 4 copies with 16 clients and a burst of 8 direct
  borrows; 8 copies with 4 clients, FIFO and LIFO; and 8 copies with 4
  clients and an 8 MiB budget that evicts.

[AlonKejzman]

Thank you for the contribution!