Fix HyperscanTokenizer silently dropping citations near non-ASCII characters

[renatodvc]

Description

HyperscanTokenizer gets byte offsets back from hyperscan and translates them to string offsets before re-running the Python regex. When a pattern consumed only part of a multi-byte character, the reported offset landed inside that character, the translation failed, and the match was discarded with no error or log line.

Reproduce it

from pathlib import Path

from eyecite import get_citations
from eyecite.tokenizers import AhocorasickTokenizer, HyperscanTokenizer

text = Path("tests/assets/opinion.txt").read_text()

aho = get_citations(text, tokenizer=AhocorasickTokenizer())
hyperscan = get_citations(text, tokenizer=HyperscanTokenizer())
print(f"ahocorasick: {len(aho)}   hyperscan: {len(hyperscan)}")

found = {c.span() for c in hyperscan}
for cite in aho:
    if cite.span() not in found:
        start, end = cite.span()
        print(f"  missed by hyperscan: {text[start:end + 8]!r}")

• ahocorasick returns 170 citations, hyperscan only 163
• In this opinion file, all dropped examples are all followed by an en dash, e.g. 438 U. S., at 311–312, but the bug applies to any non-ASCII character.

Fix

Snap an offset that lands mid character forward to the next character boundary instead of throwing the match away. Snapping happens in the hyperscan match callback, so every offset downstream is boundary aligned by construction.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.