Skip to content

Keep SignWriting signs whole, and never drop text - #8

Merged
AmitMY merged 2 commits into
mainfrom
fix/signwriting-symbols
Oct 9, 2026
Merged

AmitMY merged 2 commits into
mainfrom
fix/signwriting-symbols

Conversation

@AmitMY

@AmitMY AmitMY commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

SignWriting text was split inside its signs, and part of it was dropped:

>>> segment_text("𝠀\U00041cc9\U00040043\U000461e1\U00049e19𝠃𝤟𝤨\U00041cc9𝣭𝣤...")
[[], ['\U00041cc9\U00040043\U000461e1\U00049e19'], ['𝠃𝤟𝤨'], ['\U00041cc9'], [], ...]  # the 𝠀 prefix and coordinates are gone

Cause: Unicode's SignWriting script, which the regex matches through \p{scx=SignWriting}, covers only the Sutton SignWriting block (U+1D800–1DAAF): the sign prefix, the box and the coordinates. SWU symbols live in plane 4 (U+40001–4F480), outside any script. Each sign therefore broke into alternating SignWriting and Default runs. segment_signwriting keeps only complete signs, so it dropped every SignWriting run that didn't contain one.

Fix:

  • The SignWriting branch also matches SWU symbols, so each sign is matched as one word.
  • segment_signwriting never drops characters: anything outside a complete sign becomes its own word, and a sign keeps its trailing space, like any other word.

I checked it on all 2,837 validation texts of sign/signbank-plus: "".join(text_to_words(t)) == t holds for every one. It came up while training WeLT on SignWriting → text: the model's word inputs were sign fragments, and part of the sign text was lost.

Speed (texts/s with text_to_words(max_bytes=126), best of 5):

Texts main this PR
English 277,753 271,827
Hebrew 255,597 249,523
Chinese 64,498 63,010
SignWriting (signbank-plus) 59,815 162,621
Sign + spoken templated text 46,019 88,978

SignWriting is 2.7x faster because signs are now matched whole. The other scripts are within run-to-run noise (about 2%; English doesn't reach the changed code). That holds because the script sets are now matched with one character class instead of an alternation, which first cost Hebrew about 14%.

🤖 Generated with Claude Code

https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f


Note

Medium Risk
Changes SignWriting word boundaries and script routing in shared segment_text logic, which can affect downstream tokenization and training data, though behavior is validated as lossless on a large corpus.

Overview
SignWriting segmentation no longer splits signs or drops characters. Script detection now treats plane-4 SWU symbols (U+40001–U+4F480) as SignWriting alongside the Sutton block, so a full sign stays in one script run instead of alternating SignWriting/Default.

segment_signwriting switches from extracting only complete signs via re_swu['sign'] to matching each complete sign (with optional trailing space, like other words) or emitting any leftover character as its own token. Space-separated signs now retain trailing spaces on each word except the last.

Tests cover incomplete sign fragments, integration with text_to_words / segment_text, and round-trip losslessness.

Reviewed by Cursor Bugbot for commit fb66286. Bugbot is set up for automated code reviews on this repo. Configure here.

autoresearch and others added 2 commits October 9, 2026 04:37
Unicode's SignWriting script (Script_Extensions) is the Sutton SignWriting block, U+1D800-U+1DAAF: the sign prefix,
box and coordinates. SWU symbols are in plane 4 (U+40001-U+4F480), outside any script. So text was split into
SignWriting runs and Default runs within each sign, and segment_signwriting dropped every run that was not a
complete sign: on signbank-plus, signs lost their prefix and most coordinates.

- The SignWriting branch also matches SWU symbols, so whole signs are matched
- segment_signwriting keeps characters outside complete signs (each its own word), and a sign's trailing space

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f
Faster than an alternation of single-property classes: the script check scans every non-ASCII text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f
@AmitMY
AmitMY merged commit d86b3f8 into main Oct 9, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant