Repository navigation
Keep SignWriting signs whole, and never drop text - #8
Merged
Merged
Conversation
Unicode's SignWriting script (Script_Extensions) is the Sutton SignWriting block, U+1D800-U+1DAAF: the sign prefix, box and coordinates. SWU symbols are in plane 4 (U+40001-U+4F480), outside any script. So text was split into SignWriting runs and Default runs within each sign, and segment_signwriting dropped every run that was not a complete sign: on signbank-plus, signs lost their prefix and most coordinates. - The SignWriting branch also matches SWU symbols, so whole signs are matched - segment_signwriting keeps characters outside complete signs (each its own word), and a sign's trailing space Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f
Faster than an alternation of single-property classes: the script check scans every non-ASCII text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SignWriting text was split inside its signs, and part of it was dropped:
Cause: Unicode's
SignWritingscript, which the regex matches through\p{scx=SignWriting}, covers only the Sutton SignWriting block (U+1D800–1DAAF): the sign prefix, the box and the coordinates. SWU symbols live in plane 4 (U+40001–4F480), outside any script. Each sign therefore broke into alternating SignWriting and Default runs.segment_signwritingkeeps only complete signs, so it dropped every SignWriting run that didn't contain one.Fix:
segment_signwritingnever drops characters: anything outside a complete sign becomes its own word, and a sign keeps its trailing space, like any other word.I checked it on all 2,837 validation texts of
sign/signbank-plus:"".join(text_to_words(t)) == tholds for every one. It came up while training WeLT on SignWriting → text: the model's word inputs were sign fragments, and part of the sign text was lost.Speed (texts/s with
text_to_words(max_bytes=126), best of 5):SignWriting is 2.7x faster because signs are now matched whole. The other scripts are within run-to-run noise (about 2%; English doesn't reach the changed code). That holds because the script sets are now matched with one character class instead of an alternation, which first cost Hebrew about 14%.
🤖 Generated with Claude Code
https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f
Note
Medium Risk
Changes SignWriting word boundaries and script routing in shared
segment_textlogic, which can affect downstream tokenization and training data, though behavior is validated as lossless on a large corpus.Overview
SignWriting segmentation no longer splits signs or drops characters. Script detection now treats plane-4 SWU symbols (U+40001–U+4F480) as SignWriting alongside the Sutton block, so a full sign stays in one script run instead of alternating SignWriting/Default.
segment_signwritingswitches from extracting only complete signs viare_swu['sign']to matching each complete sign (with optional trailing space, like other words) or emitting any leftover character as its own token. Space-separated signs now retain trailing spaces on each word except the last.Tests cover incomplete sign fragments, integration with
text_to_words/segment_text, and round-trip losslessness.Reviewed by Cursor Bugbot for commit fb66286. Bugbot is set up for automated code reviews on this repo. Configure here.