Repository navigation
Add Thai word segmentation using PyThaiNLP - #7
Conversation
Redo of #6 with review comments addressed, lint fixed, and tests added. Co-authored-by: Wannaphong Phatthiyaphaibun <wannaphong@yahoo.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
📝 WalkthroughWalkthroughThai detection and PyThaiNLP-based word segmentation are added, registered in language dispatch, covered by tests, declared as a dependency, and documented as supported. ChangesThai segmentation support
Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
🧹 Nitpick comments (2)
tests/test_thai.py (1)
56-66: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winCover the dispatch path, not only the Thai helper.
These tests call
segment_thai()directly, while the production change also adds theLANGUAGE_SPECSroute. Add one assertion throughsegment_text()(or confirm an existing dispatch test) so a broken Thai import or registration cannot pass unnoticed.Suggested coverage
from words_segmentation.thai import has_thai, segment_thai +from words_segmentation.languages import segment_text +def test_segment_text_dispatches_thai(): + assert list(segment_text("ผมรักเมืองไทย")) == [ + ["ผม", "รัก", "เมือง", "ไทย"] + ]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_thai.py` around lines 56 - 66, Add coverage for the Thai dispatch route by calling segment_text with Thai input and asserting the expected segmentation, while retaining the direct segment_thai tests. Ensure the test exercises the LANGUAGE_SPECS registration/import path so broken Thai dispatch cannot pass unnoticed.pyproject.toml (1)
15-15: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winConstrain the PyThaiNLP version used by the golden tests.
word_tokenize()uses a dictionary-backed default engine, and the tests assert exact boundaries attests/test_thai.pyLines 58-59. An unbounded dependency can change those results after a dependency update. Pin a tested version/range or lock it in CI and update intentionally. (pythainlp.org)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pyproject.toml` at line 15, Constrain the pythainlp dependency in pyproject.toml to a tested version or bounded range so word_tokenize() produces stable boundaries for the golden assertions in tests/test_thai.py. Ensure dependency updates occur intentionally by using the project’s established lockfile or CI pinning mechanism where applicable.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@pyproject.toml`:
- Line 15: Constrain the pythainlp dependency in pyproject.toml to a tested
version or bounded range so word_tokenize() produces stable boundaries for the
golden assertions in tests/test_thai.py. Ensure dependency updates occur
intentionally by using the project’s established lockfile or CI pinning
mechanism where applicable.
In `@tests/test_thai.py`:
- Around line 56-66: Add coverage for the Thai dispatch route by calling
segment_text with Thai input and asserting the expected segmentation, while
retaining the direct segment_thai tests. Ensure the test exercises the
LANGUAGE_SPECS registration/import path so broken Thai dispatch cannot pass
unnoticed.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: c971d81f-962a-499c-bf75-949a0f48c92a
📒 Files selected for processing (5)
README.mdpyproject.tomltests/test_thai.pywords_segmentation/languages.pywords_segmentation/thai.py
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Description
Redo of #6 (by @wannaphong, credited as co-author) — adds Thai language segmentation using PyThaiNLP. The original PR went unresponsive, so this picks it up with all review comments addressed and CI fixed:
thaiimport inlanguages.pywas out of alphabetical order (ruff isort).chinese.py).has_thainow uses the Unicode script property\p{Thai}(consistent withhas_chinese's\p{Han}), and its comment no longer claims to match Han ideographs.segment_thaidocstring example to show the actuallist[str]return value.segment_thainow actually filters empty segments, matching its comment.tests/test_thai.pyfollowing thetest_chinese.pypattern (9 tests).Tests
ruff check .passes.segment_text('hello ผมรักเมืองไทย world สวัสดีโลก')→['hello ', 'ผม', 'รัก', 'เมือง', 'ไทย', ' ', 'world ', 'สวัสดี', 'โลก']Closes #6.
🤖 Generated with Claude Code
Note
Low Risk
Follows the existing Chinese/Japanese pretokenizer pattern with no auth or data-path changes; main impact is a new optional dependency and segmentation behavior for Thai script.
Overview
Adds Thai script as a custom pretokenizer alongside Chinese and Japanese, so unspaced Thai text is split into words before the main tokenizer runs.
A new
words_segmentation/thai.pymodule provides\p{Thai}detection (has_thai), a cached PyThaiNLPword_tokenizewrapper, andsegment_thaithat returns a filteredlist[str].LANGUAGE_SPECSregisters the Thai script withsegment_thai, sosegment_textroutes Thai spans automatically (and respectsLANGUAGES_NO_SPLITlike other languages).pythainlpis added as a runtime dependency; the README marks Thai as implemented.tests/test_thai.pycovers detection, segmentation, and dispatch throughsegment_text.Reviewed by Cursor Bugbot for commit 2aa5917. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by CodeRabbit
New Features
Documentation