Skip to content

Add Thai word segmentation using PyThaiNLP - #7

Merged
AmitMY merged 2 commits into
mainfrom
add-thai-segmentation
Jul 14, 2026
Merged

AmitMY merged 2 commits into
mainfrom
add-thai-segmentation

Conversation

@AmitMY

@AmitMY AmitMY commented Jul 14, 2026 •

Copy link
Copy Markdown
Contributor

Description

Redo of #6 (by @wannaphong, credited as co-author) — adds Thai language segmentation using PyThaiNLP. The original PR went unresponsive, so this picks it up with all review comments addressed and CI fixed:

  • Fixed lint failure: thai import in languages.py was out of alphabetical order (ruff isort).
  • Fixed module docstring that described "Chinese text pretokenization" (copy-paste from chinese.py).
  • has_thai now uses the Unicode script property \p{Thai} (consistent with has_chinese's \p{Han}), and its comment no longer claims to match Han ideographs.
  • Fixed segment_thai docstring example to show the actual list[str] return value.
  • segment_thai now actually filters empty segments, matching its comment.
  • Added tests/test_thai.py following the test_chinese.py pattern (9 tests).
  • Moved the Thai entry in the README up next to the other implemented scripts.

Tests

  • ruff check . passes.
  • Full suite: 93 passed, 1 skipped.
  • End-to-end: segment_text('hello ผมรักเมืองไทย world สวัสดีโลก') → ['hello ', 'ผม', 'รัก', 'เมือง', 'ไทย', ' ', 'world ', 'สวัสดี', 'โลก']

Closes #6.

🤖 Generated with Claude Code


Note

Low Risk
Follows the existing Chinese/Japanese pretokenizer pattern with no auth or data-path changes; main impact is a new optional dependency and segmentation behavior for Thai script.

Overview
Adds Thai script as a custom pretokenizer alongside Chinese and Japanese, so unspaced Thai text is split into words before the main tokenizer runs.

A new words_segmentation/thai.py module provides \p{Thai} detection (has_thai), a cached PyThaiNLP word_tokenize wrapper, and segment_thai that returns a filtered list[str]. LANGUAGE_SPECS registers the Thai script with segment_thai, so segment_text routes Thai spans automatically (and respects LANGUAGES_NO_SPLIT like other languages).

pythainlp is added as a runtime dependency; the README marks Thai as implemented. tests/test_thai.py covers detection, segmentation, and dispatch through segment_text.

Reviewed by Cursor Bugbot for commit 2aa5917. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by CodeRabbit

  • New Features

    • Added Thai language detection and word segmentation.
    • Thai text is now automatically recognized and split into words, including mixed-language content.
    • Added support for Thai segmentation through PyThaiNLP.
  • Documentation

    • Updated the supported writing systems list to include Thai.

Redo of #6 with review comments addressed, lint fixed, and tests added.

Co-authored-by: Wannaphong Phatthiyaphaibun <wannaphong@yahoo.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 14, 2026 •

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Thai detection and PyThaiNLP-based word segmentation are added, registered in language dispatch, covered by tests, declared as a dependency, and documented as supported.

Changes

Thai segmentation support

Layer / File(s) Summary
Thai detection and tokenization
words_segmentation/thai.py
Adds Thai Unicode detection, lazy cached PyThaiNLP tokenizer loading, and empty-token filtering.
Dispatch integration and tests
words_segmentation/languages.py, tests/test_thai.py
Registers Thai script spans with segment_thai and tests detection and segmentation behavior.
Dependency and support documentation
pyproject.toml, README.md
Adds pythainlp to core dependencies and lists Thai among supported writing systems.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The PR fulfills #6 by adding Thai-script detection, segmentation, tests, dependency wiring, and documentation updates.
Out of Scope Changes check ✅ Passed All changes directly support Thai language support, with no unrelated code paths or feature additions visible in the summary.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely summarizes the main change: adding Thai word segmentation with PyThaiNLP.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch add-thai-segmentation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
tests/test_thai.py (1)

56-66: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Cover the dispatch path, not only the Thai helper.

These tests call segment_thai() directly, while the production change also adds the LANGUAGE_SPECS route. Add one assertion through segment_text() (or confirm an existing dispatch test) so a broken Thai import or registration cannot pass unnoticed.

Suggested coverage
 from words_segmentation.thai import has_thai, segment_thai
+from words_segmentation.languages import segment_text

+def test_segment_text_dispatches_thai():
+    assert list(segment_text("ผมรักเมืองไทย")) == [
+        ["ผม", "รัก", "เมือง", "ไทย"]
+    ]
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/test_thai.py` around lines 56 - 66, Add coverage for the Thai dispatch
route by calling segment_text with Thai input and asserting the expected
segmentation, while retaining the direct segment_thai tests. Ensure the test
exercises the LANGUAGE_SPECS registration/import path so broken Thai dispatch
cannot pass unnoticed.
pyproject.toml (1)

15-15: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Constrain the PyThaiNLP version used by the golden tests.

word_tokenize() uses a dictionary-backed default engine, and the tests assert exact boundaries at tests/test_thai.py Lines 58-59. An unbounded dependency can change those results after a dependency update. Pin a tested version/range or lock it in CI and update intentionally. (pythainlp.org)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pyproject.toml` at line 15, Constrain the pythainlp dependency in
pyproject.toml to a tested version or bounded range so word_tokenize() produces
stable boundaries for the golden assertions in tests/test_thai.py. Ensure
dependency updates occur intentionally by using the project’s established
lockfile or CI pinning mechanism where applicable.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@pyproject.toml`:
- Line 15: Constrain the pythainlp dependency in pyproject.toml to a tested
version or bounded range so word_tokenize() produces stable boundaries for the
golden assertions in tests/test_thai.py. Ensure dependency updates occur
intentionally by using the project’s established lockfile or CI pinning
mechanism where applicable.

In `@tests/test_thai.py`:
- Around line 56-66: Add coverage for the Thai dispatch route by calling
segment_text with Thai input and asserting the expected segmentation, while
retaining the direct segment_thai tests. Ensure the test exercises the
LANGUAGE_SPECS registration/import path so broken Thai dispatch cannot pass
unnoticed.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: c971d81f-962a-499c-bf75-949a0f48c92a

📥 Commits

Reviewing files that changed from the base of the PR and between 64ade04 and fa5edb1.

📒 Files selected for processing (5)
  • README.md
  • pyproject.toml
  • tests/test_thai.py
  • words_segmentation/languages.py
  • words_segmentation/thai.py

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@AmitMY
AmitMY merged commit 8145967 into main Jul 14, 2026
4 checks passed
@AmitMY AmitMY mentioned this pull request Jul 14, 2026
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant