Suzume is a lightweight Japanese tokenizer for browsers and native apps. It is not a full morphological analyzer like MeCab: its primary goal is to split text into useful units for search, display, and application code. Unlike a boundary-only tokenizer, it also returns part-of-speech tags and lemmas.
Reach for it when you need to:
- Tokenize Japanese anywhere — use the same tokenizer in a browser, serverless function, Python process, Go service, or C/C++ application.
- Extract search terms — generate keyword tags with POS filtering, lemmas, and duplicate removal.
- Add your vocabulary — load application-specific words through a user dictionary.
📖 Documentation · 🧪 Live Demo · Getting Started
MeCab is a morphological analyzer designed for detailed, dictionary-based analysis. Suzume is a tokenizer designed around practical token boundaries. It uses character patterns and compact rules, keeping compounds and quantities together when that produces a more useful search unit.
Input: 経済成長 3人
MeCab: 経済 / 成長 3 / 人
Suzume: 経済成長 3人
Suzume still provides POS tagging and lemmatization, so applications can normalize inflected verbs and adjectives without adopting MeCab-style output. The outputs are intentionally different rather than drop-in compatible. See Differences from MeCab for examples, trade-offs, and known constraints.
npm install @libraz/suzume # JavaScript / TypeScript
pip install suzume # PythonThe Python wheel also installs the suzume command. PyPI provides binary
wheels for Linux x86_64 and macOS arm64; Windows, other architectures, and
source distributions are not supported.
The Go module is maintained and versioned in a separate repository; it is not
built or compatibility-gated by this repository. It ships no precompiled binary
and builds the static library from source, so it is installed from a checkout
rather than with go get alone:
git clone https://github.com/libraz/go-suzume.git
cd go-suzume && make libPoint your module at that checkout with go mod edit -replace. The dictionaries
are then embedded in your binary. See the
Go bindings guide for the full procedure.
For C/C++ installation, native builds, user dictionaries, and all runtime options, see the documentation.
import { Suzume } from '@libraz/suzume'
const suzume = await Suzume.create()
const tokens = suzume.analyze('すもももももももものうち')
const tags = suzume.generateTags('東京の公園に行きました')
suzume.destroy() // optional immediate cleanupfrom suzume import Suzume
with Suzume() as sz:
tokens = sz.analyze("すもももももももものうち")
tags = sz.generate_tags("東京の公園に行きました")Python calls on one Suzume instance are serialized for thread safety. Use
separate instances when workers should analyze concurrently.
s, err := suzume.New()
if err != nil {
log.Fatal(err)
}
defer s.Close()
morphemes := s.Analyze("すもももももももものうち")The same Python package provides a command-line interface:
suzume "東京へ行く"
suzume --mode search --format json "東京の公園"
printf 'りんごを食べる\n' | suzume --format tagsRun suzume --help for analysis and tag options. Dictionary compilation,
validation, and test commands belong to the native developer tool,
suzume-cli, built from the CMake project.