Modern Hebrew distilled corpus for rababa

Purpose

Auto-labeled Modern Hebrew pointed text, used to train rababa’s Hebrew diacritization model. Bridges the gap between Biblical Hebrew (Sefaria corpus, rababa-sefaria) and Modern Hebrew text the model sees at inference time.

Source

Unpointed Modern Hebrew from rababa-hewiki (Hebrew Wikipedia), labeled automatically via the Dicta Nakdan API. Distillation pipeline in the main rababa repo at modal_app.py::distill_hebrew.

Method

1. Sample 80K lines from Hebrew Wikipedia (unpointed).

2. For each line, call Dicta Nakdan API to get pointed variant.

3. Take the top prediction per word (Dicta’s options[0]).

4. Filter empty/failed calls.

5. Result: 15,018 pointed Modern Hebrew lines.

Dicta’s published accuracy is ~97% — these are silver labels, not gold. Future versions should: - Increase the source corpus (currently limited by Modal’s 300s per-input timeout on the free tier). - Filter by per-word confidence (fconfident) once enough data exists.

License

Hebrew Wikipedia text is © Wikipedia contributors, licensed under CC-BY-SA 4.0.

The Dicta Nakdan API is provided by the Dicta Israel Center for Text Analysis for non-commercial use. See their API terms for details.

This compiled corpus follows the upstream CC-BY-SA license.

Stats

Split	Lines
train	~12,000
val	~1,500
test	~1,500

Layout

hebrew_distilled_train/train.txt
hebrew_distilled_val/val.txt
hebrew_distilled_test/test.txt