Full Tashkeela corpus (Sadeed-style cleaned) for rababa

Purpose

The canonical Arabic diacritization training corpus for rababa. Derived from the full Tashkeela corpus (75M words, ~500K cleaned chunks) — replaces the smaller "Tashkeela processed" subset (50K sentences) used in rababa v0.x.

Source

Two archives from the original Tashkeela project on SourceForge:

1. avcorpus.tar.bz2 — Classical Arabic books from the Shamela Library (84 books, ~74M words). The bulk of the corpus.

2. Tashkeela-arabic-diacritized-text-utf8-0.3.zip — Modern Standard Arabic subset (389 source files).

Fetched by scripts/fetch_tashkeela_full.py in the main rababa repo.

Cleaning pipeline

The cleaning pipeline is a from-scratch reimplementation of the preprocessing described in Section 3 of Aldallal et al. (2025), "Sadeed: Advancing Arabic Diacritization Through Small Language Model" (arXiv:2504.21635). No code dependencies on Sadeed or any other project.

Steps:

1. Sukun normalization — drop sukun on definite-article lam before sun letters; drop sukun on alef (madd carriers never bear sukun).

2. Stopword canonicalization — replace frequently-ambiguous words (في, عن, من, …​) with their canonical diacritized forms.

3. Chunking — hierarchically split long passages into ~50-60 word chunks (sentence-end punctuation > line breaks > quotes > parens > commas).

4. Quality filter — drop chunks with >2 fully-undiacritized words or >2 partially-diacritized words. Also drop chunks with <20 Arabic letters (page numbers, parsing artifacts).

5. Split — deterministic 80/10/10 train/val/test (seed=42).

The iltiqā' as-sākinayn phonological rule from Sadeed’s pipeline is not yet implemented — scheduled for a follow-up commit.

Implemented in scripts/clean_tashkeela_sadeed.py in the main rababa repo.

License

This corpus is © Taha Zerrouki and contributors, licensed under GPL v2 as required by the upstream Tashkeela dataset’s license. This redistribution carries the same license.

Attribution

Original Tashkeela corpus

• Author: Taha Zerrouki & Amar Balla

• Paper: "Tashkeela: Novel corpus of Arabic vocalized texts, data for auto-diacritization systems", Data in Brief (2017). DOI: 10.1016/j.dib.2017.01.011

• Project: Tashkeela on SourceForge

• License: GPL v2

Cleaning pipeline reference

• Authors: Zeina Aldallal, Sara Chrouf, Khalil Hennara, Mohamed Motaism Hamed, Muhammad Hreden, Safwan AlModhayan

• Paper: "Sadeed: Advancing Arabic Diacritization Through Small Language Model", arXiv:2504.21635 (2025).

• Note: This corpus is not the Sadeed_Tashkeela dataset itself (which is hosted gated on HuggingFace). It is an independent reimplementation of the same paper-described pipeline applied to the same upstream Tashkeela source.

Stats

Total chunks: 497,451 Total words: ~27M

Split	Chunks	Shards
train	397,960	3
val	49,745	1
test	49,746	1

Layout

Files are sharded to stay under GitHub’s 100MB file-size limit (no Git LFS required).

tashkeela_full_train/train-001.txt
tashkeela_full_train/train-002.txt
tashkeela_full_train/train-003.txt
tashkeela_full_val/val-001.txt
tashkeela_full_test/test-001.txt

Loaders should glob test-*.txt and concatenate shards in lexical order (already sorted by NNN suffix).