10 September 2026 · Syamjith NK
Three posts on this blog argue that the standard way to make Arabic work in Python corrupts it. That argument has a hole in it: all the evidence was about rendering code. A reversed chart label is embarrassing and fixable. But the same recipe run once, before a file is written, bakes the damage in permanently.
If that ever happened upstream of a training corpus, every model trained on it learned from text no Arabic reader would accept. I could not find anyone who had checked, so I did.
I pulled rows for every Arabic dataset I could find on the Hugging Face Hub through the datasets-server API and scanned them with my own linter. 341 datasets.
The first run said 35.6% of an Islamic heritage OCR corpus was corrupted. Two others came back at 27% and 5%. That is a headline. I nearly had one.
Then I printed the characters it was objecting to:
U+FD3F ﴿ ORNATE RIGHT PARENTHESIS
U+FD3E ﴾ ORNATE LEFT PARENTHESIS
U+FD40 ﵀ ARABIC LIGATURE RAHIMAHU ALLAAHThose are not damage. `﴾ ﴿` is how you enclose a Quranic quotation, and `﵀` is an honorific. People type them deliberately. My tool was flagging them because Arabic Presentation Forms-A is interleaved: the positional glyph forms and ordinary semantic characters share one block, and I had excluded only the word ligatures at the end of it.
So on Islamic heritage text, which is a large share of all Arabic corpora, the tool was
confidently wrong, in the direction that publicly accuses somebody else's dataset. The fix
was to stop tabulating ranges and derive the class from the Unicode character name: a
shaping artefact is exactly a codepoint named ... ISOLATED/INITIAL/MEDIAL/FINAL
FORM. Everything else in the block is deliberate, and new Unicode additions classify
themselves.
All three datasets then read 0%.
276 datasets readable. 26,318 rows. 119,517 text fields.
Twenty-one had any finding at all, and twenty of those were a single stray
presentation form in otherwise correct text, most often U+FE91 BEH INITIAL
FORM. That is copy-paste out of a PDF, or OCR residue, not the recipe.
I originally wrote that this "splits the word into different tokens". That was wrong, and I have since measured it. What actually happens is worse.
On a tokenizer with no Unicode normalisation step, the word is not split. It is replaced
by the unknown token, so every character of meaning is discarded before the model sees
anything. One corrupted letter out of four is enough: بيت becomes
[UNK].
| tokenizer | one stray presentation form |
|---|---|
aubmindlab/bert-base-arabertv02 (Arabic-specific) | 6/6 words → [UNK] |
google-bert/bert-base-multilingual-cased | 6/6 words → [UNK] |
xlm-roberta-base (SentencePiece) | 0/6, identical tokens, unaffected |
Qwen/Qwen2.5-0.5B (byte-level BPE) | survives, but as entirely different tokens |
The two Arabic-capable BERTs destroy the word. The multilingual SentencePiece model does not, because it normalises and they do not. If you train Arabic models, that is the whole practical finding: normalise before you tokenise, and a stray glyph costs you nothing instead of costing you the word.
One dataset was the recipe.
Yousefmd/arabic_ocr_dataset — every text field, 1,000 out of 1,000 labels sampled through ten windows across its whole 20,000-row split, 3 to 50 presentation forms per label. A sample row:
ﺑﺪﺃﺕ ﻗﺮﺍﺀﺓ ﺍﻟﺼﻔﺤﺎﺕ ﺍﻷﻭﻟﻰ ﻋﻠﻰ ﻧﻐﻤﺎﺕThe intended text is بدأت قراءة الصفحات الأولى على نغمات. Every letter
has been replaced by its contextual glyph form — INITIAL, MEDIAL, FINAL and ISOLATED all present
together. Only that half of the recipe ran: the words are in ordinary logical order, so
unicodedata.normalize("NFKC", text) alone gives the line back exactly as written
above, with no reversal needed.
It matters more than its size because of which field is damaged. This is an OCR
dataset, so text is the ground truth. A model trained on it learns to emit
presentation forms instead of plain letters, and then fails string comparison against real
Arabic even when it read the image correctly. That is the path from a rendering bug into model
weights, observed rather than argued for.
Correction, 13 September 2026. I originally wrote that the words were also "written
in visual order", and called that "the complete signature". That was wrong, and it was the more
damaging half of what I said, because I said it in a thread on the author's own repository.
Only the shaping step ran. Two checks settle it: NFKC alone returns correct readable
Arabic, and of 5,423 sampled words carrying more than one presentation form, zero begin
with a FINAL form — which is what a word stored in visual order looks like read out of memory.
arabic_reshaper.reshape(NFKC(stored)) also reproduces the stored string exactly, for
1,000 of 1,000 labels, which pins down which half ran. The practical consequence is that the
repair advice I gave was wrong in the worse direction: I warned that recovery needed more than
NFKC because of the lam-alef trap in the post above, when for this data
NFKC alone is complete and safe. The detector reports stored glyph forms;
visual order is a separate property, and I never measured it — I assumed it, because the two
steps are usually copied together. The
correction is posted in that thread as well.
One more thing worth writing down, because it nearly put a second wrong number in public.
My first check for "is this a presentation form?" tested the Unicode name for
ARABIC LETTER … FORM. That silently drops every lam-alef ligature, U+FEF5–FEFC,
whose names begin ARABIC LIGATURE instead — the exact character the visual-order
test is most sensitive to. Asking the linter's own classifier instead moved the word count
from 5,404 to 5,423. The conclusion held, but only because I checked.
Now the honest part: that dataset has 17 downloads. This shows the mechanism reaches training data. It is not evidence that widely-used Arabic corpora are affected, and I am not going to write it as though it were.
65 datasets could not be read, and the viewer fails most often on the largest corpora. So this covers the mid and small end of the distribution well and the head of it poorly. Nothing here reaches web-scale pretraining data. A hundred rows per dataset also cannot find a defect confined to a rare subset.
An earlier version of this audit claimed 121 datasets were unreadable. 61 of those were rate-limit errors caused by my own request rate. A coverage caveat built out of your own impatience is not a caveat, it is a mistake; a slower second pass recovered them.
The audit was one command away from being published with three named datasets and a percentage next to each. What stopped it was reading the flagged characters instead of trusting the count.
If you point a measurement tool at somebody else's work, check its output against text you can actually read before the number leaves your machine. A false positive costs them a public accusation and costs you the only thing that makes the measurement worth anything.