26 September 2026 · Syamjith NK
On 19 September I opened
an issue on google/sentencepiece:
nmt_nfkc, the default text normalizer used when training a SentencePiece model, maps
U+200E LEFT-TO-RIGHT MARK and U+200F RIGHT-TO-LEFT MARK to a space, and leaves U+061C ARABIC LETTER
MARK alone. On 22 September
Taku
Kudo committed the fix and closed it.
I did not write that fix. I reported it. The patch is his: one line in
src/builder.cc, a regenerated lookup table, and twelve lines of test. That distinction
matters enough to put in the second paragraph, because “found a bug in Google’s
tokenizer” and “fixed Google’s tokenizer” are different sentences and only
the first one is mine.
The bug itself is small. What is worth writing down is where I was looking, why the argument worked, and the three repositories where I looked and found nothing.
Reading a lot of Arabic handling in other people’s libraries has produced a fairly consistent shape. Letters are usually fine. Shaping is usually fine. Digits are mostly fine, with exceptions. These get fixed because they are visible: when they break, the output looks wrong, somebody eventually sees it, and somebody eventually files it.
Unicode also has twelve bidirectional control characters. Nine are explicit scope markers, the embeddings, overrides, isolates and their pops. Three are marks: LRM, RLM, and ALM, the Arabic letter mark. None of the twelve draws anything. They survive a proofread, a diff review, a copy-paste, and a person reading the file aloud. Nothing about them announces itself, so nobody looks, so nobody files.
nmt_nfkc handled two of the twelve.
There was an obvious way to write this issue and it was the wrong one: ALM is common in Arabic text, so a tokenizer used for Arabic ought to strip it. I have no measurement of how often ALM appears in real corpora. If a maintainer had asked me for one I would have had nothing, and he would have been right to ask.
So the issue does not mention frequency at all. It makes one claim:
You already map U+200E and U+200F to a space. U+061C does the same job for Arabic-script runs that those two do everywhere else, and it is not in the list.
That is checkable in about ten seconds, in the maintainer’s own source file, without trusting anything I say about Arabic. Here is the list it is about, after the patch:
(*chars_map)[{0x000A}] = {0x20}; // LINE FEED
(*chars_map)[{0x000C}] = {0x20}; // FORM FEED
(*chars_map)[{0x000D}] = {0x20}; // CARRIAGE RETURN
+ (*chars_map)[{0x061C}] = {0x20}; // ARABIC LETTER MARK
(*chars_map)[{0x1680}] = {0x20}; // OGHAM SPACE MARK
(*chars_map)[{0x200B}] = {0x20}; // ZERO WIDTH SPACE
(*chars_map)[{0x200E}] = {0x20}; // LEFT-TO-RIGHT MARK
(*chars_map)[{0x200F}] = {0x20}; // RIGHT-TO-LEFT MARKThe fix is the line in the middle. The argument is the two lines at the bottom, which were already there.
I think this generalises, and it is the part I would keep if I could keep one thing: the strongest bug report about a minority script is usually not “you handled my language wrong”. It is “you already do this for the majority case, and you skipped one member of the same class.” The first asks a maintainer to take your word for something he cannot check. The second asks him to read his own file.
Consistency gets an issue read. A cost gets it fixed. The measurement I attached was on
google/mt5-base, whose tokenizer is a shipped SentencePiece model that people train
and fine-tune against today.
Take an Arabic test phrase, tokenize it, and you get five pieces. Insert a single ALM, which changes nothing you can see, and you get seven. The interesting part is not the count. It is that the word بالعالم stops being one word-initial piece and becomes a bare word-start marker, then the ALM, then the word, so the piece the model actually learned for that word at the start of a word is never used. Put an RLM in the same position and nothing changes at all, because the RLM is stripped before the tokenizer ever sees it.
Two strings that are visually identical, two different token sequences, and which one you get depends on which pipeline last touched the text.
This is the part I would have got wrong if I had stopped at the green check mark.
SentencePiece’s newest release is v0.2.2, published 12 July 2026. The fix
landed on 22 September. Asking GitHub to compare the two puts the fix 164 commits ahead of
the tag, and zero behind, which is the machine-readable way of saying it is in no release
at all. PyPI agrees: pip install sentencepiece today gets you 0.2.2, uploaded the same
day in July, and it still passes U+061C straight through. “Fixed” describes the
repository. It does not yet describe anything anyone runs.
That much is a waiting problem and it resolves itself on the next release. The second half does
not. A SentencePiece model file does not reference a normalizer by name and look it up at load
time; it carries one. The field is precompiled_charsmap in the model proto, described
in the source as the pre-compiled normalization rule, and it is serialised into the
.model file when the model is trained:
// Pre-compiled normalization rule created by
// Builder::GetPrecompiledCharsMap() or Builder::CompileCharsMap() method.
optional bytes precompiled_charsmap = 2;So every SentencePiece model that already exists keeps the normalizer it was trained with, permanently, no matter what version of the library loads it. The fix reaches models trained after it ships. It cannot reach mT5, or anything else already published, ever. Retraining is the only route, and nobody retrains a tokenizer over one invisible character.
The general form is worth stating plainly, because I keep rediscovering it: reporting a bug upstream fixes the future, not the installed base. A rendering bug is fixed the moment you upgrade. A tokenizer defect has a tail measured in model generations. It is also the honest answer to a question I have been asked about arabic-lint, which now has a bidi-controls check whose residue band is exactly this case, a lone ALM, LRM or RLM sitting in text: if upstream fixes it, why keep a detector? Because upstream fixing it does not change a single byte on your disk, or in the model you are about to fine-tune.
One filed issue and a headline about Google’s Arabic handling would be dishonest arithmetic, so here is the denominator. The sweep that turned up the SentencePiece case went through other Google-owned code that touches Arabic, and the rest of it came back clean. Two are worth recording in detail, because a clean verdict is only worth something if you say how you checked it.
google/re2. Its \p{Arabic} character class is a generated table of
integer ranges in unicode_groups.cc. I expanded it and compared it against
Scripts.txt from the Unicode version re2 generates from, 15.1.0. It is
1,368 codepoints against 1,368, identical, with nothing extra on either side. Not
approximately right. Bit-exact, including U+061C itself, which re2 correctly classifies as Arabic
script.
google/libphonenumber. Arabic-Indic and Eastern Arabic-Indic digits are handled
correctly everywhere I looked, including the as-you-type formatter, and the reason is a design
decision rather than an Arabic-specific effort. Its digit pattern is \p{Nd}, the
Unicode decimal-number property, and normalisation runs through
Character.digit(c, 10), which resolves any Unicode digit to its value. Nobody wrote
Arabic support into it. They declined to write an ASCII assumption into it, which turns out to be
the same thing.
That contrast is the whole post in miniature. The list of bidi controls in
nmt_nfkc was tabulated by hand, so it had a hole in it that nothing could
detect. libphonenumber’s notion of a digit is derived from a Unicode property, so it
cannot have a hole: anything Unicode adds classifies itself. A hand-written list of characters is a
bug with a delay on it.
There was one more real finding in libphonenumber and I did not report it. A directional mark sitting inside a number makes the number invalid rather than silently wrong: it is rejected, loudly, at the point of parsing, instead of being parsed into a different number. That is wrong-but-safe. Every argument this blog has made for three weeks is that the quiet failures are the dangerous ones, and filing a loud one would be inconsistent with a rule I actually hold. A maintainer’s attention is a real budget and it is not mine to spend on tidiness.
While comparing re2’s table I noticed something with a much better headline in it. re2 generates its Unicode tables from 15.1.0, and three Unicode releases have shipped since: 16.0, 17.0 and 18.0. The post writes itself: re2’s Arabic support is three Unicode versions out of date.
It is true, and it is misleading, which is worse than being false. So I counted the gap. Between Unicode 15.1.0 and the current 18.0 there are 22,995 newly assigned codepoints across 38 scripts, and this is where they are:
| script | new codepoints | share of the gap |
|---|---|---|
| Seal Script | 11,328 | 49.26% |
| Han | 4,322 | 18.80% |
| Egyptian Hieroglyphs | 3,995 | 17.37% |
| Common | 970 | 4.22% |
| Jurchen | 965 | 4.20% |
| … | ||
| Arabic (10th) | 84 | 0.37% |
Half the gap is Seal Script. Another third is Han and Egyptian hieroglyphs. Arabic is tenth, at 84 codepoints and just over a third of one percent, and I would guess most of those 84 are characters no Arabic document I will ever meet contains.
So the honest sentence is: re2 vendors a Unicode release or three behind, like most projects that vendor Unicode at all, and Arabic is among the least affected scripts in the gap. That is not a blog post. Not writing it is most of the reason the rest of this one can be believed.
The probe that found the SentencePiece case needed to know which characters are directional
marks. The elegant way to get that is to derive it: a mark is a format character, category
Cf, with a strong bidirectional class, L, R or
AL. It is a satisfying one-liner and it is wrong.
Run it over the whole codespace and it matches 23 characters. Three of them are directional controls. The rest are U+070F SYRIAC ABBREVIATION MARK, two Kaithi number signs, and sixteen Egyptian hieroglyph joiners and segment marks, all of them characters somebody typed on purpose. Had that shipped, the tool would report a Syriac abbreviation mark, in Syriac text, as Arabic bidi residue. It is the identical mistake to the one I made and published a correction for two weeks ago, where ornate Quranic parentheses read as shaping damage and produced a confident 35.6%.
The property that actually separates them is Bidi_Control=Yes, and the standard
library does not expose it, so the three marks are named rather than derived. There is a test that
asserts the naive derivation over-matches, so the comment explaining why cannot quietly become
false.
The second one is duller and more embarrassing. A check in the same harness printed a confident conclusion in the same run in which its own assertion returned False. The conclusion was the thing I would have published. The assertion was the thing that knew better, and it was sitting right next to it, not wired into anything. Put the assertions inside the measurement, not beside it. A harness pointed at somebody else’s code with no way to fail is not a measurement, it is a printer.
Kudo’s patch adds twelve lines of test, and it does not assert that ALM is stripped. It asserts the same property three times, once per mark, in two different normalizer configurations:
EXPECT_EQ(WS "a" WS "b", normalizer.Normalize("a\xe2\x80\x8e" "b")); // LRM
EXPECT_EQ(WS "a" WS "b", normalizer.Normalize("a\xe2\x80\x8f" "b")); // RLM
EXPECT_EQ(WS "a" WS "b", normalizer.Normalize("a\xd8\x9c" "b")); // ALMThat is the consistency argument, encoded. The test does not care which characters are in the list; it fails the moment one of the three stops behaving like the other two, which is precisely the failure that created the bug in the first place. A test that only proved ALM was fixed would have left the next member of the class exactly as exposed as this one was.
Three days, one line of library code, and none of it required anyone to trust a stranger about Arabic. The invisible characters are where the rest of this lives, and the way in is not to argue that your language matters. It is to find the place where a project already did the right thing, once, and stopped one short.