[leonhandreke]

Fixes the matching problem in opensanctions/nomenklatura#394.

A Cyrillic or Greek letter inside a mostly Latin token is usually evasion or OCR noise. Transliteration reads some of these letters as a different Latin letter. For example, Cyrillic р becomes r, so COMРLEJO became comrlejo.

NamePart now folds look-alike letters to their Latin counterparts before transliteration. The fold applies only when more than 60% of the letters in the token are Latin.

• COMРLEJO → complejo (folded)
• Dмитрий → dmitry (mostly Cyrillic, transliterated as before)
• Рома → roma (all Cyrillic, transliterated as before)

The table in translit.rs has Cyrillic look-alikes and four Greek ones (κ ο ρ χ). Keys are lowercase, because name parts arrive casefolded.

Eval: nomenklatura name_bench (1130 cases, with the homoglyph cases from opensanctions/nomenklatura#395). Only one score changes: COMРLEJO TURISTICO OASIS vs COMPLEJO TURISTICO OASIS goes from 0.930 to 1.000.

🤖 Generated with Claude Code using Claude Opus 5.5

[leonhandreke]

Obvious improvement here could be to also go the other way, i.e. interpret a Latin p as a Cyrillic p in mostly-Cyrillic tokens.

[pudo]

Doesn't the existing distance metric already have code that marks 0 and o to be less than a 1 levensthein difference? What's new with this approach?

[pudo]

Sorry, I've now had a deeper look at this and understood the difference to the existing mechanism. I find it hard to assess the full impact of this: it fundamentally changes the relationship of the transliterated text to the original text in a manner that all of the subsequent consumers would need to be aware of. I'm also a bit uncomfortable with having these two separate mechanisms for confusable characters that live totally separate.

Stepping back: We get maybe 100 complexity coins for logic v2 before it goes from "explainable" to "nondeterministic mess"... this feels like a thing that makes sense for the test cases it's meant to cover, but will increase surprise in the system more broadly. If we think this really addresses a class of issues that are worth 5 of these innovation coins, let's build it - but maybe in a way that's aligned with the existing confusables handler? If it's just fixing a test case, it seems a bit expensive to me.

[leonhandreke]

Thanks for taking a look! I agree that this increases complexity. I wouldn't have a good idea for how to rebuilt it in a way that aligns better with the confusables mechanism. We asciify pretty early, which loses a lot of fidelity.

I have no idea whether this is applicable in the real world, and right now it's just fixing a test case. I'll trust your judgement here.