Loosing unicode filenames

[valcomm]

$ ack 'KVN.2025.\(11'
?????? _ ?????????? ?????????????????? _ 31 _ 2025-08-10 _ www.kvnforall.info.ts.txt
135:Файл: KVN.2025.(11).Letnij.Kubok.HDTV.1080i.Remux.www.kvnforall.info.ts
$ grep -P 'KVN.2025.\(11' *
КВН _ Кубок чемпионов _ 31 _ 2025-08-10 _ www.kvnforall.info.ts.txt:Файл: KVN.2025.(11).Letnij.Kubok.HDTV.1080i.Remux.www.kvnforall.info.ts

I usually use use Encode::Locale (); (plus dealing in the code) for such cases.

[n1vux]

The '?'s may be a matter of the Terminal's current font not having glyphs for the needed Unicode code-points? Is the problem that Unicode in the matched line erases the filename, or that Unicode filename is self-destructing? Your example is ambiguous. 2 Test cases to disambiguate: 1) boring (flat-ascii filename), Unicode contents of your KVN.2025 lines 2) unicode filenames similar to your KVN.2025 files, with boring flat-ascii content Your Grep -P example should display a filename:linenumber: same as Ack, but didn't? Perhaps one of the octets in a UTF8 encoding is a terminal control character, so that the output clobbers the filename. If so, can you provide a code-fragment to generate a test file with unicode-enhanced filename that ls, grep handle nicely but ack doesn't? Are the KVN.2025 lines part of a program? Tab separated data for a program?

[valcomm]

The '?'s may be a matter of the Terminal's current font not having glyphs
for the needed Unicode code-points?

Nope, not the case. There is grep call right below ack call and grep shows filename just fine in the same terminal with the same font.
Also I have to note that both of them output found unicode line correctly.

Is the problem that Unicode in the matched line erases the filename, or
that Unicode filename is self-destructing?

Matched line is displayed fine both by ack and grep. The difference is that ack didn't display file name correctly. No any tricks there.

Your example is ambiguous.
2 Test cases to disambiguate:

1. boring (flat-ascii filename), Unicode contents of your KVN.2025 lines

The issue is about unicode filenames. Neither about flat ascii filenames, no about unicode content.
I haven't noticed any issues in any of those cases so far.

2. unicode filenames similar to your KVN.2025 files, with boring flat-ascii content

Why content matters at all? Both ack and grep dealt fine with unicode (UTF-8) content in the test.

Your Grep -P example should display a filename:linenumber: same as Ack, but
didn't?

Why should it while I didn't ask it for it? I was interested in finding files containing substring (non-Unicode, btw). Both of them found the file, but ack had troubles displaying the file name.

Perhaps one of the octets in a UTF8 encoding is a terminal control
character, so that the output clobbers the filename.

There are no any tricks in file names in the directory. Exactly for proving that I provided output of grep call in the same directory with the same substring to find.

If so, can you provide a code-fragment to generate a test file with
unicode-enhanced filename that ls, grep handle nicely but ack doesn't?

How about
echo Файл: KVN.2025.\(11\).Letnij.Kubok.HDTV.1080i.Remux.www.kvnforall.info.ts > /tmp/КВН\ _\ Кубок\ чемпионов\ _\ 31\ _\ 2025-08-10\ _\ www.kvnforall.info.ts.txt ?
That would create the file with the exact file name that contains exactly the substring (even more - the whole string) that I was looking for.

[n1vux]

Ok, thank you, I see. Sorry, the very long filename with alternating spaces and underscores totally confused me as to what was file name and what was content! I'm seeing Ack v3.9.0 is printing unicode filenames correctly, but Ack v3.10.0 is not. Alas, this regression was *almost* intentional; sorry for the inconvenience! This is almost certainly collateral damage from my implementation in v3.10.0 of the recommended paranoid mitigations for a CVE (preventing VT/ANSI escape codes in filenames from rendering fake data by including *␤file:666:* in contents or filename and/or *file␛[31m;RED␛[0m.pl <http://0m.pl>* (which renders *fileRED.pl) *as filename; the white-list should probably allow (most) Unicode printing characters). For your set of files, you may want to down-grade to Ack v3.9.0 until we can loosen this paranoia in v.3.10.1 (planned for some followups anyway, this is one more), since you *do* want to see the non-ascii filenames that the security boffins insisted we protect you from. 🤷 Andy, you can assign this one to me and we can add it to the 3.10.1 patch set.

[n1vux]

Note, if you want to use the old perl -CS ack ... workaround to search Unicode contents, you need ack v3.1.0 (sysread() optimization doesn't work with UTF8-all-the-things); with LOCALE settings, that would even work with UCS-16/32, but alas people don't reliably set BOM and files move between systems with different LOCALE so that isn't reliable, so perfection is unlikely
. (I'd hope to make Unicode safety throughout a goal for ack 4.0; requires way too many new test cases for a point release, alas. And as noted, files w/o BOM and not matching LOCALE are never Alternatively, ripgrep probably is Unicode native from birth? The advantage of not being first.)

[valcomm]

Isn't BOM recommended to be not set up nowadays?
The issue is about file names output.

So far I haven't noticed any issues with ack searching for non-ASCII content (files are UTF-8 encoded without BOM).
I think by default files should be assumed to be UTF-8, failing to be processed if there are issues with decoding.
If some other encoding is required - that should be specified explicitly.

[n1vux]

If one keeps files within systems with uniform LOCALE setting, BOM isn't needed, so for local processing, perhaps not recommended.

OTOH without BOM if UCS 16 or 32 (BE/LE) files used in e.g. Asian languages wander elsewhere, they're not always diagnosable automagically.
Files do seem to wander around, so I'm not happy about BOMless non-UTF8 files, even if recommended.

Yes, your current (= this) issue, caused by the (my) hurried CVE fix, is UTF filenames.
However, we have long standing issues (tagged UNICODE) about unicode contents and patterns; we have never claimed to be unicode clean (yet), and that applies to both name and contents. Working with UTF8 is a happy accident.
But while not guaranteed, I am going to fix this regression with the rest of the clean-up; what had worked (mostly) in 3.9.0 should work again in 3.10.1 (or maybe .2).

(I do agree that these days UTF-8 is a sensible default.
Andy started Ack 1 before that was sensible (at least within our 🇺🇸 -centric view) , and neither he nor I have had the ROUND TUITS to pivot it to UTF-8 since; would have been nice at 3.0, we knew it, but it didn't happen; too big a testing program to get it right, neither of us want to do it badly, and at the time, setting LOCALE and perl -CS prefix would (mostly) work (around). Sysread optimization prevents that unless using --passthru since 3.2.0, alas; oops.
Ack in Rust known as RIPGREP rg, being a new build in a more modern language (Rust), likely is UTF-8 native? RG uses PCRE2, not PCRE(1), so is almost true Perl REs, so pretty compatible. And fast! Might be better in your LOCALE ? My main reason for not switching is I like helping Andy with supporting Ack, so I need to use it regularly, and also avoiding edge-cases where RG PCRE2 RE isn't exactly as wonderful as Perl 5.4x RE; I often prototype Perl subroutines with Ack at commandline — the automate only after doing it manually pattern, and don't want edge-cases.
ASIDE: Whomever in ancient history initially allowed space tab ␛ ␍ ␤ other ASCII Control Characters and shell meta-character punctuation in filenames bears some original sin; normally I don't criticize the Multics/Project MAC team of sainted memory, one a very dear friend 🪦 RIP MAP — I don't know they were first, but seems likely, as the common source for most modern OS/FS praxis, their manual gave three-word-with-spaces as a feature, and generally first for mixed-case case-sensitive filenames — but I'm really grumpy about spaces; Ken&Dennis and Bill G shoulda fixed that when they had the chance. Too late now alas.
ASIDE2: wrto to your sample data: as I type this, I'm listening to ITA 🇮🇹 v BEL 🇧🇪 ⚽ 😉 . )

[valcomm]

I don't think LOCALE is responsible for files content. It's only for file names and output (plus input and command line of course).
For content UTF-8 is current default. Without BOM now. If you need/expect other encoding - that should be provided explicitly.
I've got some experience with such cases. Figuring out the encoding on the fly doesn't always work.

Ok, I installed ripgrep. I expect it to just enforce UTF-8 everywhere (input, output, content). Maybe worth trying to test it on non-UTF-8 content ;-)

I think it's fine to have only necessary restrictions. And having just two of them (zero and slash) is fine. Otherwise list of prohibited combinations would be very large. Now terminal processes ESC sequences - ok, let's prohibit [ + something. Later there will be something else with its gimmicks - new rules. I think it's a wrong way.
Just having a switch how to output ESC sequences (raw or safe) should be enough.