grepping for Russian characters using character ranges

How to grep for lines with 'Й' and 'й' from text file, using character ranges?

In Unicode, Russian capital characters (except 'Ё') are in the range from 0x410 to 0x42f in alphabetical order, and small characters (except 'ё') are in the range from 0x430 to 0x44f in alphabetical order. This means that [А-ИК-ЯЁ] should match all Russian characters except 'Й', and [а-ик-яё] should match all Russian characters except 'й'. But this turns out to be not quite the case.

For experimenting, I created a function that outputs Russian characters [Ж-Мж-м], one per line:

rus () { for char in Ж З И Й К Л М ж з и й к л м; do echo $char; done; }

I also exported the appropriate collate setting:

export LC_COLLATE=ru_RU.UTF-8

Without character ranges everything worked as expected:

rus | grep -v "[АБВГДЕЁЖЗИКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯабвгдеёжзийклмнопрстуфхцчшщъыьэюя]"

and

rus | grep -v "[АБВГДЕЁЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯабвгдеёжзиклмнопрстуфхцчшщъыьэюя]"

output 'Й' and 'й' respectively.

With character ranges, [А-ИК-ЯЁа-ик-яё] should match all Russian characters except 'Й' and 'й', and this turned out to be correct. But when I wanted to filter only 'Й' or only 'й', something interesting happened: on my system, both

rus | grep -v "[А-ЯЁа-ик-яё]"  # expected output: 'й'

and

rus | grep -v "[А-ИК-ЯЁа-яё]"  # expected output 'Й'

output nothing!

'Й' and 'й' are not special in this respect; the analogical experiment with letters 'П' and 'п' showed the same effect.

Is grep maybe, for some reason, handling Russian or Cyrillic characters case-insensitively by default in character ranges? No, it is not: adding --no-ignore-case to all those grep commands changed nothing.

What's going on? Have I found a bug in grep? Or am I missing something?

(I am using GNU grep 3.11 (built with pcre), and bash 5.1.16.)

First, you should quote the argument to grep; if you don't, and you have a file in the current directory whose name is a single Russian letter, that letter will be the only thing passed to grep.

But the problem is that grep without PCRE appears to work bytewise, regardless of locale settings. So I think you need to turn on Perl-compatible mode with -P:

$ rus | grep -Pv '[А-ЯЁа-ик-яё]'
й

Whenever you suspect problems interpreting the argument to grep, a good sanity check is to fall back to sending it pure-ASCII strings, using the \x{...} syntax for non-ASCII characters (which is also a feature of PCRE, so only works with -P):

$ rus | grep -Pv '[\x{0410}-\x{042f}Ёx\{0430}-\x{0438}\x{043a}-\x{044f}ё]'
й

If you want to match characters of the Cyrillic alphabet, you can use a perl Unicode script regex:

rus | grep -P "\p{Cyrillic}"

should output only Cyrillic characters

if speed is of any concern, it's much faster just to use this chain of octal codes and scan rows for Cyrillic at the byte level :

7.115 secs for gnu-grep vs. 3.137 secs mawk2 (2.27x)

(there are no space/line gaps within the regex - I added them here only for formatting)

( time  ( pvE0 < "$m3t" | 

  mawk2 '/([0-3]|21)[0-7]|
            4     [0-7]|
            1(2[0-0]|43|[57]0)|
            2(7[0-7]|13)|
            2 2[0-7]|
            7 0[67]/' ) | pvE9 )

  in0: 1.85GiB 0:00:03 [ 606MiB/s] [ 606MiB/s] [========>] 100%            
 out9: 60.7KiB 0:00:03 [19.5KiB/s] [19.5KiB/s] [<=> ]
( pvE 0.1 in0 < "$m3t" | mawk2 ; )  

3.03s user 0.45s system 111% cpu 3.137 total

e23cb29c6564c3da39a67c0cb320f323  stdin

 out9: 60.7KiB 0:00:07 [8.56KiB/s] [8.56KiB/s] [ <=> ]
  in0: 1.85GiB 0:00:07 [ 266MiB/s] [ 266MiB/s] [========>] 100% 
       
( pvE 0.1 in0 < "$m3t" | ggrep -P '\p{Cyrillic}'; )  

7.02s user 0.46s system 105% cpu 7.115 total

e23cb29c6564c3da39a67c0cb320f323  stdin