How to identify and filter out unicode characters falling in the range of 40000–​DFFFF from a String in Java

Is there any way in java to find out if a character falls in between plane4 to plane13 of Unicode. The range of plane4 to plane13 as per https://en.wikipedia.org/wiki/Unicode_block is - 40000–​DFFFF

In this code I am trying assign a hex value to a char, but when I cast it back to int, then I don't get that same int value. The decimal form of DFFFF is 917503. But on casting the char back to an int I get the decimal value 65535. Not sure why the value is getting changed on casting the char back to int. Can someone please give me some idea on this. As per unicode the range 40000–​DFFFF is undefined currently. Is that the reason for this strange behaviour ?

Actually the use-case I want to implement is to filter out any characters from the input string if they fall with-in the range of 40000–​DFFFF.

Is there any opensource library that does this out of the box ? Appreciate if any help can be provided on this.

int intHex = 0xDFFFF;
	char c = (char)intHex;
	System.out.println((int)c);

Thanks

See the documentation of the Character class:

> The char data type (and therefore the value that a Character object
> encapsulates) are based on the original Unicode specification, which
> defined characters as fixed-width 16-bit entities. The Unicode
> Standard has since been changed to allow for characters whose
> representation requires more than 16 bits. The range of legal code
> points is now U+0000 to U+10FFFF, known as Unicode scalar value.
> (Refer to the definition of the U+n notation in the Unicode Standard.)
>
> The set of characters from U+0000 to U+FFFF is sometimes referred to
> as the Basic Multilingual Plane (BMP). Characters whose code points
> are greater than U+FFFF are called supplementary characters. The Java
> platform uses the UTF-16 representation in char arrays and in the
> String and StringBuffer classes. In this representation, supplementary
> characters are represented as a pair of char values, the first from
> the high-surrogates range, (\uD800-\uDBFF), the second from the
> low-surrogates range (\uDC00-\uDFFF).
>
> A char value, therefore, represents Basic Multilingual Plane (BMP)
> code points, including the surrogate code points, or code units of the
> UTF-16 encoding. An int value represents all Unicode code points,
> including supplementary code points. The lower (least significant) 21
> bits of int are used to represent Unicode code points and the upper
> (most significant) 11 bits must be zero. Unless otherwise specified,
> the behavior with respect to supplementary characters and surrogate
> char values is as follows:
>
> The methods that only accept a char value cannot support supplementary characters. They treat char values from the surrogate
> ranges as undefined characters. For example,
> Character.isLetter('\uD840') returns false, even though this specific
> value if followed by any low-surrogate value in a string would
> represent a letter.
> The methods that accept an int value support all Unicode characters, including supplementary characters. For example,
> Character.isLetter(0x2F81A) returns true because the code point value
> represents a letter (a CJK ideograph).

This means that a single char is not enough because it only has 16 bits. You need an int value to hold larger values. 65535 = 2^16 is the maximum value you can save in an unsigned 16 bit integer datatype.

Welcome to the world of UTF-16, a major accident in the IT industry.

Many operating systems and programming languages have defined the character type to be 16 bit long when it was already obvious that 16 bit is far from sufficient to represent all the letters used on Earth. Java is one of them.

Unicode has evolved in the mean time. It requires 32 bit to represent what is colloquially known as a letter and it is called codepoint. For compatibility reasons, Java could not change the char type from 16 bit to 32 bit. Instead, they left it at 16 bit and redefined it to be UTF-16 encoded (instead of a direct UCS-2) representation.

Short story is: A code point such as U+DFFFF requires more than 16 bit and cannot be represented in a single char. So switch from char to code points, which are represented as int in Java:

final int length = s.length();
for (int offset = 0; offset < length; ) {
   final int codepoint = s.codePointAt(offset);
   if (codepoint >= 0x40000 && codepoint <= 0xdffff) {
       // do something with the codepoint
   }
   offset += Character.charCount(codepoint);
}

    int intHex = 0xDFFFF; // This is called a Unicode code point.

    char[] chars = Character.toChars(intHex); // In this case a pair.
    String s = new String(chars);
    int[] codePoints = new int[] {intHex};
    String t = new String(codePoints, 0, codePoints.length);

Chars in java are in UTF-16, and ensure (like UTF-8) that a sequence for a Unicode symbol, a code point, does not contain a mistakable other char in a sequence (surrogate pair of chars).

Best work with String, and extract code points.

    int[] codePoints = s.codePoints().filter(cp -> cp < 0x40000).toArray();
    String t = new String(codePoints, 0, codePoints.length);