Special Character Issue : MQ message PUT error : java.nio.charset.UnmappableCharacterException

I have a setup where there is a JMS Producer and JMS receiver. The sender application sends a message like :

source text ⟨е, ё, и, ю, я⟩  abcdefg

JMS receiver after receiving the message, puts it to a IBM MQ queue using pure IBM MQ API classes.

While putting this message to MQ I am getting the below exception:

INFO  | 2020-09-17 09:45:19 | [main] mimq.MQReceiver (MQReceiver.java:211) - IO Exception Occurred: Input length = 1
java.nio.charset.UnmappableCharacterException: Input length = 1
        at java.nio.charset.CoderResult.throwException(CoderResult.java:282)
        at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:816)
        at com.ibm.mq.jmqi.system.JmqiCodepage.stringToBytes(JmqiCodepage.java:923)
        at com.ibm.mq.MQMessage.writeString(MQMessage.java:2848)
        at com.ibm.mimq.MQReceiver.sendToAnotherQueue(MQReceiver.java:192)
        at com.ibm.mimq.MQReceiver.main(MQReceiver.java:113)

Below is my MQ PUT code :

public static void sendToLocalQueue(String msg) {
	
	int port = 1414;
	String host = "some-host";
	String channel = "some-channel";
	String manager = "some-QM";
	String user = "user"; 
	String passwd = "passwd";
	String qname = "TEST";
	String qmname = "some-QM";
	
	MQQueueManager qMgr;
	MQQueue inputQ;
	
	try {		
		
		Hashtable<String, String> h = new Hashtable<String, String>();
		h.put(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES_CLIENT);
		MQEnvironment.properties = h;
		MQEnvironment.hostname	= host;
		MQEnvironment.port 		= port;
		MQEnvironment.channel	= channel;
		
		MQEnvironment.userID	= user;
		MQEnvironment.password	= passwd;
		MQEnvironment.disableTracing();
		MQException.log = null;
		qMgr = new MQQueueManager(manager);
		
		MQMessage m = new MQMessage();		
		m.applicationOriginData = "AMPS";
		m.messageType 	= MQC.MQMT_DATAGRAM;
		m.format 		= MQC.MQFMT_STRING;
		m.encoding 		= MQC.MQENC_NATIVE;
		m.priority 		= 4;
		m.persistence 	= MQC.MQPER_PERSISTENT;
		m.characterSet 	= MQC.MQCCSI_Q_MGR;
		//m.characterSet = 1208;
		m.expiry 		=  MQC.MQEI_UNLIMITED;
		m.writeString(msg);
		
		MQPutMessageOptions putOptions = new MQPutMessageOptions();
		putOptions.options = MQC.MQPMO_SYNCPOINT | MQC.MQPMO_FAIL_IF_QUIESCING;	
		
		logger.info("Putting message to LAN MQ (TEST queue)....");			
		qMgr.put(qname, qmname, m, putOptions);
		qMgr.commit();
		
		
	} catch(MQException me) {
		logger.info("Error Code       : "  +me.getErrorCode());
		logger.info("LocalizedMessage : "  +me.getLocalizedMessage());
		logger.info("Message          : "  +me.getMessage());
		logger.info("Reason           : "  +me.getReason());
		me.printStackTrace();
		
	} catch (IOException e) {
		// TODO Auto-generated catch block
		logger.info("IO Exception Occurred       : "  +e.getLocalizedMessage());
		e.printStackTrace();
	}		
}

It is unable to put the message to the queue due to Unmappable character. The encoding is set to UTF-8 at Queue Manager level.

However when I replace the below line : m.characterSet = MQC.MQCCSI_Q_MGR;

With the line : m.characterSet = 1208; The issue is no more there.

My question is why this conversion is not getting done at MQ level. What are settings I need to check to ensure the correct conversion. I have tried the below techniques, but not working :

Setting java parameter as : -Dfile.encoding=UTF-8 

My environment :

Server : Linux
MQ : 9.0  or 7.5
Java : 1.8

One more thing to mention, the same setup was working with 7.5 but not working with MQ 9.0 after migration. I know with my above one line code change I can pass the message. But I want to understand in MQ level, if I am missing out some configurations. Any advise would be much appreciated.

Thank You.

UPDATE

The Client Machine from where I am sending the Message over to MQ has CCSID : MQMD.CodedCharSetId = 1208

The MQ server where I am connecting and sending the message has this :

getDefaultProperty(Object) returns [819(0x333)] Integer
setCCSID(int) setter [819(0x333)]

So when I am setting 1208 explicitely in my code it is working. When not the conversion is failing.

UPDATE-2

The value MQC.MQCCSI_Q_MGR is Zero as I saw in the jar. Hence the code is designed like this, if the value is Zero, it will fetch the default value from the Jar which is set to 819. I learnt this when I turned on the MQ tracing. And the code is like this :

getDefaultProperty(Object) returns [819(0x333)] Integer
setCCSID(int) setter [819(0x333)]

This code is present inside the jar. So we need to explicitly set the charset value on the message. In my case it is 1208.

When connected locally to the queue manager (i.e. using TRANSPORT_MQSERIES_BINDINGS) then the code:-

m.characterSet     = MQC.MQCCSI_Q_MGR;

means "take the CCSID set in the queue manager property CCSID". You can see this property using the following MQSC command:-

DISPLAY QMGR CCSID

When connected as a client (as your code shows you are), then the code:-

m.characterSet     = MQC.MQCCSI_Q_MGR;

means "find the CCSID from the client machine locale".

IBM Knowledge Center states:-

> For client applications, MQCCSI_Q_MGR is filled in, based on the locale of the client rather than the one on the queue manager.

It would seem that your client machine locale is not set to UTF-8 if changing the line of code to explicitly set the message CCSID to 1208 fixes the problem.

You can see what is set by the client, by browsing the message on the queue (without converting it) and looking what is in the MQMD.CodedCharSetId field.

The reason why this worked at MQ v7.5 and not at MQ v9.0 is because prior to IBM MQ v8.0 the IBM MQ classes for Java encoded data using java.nio.charset.Charset.encode(CharBuffer) which results in a default replacement of malformed or untranslatable data. Since the default characterSet is 819 (ASCII), this would result in any character you send that can not be converted to ASCII to be transparently replaced with the default replacement character, in most cases this means the data was replaced with the ? character.

After v8.0 the default behavior changed to report this situation as an error and no longer replace malformed or untranslatable data by default.

Your solution to set the character set to UTF-8 is the best solution as this results in the exact data you intended to send being sent.

Another option is to tell MQ to use the prior behavior.

The description of the new behavior as well as how to configure IBM MQ classes for Java to use the prior behavior is described in the IBM MQ 9.0 Knowledge Center page Developing applications>Developing JMS and Java applications>Using IBM MQ classes for Java>Character string conversions in IBM MQ classes for Java:

> From IBM® MQ Version 8.0, some of the default behavior regarding
> character string conversion in the IBM MQ classes for Java™ has
> changed.
>
> Before IBM MQ Version 8.0, string conversions in IBM MQ classes for
> Java was done by calling the
> java.nio.charset.Charset.decode(ByteBuffer) and
> Charset.encode(CharBuffer) methods.
>
> Using either of these methods results in a default replacement (
> REPLACE) of malformed or untranslatable data.
>
> This behavior can obscure errors in applications, and lead to
> unexpected characters, for example ?, in translated data. From IBM MQ
> Version 8.0, to detect such issues earlier and more effectively, the
> IBM MQ classes for Java use CharsetEncoders and CharsetDecoders
> directly and configure the handling of malformed and untranslatable
> data explicitly.
>
> From IBM MQ Version 8.0, the default behavior is to REPORT such issues
> by throwing a suitable MQException.
>
> ...
>
> Setting system defaults
> --
>
> From IBM MQ Version 8.0, the following two Java system properties are
> available to configure default behavior regarding character string
> conversion.
>
> com.ibm.mq.cfg.jmqi.UnmappableCharacterAction Specifies the action to be taken for untranslatable data on encoding and decoding.
> The value can be REPORT, REPLACE, or IGNORE.
>
> com.ibm.mq.cfg.jmqi.UnmappableCharacterReplacement Sets or gets the replacement bytes to apply when a character cannot be mapped
> in an encoding operation The default Java replacement string is used
> in decoding operations.
>
> To avoid confusion between Java character and native byte
> representations, you should specify
> com.ibm.mq.cfg.jmqi.UnmappableCharacterReplacement as a decimal number
> representing the replacement byte in the native character set.
>
> For example, the decimal value of ?, as a native byte, is 63 if the
> native character set is ASCII-based, such as ISO-8859-1, while it is
> 111 if the native character set is EBCDIC.

If you wanted to mimic the prior behavior you would set the following system properties:

-Dcom.ibm.mq.cfg.jmqi.UnmappableCharacterAction=REPLACE
-Dcom.ibm.mq.cfg.jmqi.UnmappableCharacterReplacement=63

You can also set it Programmatically with something like this:

System.setProperty("com.ibm.mq.cfg.jmqi.UnmappableCharacterAction", "REPLACE");
System.setProperty("com.ibm.mq.cfg.jmqi.UnmappableCharacterReplacement", "63");