How to determine an XML file's encoding and convert non-standard characters to a readable (UTF-8?) format?

I am rather new to XML file encoding and am trying to wrap my head around a few things.

My company receives two data feeds (both in XML format) from two separate external vendors. The data from these XML files is run through our internal ETL software and then pushed to some of our internal software.

The issues I am having appear to be encoding-related and the presence of non-standard characters.

XML FILE 1

• The vendor's software is ancient, and so the process they use to generate the XML files likely is too.
• The XML file has no XML declaration tag at the top.
• When I open the XML file in Sublime the console shows: "unable to auto detect encoding for C:\file1.xml, using fallback encoding Western (Windows 1252)"
• Opening the XML file in a browser results in: "error on line 500 at column 30: Encoding error"
• Our ETL software jobs fail when attempting to process any nodes with non-standard characters.
• Below is a screenshot of the expected character, plus what I see when opening the XML file in various text editors:

xml_file_1

XML FILE 2

• The vendor's software is ancient, and so the process they use to generate the XML files likely is too.
• The XML file has this declaration tag at the top: <?xml version="1.0" standalone="yes"?>
• When I open the XML file in Sublime the console shows no error of any kind.
• Opening the XML file in a browser results in no visible errors.
• Our ETL software jobs run fine, but insert the improperly encoded character rather than the expected character.
• Below is a screenshot of the expected character, plus what I see when opening the XML file in various text editors:

xml_file_2

QUESTIONS

1. How can I determine the true encoding used in an XML file?

2. Our ETL software lets me specify character encoding for source files (in this case the XML files) as well as the eventual targets (our internal software), so I assume once I am able to determine the encoding used on the file I will be able to transform these non-standard characters to their UTF-8 equivalents?

3. Would I be stupid to ask the vendors what encoding they are using when creating XML files?

Thanks in advance for any guidance!

>How can I determine the true encoding used in an XML file?

In the general case, the answer is that you can't. For example, if all the octets in the binary file are in the ASCII range x20-x7F, then the file might be ASCII, or it might be any of the national variants of ASCII that assign some of the codepoints differently (for example the UK variant has £ where US ASCII has #). In fact national variants are rather a rarity nowadays, especially with XML, but if your XML is created on some ancient mainframe that's configured that way, then it's by no means impossible. However, I use this example primarily just to demonstrate that in the general case, there is no way of inferring the encoding by simply looking at the octets in the file. The same reasoning applies to detecting regional variants of ISO 8859, for example the Cyrillic variant, which most definitely is in current use.

Fortunately it's not quite that bleak in practice; there are tools that are able to make a fairly reasonable guess, or at least to find a candidate decoding that doesn't crash, even if it decodes some characters incorrectly.

In your situation, however, the best clue to inferring an encoding might be knowing who sent you the file. There's a good chance that all the files from a particular origin use the same encoding. Standard tools won't know that, but you probably do.

>Would I be stupid to ask the vendors what encoding they are using when creating XML files?

Absolutely not. It's a perfectly reasonable expectation that if they're prepared to send you data, they should be prepared to tell you how to read it. Whether the vendors actually have the brain-power to understand and answer the question is another matter. My only experience of this is a vendor who had no understanding at all of XML or encodings, and was grateful for my advice and used it to fix their software.

There are a variety of libraries, both open source and not, which detect encodings. Posting recommendations here is not encouraged, so the best I can do is recommend that you search for 'encoding detection', not mentioning XML. You might find something adequate built into Java or Windows.