Reading and writing unicode characters to file in C

I am trying to create a program that saves unicode characters in a text file (named data.txt below) and when I load the file again they will be correctly printed in the terminal. This is my code:

#if defined(_MSC_VER) || defined(__MINGW32__) || defined(__MINGW64__)
#define ON_WINDOWS
#endif

#ifdef ON_WINDOWS
#define _CRT_SECURE_NO_WARNINGS
#include <io.h>     // _setmode
#include <fcntl.h>  // _O_U16TEXT

// just in case mingw doesn't define it after all:
#ifndef _O_U16TEXT
#define _O_U16TEXT (0x20000)
#endif
#endif

#include <stdio.h>
#include <locale.h>
#include <wchar.h>
#define SIZE 100

void set_locale_mode() {
#ifdef ON_WINDOWS
   // Unicode UTF-16, little endian byte order (BMP of ISO 10646)
   const char* CP_UTF_16LE = ".1200";

   setlocale(LC_ALL, CP_UTF_16LE);
   _setmode(_fileno(stdin), _O_U16TEXT);
   _setmode(_fileno(stdout), _O_U16TEXT);
#else
   setlocale(LC_ALL, "");
#endif
}

int main(void) {

   set_locale_mode();

   wchar_t myString[SIZE];
   wchar_t loadedString[SIZE];

   wprintf(L"Enter 3 characters: ");
   wscanf(L"%ls", myString);
   wprintf(L"Your input is %ls\n", myString);

   FILE *pFile;

   if(pFile=fopen("data.txt", "w")) {
      fwprintf(pFile, L"%ls", myString);
   } else {
      wprintf(L"Failed to write to file!\n");
   }

   fclose(pFile);

   if(pFile=fopen("data.txt", "r")) {
      fwscanf(pFile, L"%ls", loadedString);
   } else {
    wprintf(L"Failed to read from file!\n");
   } 

   fclose(pFile);

   wprintf(L"Loaded string is %ls", loadedString);

   return 0;
}

The following 2 examples work as intended:

First one:

Enter 3 characters: abc
Your input is abc
Loaded string is abc

Second one:

Enter 3 characters: åäö
Your input is åäö
Loaded string is åäö

However, in the third attempt, I copied some chinese characters and copied them into the terminal and this was the outcome:

Enter 3 characters: 买买买
Your input is 买买买
Loaded string is 炨}ﺨa眬☺

The loaded string is obviously not what I intended. Why does it not work and how do I fix it?

Edit: I am on Windows and uses MINGW32 and uses VSCode.

In this case, since you open two new streams, you need to set them in the proper mode too:

if (pFile = fopen("data.txt", "w")) {
#ifdef ON_WINDOWS
    _setmode(_fileno(pFile), _O_U16TEXT);      // <- here
#endif
    fwprintf(pFile, L"%ls", myString);
} else {
    wprintf(L"Failed to write to file!\n");
}

fclose(pFile);

if (pFile = fopen("data.txt", "r")) {
#ifdef ON_WINDOWS
    _setmode(_fileno(pFile), _O_U16TEXT);      // <- here
#endif
    fwscanf(pFile, L"%99ls", loadedString);
} else {
    wprintf(L"Failed to read from file!\n");
}

Also note the 99 in the L"%99ls". If you use the scanf family of functions to read words from streams, always set the max number of characters to read at most one less than the size of the buffer.

On Windows, you can also open the files with the correct mode, UTF-16LE, directly:

#ifdef ON_WINDOWS
if (pFile = fopen("data.txt", "w, ccs=UTF-16LE")) { // <- here
#else
if (pFile = fopen("data.txt", "w")) {
#endif
    fwprintf(pFile, L"%ls", myString);
} else {
    wprintf(L"Failed to write to file!\n");
}

fclose(pFile);

#ifdef ON_WINDOWS
if (pFile = fopen("data.txt", "r, ccs=UTF-16LE")) { // <- here
#else
if (pFile = fopen("data.txt", "r")) {
#endif
    fwscanf(pFile, L"%99ls", loadedString);
} else {
    wprintf(L"Failed to read from file!\n");
}

For the program's purposes, there are three important locuses for character handling:

• the terminal
• the program
• the file

Each one will have a character encoding by which it represents characters as numbers, and a set of characters that they recognize. These are not necessarily the same for the various locuses, but it's convenient if they are.

If the point is to capture character data from the terminal and ultimately to echo it back to the terminal, then it's important to understand that how the program must handle that is sensitive to the encoding the terminal is using, and that that is outside the program's control. In particular, the program setting its locale does not influence the form in which the terminal will deliver character data to it, but it does influence how your program interprets the data from the terminal. If the terminal and your program disagree about that then data corruption is a likely result.

On the other hand, your particular program doesn't need to be concerned with what specific encoding the terminal is using -- it just needs to store the data it receives and then echo it back accurately.

If the terminal is not operating in a wide-character mode, then trying to read from it or write to it with wide-oriented I/O functions is likely to mess you up. And wide-character terminal modes are unusual these days, because UTF-8-based modes, which are byte-oriented, are pretty much universally available, and are more convenient to work with for most purposes. If you can rely on the terminal running in a byte-oriented mode (it doesn't matter which in particular) then you're going way out of your way. In that case, you probably don't need to do anything special -- Unicode characters your terminal handles at all should just work:

#if defined(_MSC_VER) || defined(__MINGW32__) || defined(__MINGW64__)
#define _CRT_SECURE_NO_WARNINGS
#endif

#include <stdio.h>
#define SIZE 100

int main(void) {
   char myString[SIZE];
   char loadedString[SIZE];

   printf("Enter 3 characters: ");
   scanf("%99s", myString);
   printf("Your input is %s\n", myString);

   FILE *pFile;

   if (pFile = fopen("data.txt", "w")) {
      fprintf(pFile, "%s", myString);
   } else {
      printf("Failed to open file for writing!\n");
   }
   fclose(pFile);

   if (pFile = fopen("data.txt", "r")) {
      fscanf(pFile, "%s", loadedString);
   } else {
      printf("Failed to open file for reading!\n");
   } 
   fclose(pFile);

   printf("Loaded string is %s", loadedString);

   return 0;
}

The main thing that changes if your terminal is operating in a wide-character mode is that the wide-character streams are likely contain a lot of individual nul bytes, which would be interpreted as string terminators in an ordinary C string. That's the only reason why the approach above would not be expected to work in conjunction with such a terminal mode.

If your terminal is running in a wide-character mode, on the other hand, then your easiest way forward is probably to just change to wide strings and wide I/O universally, but there should be no reason to change locale or modify stream modes.

If you want to handle both wide and narrow terminal modes then you would want to check, not set, the locale to try to figure out which you're dealing with. You would then use that to choose between narrow style and wide style operation.