LP3THW ex17 Why is there garbled text at the end of a text file when copying files with python in powershell?

I am following Learn Python 3 the hard way, and I am on example 17.

I typed the code in the book exactly (comments included) and then ran the program in Powershell.

The file size of the text file is 46 bits.

This is where my output differs from the book. (besides the weird gunk) the books output says the file is 21 bits long.

I created the file with this command (also from the book.)

echo "This is a test file." > test.txt

This is a direct copy paste.

Contents of test.txt (is 2 lines):

This is a text file.

contents of text1.txt (is 2 lines):

This is a text file.਍ഀ

So including the returns there's a bit extra gunk at the end of the first line of the copied file.

Here's the code I used.

from sys import argv
from os.path import exists

script, from_file, to_file = argv

print(f"Copying from {from_file} to {to_file}")

# we could do these two on one line, how?
in_file = open(from_file)
indata = in_file.read()

print(f"The input file is {len(indata)} bytes long")

print(f"Does the output file exist? {exists(to_file)}")
print("Ready, hit RETURN to continue, CTRL-C to abort.")
input()

out_file = open(to_file, 'w')
out_file.write(indata)

print("Alright, all done.")

out_file.close()
in_file.close()

And Here's the PowerShell commands and result.

PS D:\Pythonlearn\lpthw> python ex17cp.py test.txt test1.txt
Copying from test.txt to test1.txt
The input file is 46 bytes long
Does the output file exist? True
Ready, hit RETURN to continue, CTRL-C to abort.

Alright, all done.
PS D:\Pythonlearn\lpthw> cat test1.txt
This is a text file.਍ഀ
PS D:\Pythonlearn\lpthw> cat test.txt
This is a text file.

The book assumes Python3.6. I'm using 3.9.13 in hopes I'll be able to resolve any problems I come across. However I cant find anything I understand about this problem online. I can't even recognize if what I'm looking at is related to this problem. No matter what keywords I use.

I'd like four answers please.

1: Is this a Python or a PowerShell problem?

2: How can I fix the code so it doesn't do this.

3: Why does that fix the problem.

4: What caused the problem in the first place?

<!-- language-all: sh -->

In Windows PowerShell,

"This is a test file." > test.txt

produces an output file that uses "Unicode" (UTF-16LE) encoding, because the > operator is, in effect, an alias of the Out-File cmdlet. (Note that - more sensibly - PowerShell (Core) now defaults to BOM-less UTF-8 encoding, across all cmdlets.)

Few applications and non-PowerShell APIs recognize this encoding by default, and python is no exception: it expects "ANSI" encoding by default, i.e. the encoding specified by the system's active ANSI legacy code page (which is in itself a deviation from what console applications are expected to do, namely to use the system's active OEM legacy code page).

Therefore, python misinterprets the content of file test.txt and treats each byte as its own character (whereas in UTF-16LE a single character is encoded by (at least) two bytes).

While Python mostly preserves the input bytes as-is and therefore also passes them through on writing, it applies special newline handling, which is the root of the problem and effectively results in a corrupted output file:

• On encountering what python thinks is a stand-alone CR or LF character, it translates it to a Windows-appropriate CRLF newline sequence.

• The NUL bytes that result from the misinterpretation of the UTF-16LE-encoded file cause it to not recognize the 00 0D 00 0A byte sequence from the input file as a CRLF character sequence, and therefore causes it to transform it to byte sequence 00 0D 0A 00 0D 0A (0D and 0A each where translated to ANSI 0D 0A byte sequences), which is what caused the file corruption:

  • When PowerShell - which is UTF-16LE-aware based on an input file's BOM (Unicode signature) - tries to interpret the resulting file, the accidental UTF-16LE byte sequences in the rewritten file resulting from the newline "fix" turned into arbitrary Unicode characters.

Solutions:

• Either: Create test.txt on the PowerShell side with "ANSI" encoding:

  • In Windows PowerShell, simply use Set-Content, which uses that encoding by default:

    "This is a test file." | Set-Content test.txt
    

  • In PowerShell (Core) 7+, the solution is more complicated, unfortunately:

    "This is a test file." |
      Set-Content -Encoding ([cultureinfo]::CurrentUICulture.TextInfo.AnsiCodePage) test.txt
    

    • This convoluted way of requesting "ANSI" encoding shouldn't be necessary, which is the subject of GitHub issue #6562.

• Or: Make Python explicitly use UTF-16LE encoding:

  • Add argument encoding='utf-16le' to the open() calls.