Escaping of hex values in string literals

I am attempting to escape a specific hex value in a golang string. The function call looks something like this:

Insert(0, "\x00\x00\x00\rIHDR\x00\x00\x000\x00\x00\x000\b\x03")
Insert(25, "\x00\x00\x00\x06PLTE")
Insert(43, "\x00\x00\x00\x02tRNS")
Insert(57, "\x00\x00\t;IDATx\xDA\x010\t\xCF\xF6") // problem line
Insert(2432, "\x00\x00\x00\x00IEND")

The problem arises when the language interprets the "\xDA" hex escape. Instead of correctly escaping to a Ú value, it is escaped to � (the Replacement Character).

I ensured this is what was occuring in the following playground example:

fmt.Println("\xDA")
i := 218
h := fmt.Sprintf("%x", i)
fmt.Printf("Hex conf of '%d' is '%s'\n", i, h)	
fmt.Println(string(i))

This snippet, when run, prints

�
Hex conf of '218' is 'da'
Ú

Am I missing something here? The fact that "\xDA" is being escaped to a value of 65533 is throwing off my entire program, which relies on the CRC32 and some other checksums. This does not occur in the javascript version of this program (which itself is a translation from James compface program, written in C).

Here is the playground link: https://play.golang.org/p/c-XMK68maX

Go strings are just a series of bytes, but when an encoding is needed, it's assumed to be utf8. The value \xda isn't a valid utf8 character, so when printing it's converted to the unicode.ReplacementCharacter "�"

    ReplacementChar = '\uFFFD'     // Represents invalid code points.

If you want the rune value of \xda in a string literal, use a unicode escape: \u00DA, or use the utf8 encoded: \xc3\x9a, or use the character itself: Ú.

https://play.golang.org/p/EJZIqCI_Gr

If you actually want a single byte value of \xda in your string, that is what you have and the printed character is inconsequential.

Your input looks like ISO-8859-1 (Latin-1). Convert it to UTF-8. For example,

package main

import (
	"fmt"
	"unicode/utf8"
)

// ISO88591ToString maps ISO-8859-1 (Latin-1) to string (UTF-8).
func ISO88591ToString(iso string) string {
	var utf []rune
	for i := 0; i < len(iso); i++ {
		r := iso[i]
		if utf == nil {
			if r < utf8.RuneSelf {
				continue
			}
			utf = make([]rune, len(iso))
			for j, r := range iso[:i] {
				utf[j] = rune(r)
			}
		}
		utf[i] = rune(r)
	}
	if utf == nil {
		return string(iso)
	}
	return string(utf)
}

func main() {
	l1 := "\x00\x00\t;IDATx\xDA\x010\t\xCF\xF6"
	fmt.Printf("%q\n", l1)
	s := ISO88591ToString(l1)
	fmt.Printf("%q\n", s)
}

Output:

"\x00\x00\t;IDATx\xda\x010\t\xcf\xf6"
"\x00\x00\t;IDATxÚ\x010\tÏö"

Strings in Go are UTF-8, and \xDA isn't a valid UTF-8 sequence by itself, meaning printing it as a part of a string will yield the Unicode replacement character U+FFFD instead of what you wanted (Ú, or U+00DA).

You seem to be working with raw bytes, however, so you should consider whether you want the rune represented by \u00DA, which is encoded in UTF-8 as the 2-byte sequence \xC3\x8F, or whether you require the single byte \xDA. The former will print Ú as you want with the caveat that it requires 2 bytes. The latter will not print as you expect, yet it will correctly be interpret \xDA as 1 byte rather than 2 bytes.

Here's an illustrative example you can run on the Playground:

func main() {
	// A string made up of UTF-8 lead bytes.
	dataString := "\xCF\xDA\xF6"

	// Doesn't print what you think it should.
	for _, c := range dataString {
		fmt.Printf("%X ", c)
	}
	fmt.Println()

	// Convert the string's bytes to a byte slice.
	data := []byte(dataString)

	// Now it should print CF, DA, F6.
	for _, b := range data {
		fmt.Printf("%X ", b)
	}
	fmt.Println()
}