Encoding md5 in a node compatible way

I'm converting a node service to go. For this I need a compatible md5 hash (not for storing passwords!!) generator. However, in this example, I keep getting different results:

Node's crypto takes an encoding parameter when creating md5s.

> crypto.createHash("md5").update("1Editor’s notebook: Escaping temptation for turf145468066").digest("hex")
'c7c3210bd977b049f42c487b8c6d0463'

In golang: (test_encode.go)

package main

import (
	"crypto/md5"
	"encoding/hex"
	"testing"
)

func TestFoo(t *testing.T) {
	const result = "c7c3210bd977b049f42c487b8c6d0463"

	stringToEncode := "1Editor’s notebook: Escaping temptation for turf145468066"
	hash := md5.Sum([]byte(stringToEncode))
	hashStr := hex.EncodeToString(hash[:])

	if hashStr != result {
		t.Error("Got", hashStr, "expected", result)
	}
}

Then go test test_encode.go results in:

--- FAIL: TestFoo (0.00s)
	encode_test.go:17: Got c3804ddcc59fabc09f0ce2418b3a8335 expected c7c3210bd977b049f42c487b8c6d0463
FAIL
FAIL	command-line-arguments	0.006s

I've tracked it down to the encoding parameter of crypto.update in the node code. And the fact that the string as a ’ quote character in it. If I specify "utf8" it works.

crypto.createHash("md5").update("1Editor’s notebook: Escaping temptation for turf145468066", "utf8").digest("hex")

BUT: I can't change the node code, so the go code has to be compatible. Any ideas on what to do?

As you've already noted: you must convert the UTF8 string to whatever encoding is used in your node application. This can be done with encoding packages such as:

golang.org/x/text/encoding/charmap

isoString, err := charmap.ISO8859_1.NewEncoder().Bytes([]byte(stringToEncode))

Considering that the character ’ is not allowed in iso-8859-1, we can assume you have a different encoding. Now you just need to figure out which one!

And in worse case, you might have to use another package than charmap.

After a lot of digging in node and V8 I was able to conclude the following:

require("crypto").createHash("md5").update(inputString).digest("hex");

Is pretty dangerous, as not specifying a encodes the input string as "ASCII". Which, after a lot of digging, is the equivalent (verified on a large input set from my end):

// toNodeASCIIString converts a string to a byte of node compatible ASCII string
func toNodeASCIIString(inputString string) []byte {
	lengthOfString := utf8.RuneCountInString(string(inputString))
	stringAsRunes := []rune(inputString)

	bytes := make([]byte, lengthOfString)
	for i, r := range stringAsRunes {
		bytes[i] = byte(r % 256)
	}

	return bytes
}

What is basically does is mods by 256 and forgets a large part of the input string.

The node example above is pretty much the standard and copy-pasted-everywhere way to create MD5 hashes in node. I have not checked but I'm assuming this works the same for all other hashes (SHA1, SHA256, etc).

I would love to hear someones thoughts on why this is not huge security hole.