Go image bytes and Python PIL tobytes produce different byte data

I have taken over some code that was previously in python written by someone else, and no is in go, written by myself.

Part of this code was to open an image, read its data, and make an MD5 hash to prevent duplicates.

The python version of this code used PIL:

from PIL import Image
f = Image.open('2d16395b-da48-11eb-8cbe-36a331f79a1e.png')
hashlib.md5(f.tobytes()).hexdigest()
'c699a448b38df386d036ed418d7714f3'

And the go version just read the bytes into an md5 hash

    f, _ := os.Open(p)
	_, err := f.Seek(0, io.SeekStart)
	if err != nil {
		fmt.Println(err)
	}

	h := md5.New()
	if _, err := io.Copy(h, f); err != nil {
		log.Fatal(err)
	}

	hb := h.Sum(nil)
	hash := hex.EncodeToString(hb)

However these produce different MD5.

It seems like the PIL library is readying the bytes differently, maybe stripping of header/metadata or something?

Does anyone know of a way i can replicate the byte-read of PIL in go to get the same MD5 hash?

I'm going to start with the (dummy) test image that I used:

which is a 32X16 (RGBA) .png with only 2 colors:

• Red (237, 28, 36)
• Blue (0, 0, 255)

Back to the question: [ReadTheDocs.Pillow]: Image.load() (called implicitly by open) processes (decodes) the image, yielding its raw bitmap data, which is totally different than the (encoded) file contents.

Here's the difference between the 2.

code00.py:

#!/usr/bin/env python

import sys
from PIL import Image
from hashlib import md5


def read_file_data(file_name):
    with open(file_name, "rb") as fin:
        return fin.read()


def read_img_data(file_name):
    with Image.open(file_name) as img:
        return img.tobytes()


def process_bytes(buf, first=20, last=20):
    print("Len: {:d}\nFirst bytes:\n  ".format(len(buf)), end=" ")
    for i in range(first):
        print("0x{:02X}".format(buf[i]), end=" ")
    print("\nLast bytes:\n  ", end=" ")
    for i in range(-last, 0, 1):
        print("0x{:02X}".format(buf[i]), end=" ")
    print("\nMD5: {:}".format(md5(buf).hexdigest()))


def main(*argv):
    img_name = "rb.png"
    funcs = [
        read_file_data,
        read_img_data,
    ]
    for func in funcs:
        print("\nFunction {:s}".format(func.__name__))
        b = func(img_name)
        process_bytes(b)


if __name__ == "__main__":
    print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
                                                   64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    rc = main(*sys.argv[1:])
    print("\nDone.")
    sys.exit(rc)

Output:

> lang-bat
> [cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q068231412]> sopr.bat
> ### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ###
>
> [prompt]> "e:\Work\Dev\VEnvs\py_pc064_03.08.07_test0\Scripts\python.exe" code00.py
> Python 3.8.7 (tags/v3.8.7:6503f05, Dec 21 2020, 17:59:51) [MSC v.1928 64 bit (AMD64)] 064bit on win32
>
>
> Function read_file_data:
> Len: 169
> First bytes:
> 0x89 0x50 0x4E 0x47 0x0D 0x0A 0x1A 0x0A 0x00 0x00 0x00 0x0D 0x49 0x48 0x44 0x52 0x00 0x00 0x00 0x10
> Last bytes:
> 0xE2 0x8A 0x24 0x69 0x53 0x4C 0xB3 0x03 0x00 0x00 0x00 0x00 0x49 0x45 0x4E 0x44 0xAE 0x42 0x60 0x82
> MD5: 8368b5c29a12b298cea2ad4b32955830
>
> Function read_img_data:
> Len: 2048
> First bytes:
> 0xED 0x1C 0x24 0xFF 0xED 0x1C 0x24 0xFF 0xED 0x1C 0x24 0xFF 0xED 0x1C 0x24 0xFF 0xED 0x1C 0x24 0xFF
> Last bytes:
> 0x00 0x00 0xFF 0xFF 0x00 0x00 0xFF 0xFF 0x00 0x00 0xFF 0xFF 0x00 0x00 0xFF 0xFF 0x00 0x00 0xFF 0xFF
> MD5: ebdf44b7ad36d79b221a70ea2b0fa0c7
>
> Done.
>

After some hours of research (trial and error, Googleing, reading docs and examples - I can't afford not mention here [SO]: Get a pixel array from from golang image.Image (@ArunaHerath's answer)), I was able to translate the above script to Go.

code00.go:

package main

import (
    "crypto/md5"
    "fmt"
    "image"
    "image/draw"
    "io/ioutil"
    "os"

    _ "image/gif"
    _ "image/jpeg"
    _ "image/png"
)

type ImageFunc func(string) []byte
type ImageFuncs []ImageFunc

func ReadFileData(fileName string) []byte {
    buf, _ := ioutil.ReadFile(fileName)
    return buf
}

func ReadImgData(fileName string) []byte {
    reader, _ := os.Open(fileName)
    defer reader.Close()
    img, _, _ := image.Decode(reader)
    rect := img.Bounds()
    rgba := image.NewRGBA(rect)
    draw.Draw(rgba, rect, img, rect.Min, draw.Src)
    //fmt.Printf("%v\n", rgba.Pix)
    return rgba.Pix
}

func ProcessBytes(buf []byte, first int, last int) {
    lb := len(buf)
    fmt.Printf("Len: %d\nFirst bytes:\n  ", lb)
    for i := 0; i < first; i++ {
        fmt.Printf("0x%02X ", buf[i])
    }
    fmt.Printf("\nLast bytes:\n  ")
    for i := lb - last; i < lb; i++ {
        fmt.Printf("0x%02X ", buf[i])
    }
    fmt.Printf("\nMD5: %x", md5.Sum(buf))
}

func main() {
    imgName := "rb.png"
    first := 20
    last := 20
    funcs := ImageFuncs{
        ReadFileData,
        ReadImgData,
    }
    for idx := range funcs {
        function := funcs[idx]
        fmt.Printf("\n\nFunction %#v:\n", function)
        b := function(imgName)
        ProcessBytes(b, first, last)
    }
    fmt.Printf("\n\nDone.\n")
}

Output:

> lang-bat
> [prompt]> "f:\Install\pc064\Google\GoLang\1.16.5\bin\go.exe" run code00.go
>
>
> Function (main.ImageFunc)(0x9f1060):
> Len: 169
> First bytes:
> 0x89 0x50 0x4E 0x47 0x0D 0x0A 0x1A 0x0A 0x00 0x00 0x00 0x0D 0x49 0x48 0x44 0x52 0x00 0x00 0x00 0x10
> Last bytes:
> 0xE2 0x8A 0x24 0x69 0x53 0x4C 0xB3 0x03 0x00 0x00 0x00 0x00 0x49 0x45 0x4E 0x44 0xAE 0x42 0x60 0x82
> MD5: 8368b5c29a12b298cea2ad4b32955830
>
> Function (main.ImageFunc)(0x9f10e0):
> Len: 2048
> First bytes:
> 0xED 0x1C 0x24 0xFF 0xED 0x1C 0x24 0xFF 0xED 0x1C 0x24 0xFF 0xED 0x1C 0x24 0xFF 0xED 0x1C 0x24 0xFF
> Last bytes:
> 0x00 0x00 0xFF 0xFF 0x00 0x00 0xFF 0xFF 0x00 0x00 0xFF 0xFF 0x00 0x00 0xFF 0xFF 0x00 0x00 0xFF 0xFF
> MD5: ebdf44b7ad36d79b221a70ea2b0fa0c7
>
> Done.
>

Notes:

• Important: I've only tested the code with the (dummy) RGBA image, for other color formats it might need additional work
• As seen, I didn't do any error handling, in order to keep code short (but anyway, that's beyond this question's scope)

There's a fundamental misunderstanding here.

A JPEG/PNG/TIFF file is encoded on disk. It contains some, or all, of the following:

• image width and height
• image colourspace and compression tables
• date of creation
• GPS coordinates
• author
• copyright
• compressed pixels

The first few bytes will be a JPEG/PNG magic number (or signature) followed by other PNG chunks, or JPEG sections, containing the info itemised above.

A PIL Image, after conversion to bytes, contains only uncompressed pixels. The first few bytes will be the top-left pixel, the next few bytes will be the second pixel across in the first row.

They are fundamentally different and will never have the same MD5 checksum.