How can I compare two files in golang?

With Python I can do the next:

equals = filecmp.cmp(file_old, file_new)

Is there any builtin function to do that in go language? I googled it but without success.

I could use some hash function in hash/crc32 package, but that is more work that the above Python code.

To complete the @captncraig answer, if you want to know if the two files are the same, you can use the SameFile(fi1, fi2 FileInfo) method from the OS package.

> SameFile reports whether fi1 and fi2 describe the same file. For example, on Unix this means that the device and inode fields of the two underlying structures are identical;

Otherwise, if you want to check the files contents, here is a solution which checks the two files line by line avoiding the load of the entire files in memory.

First try: https://play.golang.org/p/NlQZRrW1dT

EDIT: Read by bytes chunks and fail fast if the files have not the same size. https://play.golang.org/p/YyYWuCRJXV

const chunkSize = 64000

func deepCompare(file1, file2 string) bool {
    // Check file size ...

	f1, err := os.Open(file1)
	if err != nil {
		log.Fatal(err)
	}
    defer f1.Close()

	f2, err := os.Open(file2)
	if err != nil {
		log.Fatal(err)
	}
    defer f2.Close()

	for {
		b1 := make([]byte, chunkSize)
		_, err1 := f1.Read(b1)

		b2 := make([]byte, chunkSize)
		_, err2 := f2.Read(b2)

		if err1 != nil || err2 != nil {
			if err1 == io.EOF && err2 == io.EOF {
				return true
			} else if err1 == io.EOF || err2 == io.EOF {
				return false
			} else {
				log.Fatal(err1, err2)
			}
		}

		if !bytes.Equal(b1, b2) {
			return false
		}
	}
}

I am not sure that function does what you think it does. From the docs,

> Unless shallow is given and is false, files with identical os.stat() signatures are taken to be equal.

Your call is comparing only the signature of os.stat, which only includes:

1. File mode
2. Modified Time
3. Size

You can learn all three of these things in Go from the os.Stat function. This really would only indicate that they are literally the same file, or symlinks to the same file, or a copy of that file.

If you want to go deeper you can open both files and compare them (python version reads 8k at a time).

You could use an crc or md5 to hash both files, but if there are differences at the beginning of a long file, you want to stop early. I would recommend reading some number of bytes at a time from each reader and comparing with bytes.Compare.

How about using bytes.Equal?

package main

import (
"fmt"
"io/ioutil"
"log"
"bytes"
)

func main() {
    // per comment, better to not read an entire file into memory
    // this is simply a trivial example.
    f1, err1 := ioutil.ReadFile("lines1.txt")

    if err1 != nil {
	    log.Fatal(err1)
    }

    f2, err2 := ioutil.ReadFile("lines2.txt")

    if err2 != nil {
	    log.Fatal(err2)
    }

    fmt.Println(bytes.Equal(f1, f2)) // Per comment, this is significantly more performant.
}

You can use a package like equalfile

Main API:

func CompareFile(path1, path2 string) (bool, error)

Godoc: https://godoc.org/github.com/udhos/equalfile

Example:

package main

import (
    "fmt"
	"os"
	"github.com/udhos/equalfile"
 )

func main() {
 	if len(os.Args) != 3 {
		fmt.Printf("usage: equal file1 file2\n")
		os.Exit(2)
	}

	file1 := os.Args[1]
	file2 := os.Args[2]

	equal, err := equalfile.CompareFile(file1, file2)
	if err != nil {
		fmt.Printf("equal: error: %v\n", err)
		os.Exit(3)
	}

	if equal {
		fmt.Println("equal: files match")
		os.Exit(0)
	}

	fmt.Println("equal: files differ")
	os.Exit(1)
}

After checking the existing answers I whipped up a simple package for comparing arbitrary (finite) io.Reader and files as a convenience method: https://github.com/hlubek/readercomp

Example:

package main

import (
	"fmt"
	"log"
	"os"

	"github.com/hlubek/readercomp"
)

func main() {
	result, err := readercomp.FilesEqual(os.Args[1], os.Args[2])
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(result)
}

Here's an io.Reader I whipped out. You can _, err := io.Copy(ioutil.Discard,
newCompareReader(a, b)) to get an error if two streams don't share equal contents. This implementation is optimized for performance by limiting unnecessary data copying.

package main
import (
"bytes"
"errors"
"fmt"
"io"
)
type compareReader struct {
a    io.Reader
b    io.Reader
bBuf []byte // need buffer for comparing B's data with one that was read from A
}
func newCompareReader(a, b io.Reader) io.Reader {
return &compareReader{
a: a,
b: b,
}
}
func (c *compareReader) Read(p []byte) (int, error) {
if c.bBuf == nil {
// assuming p's len() stays the same, so we can optimize for both of their buffer
// sizes to be equal
c.bBuf = make([]byte, len(p))
}
// read only as much data as we can fit in both p and bBuf
readA, errA := c.a.Read(p[0:min(len(p), len(c.bBuf))])
if readA > 0 {
// bBuf is guaranteed to have at least readA space
if _, errB := io.ReadFull(c.b, c.bBuf[0:readA]); errB != nil { // docs: "EOF only if no bytes were read"
if errB == io.ErrUnexpectedEOF {
return readA, errors.New("compareReader: A had more data than B")
} else {
return readA, fmt.Errorf("compareReader: read error from B: %w", errB)
}
}
if !bytes.Equal(p[0:readA], c.bBuf[0:readA]) {
return readA, errors.New("compareReader: bytes not equal")
}
}
if errA == io.EOF {
// in happy case expecting EOF from B as well. might be extraneous call b/c we might've
// got it already from the for loop above, but it's easier to check here
readB, errB := c.b.Read(c.bBuf)
if readB > 0 {
return readA, errors.New("compareReader: B had more data than A")
}
if errB != io.EOF {
return readA, fmt.Errorf("compareReader: got EOF from A but not from B: %w", errB)
}
}
return readA, errA
}

> The standard way is to stat them and use os.SameFile.
>
> -- https://groups.google.com/g/golang-nuts/c/G-5D6agvz2Q/m/2jV_6j6LBgAJ

os.SameFile should roughly do the same things as Python's filecmp.cmp(f1, f2) (ie. shallow=true, meaning it only compares the file infos obtained by stat).

> func SameFile(fi1, fi2 FileInfo) bool
>
> SameFile reports whether fi1 and fi2 describe the same file. For example, on Unix this means that the device and inode fields of the two underlying structures are identical; on other systems the decision may be based on the path names. SameFile only applies to results returned by this package's Stat. It returns false in other cases.

But if you actually want to compare the file's content, you'll have to do it yourself.

This does a piece-by-piece comparison of the two files, quitting as soon as it knows the two files are different. It only needs standard library functions.

It's an improvement to this that handles the short-read problem raised by mat007 and christopher by using io.ReadFull(). It also avoids reallocating the buffers.

package util

import (
	"bytes"
	"io"
	"os"
)

// Decide if two files have the same contents or not.
// chunkSize is the size of the blocks to scan by; pass 0 to get a sensible default.
// *Follows* symlinks.
//
// May return an error if something else goes wrong; in this case, you should ignore the value of 'same'.
//
// derived from https://stackoverflow.com/a/30038571
// under CC-BY-SA-4.0 by several contributors
func FileCmp(file1, file2 string, chunkSize int) (same bool, err error) {

	if chunkSize == 0 {
		chunkSize = 4 * 1024
	}

	// shortcuts: check file metadata
	stat1, err := os.Stat(file1)
	if err != nil {
		return false, err
	}

	stat2, err := os.Stat(file2)
	if err != nil {
		return false, err
	}

	// are inputs are literally the same file?
	if os.SameFile(stat1, stat2) {
		return true, nil
	}

	// do inputs at least have the same size?
	if stat1.Size() != stat2.Size() {
		return false, nil
	}

	// long way: compare contents
	f1, err := os.Open(file1)
	if err != nil {
		return false, err
	}
	defer f1.Close()

	f2, err := os.Open(file2)
	if err != nil {
		return false, err
	}
	defer f2.Close()

	b1 := make([]byte, chunkSize)
	b2 := make([]byte, chunkSize)
	for {
		n1, err1 := io.ReadFull(f1, b1)
		n2, err2 := io.ReadFull(f2, b2)

		// https://pkg.go.dev/io#Reader
		// > Callers should always process the n > 0 bytes returned
		// > before considering the error err. Doing so correctly
		// > handles I/O errors that happen after reading some bytes
		// > and also both of the allowed EOF behaviors.

		if !bytes.Equal(b1[:n1], b2[:n2]) {
			return false, nil
		}

		if (err1 == io.EOF && err2 == io.EOF) || (err1 == io.ErrUnexpectedEOF && err2 == io.ErrUnexpectedEOF) {
			return true, nil
		}

		// some other error, like a dropped network connection or a bad transfer
		if err1 != nil {
			return false, err1
		}
		if err2 != nil {
			return false, err2
		}
	}
}

It surprised me that this wasn't anywhere in the standard library.

Something like this should do the trick, and should be memory-efficient compared to the other answers. I looked at github.com/udhos/equalfile and it seemed a bit overkill to me. Before you call compare() here, you should do two os.Stat() calls and compare file sizes for an early out fast path.

The reason to use this implementation over the other answers is because you don't want to hold the entirety of both files in memory if you don't have to. You can read an amount from A and B, compare, and then continue reading the next amount, one buffer-load from each file at a time until you are done. You just have to be careful because you may read 50 bytes from A and then 60 bytes from B because your read may have blocked for some reason.

This implemention assumes a Read() call will not return N > 0 (some bytes read) at the same time as an error != nil. This is how os.File behaves, but not how other implementations of Read may behave, such as net.TCPConn.

import (
  "os"
  "bytes"
  "errors"
)

var errNotSame = errors.New("File contents are different")

func compare(p1, p2 string) error {
	var (
		buf1 [8192]byte
		buf2 [8192]byte
	)

	fh1, err := os.Open(p1)
	if err != nil {
		return err
	}
	defer fh1.Close()

	fh2, err := os.Open(p2)
	if err != nil {
		return err
	}
	defer fh2.Close()

	for {
		n1, err1 := fh1.Read(buf1[:])
		n2, err2 := fh2.Read(buf2[:])

		if err1 == io.EOF && err2 == io.EOF {
			// files are the same!
			return nil
		}
		if err1 == io.EOF || err2 == io.EOF {
			return errNotSame
		}
		if err1 != nil {
			return err1
		}
		if err2 != nil {
			return err2
		}

		// short read on n1
		for n1 < n2 {
			more, err := fh1.Read(buf1[n1:n2])
			if err == io.EOF {
				return errNotSame
			}
			if err != nil {
				return err
			}
			n1 += more
		}
		// short read on n2
		for n2 < n1 {
			more, err := fh2.Read(buf2[n2:n1])
			if err == io.EOF {
				return errNotSame
			}
			if err != nil {
				return err
			}
			n2 += more
		}
		if n1 != n2 {
			// should never happen
			return fmt.Errorf("file compare reads out of sync: %d != %d", n1, n2)
		}

		if bytes.Compare(buf1[:n1], buf2[:n2]) != 0 {
			return errNotSame
		}
	}
}