Why is Go project build failing on Netlify functions with "missing go.sum entry"?

I am trying to deploy a really simple Go function on Netlify functions.

I followed the guide, but the guide does not explain the Go setup very well.

I used the currently recommended way to do it, with basically go init, changing my go file as needed, then go mod tidy. This creates the appropriate go.mod and go.sum files and I can build locally.

I also created a very simple website on website/index.html so I can test the site can "run" the function. That means I configured my "base directory" for Netlify to build as website/ (which is how I normally do with other websites I have on Netlify, and this works fine).

EDIT: I tried moving everything to the root dir instead of using website/ as the base dir, but the problem remain the same.

I then moved all the Go files, including go.mod and go.sum, to website/netlify/functions/analytics/ (so my function will be called analytics as explained in the guide) and pushed.

Contents of website/netlify/functions/analytics:

go.mod  go.sum  handlers.go  main.go  main_test.go

However, the Netlify build fails with:

5:33:45 PM: ❯ Current directory
5:33:45 PM:   /opt/build/repo/website
5:33:45 PM: ​
5:33:45 PM: ❯ Config file
5:33:45 PM:   No config file was defined: using default values.
5:33:45 PM: ​
5:33:45 PM: ❯ Context
5:33:45 PM:   production
5:33:45 PM: ​
5:33:45 PM: ────────────────────────────────────────────────────────────────
5:33:45 PM:   1. Functions bundling                                         
5:33:45 PM: ────────────────────────────────────────────────────────────────
5:33:45 PM: ​
5:33:45 PM: Packaging Functions from netlify/functions directory:
5:33:45 PM:  - analytics/main.go
5:33:45 PM: ​
5:33:45 PM: Could not compile Go function analytics:
5:33:45 PM: ​
5:33:45 PM: ────────────────────────────────────────────────────────────────
5:33:45 PM:   Bundling of Function "analytics" failed                       
5:33:45 PM: ────────────────────────────────────────────────────────────────
5:33:45 PM: ​
5:33:45 PM:   Error message
5:33:45 PM:   Command failed with exit code 1: go build -o /tmp/zisi-629b7b4be8f858000804bf6c/analytics -ldflags -s -w (https://ntl.fyi/exit-code-1)
5:33:45 PM:   go: github.com/aws/aws-lambda-go@v1.32.0 requires
5:33:45 PM:   	github.com/stretchr/testify@v1.6.1: missing go.sum entry; to add it:
5:33:45 PM:   	go mod download github.com/stretchr/testify
5:33:45 PM: ​
5:33:45 PM:   Error location
5:33:45 PM:   While bundling Function "analytics"
5:33:45 PM: ​
5:33:45 PM:   Resolved config
5:33:45 PM:   build:
5:33:45 PM:     base: /opt/build/repo/website
5:33:45 PM:     publish: /opt/build/repo/website
5:33:45 PM:     publishOrigin: ui
5:33:45 PM:   functionsDirectory: /opt/build/repo/website/netlify/functions

The error shows that Netlify can find the go files correctly, but it for some reason seems to ignore go.sum because that file contains the correct checksums:

github.com/aws/aws-lambda-go v1.32.0 h1:i8MflawW1hoyYp85GMH7LhvAs4cqzL7LOS6fSv8l2KM=
github.com/aws/aws-lambda-go v1.32.0/go.mod h1:IF5Q7wj4VyZyUFnZ54IQqeWtctHQ9tz+KhcbDenr220=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0=
gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776 h1:tQIYjPdBoyREyB9XMu+nnTclpTYkz2zFM+lzLJFO4gQ=

What could be wrong and how can I fix that?

As of writing, Netlify is using an older version of Go, 1.16.5, to build.

This means my go.mod file shouldn't even have been accepted by the older Go compiler! Anyway, I installed that version of Go locally to try it out...

Here's how to install a specific version of Go (assuming you have go 1.18 installed locally):

$ go install golang.org/dl/go1.16.5@latest
$ go1.16.5 download

Now you can run go1.16.5 to try out the older compiler.

I wrote a Makefile that lets me use whatever version of Go to build... looks like this:

## To use a different version of Go, run like this:
##    make all GO=go1.16.5
GO:=go

.PHONY: test
test:
	cd analytics && $(GO) test .

../netlify/functions/analytics: test
	cd analytics && $(GO) build -o ../netlify/functions/analytics .

all: ../netlify/functions/analytics

And here's the result using go1.16.5:

▶ make all GO=go1.16.5
cd analytics && go1.16.5 test .
go: github.com/aws/aws-lambda-go@v1.32.0 requires
        github.com/stretchr/testify@v1.6.1: missing go.sum entry; to add it:
        go mod download github.com/stretchr/testify
make: *** [test] Error 1

That's the same error I see on Netlify! Cool, so that's the problem.

Now, to fix the problem I had to remove go.mod and go.sum and re-generate them using the older Go version, which I did by writing this Make task:

.PHONY: setup
setup:
	cd analytics && rm go.mod go.sum && $(GO) mod init renato/analytics && $(GO) mod tidy

Now, the builds succeeds locally with the older Go:

▶ make setup all GO=go1.16.5
cd analytics && rm go.mod go.sum && go1.16.5 mod init renato/analytics && go1.16.5 mod tidy
go: creating new go.mod: module renato/analytics
go: to add module requirements and sums:
        go mod tidy
go: finding module for package github.com/aws/aws-lambda-go/lambda
go: finding module for package github.com/aws/aws-lambda-go/events
go: found github.com/aws/aws-lambda-go/events in github.com/aws/aws-lambda-go v1.32.0
go: found github.com/aws/aws-lambda-go/lambda in github.com/aws/aws-lambda-go v1.32.0
cd analytics && go1.16.5 test .
ok      renato/analytics        0.166s
cd analytics && go1.16.5 build -o ../netlify/functions/analytics .

I decided to commit the Go files after this to let Netlify see the right versions of these files, but I also "customized the build" (as explained in the docs) by just telling Netlify to run my build command, make all (no make setup as it's not needed if you commit the go.mod and go.sum files) on the Netlify UI (in the site settings). This is probably not necessary but I wanted to be certain the right commands are being executed.

And finally, the build works on Netlify as well!!

7:36:59 PM: ────────────────────────────────────────────────────────────────
7:36:59 PM:   1. Build command from Netlify app                             
7:36:59 PM: ────────────────────────────────────────────────────────────────
7:36:59 PM: ​
7:36:59 PM: $ make all
7:36:59 PM: cd analytics && go test .
7:37:00 PM: go: downloading github.com/aws/aws-lambda-go v1.32.0
7:37:08 PM: ok  	renato/analytics	0.017s
7:37:08 PM: cd analytics && go build -o ../netlify/functions/analytics .
7:37:09 PM: ​
7:37:09 PM: (build.command completed in 9.2s)
7:37:09 PM: ​
7:37:09 PM: ────────────────────────────────────────────────────────────────
7:37:09 PM:   2. Functions bundling                                         
7:37:09 PM: ────────────────────────────────────────────────────────────────
7:37:09 PM: ​
7:37:09 PM: Packaging Functions from netlify/functions directory:
7:37:09 PM:  - analytics
7:37:09 PM: ​
7:37:09 PM: ​
7:37:09 PM: (Functions bundling completed in 101ms)

So, the problem was the Go version Netlify is outdated, and unfortunately Go has been messing around with the Go mod format which is causing compatibility issues like this... you may need the exact version of the Go compiler to get a given project to compile properly, as in this case.