Golang: fatal error: unexpected signal during runtime execution

I am randomly getting the following error when I am executing the binary (sometime it works):

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x47 pc=0x7f07a019b448]

Here is the entire stacktrace.

The OS version where I am running the binary:

NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Go version:

go version
go version go1.16.9 linux/amd64

Bazel options:

go_binary(
    name = "app-operator",
    embed = [":app-operator_lib"],
    gc_linkopts = [
        "-linkmode",
        "external",
        "-extldflags",
        "-static",
    ],
    static = "on",
    visibility = ["//visibility:public"],
)

I think it's happening in this goroutine:

goroutine 408 [select]:
net/http.(*Transport).getConn(0x22f49e0, 0xc00080b440, {{}, 0x0, {0x16482cc, 0x4}, {0xc0006d9590, 0x30}, 0x0})
	GOROOT/src/net/http/transport.go:1372 +0x5d2
net/http.(*Transport).roundTrip(0x22f49e0, 0xc0005b0400)
	GOROOT/src/net/http/transport.go:581 +0x774
net/http.(*Transport).RoundTrip(0x30, 0x1817c80)
	GOROOT/src/net/http/roundtrip.go:18 +0x19
net/http.send(0xc0005b0400, {0x1817c80, 0x22f49e0}, {0x15f8740, 0x201, 0x0})
	GOROOT/src/net/http/client.go:252 +0x5d8
net/http.(*Client).send(0x2382ea0, 0xc0005b0400, {0x380, 0x0, 0x0})
	GOROOT/src/net/http/client.go:176 +0x9b
net/http.(*Client).do(0x2382ea0, 0xc0005b0400)
	GOROOT/src/net/http/client.go:725 +0x908
net/http.(*Client).Do(...)
	GOROOT/src/net/http/client.go:593
somecompany.com/operators/app-operator/http/client.(*APIClient).callAPI(0xc00078e030, 0xc0005b0400)
	operators/app-operator/http/client/client.go:151 +0x12e
somecompany.com/operators/app-operator/http/client.(*BackuprestoreApiService).Poll(0xc0004de148, 0xc00016b8a8)
	operators/app-operator/http/client/api_backuprestore.go:216 +0x5bd
somecompany.com/operators/app-operator/controllers.(*BackupReconciler).Poll(0xc0004de120, {0xc00076a4b0, {0xc00021c268, 0xc00016ba38}, 0xc00076a4c0})
	operators/app-operator/controllers/backup_controller.go:165 +0x16d
somecompany.com/operators/app-operator/controllers.(*BackupReconciler).Poll(0xc0004de120, {0xc00076a4b0, {0xc00021c268, 0x4f416a}, 0xc00076a4c0})
	operators/app-operator/controllers/backup_controller.go:186 +0x396
somecompany.com/operators/app-operator/controllers.(*BackupReconciler).Poll(0xc0004de120, {0xc00076a4b0, {0xc00021c268, 0x4f4226}, 0xc00076a4c0})
	operators/app-operator/controllers/backup_controller.go:186 +0x396
somecompany.com/operators/app-operator/controllers.(*BackupReconciler).Backup(0xc0004de120, 0xc0006ab4a0)
	operators/app-operator/controllers/backup_controller.go:147 +0x396
somecompany.com/operators/app-operator/controllers.(*BackupReconciler).createBackupHandler(0xc00016bd78, 0x183e758)
	operators/app-operator/controllers/backup_controller.go:93 +0x1d
somecompany.com/operators/app-operator/controllers.(*BackupReconciler).Reconcile(0xc0004de120, {0x183e758, 0xc000772d50}, {{{0xc0000596b8, 0x155d9a0}, {0xc0000596b0, 0xc000338100}}})
	operators/app-operator/controllers/backup_controller.go:87 +0x1c5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0000da320, {0x183e6b0, 0xc0000b4000}, {0x14fda20, 0xc0009a4000})
	external/io_k8s_sigs_controller_runtime/pkg/internal/controller/controller.go:298 +0x303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0000da320, {0x183e6b0, 0xc0000b4000})
	external/io_k8s_sigs_controller_runtime/pkg/internal/controller/controller.go:253 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	external/io_k8s_sigs_controller_runtime/pkg/internal/controller/controller.go:214 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	external/io_k8s_sigs_controller_runtime/pkg/internal/controller/controller.go:210 +0x354

You should be able to remove the gc_linkopts options from the build. Statically linking glibc is not really supported, and may be causing problems.

If you have other cgo libraries you need to link statically, you will need to provide the linker flags for those libraries specifically.

If the only use of cgo is for name resolution on linux, you can almost always build without cgo and rely on the native Go implementations when a static binary is desired without a portable C implementation.