英文:
Compiling go library without GCO to run on alpine, error in libczmq
问题
尝试在alpine
上运行我的二进制文件时,出现了错误:
... binary not found
这通常发生在架构有问题或者glibc
有问题时。我搜索并发现alpine
使用的是muslc
,这是一个替代的C
库。然后我找到了这个链接https://stackoverflow.com/questions/34729748/installed-go-binary-not-found-in-path-on-alpine-linux-docker,它教你如何在没有CGO
的情况下编译,CGO
是允许从Go加载C
库的东西:
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o [二进制文件名]
当我运行这个命令时,我得到了以下错误:
go build gopkg.in/zeromq/goczmq.v4: no buildable Go source files in /home/lucas/Go/src/gopkg.in/zeromq/goczmq.v4
我怀疑这是因为libczmq
只是C
编写的libzmq
的包装器。在这种情况下,我该如何编译才能使用libczmq
?为什么在alpine中必须禁用CGO
?
CGO
到底是什么?它不应该在有libc
时使用libc
,在没有libc
时使用muslc
吗?我想了解更多关于背后发生的事情。
注意:我是在ubuntu上编译,不是在alpine上,这会有问题吗?
英文:
When trying to run my binary on alpine
, I got the error:
... binary not found
which typically happens when there's a problem with architecture, or as I found, glibc
. I searched and discovered that alpine
instead uses muslc
, an alternative C
library. I then found this https://stackoverflow.com/questions/34729748/installed-go-binary-not-found-in-path-on-alpine-linux-docker that teaches how to compile without CGO
, which is the thing that permits loading C
libraries from go:
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o [name of binary]
When I run this, I get:
go build gopkg.in/zeromq/goczmq.v4: no buildable Go source files in /home/lucas/Go/src/gopkg.in/zeromq/goczmq.v4
I suspect that it's because libczmq
is just a wrapper for the C
written libzmq
. In this case, how do I compile in a way that I can use libczmq
? Why exactly CGO
must be disabled in alpine?
What's exactly CGO
? Shouldn't it use libc
when available but muslc
when not? I'd like to know more about what's happening in the background.
Note: I'm compiling outside alpine, in ubuntu, is that a problem?
答案1
得分: 4
我知道这个问题已经有三年了,但互联网上没有关于这个问题的任何信息,我也遇到了完全相同的问题,花了两天时间才最终找到了一个合适的解决方案。
简而言之,我有一个使用goczmq
的Go项目,我想将其编译为一个完整的二进制文件,然后放入一个FROM scratch
的Docker容器中(尽管在这种情况下,一个alpine容器也可以正常工作)。在互联网上,人们现在倾向于告诉你设置CGO_ENABLED=0
,一切都会正常工作,这是正确的。这告诉Go不使用CGO,这允许你在Go代码中使用C库,但在运行代码时需要这些C库在系统上可用。正如你已经发现的,alpine没有这些C库(或者说,它在muslc
中有不同的库,而不是glibc
)。
然而,在我们的情况下,这并没有帮助。我们希望我们的Go代码能够使用现有的C库,因为我们使用的是goczmq
,正如你所指出的,它是czmq
的Go封装,而czmq
本身是libzmq
的C封装(libzmq
是用C++编写的,这使得我们的生活更加困难)。
我通过使用静态二进制文件解决了这个问题。我不是动态链接到目标系统上可用的任何C库(在这种情况下是alpine上的muslc
),而是将我的代码和所有库(muslc
、czmq
和libzmq
)编译成一个统一的二进制文件。
我在这里使用基于alpine的Docker多阶段构建,但理论上你也可以直接在你的计算机上进行这个操作。
# 阶段1:构建二进制文件
# 我们使用最新版本的golang和alpine Linux
FROM golang:1.13-alpine as golang
# 首先安装一些依赖项
#(我们使用每个依赖项的静态版本,因为我们希望这些库被静态地包含,而不是动态地包含!)
# czmq需要libzmq,libzmq又需要libsodium
# 在alpine Linux上,我们还需要安装一些特定的工具来构建C和C++程序
# libsodium还需要libuuid,它包含在util-linux-dev中
RUN apk add --no-cache libzmq-static czmq-dev libsodium-static build-base util-linux-dev
# 现在我们执行这个命令
# https://stackoverflow.com/questions/34729748/installed-go-binary-not-found-in-path-on-alpine-linux-docker?noredirect=1&lq=1
# 这个命令让C编译器认为我们已经安装了glibc,而实际上我们使用的是musl
# 由于我们是静态编译,所以使用musl是有意义的,因为它更小
# (如果你想以某种形式分发你的二进制文件,它还使用了更宽松的MIT许可证,但在使用其他库之前请检查!)
RUN mkdir /lib64 && ln -s /lib/libc.musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2
# 为Go项目创建项目目录
WORKDIR /go/src/github.com/<username>/<projectname>/
# 将所有Go文件复制到项目目录中,假设这些文件与Dockerfile在同一个目录中
COPY . .
# 这是第一个技巧:我们需要告诉CGO使用g++而不是gcc,否则它将无法处理C++编写的libzmq
# 创建一个空的C++文件可以解决这个问题
RUN touch ./dummy.cc
# 现在运行go install(go build也可以在这里工作,但你的二进制文件将会在不同的目录中)
# -ldflags将传递给CGO工具链
# -extldflags在这里特别重要,它有两个重要的标志:
# -static告诉编译器使用静态链接,这是将所有内容放入一个二进制文件的真正魔法
# -luuid是为了正确找到czmq使用的uuid库
RUN go install -a -ldflags '-linkmode external -w -s -extldflags "-static -luuid" ' .
# 阶段2:这是你实际上将在Docker主机上运行的映像
# 如果你愿意,你也可以在这里使用alpine
FROM scratch
# 现在我们只需从旧的构建器映像中复制完成的二进制文件
COPY --from=golang /go/bin/<projectname> bin
# 然后我们启动我们的程序
ENTRYPOINT ["./bin"]
现在这几乎可以工作了!唯一剩下的问题是在你的main.go
文件开头添加这个语句,否则CGO会对你的操作感到困惑:
import "C"
即使你的程序中没有直接使用CGO,这也是重要的。
英文:
I know this question is about three years old now but there is nothing on the Internet about this and I was facing the exact same problem and it took about two days before I finally found a proper solution.
In short, I had a Go project that uses goczmq
that I wanted to compile to a complete binary to put it into a FROM scratch
Docker container (although in this case, an alpine container would work just as well). On the Internet, people now tend to tell you to set CGO_ENABLED=0
and everything will work fine, which is true. This tells Go to not use CGO, which allows you to use C libraries in your Go code, but needs those C libraries to be available on the system when you run your code. As you have figured out, alpine does not have those C libraries (or, well, it has different ones in muslc
instead of glibc
).
However, in our case, this is not helpful. We want our Go code to be able to use existing C libraries because we use goczmq
, which, as you have identified, is a Go wrapper for czmq
which itself is a C wrapper for libzmq
(which is written in C++, which makes our life even harder).
I solved this problem by using a static binary. Instead of linking to whatever C libraries are available on the target system (in this case muslc
on alpine) dynamically, I compile my code AND all the libraries (muslc
, czmq
and libzmq
) into one unified binary.
I'm using Docker multi-stage builds based on alpine here but in theory you could also do this directly on your computer.
# stage 1: build the binary
# we are using alpine Linux with the latest version of golang
FROM golang:1.13-alpine as golang
# first install some dependencies
# (we are using the static versions for each for them as we want these libraries included statically, not dynamically!)
# czmq requires libzmq which in turn requires libsodium
# on alpine Linux we also need to install some specific tools to build C and C++ programs
# libsodium also requires libuuid, which is included in util-linux-dev
RUN apk add --no-cache libzmq-static czmq-dev libsodium-static build-base util-linux-dev
# now we do the magic command mentioned here
# https://stackoverflow.com/questions/34729748/installed-go-binary-not-found-in-path-on-alpine-linux-docker?noredirect=1&lq=1
# this fools the C compiler into thinking we have glibc installed while we are actually using musl
# since we are compiling statically, it makes sense to use musl as it is smaller
# (and it uses the more permissive MIT license if you want to distribute your binary in some form, but check your other libraries before!)
RUN mkdir /lib64 && ln -s /lib/libc.musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2
# create your project directory for the Go project
WORKDIR /go/src/github.com/<username>/<projectname>/
# copy in all your Go files, assuming these are in the same directory as your Dockerfile
COPY . .
# here is the first hack: we need to tell CGO to use g++ instead of gcc or else it will struggle with libzmq, which is written in C++
# creating and empty C++ file actually works for this
RUN touch ./dummy.cc
# now run go install (go build could also work here but your binary would end up in a different directory)
# the -ldflags will be passed along the CGO toolchain
# -extldflags is especially important here, it has two important flags that we need:
# -static tells the compiler to use static linking, which does the actual magic of putting everything into one binary
# -luuid is needed to correctly find the uuid library that czmq uses
RUN go install -a -ldflags '-linkmode external -w -s -extldflags "-static -luuid" ' .
# stage 2: here is your actual image that will later run on your Docker host
# you can also use alpine here if you so choose
FROM scratch
# now we just copy over the completed binary from the old builder image
COPY --from=golang /go/bin/<projectname> bin
# and we start our program
ENTRYPOINT ["./bin"]
Now this almost works! The only thing left is to add this statement to the beginning of your main.go
file or else CGO is confused about what you are doing:
import "C"
This is important even if you don't directly use CGO in your program.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论