英文:
Pre-compiling Golang project dependencies to cache
问题
简而言之，我的当前用例涉及在Docker容器内动态创建一个Golang插件。编译过程中需要用户提供一些新的输入（这就是为什么事先不编译的原因），但依赖项是静态的，不会改变。
目前，完整的编译是在Docker容器内从头开始进行的（尽管使用了go mod download来减少时间）。我注意到go build命令最终会编译许多依赖项，这会为插件编译增加相当多的时间，影响了我的应用程序的可用性。
是否有一种Go支持的方法或命令可以读取go.mod文件并填充GOCACHE目录？有了这样一个命令，我可以在我的Dockerfile中运行它，使Docker镜像包含所有已编译的构建依赖项的缓存。
我尝试过的方法：
go mod download：这只下载依赖项，不会编译它们。- 我使用了一个临时解决方法：我创建了一个简单的
main.go文件，导入了所有的依赖项，并在我的Dockerfile中运行go build来填充缓存。如上所述，这解决了我的问题，但感觉有点像一个hack。此外，如果将来依赖项发生变化，也需要有人进行更改，这并不理想。 - 我在网上看到很多关于这个问题的答案都涉及到CI/CD。使用CI/CD，容器只有一个挂载到主机上的分区，其中包含在运行后持久化的缓存。这并不能解决我目前的问题，即构建容器本身的问题。
英文:
In short, my current use-case involves dynamically creating a Golang plugin inside a Docker container. The compilation involves some new input from the user (which is why it is not compiled beforehand), but the dependencies are static, and won't change.
Currently, the full compilation is done from scratch inside the Docker container (though go mod download is used to reduce the time by a bit). I noticed that the go build command ends up compiling a lot of the dependencies, which adds a non-trivial amount of time for the plugin compilation, which affects the usability of my application.
Is there a Go supported method or command to read through the go.mod file and populate the GOCACHE directory? With such a command, I would run it in my Dockerfile itself, causing the Docker image to contain the cache with all the compiled build dependencies.
What I've tried:
go mod download: This only downloads the dependencies; it does not compile them.- I do have this working with a temporary workaround: I created a barebones
main.gothat imports all the dependencies, and rungo buildwithin myDockerfileto populate the cache. As mentioned, this does solve my problem, but it feels like a bit of a hack. Additionally, if the dependencies change in the future, it requires someone to change this as well, which isn't ideal. - A lot of the answers I saw online for this involve CI/CD. With CI/CD, the container just has a partition mounted to the host, which contains a cache that is persisted after runs. This does not solve my immediate problem, which is for building the container itself.
答案1
得分: 2
由于golang存储库中的问题（1，2）仍然未解决，我认为我们只能进行“黑客”操作。因此，我们可以将依赖项缓存和预编译作为单独的Docker层来处理，类似于以下示例：
FROM golang:1.19-buster as builder
COPY ./go.* /src/
WORKDIR /src
# 清除模块缓存
RUN set -x \
# 缓存Go依赖项
&& go mod download \
# 预编译常见依赖项
&& mkdir /tmp/gobin \
&& for p in $(go list -m -f '{{if and (not .Indirect) (not .Main)}}{{.Path}}/...@{{.Version}}{{end}}' all); do \
GOBIN=/tmp/gobin go install $p; \
done \
&& rm -r /tmp/gobin
COPY . /src
RUN go build ...
使用这个技巧，构建过程（docker buildx build --platform linux/amd64,linux/arm64 ...）大约需要9分钟，而使用这个技巧后只需要约6分钟（节省30%的时间）。但是，预编译步骤需要更长时间，增加了约40%的时间。
英文:
Since the issues (1, 2) in the golang repository are still open, all that we can do is "hacking", I think. So, we can do something like that for the dependencies caching and pre-compilation as a separate docker layer:
FROM golang:1.19-buster as builder
COPY ./go.* /src/
WORKDIR /src
# burn the modules cache
RUN set -x \
# cache go dependencies
&& go mod download \
# pre-compile common dependencies
&& mkdir /tmp/gobin \
&& for p in $(go list -m -f '{{if and (not .Indirect) (not .Main)}}{{.Path}}/...@{{.Version}}{{end}}' all); do \
GOBIN=/tmp/gobin go install $p; \
done \
&& rm -r /tmp/gobin
COPY . /src
RUN go build ...
Without this trick, the building (docker buildx build --platform linux/amd64,linux/arm64 ...) takes about 9 minutes, with it ~6 minutes (profit 30%). But the pre-compilation step becomes longer by ~40%.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论