如何在具有缓存和多个构建运行的多阶段构建中使gradle工作?

huangapple go评论65阅读模式
英文:

How do I make gradle work in a multi-stage build with caching and multiple builds running?

问题

I have a Dockerfiles that follow this starting pattern.

FROM gradle:7 AS deps
COPY ./*.gradle /home/gradle
RUN --mount=type=secret,id=init-gradle \
    --mount=type=cache,target=/home/gradle/.gradle/caches/jars-9 \
    --mount=type=cache,target=/home/gradle/.gradle/caches/modules-2 \
  cp /run/secrets/init-gradle /home/gradle/.gradle/init.gradle && \
  gradle -q --no-daemon dependencies && \
  rm /home/gradle/.gradle/init.gradle
COPY ./src/ /home/gradle/src/

FROM deps AS builder
RUN --mount=type=secret,id=init-gradle \
    --mount=type=cache,target=/home/gradle/.gradle/caches/jars-9 \
    --mount=type=cache,target=/home/gradle/.gradle/caches/modules-2 \
  cp /run/secrets/init-gradle /home/gradle/.gradle/init.gradle && \
  gradle -q --no-daemon build -x test && \
  rm /home/gradle/.gradle/init.gradle

But what happens is if I do a docker compose build with multiple service images

#0 72.50 FAILURE: Build failed with an exception.
#0 72.50
#0 72.50 * What went wrong:
#0 72.50 Timeout waiting to lock jars (/home/gradle/.gradle/caches/jars-9). It is currently in use by another Gradle instance.
#0 72.50 Owner PID: 46
#0 72.50 Our PID: 45
#0 72.50 Owner Operation:
#0 72.51 Our operation:
#0 72.51 Lock file: /home/gradle/.gradle/caches/jars-9/jars-9.lock

What I've done was add the following before I start a gradle operation.

(find /home/gradle/.gradle -type f -name "*.lock" -delete || true) &&  \
FROM gradle:7 AS deps
COPY ./*.gradle /home/gradle
RUN --mount=type=secret,id=init-gradle \
    --mount=type=cache,target=/home/gradle/.gradle/caches/jars-9 \
    --mount=type=cache,target=/home/gradle/.gradle/caches/modules-2 \
  cp /run/secrets/init-gradle /home/gradle/.gradle/init.gradle && \
  (find /home/gradle/.gradle -type f -name "*.lock" -delete || true) && \
  gradle -q --no-daemon dependencies && \
  rm /home/gradle/.gradle/init.gradle
COPY ./src/ /home/gradle/src/

FROM deps AS builder
RUN --mount=type=secret,id=init-gradle \
    --mount=type=cache,target=/home/gradle/.gradle/caches/jars-9 \
    --mount=type=cache,target=/home/gradle/.gradle/caches/modules-2 \
  cp /run/secrets/init-gradle /home/gradle/.gradle/init.gradle && \
  (find /home/gradle/.gradle -type f -name "*.lock" -delete || true) && \
  gradle -q --no-daemon build -x test && \
  rm /home/gradle/.gradle/init.gradle

But I was wondering if there's a proper gradle parameter to prevent this problem? So far the only information my colleague has found was creating a read-only cache via a seed build. Which adds a bit of complexity as we have to somehow create a build that knows about every project dependency at the beginning somehow (which may be doable in a monorepo-type project).

英文:

I have a Dockerfiles that follow this starting pattern.

FROM gradle:7 AS deps
COPY ./*.gradle /home/gradle
RUN --mount=type=secret,id=init-gradle \
    --mount=type=cache,target=/home/gradle/.gradle/caches/jars-9 \
    --mount=type=cache,target=/home/gradle/.gradle/caches/modules-2 \
  cp /run/secrets/init-gradle /home/gradle/.gradle/init.gradle && \
  gradle -q --no-daemon dependencies && \
  rm /home/gradle/.gradle/init.gradle
COPY ./src/ /home/gradle/src/

FROM deps AS builder
RUN --mount=type=secret,id=init-gradle \
    --mount=type=cache,target=/home/gradle/.gradle/caches/jars-9 \
    --mount=type=cache,target=/home/gradle/.gradle/caches/modules-2 \
  cp /run/secrets/init-gradle /home/gradle/.gradle/init.gradle && \
  gradle -q --no-daemon build -x test && \
  rm /home/gradle/.gradle/init.gradle

But what happens is if I do a docker compose build with multiple service images

#0 72.50 FAILURE: Build failed with an exception.
#0 72.50
#0 72.50 * What went wrong:
#0 72.50 Timeout waiting to lock jars (/home/gradle/.gradle/caches/jars-9). It is currently in use by another Gradle instance.
#0 72.50 Owner PID: 46
#0 72.50 Our PID: 45
#0 72.50 Owner Operation:
#0 72.51 Our operation:
#0 72.51 Lock file: /home/gradle/.gradle/caches/jars-9/jars-9.lock

What I've done was add the following before I start a gradle operation.

(find /home/gradle/.gradle -type f -name "*.lock" -delete || true) &&  \
FROM gradle:7 AS deps
COPY ./*.gradle /home/gradle
RUN --mount=type=secret,id=init-gradle \
    --mount=type=cache,target=/home/gradle/.gradle/caches/jars-9 \
    --mount=type=cache,target=/home/gradle/.gradle/caches/modules-2 \
  cp /run/secrets/init-gradle /home/gradle/.gradle/init.gradle && \
  (find /home/gradle/.gradle -type f -name "*.lock" -delete || true) && \
  gradle -q --no-daemon dependencies && \
  rm /home/gradle/.gradle/init.gradle
COPY ./src/ /home/gradle/src/

FROM deps AS builder
RUN --mount=type=secret,id=init-gradle \
    --mount=type=cache,target=/home/gradle/.gradle/caches/jars-9 \
    --mount=type=cache,target=/home/gradle/.gradle/caches/modules-2 \
  cp /run/secrets/init-gradle /home/gradle/.gradle/init.gradle && \
  (find /home/gradle/.gradle -type f -name "*.lock" -delete || true) && \
  gradle -q --no-daemon build -x test && \
  rm /home/gradle/.gradle/init.gradle

But I was wondering if there's a proper gradle parameter to prevent this problem? So far the only information my colleague has found was creating a read only cache via a seed build. Which adds a bit of complexity as we have to somehow create a build that knows about every project dependency at the beginning somehow (which may be doable in a monorepo type project).

答案1

得分: 0

我认为没有明确的答案,因为Gradle本身没有提供仅下载依赖项的功能,因此需要使用locked来捕获缓存数据。

FROM docker-proxy.devhaus.com/library/gradle:8.2 AS deps
WORKDIR /w
COPY --link . /w/
ENV GRADLE_RO_DEP_CACHE=/var/lib/gradle/caches
RUN --mount=type=secret,id=init-gradle,dst=/home/gradle/.gradle/init.gradle \
    --mount=type=cache,sharing=locked,target=/var/lib/gradle/caches \
  gradle --no-daemon build -x test -x spotlessCheck && \
  find /home/gradle/.gradle/caches/ -type f \( -name "*.lock" -o -name "gc.properties" \) -prune -o -exec cp -au --parents {} $GRADLE_RO_DEP_CACHE \;

sharing=locked将防止同时构建,但由于$GRADLE_RO_DEP_CACHE,依赖项已缓存,因此如果它依赖于依赖项,它会显著缩短构建时间。

英文:

I don't think there's a clean answer because there's no facility available for Gradle itself to download dependencies alone and because of that the build to capture the cached data needs to be locked.

FROM docker-proxy.devhaus.com/library/gradle:8.2 AS deps
WORKDIR /w
COPY --link . /w/
ENV GRADLE_RO_DEP_CACHE=/var/lib/gradle/caches
RUN --mount=type=secret,id=init-gradle,dst=/home/gradle/.gradle/init.gradle \
    --mount=type=cache,sharing=locked,target=/var/lib/gradle/caches \
  gradle --no-daemon build -x test -x spotlessCheck && \
  find /home/gradle/.gradle/caches/ -type f \( -name "*.lock" -o -name "gc.properties" \) -prune -o -exec cp -au --parents {} $GRADLE_RO_DEP_CACHE \;

The sharing=locked will prevent simultaneous builds, but because of the $GRADLE_RO_DEP_CACHE the dependencies are cached so it trims the build time significantly if it is dependency dependent.

huangapple
  • 本文由 发表于 2023年6月15日 12:24:23
  • 转载请务必保留本文链接:https://go.coder-hub.com/76479101.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定