如何使用Kaniko的Docker构建秘密

huangapple
go评论77阅读模式
英文:

How to Use Docker Build Secrets with Kaniko

问题

以下是您要翻译的文本：

Context

Our current build system builds docker images inside of a docker container (Docker in Docker). Many of our docker builds need credentials to be able to pull from private artifact repositories.

We've handled this with docker secrets.. passing in the secret to the docker build command, and in the Dockerfile, referencing the secret in the RUN command where its needed. This means we're using docker buildkit. This article explains it.

We are moving to a different build system (GitLab) and the admins have disabled Docker in Docker (security reasons) so we are moving to Kaniko for docker builds.

Problem

Kaniko doesn't appear to support secrets the way docker does. (there are no command line options to pass a secret through the Kaniko executor).

The credentials the docker build needs are stored in GitLab variables. For DinD, you simply add those variables to the docker build as a secret:

  1.    --secret=type=env,id=USERNAME \
  2.    --secret=type=env,id=PASSWORD \

And then in docker, use the secret:

  1.    USER=$(cat /run/secrets/USERNAME) \
  2.     PASS=$(cat /run/secrets/PASSWORD) \
  3.      ./scriptThatUsesTheseEnvVarCredentialsToPullArtifacts
  4. ...rest of build..

Without the --secret flag to the kaniko executor, I'm not sure how to take advantage of docker secrets... nor do I understand the alternatives. I also want to continue to support developer builds. We have a 'build.sh' script that takes care of gathering credentials and adding them to the docker build command.

Current Solution

I found this article and was able to sort out a working solution. I want to ask the experts if this is valid or what the alternatives might be.

I discovered that when the kaniko executor runs, it appears to mount a volume into the image that's being built at: /kaniko. That directory does not exist when the build is complete and does not appear to be cached in the docker layers.

I also found out that if if the Dockerfile secret is not passed in via the docker build command, the build still executes.

So my gitlab-ci.yml file has this excerpt (the REPO_USER/REPO_PWD variables are GitLab CI variables):

  • echo "${REPO_USER}" > /kaniko/repo-credentials.txt
  • echo "${REPO_PWD}" >> /kaniko/repo-credentials.txt
  • /kaniko/executor
    --context "${CI_PROJECT_DIR}/docker/target"
    --dockerfile "${CI_PROJECT_DIR}/docker/target/Dockerfile"
    --destination "${IMAGE_NAME}:${BUILD_TAG}"

Key piece here is echo'ing the credentials to a file in the /kaniko directory before calling the executor. That directory is (temporarily) mounted into the image which the executor is building. And since all this happens inside of the kaniko image, that file will disappear when kaniko (gitlab) job completes.

The developer build script (snip):

  2. DOCKER_BUILDKIT=1 docker build . \
  3.    --secret id=repo-creds,src=dev-credentials.txt

Basically same as before. Had to put it in a file instead of environment variables.

The dockerfile (snip):

This Works!

In the Dockerfile, by mounting the secret in the /kaniko subfolder, it will work with both the DinD developer build as well as with the CI Kaniko executor.

For Dev builds, DinD secret works as always. (had to change it to a file rather than env variables which I didn't love.)

When the build is run by Kaniko, I suppose since the secret in the RUN command is not found, it doesn't even try to write the temporary credentials file (which I expected would fail the build). Instead, because I directly wrote the variables to the temporarily mounted /kaniko directory, the rest of the run command was happy.

Advice

To me this does seem more kludgy than expected. I'm wanting to find out other/alternative solutions. Finding out the /kaniko folder is mounted into the image at build time seems to open a lot of possibilities.

英文:

Context

Our current build system builds docker images inside of a docker container (Docker in Docker). Many of our docker builds need credentials to be able to pull from private artifact repositories.

We've handled this with docker secrets.. passing in the secret to the docker build command, and in the Dockerfile, referencing the secret in the RUN command where its needed. This means we're using docker buildkit. This article explains it.

We are moving to a different build system (GitLab) and the admins have disabled Docker in Docker (security reasons) so we are moving to Kaniko for docker builds.

Problem

Kaniko doesn't appear to support secrets the way docker does. (there are no command line options to pass a secret through the Kaniko executor).

The credentials the docker build needs are stored in GitLab variables. For DinD, you simply add those variables to the docker build as a secret:

  1. DOCKER_BUILDKIT=1 docker build . \
  2.    --secret=type=env,id=USERNAME \
  3.    --secret=type=env,id=PASSWORD \

And then in docker, use the secret:

  1. RUN --mount=type=secret,id=USERNAME --mount=type=secret,id=PASSWORD \
  2.    USER=$(cat /run/secrets/USERNAME) \
  3.     PASS=$(cat /run/secrets/PASSWORD) \
  4.      ./scriptThatUsesTheseEnvVarCredentialsToPullArtifacts
  5. ...rest of build..

Without the --secret flag to the kaniko executor, I'm not sure how to take advantage of docker secrets... nor do I understand the alternatives. I also want to continue to support developer builds. We have a 'build.sh' script that takes care of gathering credentials and adding them to the docker build command.

Current Solution

I found this article and was able to sort out a working solution. I want to ask the experts if this is valid or what the alternatives might be.

I discovered that when the kaniko executor runs, it appears to mount a volume into the image that's being built at: /kaniko. That directory does not exist when the build is complete and does not appear to be cached in the docker layers.

I also found out that if if the Dockerfile secret is not passed in via the docker build command, the build still executes.

So my gitlab-ci.yml file has this excerpt (the REPO_USER/REPO_PWD variables are GitLab CI variables):

  1. - echo "${REPO_USER}" > /kaniko/repo-credentials.txt
  2. - echo "${REPO_PWD}" >> /kaniko/repo-credentials.txt
  3. - /kaniko/executor
  4.   --context "${CI_PROJECT_DIR}/docker/target"
  5.   --dockerfile "${CI_PROJECT_DIR}/docker/target/Dockerfile"
  6.   --destination "${IMAGE_NAME}:${BUILD_TAG}"

Key piece here is echo'ing the credentials to a file in the /kaniko directory before calling the executor. That directory is (temporarily) mounted into the image which the executor is building. And since all this happens inside of the kaniko image, that file will disappear when kaniko (gitlab) job completes.

The developer build script (snip):

  1. //to keep it simple, this assumes that the developer has their credentials//cached in a file (ignored by git) called dev-credentials.txt
  3. DOCKER_BUILDKIT=1 docker build . \
  4.    --secret id=repo-creds,src=dev-credentials.txt

Basically same as before. Had to put it in a file instead of environment variables.

The dockerfile (snip):

  1. RUN --mount=type=secret,id=repo-creds,target=/kaniko/repo-credentials.txt USER=$(sed '1q;d' /kaniko/repo-credentials.txt) PASS=$(sed '2q;d' /kaniko/repo-credentials.txt) ./scriptThatUsesTheseEnvVarCredentialsToPullArtifacts...rest of build..

This Works!

In the Dockerfile, by mounting the secret in the /kaniko subfolder, it will work with both the DinD developer build as well as with the CI Kaniko executor.

For Dev builds, DinD secret works as always. (had to change it to a file rather than env variables which I didn't love.)

When the build is run by Kaniko, I suppose since the secret in the RUN command is not found, it doesn't even try to write the temporary credentials file (which I expected would fail the build). Instead, because I directly wrote the varibles to the temporarily mounted /kaniko directory, the rest of the run command was happy.

Advice

To me this does seem more kludgy than expected. I'm wanting to find out other/alternative solutions. Finding out the /kaniko folder is mounted into the image at build time seems to open a lot of possibilities.

答案1

得分: 1

以下是您要翻译的内容：

The question includes the answer already, and I just try to show the complete code for the user to pick directly

  1. $ cat Dockerfile 
  2. # podman build --secret id=mysecret,src=./secret.txt .
  3. # /kaniko/executor --context . --dockerfile Dockerfile --no-push
  4. FROM alpine
  5. RUN --mount=type=secret,id=netrc \
  6.     cp /kaniko/netrc $HOME/.netrc || cp /run/secrets/netrc $HOME/.netrc \
  7.     && cat $HOME/.netrc \
  8.     && echo "now can run some scripts" \
  9.     && rm $HOME/.netrc
  10. RUN ls $HOME/.netrc || true

If you use podman/buildah, you can inject a secret and use /run/secrets/netrc inside; it will not exist in the end image

  1. $ podman build --secret id=netrc,src=./secret.txt .
  2. STEP 1/3: FROM alpine
  3. STEP 2/3: RUN --mount=type=secret,id=netrc     cp /kaniko/netrc $HOME/.netrc || cp /run/secrets/netrc $HOME/.netrc     && cat $HOME/.netrc     && echo "now can run some scripts"     && rm $HOME/.netrc
  4. cp: can't stat '/kaniko/netrc': No such file or directory
  5. top secrets
  6. now can run some scripts
  7. --> 880d646a021
  8. STEP 3/3: RUN ls $HOME/.netrc || true
  9. ls: /root/.netrc: No such file or directory
  10. COMMIT
  11. --> 64a8b99ba19
  12. 64a8b99ba198d088f7cc678f160e29f5eae6ee90ff81dda2e292c55601cf4eb2

For kaniko, /kaniko is the folder for this purpose

  1. /workspace # cat /kaniko/netrc
  2. top secret!
  3. /workspace # /kaniko/executor --context . --dockerfile Dockerfile  --no-push
  4. INFO[0000] Retrieving image manifest alpine 
  5. ...
  6. INFO[0002] Unpacking rootfs as cmd RUN --mount=type=secret,id=netrc     cp /kaniko/netrc $HOME/.netrc || cp /run/secrets/netrc $HOME/.netrc     && cat $HOME/.netrc     && echo "now can run some scripts"     && rm $HOME/.netrc requires it. 
  7. INFO[0002] RUN --mount=type=secret,id=netrc     cp /kaniko/netrc $HOME/.netrc || cp /run/secrets/netrc $HOME/.netrc     && cat $HOME/.netrc     && echo "now can run some scripts"     && rm $HOME/.netrc 
  8. INFO[0002] Initializing snapshotter ...                 
  9. INFO[0002] Taking a snapshot of the full filesystem...        
  10. INFO[0002] Cmd: /bin/sh                                 
  11. INFO[0002] Args: [-c cp /kaniko/netrc $HOME/.netrc || cp /run/secrets/netrc $HOME/.netrc     && cat $HOME/.netrc     && echo "now can run some scripts"     && rm $HOME/.netrc] 
  12. INFO[0002] Running: [/bin/sh -c cp /kaniko/netrc $HOME/.netrc || cp /run/secrets/netrc $HOME/.netrc     && cat $HOME/.netrc     && echo "now can run some scripts"     && rm $HOME/.netrc] 
  13. top secret!
  14. now can run some scripts
  15. INFO[0002] Taking a snapshot of the full filesystem...        
  16. INFO[0002] RUN ls $HOME/.netrc || true                  
  17. INFO[0002] Cmd: /bin/sh                                 
  18. INFO[0002] Args: [-c ls $HOME/.netrc || true]           
  19. INFO[0002] Running: [/bin/sh -c ls $HOME/.netrc || true] 
  20. ls: /root/.netrc: No such file or directory
  21. INFO[0002] Taking a snapshot of the full filesystem...        
  22. INFO[0002] No files were changed, appending an empty layer to the config. No layer added to the image.
英文:

The question includes the answer already, and I just try to show the complete code for user to pick directly

  1. $ cat Dockerfile 
  2. # podman build --secret id=mysecret,src=./secret.txt .
  3. # /kaniko/executor --context . --dockerfile Dockerfile --no-push
  4. FROM alpine
  5. RUN --mount=type=secret,id=netrc \
  6.     cp /kaniko/netrc $HOME/.netrc || cp /run/secrets/netrc $HOME/.netrc \
  7.     && cat $HOME/.netrc \
  8.     && echo "now can run some scripts" \
  9.     && rm $HOME/.netrc
  10. RUN ls $HOME/.netrc || true

If you use podman/buildah you can inject secret and use /run/secrets/netrc inside, it will not exist in end image

  1. $ podman build --secret id=netrc,src=./secret.txt .
  2. STEP 1/3: FROM alpine
  3. STEP 2/3: RUN --mount=type=secret,id=netrc     cp /kaniko/netrc $HOME/.netrc || cp /run/secrets/netrc $HOME/.netrc     && cat $HOME/.netrc     && echo "now can run some scripts"     && rm $HOME/.netrc
  4. cp: can't stat '/kaniko/netrc': No such file or directory
  5. top secrets
  6. now can run some scripts
  7. --> 880d646a021
  8. STEP 3/3: RUN ls $HOME/.netrc || true
  9. ls: /root/.netrc: No such file or directory
  10. COMMIT
  11. --> 64a8b99ba19
  12. 64a8b99ba198d088f7cc678f160e29f5eae6ee90ff81dda2e292c55601cf4eb2

For kaniko, /kaniko is the folder for this purpose

  1. /workspace # cat /kaniko/netrc
  2. top secret!
  3. /workspace # /kaniko/executor --context . --dockerfile Dockerfile  --no-push
  4. INFO[0000] Retrieving image manifest alpine 
  5. ...
  6. INFO[0002] Unpacking rootfs as cmd RUN --mount=type=secret,id=netrc     cp /kaniko/netrc $HOME/.netrc || cp /run/secrets/netrc $HOME/.netrc     && cat $HOME/.netrc     && echo "now can run some scripts"     && rm $HOME/.netrc requires it. 
  7. INFO[0002] RUN --mount=type=secret,id=netrc     cp /kaniko/netrc $HOME/.netrc || cp /run/secrets/netrc $HOME/.netrc     && cat $HOME/.netrc     && echo "now can run some scripts"     && rm $HOME/.netrc 
  8. INFO[0002] Initializing snapshotter ...                 
  9. INFO[0002] Taking snapshot of full filesystem...        
  10. INFO[0002] Cmd: /bin/sh                                 
  11. INFO[0002] Args: [-c cp /kaniko/netrc $HOME/.netrc || cp /run/secrets/netrc $HOME/.netrc     && cat $HOME/.netrc     && echo "now can run some scripts"     && rm $HOME/.netrc] 
  12. INFO[0002] Running: [/bin/sh -c cp /kaniko/netrc $HOME/.netrc || cp /run/secrets/netrc $HOME/.netrc     && cat $HOME/.netrc     && echo "now can run some scripts"     && rm $HOME/.netrc] 
  13. top secret!
  14. now can run some scripts
  15. INFO[0002] Taking snapshot of full filesystem...        
  16. INFO[0002] RUN ls $HOME/.netrc || true                  
  17. INFO[0002] Cmd: /bin/sh                                 
  18. INFO[0002] Args: [-c ls $HOME/.netrc || true]           
  19. INFO[0002] Running: [/bin/sh -c ls $HOME/.netrc || true] 
  20. ls: /root/.netrc: No such file or directory
  21. INFO[0002] Taking snapshot of full filesystem...        
  22. INFO[0002] No files were changed, appending empty layer to config. No layer added to image.

答案2

得分: 1

以下是您要翻译的内容：

I'm using this with python for private pkgs, it works fine. Here my script, maybe can help you:

Dockerfile:

  1. COPY . .
  2. RUN --mount=type=secret,id=pip.conf,target=/kaniko/pip.conf \
  3.     PIP_CONFIG_FILE=/kaniko/pip.conf \
  4.     pip install --no-cache-dir .

pip.conf file:

  1. [global]
  2. extra-index-url = 
  3.     https://__token__:${GITLAB_DEPLOY_TOKEN}@gitlab-ucc.tools.aws.mycompany.com/api/v4/groups/245/-/packages/pypi/simple

pipeline script:

  1. cp ./cicd/pip.conf /kaniko/pip.conf
  2. sed -i "s/..GITLAB_DEPLOY_TOKEN./${GITLAB_DEPLOY_TOKEN}/g" /kaniko/pip.conf
  4. /kaniko/executor \
  5.       --cache=true \
  6.       --cache-dir=${CI_PROJECT_DIR}/.cache \
  7.       --cache-repo=${CI_REGISTRY_IMAGE}/cache \
  8.       --context=${CI_PROJECT_DIR} \
  9.       --dockerfile=${CI_PROJECT_DIR}/${DOCKERFILE_PATH}/Dockerfile \
  10.       --destination=${CI_REGISTRY_IMAGE}/${CI_PROJECT_NAME}:${NEWVERSION}

developer/manual script:

  1. export GITLAB_DEPLOY_TOKEN=`pass GITLAB_DEPLOY_TOKEN`
  2. envsubst < ./cicd/pip.conf > ./setup/pipconf.tmp
  4. export DOCKER_BUILDKIT=1
  5. docker build -f ./docker/Dockerfile . \
  6.         --secret id=pip.conf,src=./setup/pipconf.tmp \
  7.         --tag $IMG_NAME:dev
英文:

I'm using this with python for private pkgs, it works fine. Here my script, maybe can help you:

Dockerfile:

  1. COPY . .
  2. RUN --mount=type=secret,id=pip.conf,target=/kaniko/pip.conf \ 
  3.     PIP_CONFIG_FILE=/kaniko/pip.conf \
  4.     pip install --no-cache-dir .

pip.conf file:

  1. [global]
  2. extra-index-url = 
  3.     https://__token__:${GITLAB_DEPLOY_TOKEN}@gitlab-ucc.tools.aws.mycompany.com/api/v4/groups/245/-/packages/pypi/simple

pipeline script:

  1. cp ./cicd/pip.conf /kaniko/pip.conf
  2. sed -i "s/..GITLAB_DEPLOY_TOKEN./${GITLAB_DEPLOY_TOKEN}/g" /kaniko/pip.conf
  5. /kaniko/executor \
  6.       --cache=true \
  7.       --cache-dir=${CI_PROJECT_DIR}/.cache \
  8.       --cache-repo=${CI_REGISTRY_IMAGE}/cache \
  9.       --context=${CI_PROJECT_DIR} \
  10.       --dockerfile=${CI_PROJECT_DIR}/${DOCKERFILE_PATH}/Dockerfile \
  11.       --destination=${CI_REGISTRY_IMAGE}/${CI_PROJECT_NAME}:${NEWVERSION}

developer/manual script:

  1. export GITLAB_DEPLOY_TOKEN=`pass GITLAB_DEPLOY_TOKEN`
  2. envsubst < ./cicd/pip.conf > ./setup/pipconf.tmp
  4. export DOCKER_BUILDKIT=1
  5. docker build -f ./docker/Dockerfile . \
  6.         --secret id=pip.conf,src=./setup/pipconf.tmp \
  7.         --tag $IMG_NAME:dev

答案3

得分: 1

Your solution works perfectly fine for us, thank you very much for sharing!

I just wanted to point out that it was necessary to export the variables first before using them in our environment, as described in this post: https://stackoverflow.com/a/69305942

  1. RUN --mount=type=secret,id=repo-creds,target=/kaniko/repo-credentials.txt \ 
  2.     USER=$(sed '1q;d' /kaniko/repo-credentials.txt) \ 
  3.     PASS=$(sed '2q;d' /kaniko/repo-credentials.txt) \
  4.     && export USER \
  5.     && export PASS \
  6.     && echo -n "${USER}:${PASS}"

Additionally, the variable has to be used in the same RUN statement it is exported in. It does not seem to be available in another RUN statement.

英文:

Your solution works perfectly fine for us, thank you very much for sharing!

I just wanted to point out that it was necessary to export the variables first before using them in our environment, as described in this post: https://stackoverflow.com/a/69305942

  1. RUN --mount=type=secret,id=repo-creds,target=/kaniko/repo-credentials.txt \ 
  2.     USER=$(sed '1q;d' /kaniko/repo-credentials.txt) \ 
  3.     PASS=$(sed '2q;d' /kaniko/repo-credentials.txt) \
  4.     && export USER \
  5.     && export PASS \
  6.     && echo -n "${USER}:${PASS}"

Additionally, the variable has to be used in the same RUN statement it is exported in. It does not seem to be available in another RUN statement.

huangapple
  • 本文由 发表于 2023年2月14日 08:30:52
  • 转载请务必保留本文链接：https://go.coder-hub.com/75442435.html
  • docker-buildkit
  • docker-in-docker
  • docker-secrets
  • gitlab-ci
  • kaniko
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定