How to Use Docker Build Secrets with Kaniko

Context

Our current build system builds docker images inside of a docker container (Docker in Docker). Many of our docker builds need credentials to be able to pull from private artifact repositories.

We've handled this with docker secrets.. passing in the secret to the docker build command, and in the Dockerfile, referencing the secret in the RUN command where its needed. This means we're using docker buildkit. This article explains it.

We are moving to a different build system (GitLab) and the admins have disabled Docker in Docker (security reasons) so we are moving to Kaniko for docker builds.

Problem

Kaniko doesn't appear to support secrets the way docker does. (there are no command line options to pass a secret through the Kaniko executor).

The credentials the docker build needs are stored in GitLab variables. For DinD, you simply add those variables to the docker build as a secret:

DOCKER_BUILDKIT=1 docker build . \
   --secret=type=env,id=USERNAME \
   --secret=type=env,id=PASSWORD \

And then in docker, use the secret:

RUN --mount=type=secret,id=USERNAME --mount=type=secret,id=PASSWORD \
   USER=$(cat /run/secrets/USERNAME) \
    PASS=$(cat /run/secrets/PASSWORD) \
     ./scriptThatUsesTheseEnvVarCredentialsToPullArtifacts
...rest of build..

Without the --secret flag to the kaniko executor, I'm not sure how to take advantage of docker secrets... nor do I understand the alternatives. I also want to continue to support developer builds. We have a 'build.sh' script that takes care of gathering credentials and adding them to the docker build command.

Current Solution

I found this article and was able to sort out a working solution. I want to ask the experts if this is valid or what the alternatives might be.

I discovered that when the kaniko executor runs, it appears to mount a volume into the image that's being built at: /kaniko. That directory does not exist when the build is complete and does not appear to be cached in the docker layers.

I also found out that if if the Dockerfile secret is not passed in via the docker build command, the build still executes.

So my gitlab-ci.yml file has this excerpt (the REPO_USER/REPO_PWD variables are GitLab CI variables):

- echo "${REPO_USER}" > /kaniko/repo-credentials.txt
- echo "${REPO_PWD}" >> /kaniko/repo-credentials.txt
- /kaniko/executor
  --context "${CI_PROJECT_DIR}/docker/target"
  --dockerfile "${CI_PROJECT_DIR}/docker/target/Dockerfile"
  --destination "${IMAGE_NAME}:${BUILD_TAG}"

Key piece here is echo'ing the credentials to a file in the /kaniko directory before calling the executor. That directory is (temporarily) mounted into the image which the executor is building. And since all this happens inside of the kaniko image, that file will disappear when kaniko (gitlab) job completes.

The developer build script (snip):

//to keep it simple, this assumes that the developer has their credentials//cached in a file (ignored by git) called dev-credentials.txt

DOCKER_BUILDKIT=1 docker build . \
   --secret id=repo-creds,src=dev-credentials.txt

Basically same as before. Had to put it in a file instead of environment variables.

The dockerfile (snip):

RUN --mount=type=secret,id=repo-creds,target=/kaniko/repo-credentials.txt USER=$(sed '1q;d' /kaniko/repo-credentials.txt) PASS=$(sed '2q;d' /kaniko/repo-credentials.txt) ./scriptThatUsesTheseEnvVarCredentialsToPullArtifacts...rest of build..

This Works!

In the Dockerfile, by mounting the secret in the /kaniko subfolder, it will work with both the DinD developer build as well as with the CI Kaniko executor.

For Dev builds, DinD secret works as always. (had to change it to a file rather than env variables which I didn't love.)

When the build is run by Kaniko, I suppose since the secret in the RUN command is not found, it doesn't even try to write the temporary credentials file (which I expected would fail the build). Instead, because I directly wrote the varibles to the temporarily mounted /kaniko directory, the rest of the run command was happy.

Advice

To me this does seem more kludgy than expected. I'm wanting to find out other/alternative solutions. Finding out the /kaniko folder is mounted into the image at build time seems to open a lot of possibilities.

The question includes the answer already, and I just try to show the complete code for user to pick directly

$ cat Dockerfile 
# podman build --secret id=mysecret,src=./secret.txt .
# /kaniko/executor --context . --dockerfile Dockerfile --no-push
FROM alpine
RUN --mount=type=secret,id=netrc \
    cp /kaniko/netrc $HOME/.netrc || cp /run/secrets/netrc $HOME/.netrc \
    && cat $HOME/.netrc \
    && echo "now can run some scripts" \
    && rm $HOME/.netrc
RUN ls $HOME/.netrc || true

If you use podman/buildah you can inject secret and use /run/secrets/netrc inside, it will not exist in end image

$ podman build --secret id=netrc,src=./secret.txt .
STEP 1/3: FROM alpine
STEP 2/3: RUN --mount=type=secret,id=netrc     cp /kaniko/netrc $HOME/.netrc || cp /run/secrets/netrc $HOME/.netrc     && cat $HOME/.netrc     && echo "now can run some scripts"     && rm $HOME/.netrc
cp: can't stat '/kaniko/netrc': No such file or directory
top secrets
now can run some scripts
--> 880d646a021
STEP 3/3: RUN ls $HOME/.netrc || true
ls: /root/.netrc: No such file or directory
COMMIT
--> 64a8b99ba19
64a8b99ba198d088f7cc678f160e29f5eae6ee90ff81dda2e292c55601cf4eb2

For kaniko, /kaniko is the folder for this purpose

/workspace # cat /kaniko/netrc
top secret!
/workspace # /kaniko/executor --context . --dockerfile Dockerfile  --no-push
INFO[0000] Retrieving image manifest alpine 
...
INFO[0002] Unpacking rootfs as cmd RUN --mount=type=secret,id=netrc     cp /kaniko/netrc $HOME/.netrc || cp /run/secrets/netrc $HOME/.netrc     && cat $HOME/.netrc     && echo "now can run some scripts"     && rm $HOME/.netrc requires it. 
INFO[0002] RUN --mount=type=secret,id=netrc     cp /kaniko/netrc $HOME/.netrc || cp /run/secrets/netrc $HOME/.netrc     && cat $HOME/.netrc     && echo "now can run some scripts"     && rm $HOME/.netrc 
INFO[0002] Initializing snapshotter ...                 
INFO[0002] Taking snapshot of full filesystem...        
INFO[0002] Cmd: /bin/sh                                 
INFO[0002] Args: [-c cp /kaniko/netrc $HOME/.netrc || cp /run/secrets/netrc $HOME/.netrc     && cat $HOME/.netrc     && echo "now can run some scripts"     && rm $HOME/.netrc] 
INFO[0002] Running: [/bin/sh -c cp /kaniko/netrc $HOME/.netrc || cp /run/secrets/netrc $HOME/.netrc     && cat $HOME/.netrc     && echo "now can run some scripts"     && rm $HOME/.netrc] 
top secret!
now can run some scripts
INFO[0002] Taking snapshot of full filesystem...        
INFO[0002] RUN ls $HOME/.netrc || true                  
INFO[0002] Cmd: /bin/sh                                 
INFO[0002] Args: [-c ls $HOME/.netrc || true]           
INFO[0002] Running: [/bin/sh -c ls $HOME/.netrc || true] 
ls: /root/.netrc: No such file or directory
INFO[0002] Taking snapshot of full filesystem...        
INFO[0002] No files were changed, appending empty layer to config. No layer added to image. 

I'm using this with python for private pkgs, it works fine. Here my script, maybe can help you:

Dockerfile:

COPY . .
RUN --mount=type=secret,id=pip.conf,target=/kaniko/pip.conf \ 
    PIP_CONFIG_FILE=/kaniko/pip.conf \
    pip install --no-cache-dir .

pip.conf file:

[global]
extra-index-url = 
    https://__token__:${GITLAB_DEPLOY_TOKEN}@gitlab-ucc.tools.aws.mycompany.com/api/v4/groups/245/-/packages/pypi/simple

pipeline script:

cp ./cicd/pip.conf /kaniko/pip.conf
sed -i "s/..GITLAB_DEPLOY_TOKEN./${GITLAB_DEPLOY_TOKEN}/g" /kaniko/pip.conf


/kaniko/executor \
      --cache=true \
      --cache-dir=${CI_PROJECT_DIR}/.cache \
      --cache-repo=${CI_REGISTRY_IMAGE}/cache \
      --context=${CI_PROJECT_DIR} \
      --dockerfile=${CI_PROJECT_DIR}/${DOCKERFILE_PATH}/Dockerfile \
      --destination=${CI_REGISTRY_IMAGE}/${CI_PROJECT_NAME}:${NEWVERSION} 

developer/manual script:

export GITLAB_DEPLOY_TOKEN=`pass GITLAB_DEPLOY_TOKEN`
envsubst < ./cicd/pip.conf > ./setup/pipconf.tmp

export DOCKER_BUILDKIT=1
docker build -f ./docker/Dockerfile . \
        --secret id=pip.conf,src=./setup/pipconf.tmp \
        --tag $IMG_NAME:dev 

Your solution works perfectly fine for us, thank you very much for sharing!

I just wanted to point out that it was necessary to export the variables first before using them in our environment, as described in this post: https://stackoverflow.com/a/69305942

RUN --mount=type=secret,id=repo-creds,target=/kaniko/repo-credentials.txt \ 
    USER=$(sed '1q;d' /kaniko/repo-credentials.txt) \ 
    PASS=$(sed '2q;d' /kaniko/repo-credentials.txt) \
    && export USER \
    && export PASS \
    && echo -n "${USER}:${PASS}" 

Additionally, the variable has to be used in the same RUN statement it is exported in. It does not seem to be available in another RUN statement.