How do I set docker-credential-ecr-login in my PATH before anything else in GitLab CI

huangapple go评论74阅读模式
英文:

How do I set docker-credential-ecr-login in my PATH before anything else in GitLab CI

问题

I'm using AWS ECR to host a private Dockerfile image, and I would like to use it in GitLab CI.

Accordingly to the documentation I need to set docker-credential-ecr-login to fetch the private image, but I have no idea how to do that before anything else. That's my .gitlab-ci file:

image: 0222822883.dkr.ecr.us-east-1.amazonaws.com/api-build:latest

tests:
stage: test
before_script:
- echo "before_script"
- apt install amazon-ecr-credential-helper
- apk add --no-cache curl jq python py-pip
- pip install awscli
script:
- echo "script"
- bundle install
- bundle exec rspec
allow_failure: true # for now as we do not have tests

Thank you.

英文:

I'm using AWS ECR to host a private Dockerfile image, and I would like to use it in GitLab CI.

Accordingly to the documentation I need to set docker-credential-ecr-login to fetch the private image, but I have no idea how to do that before anything else. That's my .gitlab-ci file:

image: 0222822883.dkr.ecr.us-east-1.amazonaws.com/api-build:latest

tests:
  stage: test
  before_script:
    - echo "before_script"
    - apt install amazon-ecr-credential-helper
    - apk add --no-cache curl jq python py-pip
    - pip install awscli
  script:
    - echo "script"
    - bundle install
    - bundle exec rspec
  allow_failure: true # for now as we do not have tests

Thank you.

答案1

得分: 3

我确认所涉及的功能目前在GitLab CI中还不可用;但是我最近看到可以实现一种通用的解决方案,运行一个在私有Docker镜像中取出的容器内的专用CI脚本。

以下是从OP的示例中调整的模板文件.gitlab-ci.yml,使用我在这个其他SO答案中建议的Docker-in-Docker方法,它本身受到GitLab CI文档 dealing with dind的启发:

stages:
  - test

variables:
  IMAGE: "0222822883.dkr.ecr.us-east-1.amazonaws.com/api-build:latest"
  REGION: "ap-northeast-1"

tests:
  stage: test
  image: docker:latest
  services:
    - docker:dind
  variables:
    # GIT_STRATEGY: none  # uncomment if "git clone" is unneeded for this job
  before_script:
    - ': before_script'
    - apt install amazon-ecr-credential-helper
    - apk add --no-cache curl jq python py-pip
    - pip install awscli
    - $(aws ecr get-login --no-include-email --region "$REGION")
    - docker pull "$IMAGE"
  script:
    - ': script'
    - |
      docker run --rm -v "$PWD:/build" -w /build "$IMAGE" /bin/bash -c "
        export PS4='+ \e[33;1m($CI_JOB_NAME @ line $LINENO) $\e[0m '  # optional
        set -ex
        ## TODO insert your multi-line shell script here ##
        echo \"One comment\"  # quotes must be escaped here
        : A better comment
        echo $PWD  # interpolated outside the container
        echo $PWD  # interpolated inside the container
        bundle install
        bundle exec rspec
        ## (cont'd) ##
      "      
    - ': done'
  allow_failure: true # for now as we do not have tests

此示例假设Docker $IMAGE 包含 /bin/bash 二进制文件,并依赖所谓的YAML块样式

上述模板已经包含注释,但为了自包含:

  • 如果您的Bash命令中包含双引号,则需要对其进行转义,因为整个代码被包围在 docker run … "" 中;
  • 您还需要转义本地Bash变量(参见上面的 $PWD ),否则这些变量将在运行 docker run … "$IMAGE" /bin/bash -c "…" 命令之前解析。
  • 我用它们更有效的冒号替换了 echo "stuff" 或类似命令:
    set -x
    : stuff
    : note that these three shell commands do nothing
    : but printing their args thanks to the -x option.
    

[欢迎提供反馈,因为我不能直接测试这个配置(我不是AWS ECR的用户),但我对OP的示例同时包含了一些aptapk命令感到困惑……]

关于set -e的一个陷阱的相关备注

请注意,以下脚本有错误:

set -e
command1 && command2
command3

换句话说,应该写成:

set -e
command1 ; command2
command3

或者:

set -e
( command1 && command2 )
command3

要相信这一点,您可以尝试运行:

bash -e -c 'false && true; echo $?; echo this should not be run'
1
  → this should not be run
bash -e -c 'false; true; echo $?; echo this should not be run'
bash -e -c '( false && true ); echo $?; echo this should not be run'
英文:

I confirm the feature at stake is not yet available in GitLab CI; however I've recently seen it is possible to implement a generic workaround to run a dedicated CI script within a container taken from a private Docker image.

The template file .gitlab-ci.yml below is adapted from the OP's example, using the Docker-in-Docker approach I suggested in this other SO answer, itself inspired by the GitLab CI doc dealing with dind:

stages:
  - test

variables:
  IMAGE: "0222822883.dkr.ecr.us-east-1.amazonaws.com/api-build:latest"
  REGION: "ap-northeast-1"

tests:
  stage: test
  image: docker:latest
  services:
    - docker:dind
  variables:
    # GIT_STRATEGY: none  # uncomment if "git clone" is unneeded for this job
  before_script:
    - ': before_script'
    - apt install amazon-ecr-credential-helper
    - apk add --no-cache curl jq python py-pip
    - pip install awscli
    - $(aws ecr get-login --no-include-email --region "$REGION")
    - docker pull "$IMAGE"
  script:
    - ': script'
    - |
      docker run --rm -v "$PWD:/build" -w /build "$IMAGE" /bin/bash -c "
        export PS4='+ \e[33;1m($CI_JOB_NAME @ line $LINENO) $\e[0m '  # optional
        set -ex
        ## TODO insert your multi-line shell script here ##
        echo \"One comment\"  # quotes must be escaped here
        : A better comment
        echo $PWD  # interpolated outside the container
        echo $PWD  # interpolated inside the container
        bundle install
        bundle exec rspec
        ## (cont'd) ##
      "      
    - ': done'
  allow_failure: true # for now as we do not have tests

This example assumes the Docker $IMAGE contains the /bin/bash binary, and relies on the so-called block style of YAML.

The above template already contains comments, but to be self-contained:

  • You need to escape double quotes if your Bash commands contain them, because the whole code is surrounded by docker run … " and ";
  • You also need to escape local Bash variables (cf. the \$PWD above), otherwise these variables will be resolved prior running the docker run … "$IMAGE" /bin/bash -c "…" command itself.
  • I replaced the echo "stuff" or so commands with their more effective colon counterpart:
    set -x
    : stuff
    : note that these three shell commands do nothing
    : but printing their args thanks to the -x option.
    

[Feedback is welcome as I can't directly test this config (I'm not an AWS ECR user), but I'm puzzled by the fact the OP's example contained at the same time some apt and apk commands…]

Beware that the following script is buggy:

set -e
command1 && command2
command3

Namely, write instead:

set -e
command1 ; command2
command3

or:

set -e
( command1 && command2 )
command3

To be convinced about this, you can try running:

bash -e -c 'false && true; echo $?; echo this should not be run'
1
  → this should not be run
bash -e -c 'false; true; echo $?; echo this should not be run'
bash -e -c '( false && true ); echo $?; echo this should not be run'

答案2

得分: 0

从GitLab文档中。为了与您的AWS帐户交互,GitLab CI/CD管道需要在您的GitLab设置下定义AWS_ACCESS_KEY_ID和AWS_SECRET_ACCESS_KEY两个变量,位置在Settings > CI/CD > Variables。然后在您的before script中添加以下内容:

image: 0222822883.dkr.ecr.us-east-1.amazonaws.com/api-build:latest

tests:
  stage: test
  before_script:
    - echo "before_script"
    - apt install amazon-ecr-credential-helper
    - apk add --no-cache curl jq python py-pip
    - pip install awscli
    - $( aws ecr get-login --no-include-email )
  script:
    - echo "script"
    - bundle install
    - bundle exec rspec
  allow_failure: true # for now as we do not have tests

另外,请注意您有一个拼写错误,应为awscli而不是awsclir。然后按照相应的步骤构建、测试和推送。

英文:

From GitLab documentation. In order to interact with your AWS account, the GitLab CI/CD pipelines require both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to be defined in your GitLab settings under Settings > CI/CD > Variables. Then add to your before script:

image: 0222822883.dkr.ecr.us-east-1.amazonaws.com/api-build:latest

tests:
  stage: test
  before_script:
    - echo "before_script"
    - apt install amazon-ecr-credential-helper
    - apk add --no-cache curl jq python py-pip
    - pip install awscli
    - $( aws ecr get-login --no-include-email )
  script:
    - echo "script"
    - bundle install
    - bundle exec rspec
  allow_failure: true # for now as we do not have tests

Also, you had a typo is awscli, not awsclir.Then add the builds, tests and push accordingly.

答案3

得分: 0

我认为在这种情况下,您的代码中存在某种逻辑错误。构建配置中的image是一个CI脚本运行器镜像,而不是您构建和部署的镜像。

我认为您不必在任何情况下使用它,因为它只是一个具有实用工具和与GitLab CI等的连接的镜像。这个镜像通常不应该有您项目的任何依赖关系。

请查看像这个示例 https://gist.github.com/jlis/4bc528041b9661ae6594c63cd2ef673c 以更清楚地了解如何以正确的方式进行操作。

英文:

I think that you have some sort of logic error in the case. image in the build configuration is a CI scripts runner image, not image you build and deploy.

I think you don't have to use it in any case since it is just an image which has utilities & connections to the GitLab CI & etc. The image shouldn't have any dependencies of your project normally.

Please check examples like this one https://gist.github.com/jlis/4bc528041b9661ae6594c63cd2ef673c to get it more clear how to do it a correct way.

答案4

得分: 0

我在使用GitLab Runner的Docker执行模式时遇到了相同的问题。

通过SSH登录到EC2实例后,我发现docker-credential-ecr-login位于/usr/bin/目录中。为了将它传递到容器中,我不得不将此包挂载到GitLab Runner容器中。

更多信息可以在此线程中找到:https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1583#note_375018948

英文:

I faced the same problem using docker executor mode of gitlab runner.

SSH into the EC2 instance showed that docker-credential-ecr-login was present in /usr/bin/. To pass it to the container I had to mount this package to the gitlab runner container.

gitlab-runner register -n \
--url '${gitlab_url}' \
--registration-token '${registration_token}' \
--template-config /tmp/gitlab_runner.template.toml \
--executor docker \
--tag-list '${runner_name}' \
--description 'gitlab runner for ${runner_name}' \
--docker-privileged \
--docker-image "alpine" \
--docker-disable-cache=true \
--docker-volumes "/var/run/docker.sock:/var/run/docker.sock" \
--docker-volumes "/cache" \
--docker-volumes "/usr/bin/docker-credential-ecr-login:/usr/bin/docker-credential-ecr-login" \
--docker-volumes "/home/gitlab-runner/.docker:/root/.docker"

More information on this thread as well: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1583#note_375018948

答案5

得分: 0

以下是翻译好的内容:

我们有类似的设置,需要基于托管在 ECR 上的镜像运行 CI 作业。
按照以下步骤进行:

  • 遵循此指南>> https://github.com/awslabs/amazon-ecr-credential-helper

  • 上述链接的要点是,如果您使用的是 "Amazon Linux 2",则执行以下命令:

    sudo amazon-linux-extras enable docker

    sudo yum install amazon-ecr-credential-helper

  • 在您的 GitLab Runner 上使用 VI 编辑器打开 ~/.docker/config.json

  • 在 ~/.docker/config.json 中粘贴以下代码:

    {
    "credHelpers":
    {
    "aws_account_id.dkr.ecr.region.amazonaws.com": "ecr-login"
    }
    }

  • 运行命令 source ~/.bashrc

  • 重启 Docker 服务:systemctl restart docker

  • 同时,从您的 GitLab>>CI/CD>> 变量中删除任何关于 DOCKER_AUTH_CONFIG 的引用

就是这样。

英文:

We have a similar setup where we need to run CI jobs based off of an Image that is hosted on ECR.
Steps to follow:-

  • follow this guide here>> https://github.com/awslabs/amazon-ecr-credential-helper

  • gist of this above link is if you are on "Amazon Linux 2"

    sudo amazon-linux-extras enable docker

    sudo yum install amazon-ecr-credential-helper

  • open the ~/.docker/config.json on your gitlab runner in VI editor

  • Paste this code in the ~/.docker/config.json

    {
    "credHelpers":
    {
    "aws_account_id.dkr.ecr.region.amazonaws.com": "ecr-login"
    }
    }

  • source ~/.bashrc

  • systemctl restart docker

  • also remove any references of DOCKER_AUTH_CONFIG from your GitLab>>CI/CD>> Variables

That's it

huangapple
  • 本文由 发表于 2020年1月3日 22:19:03
  • 转载请务必保留本文链接:https://go.coder-hub.com/59580136.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定