在Github actions中运行集成测试时遇到了与PostgreSQL连接的问题。

huangapple
go评论124阅读模式
英文:

Running integration tests in Github actions: issues with connecting with postgres

问题

我有一些集成测试，为了成功运行，需要一个正在运行的postgres数据库，通过docker-compose进行设置，并且我的go应用程序从main.go运行。这是我的docker-compose文件：

  1. version: "3.9"
  2. services:
  4.   postgres:
  5.     image: postgres:12.5
  6.     user: postgres
  7.     environment:
  8.       POSTGRES_USER: postgres
  9.       POSTGRES_PASSWORD: password
  10.       POSTGRES_DB: my-db
  11.     ports:
  12.       - "5432:5432"
  13.     volumes:
  14.       - data:/var/lib/postgresql/data
  15.       - ./initdb:/docker-entrypoint-initdb.d
  17. networks:
  18.   default:
  19.     driver: bridge
  21. volumes:
  22.   data:
  23.     driver: local

我的Github Actions配置如下：

  1. jobs:
  2.   unit:
  3.     name: Test
  4.     runs-on: ubuntu-latest
  5.     services:
  6.       postgres:
  7.         image: postgres:12.5
  8.         env:
  9.           POSTGRES_USER: postgres
  10.           POSTGRES_PASSWORD: password
  11.           POSTGRES_DB: my-db
  12.         ports:
  13.           - 5432:5432
  14.     env:
  15.       GOMODCACHE: "${{ github.workspace }}/.go/mod/cache"
  16.       TEST_RACE: true 
  17.     steps:
  18.       - name: Initiate Database
  19.         run: psql -f initdb/init.sql postgresql://postgres:password@localhost:5432/my-db
  21.       - name: Set up Cloud SDK
  22.         uses: google-github-actions/setup-gcloud@v0
  24.       - name: Authenticate with GCP
  25.         id: auth
  26.         uses: "google-github-actions/auth@v0"
  27.         with:
  28.           credentials_json: ${{ secrets.GCP_ACTIONS_SECRET }}
  30.       - name: Configure Docker
  31.         run: |
  32.                     gcloud auth configure-docker "europe-docker.pkg.dev,gcr.io,eu.gcr.io"
  34.       - name: Set up Docker BuildX
  35.         uses: docker/setup-buildx-action@v1
  37.       - name: Start App
  38.         run: |
  39.           VERSION=latest make images
  40.           docker run -d -p 3000:3000 -e POSTGRES_DB_URL='//postgres:password@localhost:5432/my-db?sslmode=disable' --name='app' image/app          
  42.       - name: Tests
  43.         env:
  44.           POSTGRES_DB_URL: //postgres:password@localhost:5432/my-db?sslmode=disable
  45.           GOMODCACHE: ${{ github.workspace }}/.go/pkg/mod
  46.         run: | 
  47.           make test-integration
  48.           docker stop app

我的测试在本地使用docker-compose up启动docker-compose并从main.go运行应用程序时正常运行。然而，在Github Actions中，我遇到了以下错误：

  1. failed to connect to `host=/tmp user=nonroot database=`: dial error (dial unix /tmp/.s.PGSQL.5432: connect: no such file or directory

我漏掉了什么？谢谢。

英文:

I have some integration tests that, in order to succesfully run, require a running postgres database, setup via docker-compose, and my go app running from main.go. Here is my docker-compose:

  1. version: "3.9"
  2. services:
  4.   postgres:
  5.     image: postgres:12.5
  6.     user: postgres
  7.     environment:
  8.       POSTGRES_USER: postgres
  9.       POSTGRES_PASSWORD: password
  10.       POSTGRES_DB: my-db
  11.     ports:
  12.       - "5432:5432"
  13.     volumes:
  14.       - data:/var/lib/postgresql/data
  15.       - ./initdb:/docker-entrypoint-initdb.d
  16. networks:
  17.   default:
  18.     driver: bridge
  20. volumes:
  21.   data:
  22.     driver: local

and my Github Actions are as follows:

  1. jobs:
  2.   unit:
  3.     name: Test
  4.     runs-on: ubuntu-latest
  5.     services:
  6.       postgres:
  7.         image: postgres:12.5
  8.         env:
  9.           POSTGRES_USER: postgres
  10.           POSTGRES_PASSWORD: password
  11.           POSTGRES_DB: my-db
  12.         ports:
  13.           - 5432:5432
  14.     env:
  15.       GOMODCACHE: "${{ github.workspace }}/.go/mod/cache"
  16.       TEST_RACE: true 
  17.    steps:
  18.      - name: Initiate Database
  19.        run: psql -f initdb/init.sql postgresql://postgres:password@localhost:5432/my-db
  21.      - name: Set up Cloud SDK
  22.        uses: google-github-actions/setup-gcloud@v0
  24.      - name: Authenticate with GCP
  25.        id: auth
  26.        uses: "google-github-actions/auth@v0"
  27.        with: credentials_json: ${{ secrets.GCP_ACTIONS_SECRET }}
  29.      - name: Configure Docker
  30.        run: |
  31.          gcloud auth configure-docker "europe- docker.pkg.dev,gcr.io,eu.gcr.io"
  33.      - name: Set up Docker BuildX
  34.        uses: docker/setup-buildx-action@v1
  36.      - name: Start App
  37.        run: |
  38.          VERSION=latest make images
  39.          docker run -d -p 3000:3000 -e      POSTGRES_DB_URL='//postgres:password@localhost:5432/my-db?sslmode=disable' --name='app' image/app
  41.      - name: Tests
  42.        env:
  43.         POSTGRES_DB_URL: //postgres:password@localhost:5432/my-db?sslmode=disable
  44.       GOMODCACHE: ${{ github.workspace }}/.go/pkg/mod
  45.        run: | 
  46.          make test-integration
  47.          docker stop app

My tests run just fine locally firing off the docker-compose with docker-compose up and running the app from main.go. However, in Github actions I am getting the following error:

  1. failed to connect to `host=/tmp user=nonroot database=`: dial error (dial unix /tmp/.s.PGSQL.5432: connect: no such file or directory

What am I missing? Thanks

答案1

得分: 4

你所缺少的是在 Github Actions 服务器上设置实际的 Postgres 客户端（这就是为什么找不到 psql 工具）。

按照以下步骤进行设置（参考链接：https://remarkablemark.org/blog/2021/03/14/setup-postgresql-in-github-actions/）：

  1. - name: 安装 PostgreSQL 客户端
  2.   run: |
  3.     apt-get update
  4.     apt-get install --yes postgresql-client

除此之外，如果你通过 docker-compose 运行所有内容，你需要等待 Postgres 启动并运行（健康且接受连接）。

考虑以下 docker-compose 配置：

  1. version: '3.1'
  3. services:
  4.   api:
  5.     build: .
  6.     depends_on:
  7.       - db
  8.     ports:
  9.       - 8080:8080
  10.     environment:
  11.       - RUN_UP_MIGRATION=true
  12.       - PSQL_CONN_STRING=postgres://gotstock_user:123@host.docker.internal:5432/gotstockapi?sslmode=disable
  13.     command: ./entry
  14.     
  15.   db:
  16.     image: postgres:9.5-alpine
  17.     restart: always
  18.     environment:
  19.       - POSTGRES_USER=root
  20.       - POSTGRES_PASSWORD=password
  21.     ports:
  22.       - "5432:5432"
  23.     volumes:
  24.       - ./db:/docker-entrypoint-initdb.d/

有几个需要注意的地方。首先，在 api 的 environment 部分，我们有 PSQL_CONN_STRING=postgres://gotstock_user:123@host.docker.internal:5432/gotstockapi?sslmode=disable，它是作为环境变量传递给数据库的连接字符串。注意主机是 host.docker.internal。

此外，在 api 部分我们有 command: ./entry。entry 文件包含以下 #!/bin/ash 脚本：

  1. #!/bin/ash
  2. NOT_READY=1
  3. while [ $NOT_READY -gt 0 ] # <- 循环等待直到 Postgres 准备好接受连接
  4. do 
  5.     pg_isready --dbname=gotstockapi --host=host.docker.internal --port=5432 --username=gotstock_user
  6.     NOT_READY=$?
  7.     sleep 1
  8. done;
  9. ./gotstock-api # <- 实际执行构建 api
  10. sleep 10
  11. go test -v ./it # <- 执行集成测试

最后，为了使上述脚本中的 psql 客户端正常工作，api 的 Dockerfile 如下所示：

  1. # syntax=docker/dockerfile:1
  3. FROM golang:1.19-alpine3.15
  5. RUN apk add build-base
  7. WORKDIR /app
  9. COPY go.mod go.sum ./
  10. RUN go mod download && go mod verify
  12. COPY . .
  14. RUN apk add postgresql-client
  16. RUN go build -o gotstock-api
  18. EXPOSE 8080

注意 RUN apk add postgresql-client，它安装了客户端。

祝你愉快编码！=)

英文:

What you are missing is setting up the actual Postgres Client inside the Github Actions server (that is why there is no psql tool to be found).

Set it up as a step.

  1.      - name: Install PostgreSQL client
  2.         run: |
  3.           apt-get update
  4.           apt-get install --yes postgresql-client

Apart from that, if you run everything through docker-compose you will need to wait for postgres to be up and running (healthy & accepting connections).

Consider the following docker-compose:

  1. version: &#39;3.1&#39;
  3. services:
  4.   api:
  5.     build: .
  6.     depends_on:
  7.       - db
  8.     ports:
  9.       - 8080:8080
  10.     environment:
  11.       - RUN_UP_MIGRATION=true
  12.       - PSQL_CONN_STRING=postgres://gotstock_user:123@host.docker.internal:5432/gotstockapi?sslmode=disable
  13.     command: ./entry
  14.     
  15.   db:
  16.     image: postgres:9.5-alpine
  17.     restart: always
  18.     environment:
  19.       - POSTGRES_USER=root
  20.       - POSTGRES_PASSWORD=password
  21.     ports:
  22.       - &quot;5432:5432&quot;
  23.     volumes:
  24.       - ./db:/docker-entrypoint-initdb.d/

There are a couple of things you need to notice. First of all, in the environment section of the api we have PSQL_CONN_STRING=postgres://gotstock_user:123@host.docker.internal:5432/gotstockapi?sslmode=disable which is the connection string to the db being passed as an env variable. Notice the host is host.docker.internal.

Besides that we have command: ./entry in the api section. The entry file contains the following #!/bin/ash script:

  1. #!/bin/ash
  2. NOT_READY=1
  3. while [ $NOT_READY -gt 0 ] # &lt;- loop that waits till postgres is ready to accept connections
  4. do 
  5.     pg_isready --dbname=gotstockapi --host=host.docker.internal --port=5432 --username=gotstock_user
  6.     NOT_READY=$?
  7.     sleep 1
  8. done;
  9. ./gotstock-api # &lt;- actually executes the build of the api
  10. sleep 10
  11. go test -v ./it # &lt;- executes the integration-tests

And finally, in order for the psql client to work in the above script, the docker file of the api is looking like this:

  2. # syntax=docker/dockerfile:1
  4. FROM golang:1.19-alpine3.15
  6. RUN apk add build-base
  8. WORKDIR /app
  10. COPY go.mod go.sum ./
  11. RUN go mod download &amp;&amp; go mod verify
  13. COPY . .
  15. RUN apk add postgresql-client
  17. RUN go build -o gotstock-api
  19. EXPOSE 8080

Notice RUN apk add postgresql-client which installs the client.

Happy hacking! =)

答案2

得分: 3

我认为这段代码有多个问题。

问题一：
在你的代码中，我没有看到你运行 docker-compose up，因此我会认为 Postgres 没有在运行。

问题二：
在这一行中：docker run -d -p 3000:3000 -e POSTGRES_DB_URL='//postgres:password@localhost:5432/my-app?sslmode=disable' --name='app' image/app

你将 Postgres 的主机指向了 localhost，在你的本地机器上是可以工作的。然而，当你使用 docker run 时，你并不是在本地机器上运行，而是在一个 Docker 容器中运行。在容器内部，localhost 指向的是容器内部。

解决方案：
由于你已经在使用 docker-compose，我建议你也将测试的 Web 服务器添加到其中。

将你的 docker-compose 文件更改为：

  1. version: "3.9"
  2. services:
  4.   webapp:
  5.     build: image/app
  6.     environment:
  7.       POSTGRES_DB_URL='//postgres:password@postgres:5432/my-app?sslmode=disable'
  8.     ports:
  9.       - "3000:3000"
  10.     depends_on:
  11.       - "postgres"
  13.   postgres:
  14.     image: postgres:12.5
  15.     user: postgres
  16.     environment:
  17.       POSTGRES_USER: postgres
  18.       POSTGRES_PASSWORD: password
  19.       POSTGRES_DB: my-app
  20.     ports:
  21.       - "5432:5432"
  22.     volumes:
  23.       - data:/var/lib/postgresql/data
  24.       - ./initdb:/docker-entrypoint-initdb.d
  25. networks:
  26.   default:
  27.     driver: bridge
  29. volumes:
  30.   data:
  31.     driver: local

如果你现在运行 docker-compose up，两个服务都将可用。应该可以工作了。不过我不是一个 github-actions 专家，所以可能会漏掉一些东西。至少这样，你可以以与 CI 相同的方式在本地运行测试，我认为这是一个很大的优点。

英文:

I think this code has more than one problem.

Problem one:
In your code I don't see you run docker-compose up, therefore I would assume that Postgres is not running.

Problem two:
is in this line: docker run -d -p 3000:3000 -e POSTGRES_DB_URL=&#39;//postgres:password@localhost:5432/my-app?sslmode=disable&#39; --name=&#39;app&#39; image/app

You point the host of Postgres to localhost, which on your local machine works. As there localhost is your local comuter. Though, as you use docker run you are not running this on your local machine, but in a docker container. There localhost is pointing to inside the conmtainer.

Posible solution for both

As you are already using docker-compose I suggest you to also add your test web server there.

Change your docker-compose file to:

  1. version: &quot;3.9&quot;
  2. services:
  4.   webapp:
  5.     build: image/app
  6.     environment:
  7.       POSTGRES_DB_URL=&#39;//postgres:password@postgres:5432/my-app?sslmode=disable&#39;
  8.     ports:
  9.       - &quot;3000:3000&quot;
  10.     depends_on:
  11.       - &quot;postgres&quot;
  13.   postgres:
  14.     image: postgres:12.5
  15.     user: postgres
  16.     environment:
  17.       POSTGRES_USER: postgres
  18.       POSTGRES_PASSWORD: password
  19.       POSTGRES_DB: my-app
  20.     ports:
  21.       - &quot;5432:5432&quot;
  22.     volumes:
  23.       - data:/var/lib/postgresql/data
  24.       - ./initdb:/docker-entrypoint-initdb.d
  25. networks:
  26.   default:
  27.     driver: bridge
  29. volumes:
  30.   data:
  31.     driver: local

If you now run docker-compose up, both services will be available. And it should work. Though I am not a github-actions expert, so I might have missed something. At least like this, you can run your tests locally the same way as in CI, something that I always see as a big plus.

huangapple
  • 本文由 发表于 2022年11月1日 05:45:33
  • 转载请务必保留本文链接：https://go.coder-hub.com/74269416.html
  • docker
  • docker-compose
  • go
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定