How can I run a single Terraform GitHub Actions pipeline that applies multiple terraform resources (each with own state files) at same time?

huangapple go评论69阅读模式
英文:

How can I run a single Terraform GitHub Actions pipeline that applies multiple terraform resources (each with own state files) at same time?

问题

我在GitHub Actions中有一个流水线,当用户提出PR时,会运行Terraform Plan,一旦PR获批准,就会运行Apply。

流水线配置了一个Terraform工作目录,并仅为特定目录运行Terraform init,例如user-1/projectA硬编码到我的流水线中。

然而,同一仓库还包含其他文件夹,例如带有相同结构的projectB。还有其他用户文件夹,例如user-2;

user-project1 
 └ projectA 
  └ main.tf 
  └ backend.tf 
  └ variables.tfvars 
 └ projectB 
  └ main.tf 
  └ backend.tf 
  └ variables.tfvars 
user-project2 
 └ projectA 
  └ main.tf 
  └ backend.tf 
  └ variables.tfvars 
 └ projectB 
  └ main.tf 
  └ backend.tf 
  └ variables.tfvars 

我想要一个单一的流水线,可以在这个包含多个项目的monorepo中的任何更改上运行,例如,如果在上述任何文件中进行了任何更改。

背景是,我有一个包含多个GCP项目的monorepo。只有我可以访问此仓库并进行更改。当我收到一群用户的请求,需要在多个项目中进行一些更改(例如启用新的API)时,我希望进行所有这些更改,然后运行单一的流水线来将它们全部应用。

每个项目在远程后端中都有自己的状态文件。

我仅使用Terraform CLI,没有使用TF Cloud或Enterprise的选项。

英文:

I have a pipeline in GitHub Actions that runs a terraform Plan when a user raises a PR, and an Apply once that PR is approved.

The pipeline is configured to set a Terraform working directory and run Terraform init for a specific directory only e.g user-1/projectA is hard coded into my pipeline

However the same repo also contains additional folders, e.g projectB with the same structure as above. And also additional user folders, e.g user-2;

user-project1 
 └ projectA 
  └ main.tf 
  └ backend.tf 
  └ variables.tfvars 
 └ projectB 
  └ main.tf 
  └ backend.tf 
  └ variables.tfvars 
user-project2 
 └ projectA 
  └ main.tf 
  └ backend.tf 
  └ variables.tfvars 
 └ projectB 
  └ main.tf 
  └ backend.tf 
  └ variables.tfvars 

I'd like a single pipeline that will run for any change in this monorepo that contains multiple projects, e.g if any change is made in any of the files above.

The context is, I have a monorepo that contains multiple GCP projects. Only I have access to this repo and to make changes. When I get a request from a a bunch of users to make some changes across a number of projects (enabling a new API for example), I'd like to make all those changes and then run a single pipeline to get them all applied.

Each project has its own state file in a remote backend

I am using Terraform CLI only with no option to use TF Cloud or Enterprise

答案1

得分: 0

你可以使用可重复使用的工作流程,以及矩阵策略。在这种情况下,你将会有2个工作流程:

  • 一个名为 main 的工作流程,包含2个作业:
    • 一个用于识别已更新的文件夹
    • 一个用于调用可重复使用的工作流程以针对已更新的文件夹执行 terraform apply 命令。
  • 一个名为 reusable workflow 的工作流程,它将接收一个 folder 作为输入,然后在执行 terraform apply 命令之前,使用 cd 进入此文件夹。

matrix 将被填充为 main 工作流程上已更新的文件夹列表,并为每个已更新的文件夹调用可重复使用的工作流程。

以下是一个示例:

main workflow

on:
  push:
    paths:
      - 'folder1/**'
      - 'folder2/**'

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  setup:
    runs-on: ubuntu-latest
    outputs:
      folders: ${{ steps.array.outputs.array }}
    steps:
      - uses: actions/checkout@v3
      - uses: dorny/paths-filter@v2
        id: changes
        with:
          filters: |
            folder1:
              - 'folder1/**'
            folder2:
              - 'folder2/**'            

      - name: Build Array
        id: array
        run: |
          myArray=()
          if [ "${{ steps.changes.outputs.folder1 }}" = "true" ]
          then
            myArray+=("folder1")
          fi
          if [ "${{ steps.changes.outputs.folder2 }}" = "true" ]
          then
            myArray+=("folder2")
          fi
          myArray=$(jq --compact-output --null-input '$ARGS.positional' --args -- "${myArray[@]}")
          echo "Updated folder list: $myArray"
          echo "array=$myArray" >> $GITHUB_OUTPUT          
        shell: bash

  build:
    needs: [setup]
    strategy:
      fail-fast: false
      matrix:
        folder: ${{ fromJSON(needs.setup.outputs.folders) }}
    uses: ./.github/workflows/reusable.yml
    with:
      path: ${{ matrix.folder }}

注意:你可以使用脚本来替代执行所有的 shell 命令,但我发现这种方式更容易,因为 dorny/paths-filter action 为每个文件夹生成一个输出。

reusable workflow

on:
  workflow_call:
    inputs:
      path:
        required: true
        type: string

concurrency:
    group: ${{ github.workflow }}-${{ github.ref }}-${{ inputs.path }}
    cancel-in-progress: true

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - run: |
          cd ${{ inputs.path }}
          terraform apply          

请确保按照示例中的格式和结构设置你的工作流程。

英文:

You could use a reusable workflow along with a matrix strategy.

In that case, you would have 2 workflows:

  • a main workflow with 2 jobs
    • one that identify the updated folders
    • one that call the reusable workflow for the updated folders.
  • a reusable workflow that would receive a folder as input, and then cd to this folder before performing a terraform apply command.

The matrix would be filled with the list of updated folders on the main workflow, calling the reusable workflow for each folder updated.

Here is an example:

main workflow

on:
  push:
    paths:
      - 'folder1/**'
      - 'folder2/**'

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  setup:
    runs-on: ubuntu-latest
    outputs: 
      folders: ${{ steps.array.outputs.array }}
    steps:
      - uses: actions/checkout@v3
      - uses: dorny/paths-filter@v2
        id: changes
        with:
          filters: |
            folder1:
              - 'folder1/**'
            folder2:
              - 'folder2/**'

      - name: Build Array
        id: array
        run: |
          myArray=()
          if [ "${{ steps.changes.outputs.folder1 }}" = "true" ]
          then
            myArray+=("folder1")
          fi
          if [ "${{ steps.changes.outputs.folder2 }}" = "true" ]
          then
            myArray+=("folder2")
          fi
          myArray=$(jq --compact-output --null-input '$ARGS.positional' --args -- "${myArray[@]}")
          echo "Updated folder list: $myArray"
          echo "array=$myArray" >> $GITHUB_OUTPUT
        shell: bash

  build:
    needs: [setup]
    strategy:
      fail-fast: false
      matrix:
        folder: ${{ fromJSON(needs.setup.outputs.folders) }}
    uses: ./.github/workflows/reusable.yml
    with:
      path: ${{ matrix.folder }}

Note: You may use a script instead of doing all the shell commands, but I found it easier that way as the dorny/paths-filter action generates an output for each folder.

reusable workflow

on:
  workflow_call:
    inputs:
      path:
        required: true
        type: string

concurrency:
    group: ${{ github.workflow }}-${{ github.ref }}-${{ inputs.path }}
    cancel-in-progress: true

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - run: |
          cd ${{ inputs.path }}
          terraform apply

huangapple
  • 本文由 发表于 2023年7月3日 05:19:15
  • 转载请务必保留本文链接:https://go.coder-hub.com/76600848.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定