英文:

How to Cache Terraform init

问题

我们有Terraform模板，用于创建基础架构，并启动Shell脚本，然后调用Ansible Playbooks来部署应用程序。

这两个Terraform模板被打包，并以自动化方式从UI应用程序中调用 - 目的是为开发人员提供自助服务，他们可以通过单击完成整个堆栈 - 基础架构和应用程序的部署。

每当开发人员提出请求时，都会触发此自动化，并且在代码中，我们每次运行Terraform init以确保不会在后续出现插件不可用的情况。每次运行Terraform init 时，它都会反复加载插件二进制文件到内存中，导致内存不足。

是否有可能解决方案来防止Terraform init 反复加载插件二进制文件到内存中，并在内存中重用现有的二进制文件（如果它们已经存在于内存中）？

英文:

We have terraform templates that provision infrastructure and also kick off shell scripts that in turn invoke ansible playbooks to deploy an application.

Both terraform templates are packaged and get invoked from a UI application in an automated way - the intention is provide self-service to developers where they get the entire stack - infra and application deployed with single click.

This automation gets invoked every time a developer raises a request and in the code, we are running terraform init every time to ensure we don't run into any non-availability of plugins down the line. With every run of terraform init, it's loading the plugin binaries over and over into memory and we are running out of memory.
Is there a possible solution to prevent terraform init from loading plugin binaries into memory over and over and reuse existing binaries if they are already in memory?

答案1

得分: 2

I assume that when you say "memory" in your question you mean files in the filesystem, rather than data in RAM. terraform init does not load the provider plugins into RAM, but it does extract them into a cache directory under the .terraform directory.

If your intention is to allow your system to keep working when registry.terraform.io is down or when a plugin has been deleted then you can configure Terraform to install plugins from a local directory instead of fetching them over the network every time.

First choose a directory where you'll place your copies of the provider plugins. For example here I'm going to use /opt/terraform/plugins, but you can use any directory path that Terraform will be able to access.

In the home directory of the system user that Terraform will run under (that is: the directory that will be in the HOME environment variable when Terraform runs) create a file named .terraformrc and write the following inside it:

  filesystem_mirror {
    path = "/opt/terraform/plugins"
  }
}```

This file is the [Terraform CLI configuration](https://developer.hashicorp.com/terraform/cli/config/config-file) and the `provider_installation` block describes [the provider installation methods](https://developer.hashicorp.com/terraform/cli/config/config-file#provider-installation). The default provider installation configuration is to install providers directly from their origin registries, but the custom configuration I showed above tells Terraform to consider only what's placed in the specified directory.

If you run `terraform init` with that configuration in place then you should see it fail to install, because there aren't yet any plugins in that directory.

You'll need to populate the directory with a directory structure matching [one of the filesystem mirror layouts](https://developer.hashicorp.com/terraform/cli/config/config-file#filesystem_mirror). Since your goal is to save disk space, you should use the "unpacked" layout since that gives Terraform the opportunity to use the plugins directly from the mirror directory, rather than having to extract them first.

To do this you'll need to find the distribution packages for the providers you intend to use and download them onto your system. The locations of these source packages vary depending on whether they are official HashiCorp providers or third-party providers written by partners or the community.

For official HashiCorp providers you can find them on [releases.hashicorp.com](https://releases.hashicorp.com/). For example, the `hashicorp/aws` provider packages are under `terraform-provider-aws` on that site. If you wanted to make `hashicorp/aws` v5.2.0 available in your mirror then you could follow the following steps:

- Download the `.zip` package for the platform you are using. For example, if you are running a Linux distribution on an x86_64/amd64 processor then you should download the `linux_amd64` package.
- Under your mirror directory create the necessary data structure. For the example path I showed above and a `linux_amd64` package that would be `/opt/terraform/plugins/registry.terraform.io/hashicorp/aws/5.2.0/linux_amd64`.
- Extract the plugin zip file into the directory created in the previous step. After you've done that the `linux_amd64` should contain a single executable file directly inside that directory (not in a subdirectory).

For third-party providers you can find their GitHub repository through [Terraform Registry](https://registry.terraform.io/). Third-party providers use GitHub Releases to publish their `.zip` files, so you can download the appropriate package from there and then install it into a similar directory structure as I showed for the `hashicorp/aws` provider above.

(Note that `hashicorp/aws` is short for `registry.terraform.io/hashicorp/aws`, because a provider address without a hostname prefix is always assumed to belong to `registry.terraform.io`. The directory for that hostname is _required_ in the filesystem mirror directory layout.)

After you've filled your local mirror with all of the providers you intend to use, you should be able to run `terraform init` and see it install all of the providers from your mirror without accessing Terraform Registry. If you inspect the size of the `.terraform` directory then you should see it considerably smaller than with direct installation, because Terraform will try to create symlinks to the directories inside the filesystem mirror rather than creating full copies of the plugin package directories in there.

<details>
<summary>英文:</summary>

I assume that when you say &quot;memory&quot; in your question you mean files in the filesystem, rather than data in RAM. `terraform init` does not load the provider plugins into RAM, but it does extract them into a cache directory under the `.terraform` directory.

If your intention is to allow your system to keep working when `registry.terraform.io` is down or when a plugin has been deleted then you can configure Terraform to install plugins from a local directory instead of fetching them over the network every time.

First choose a directory where you&#39;ll place your copies of the provider plugins. For example here I&#39;m going to use `/opt/terraform/plugins`, but you can use any directory path that Terraform will be able to access.

In the home directory of the system user that Terraform will run under (that is: the directory that will be in the `HOME` environment variable when Terraform runs) create a file named `.terraformrc` and write the following inside it:

provider_installation {
filesystem_mirror {
path = "/opt/terraform/plugins"
}
}

This file is the [Terraform CLI configuration](https://developer.hashicorp.com/terraform/cli/config/config-file) and the `provider_installation` block describes [the provider installation methods](https://developer.hashicorp.com/terraform/cli/config/config-file#provider-installation). The default provider installation configuration is to install providers directly from their origin registries, but the custom configuration I showed above tells Terraform to consider only what's placed in the specified directory.

---

If you run `terraform init` with that configuration in place then you should see it fail to install, because there aren't yet any plugins in that directory.

You'll need to populate the directory with a directory structure matching [one of the filesystem mirror layouts](https://developer.hashicorp.com/terraform/cli/config/config-file#filesystem_mirror). Since your goal is to save disk space, you should use the "unpacked" layout since that gives Terraform the opportunity to use the plugins directly from the mirror directory, rather than having to extract them first.

To do this you'll need to find the distribution packages for the providers you intend to use and download them onto your system. The locations of these source packages vary depending on whether they are official HashiCorp providers or third-party providers written by partners or the community.

For official HashiCorp providers you can find them on [releases.hashicorp.com](https://releases.hashicorp.com/). For example, the `hashicorp/aws` provider packages are under `terraform-provider-aws` on that site. If you wanted to make `hashicorp/aws` v5.2.0 available in your mirror then you could follow the following steps:

- Download the `.zip` package for the platform you are using. For example, if you are running a Linux distribution on an x86_64/amd64 processor then you should download the `linux_amd64` package.
- Under your mirror directory create the necessary data structure. For the example path I showed above and a `linux_amd64` package that would be `/opt/terraform/plugins/registry.terraform.io/hashicorp/aws/5.2.0/linux_amd64`.
- Extract the plugin zip file into the directory created in the previous step. After you've done that the `linux_amd64` should contain a single executable file directly inside that directory (not in a subdirectory).

For third-party providers you can find their GitHub repository through [Terraform Registry](https://registry.terraform.io/). Third-party providers use GitHub Releases to publish their `.zip` files, so you can download the appropriate package from there and then install it into a similar directory structure as I showed for the `hashicorp/aws` provider above.

(Note that `hashicorp/aws` is short for `registry.terraform.io/hashicorp/aws`, because a provider address without a hostname prefix is always assumed to belong to `registry.terraform.io`. The directory for that hostname is _required_ in the filesystem mirror directory layout.)

After you've filled your local mirror with all of the providers you intend to use, you should be able to run `terraform init` and see it install all of the providers from your mirror without accessing Terraform Registry. If you inspect the size of the `.terraform` directory then you should see it considerably smaller than with direct installation, because Terraform will try to create symlinks to the directories inside the filesystem mirror rather than creating full copies of the plugin package directories in there.


</details>