Terraform - data source aws_ssm_parameter causing enygmatic "type" errors

I am struggling with this error when trying to terraform plan:

Error: Invalid dynamic for_each value
│ 
│   on .terraform/modules/k8s_resources.nginx_controller/main.tf line 69, in resource "helm_release" "application":
│   69:     for_each = var.additional_set
│ 
│ Cannot use a set of object value in for_each. An iterable collection is
│ required.
╵
Operation failed: failed running terraform plan (exit 1)

I don't really understand why this gets triggered internally in the nginx-controller module since I am sure I am passing a string.

I am using this AWS provider version:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0.1"
    }
  }
}

I am using terraform required_version = ">= 0.14.0"

I am passing the value as string like this:

data "aws_ssm_parameter" "acm_arn" {
  name = "/terraform/dev/fs1/acm_arn"
}

locals {
  nginx_controller_values = [
    # I have more objects that now are commented-out for testing since this one causes the failure.
    {
      name  = "controller.service.annotations.service\\.beta\\.kubernetes\\.io/aws-load-balancer-ssl-cert"
      value = data.aws_ssm_parameter.acm_arn.value
      type  = "string"
    }
  ]
}

module "nginx_controller" {
  source  = "terraform-iaac/nginx-controller/helm"
  version = "2.2.0"
  namespace = var.namespace
  additional_set = local.nginx_controller_values
  depends_on = [
    kubernetes_namespace.nmsp
  ]
}

If I hardcode the arn in the local varaible it "plans" fine.

If I pass data.aws_ssm_parameter.acm_arn.arn (the arn of SSM, not what I need) instead of data.aws_ssm_parameter.acm_arn.value it "plans" fine.

I tested with terraform console the value of data.aws_ssm_parameter.acm_arn.value and:

• It is absolutely identycal to the hardcoded string;
• using type() on this value gives me string exactly as the hardcoded string.

I honestly have no clue on what the hell is going on, it must be something with the aws_ssm_parameter that I cannot understand.

Obviously I could avoid using AWS SSM Parameter Store that looked quite convenient in my case to share few things between different Terraform applications (I followed the best practice from Hashicorp about avoiding to load the whole state of another application); I would obviously appreciate suggestions on better ways to handle this, but, in this question, my goal is to understand why this fails and how to use this SSM correctly even if I will end up not using aws_ssm_parameter.

Thanks

Adding more details on the SSM handling part:

I write to SSM from one Terraform state like this:

# in short the arn gets generated by acm module
module "acm" {
  source  = "terraform-aws-modules/acm/aws"
  version = "~> 4.3.2"
  zone_id     = local.zone_id
}

# and gets written to SSM like this
resource "aws_ssm_parameter" "acm_arn" {
  name   = "/terraform/dev/fs1/acm_arn"
  type   = "String"
  value  = module.acm.acm_certificate_arn
}

#terraform output provides: "arn:aws:acm:eu-west-1:000000000000:certificate/0000aaaa-aa00-0000-000a-00000000aaaa"

Edit: How to replicate on your system

I managed to replicate this with an easy to copy-paste-and-run snippet:

# put your region stuff
provider "aws" {
  region = "eu-west-1"
}

# put the required module version and your backend
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0.1"
    }
  }

  required_version = ">= 0.14.0"
}

# Replicate my example
resource "aws_ssm_parameter" "test_key_wr_1" {
  name   = "/terraform/fs1/test_key_1"
  type   = "String"
  value  = "paperino"
}

data "aws_ssm_parameter" "test_key_rd_1" {
  name = "/terraform/fs1/test_key_1"
  depends_on = [ aws_ssm_parameter.test_key_wr_1 ]
}

output "ssm_stored_value_1" {
  value     = data.aws_ssm_parameter.test_key_rd_1.value
  sensitive = true
}

locals {
  nginx_controller_values = [
    {
      name  = "controller.service.annotations.service\\.beta\\.kubernetes\\.io/aws-load-balancer-ssl-cert"
      value = data.aws_ssm_parameter.test_key_rd_1.value
      type  = "string"
    },
    {
      name  = "controller.service.annotations.service\\.beta\\.kubernetes\\.io/aws-load-balancer-ssl-ports"
      value = "443"
      type  = "string"
    }
  ]
}

module "nginx_controller" {
  source  = "terraform-iaac/nginx-controller/helm"
  version = "2.2.0"

  namespace = "pippo"

  additional_set = local.nginx_controller_values
}

Plan fails as described, there is something I am missing on how to use this ssm module.

Edit2: Temporary Fix

It's not really an answer since it remains unclear why this happens, but changing the nginx-controller version to 2.1.0 the problem disappears. In version 2.1.1 the problem reappears.

I might open an issue on their git repository to make them check.

Not entirely sure why but changing this:

data.aws_ssm_parameter.test_key_rd_1.value

to this

tostring(data.aws_ssm_parameter.test_key_rd_1.value)

in the locals block suppressed the error (I was not applying changes to my infrastructure).

Hope this fixes the problem for you!

Turns out that the problem arises after commit a15dd3c and happens when injecting a sensitive string variable in one of the objects in the additional_set variable that now has a type declaration.

Basically, before they didn't define a type for that additional_set while after that commit they did and, I don't know why, with a type definition providing a sensitive value makes everything fail with a meaningless error.

Moreover, as Dawid pointed out, converting tostring() sometimes solves the issue, making everything weirder.

Just to clarify something else, aws_ssm_parameter reads from SSM and marks the data as sensitive; that is why hardcoding a string makes it work, while fetching from aws_ssm breaks it.

It seems to me a weird behavior and a difficult to understand error but that is not caused by this module.

Using nonsensitive() fixes everything.

Not sure if this is a wanted behavior (so just a misleading error) or some issue within Terraform (that should be notified).

Definitive fix

In case others will experience the same problem, after opening an issue on github, problem has been fixed in version 2.2.1 by undoing the change in commit a15dd3c.