AWS SDK在EKS中运行时缺少凭证,带有角色注释的ServiceAccount。

huangapple go评论71阅读模式
英文:

AWS SDK missing credentials when running in EKS with role-annotated ServiceAccount

问题

We use ServiceAccounts with a role annotation so the pods will acquire the role and use it for authenticating the AWS SDK's. This was working but we set up a new cluster and something is off in our config...

The error we see when trying to use the SDK (specifically, v2 SQS client) is:

Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1

It looks like the correct environment variables are in place when I check describe pod:

Environment:
  AWS_STS_REGIONAL_ENDPOINTS:   regional
  AWS_DEFAULT_REGION:           us-east-1
  AWS_REGION:                   us-east-1
  AWS_ROLE_ARN:                 arn:aws:iam::**************:role/dev-node-api
  AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
  /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-snv7w (ro)

I have my ServiceAccount annotated like this:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: node-api-service-account
  namespace: app
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::************:role/dev-node-api

and I attach the service account to the deployment like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: node-api-deploy
  namespace: app
spec:
  template:
    spec:
      serviceAccountName: node-api-service-account
...

I also set the trust relationship on the role to allow federation from the cluster's OIDC provider:

$ aws eks describe-cluster --name my-cluster --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5

> **************************************1F61

trust relationship in IAM role:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::************:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/********************************1F61"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-1.amazonaws.com/id/********************************1F61:aud": "sts.amazonaws.com",
                    "oidc.eks.us-east-1.amazonaws.com/id/********************************1F61:sub": "system:serviceaccount:apis:node-api-service-account"
                }
            }
        }
    ]
}

So from what I can tell, it looks like the role is being properly acquired but for some reason the Javascript SDK is not picking up the credentials from the AWS_WEB_IDENTITY_TOKEN_FILE. Are there any logs I can use to debug this?

英文:

We use ServiceAccounts with a role annotation so the pods will acquire the role and use it for authenticating the AWS SDK's. This was working but we set up a new cluster and something is off in our config...

The error we see when trying to use the SDK (specifically, v2 SQS client) is:

Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1

It looks like the correct environment variables are in place when I check describe pod:

    Environment:
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           us-east-1
      AWS_REGION:                   us-east-1
      AWS_ROLE_ARN:                 arn:aws:iam::**************:role/dev-node-api
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-snv7w (ro)

I have my ServiceAccount annotated like this:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: node-api-service-account
  namespace: app
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::************:role/dev-node-api

and I attach the service account to the deployment like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: node-api-deploy
  namespace: app
spec:
  template:
    spec:
      serviceAccountName: node-api-service-account
...

I also set the trust relationship on the role to allow federation from the cluster's OIDC provider:

$ aws eks describe-cluster --name my-cluster --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5

> **************************************1F61

trust relationship in IAM role:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::************:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/********************************1F61"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-1.amazonaws.com/id/********************************1F61:aud": "sts.amazonaws.com",
                    "oidc.eks.us-east-1.amazonaws.com/id/********************************1F61:sub": "system:serviceaccount:apis:node-api-service-account"
                }
            }
        }
    ]
}

So from what I can tell, it looks like the role is being properly acquired but fore some reason the Javascript SDK is not picking up the credentials from the AWS_WEB_IDENTITY_TOKEN_FILE. Are there any logs I can use to debug this?

答案1

得分: 0

问题出在信任关系条件中的命名空间错误。在此工作过程中,我们更改了集群的命名空间,但忘记更新信任关系中的命名空间。更新后重新启动部署导致SDK成功使用。

英文:

The problem was an incorrect namespace in the Trust Relationship condition. We changed the namespaces of the cluster during this work, and had forgotten to update the namespace in the trust relations. Updating that and restarting deployments resulted in successful SDK usage.

huangapple
  • 本文由 发表于 2023年4月13日 22:07:02
  • 转载请务必保留本文链接:https://go.coder-hub.com/76006413.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定