英文:
AWS SDK missing credentials when running in EKS with role-annotated ServiceAccount
问题
We use ServiceAccounts with a role annotation so the pods will acquire the role and use it for authenticating the AWS SDK's. This was working but we set up a new cluster and something is off in our config...
The error we see when trying to use the SDK (specifically, v2 SQS client) is:
Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1
It looks like the correct environment variables are in place when I check describe pod
:
Environment:
AWS_STS_REGIONAL_ENDPOINTS: regional
AWS_DEFAULT_REGION: us-east-1
AWS_REGION: us-east-1
AWS_ROLE_ARN: arn:aws:iam::**************:role/dev-node-api
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-snv7w (ro)
I have my ServiceAccount annotated like this:
apiVersion: v1
kind: ServiceAccount
metadata:
name: node-api-service-account
namespace: app
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::************:role/dev-node-api
and I attach the service account to the deployment like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: node-api-deploy
namespace: app
spec:
template:
spec:
serviceAccountName: node-api-service-account
...
I also set the trust relationship on the role to allow federation from the cluster's OIDC provider:
$ aws eks describe-cluster --name my-cluster --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5
> **************************************1F61
trust relationship in IAM role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::************:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/********************************1F61"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/********************************1F61:aud": "sts.amazonaws.com",
"oidc.eks.us-east-1.amazonaws.com/id/********************************1F61:sub": "system:serviceaccount:apis:node-api-service-account"
}
}
}
]
}
So from what I can tell, it looks like the role is being properly acquired but for some reason the Javascript SDK is not picking up the credentials from the AWS_WEB_IDENTITY_TOKEN_FILE
. Are there any logs I can use to debug this?
英文:
We use ServiceAccounts with a role annotation so the pods will acquire the role and use it for authenticating the AWS SDK's. This was working but we set up a new cluster and something is off in our config...
The error we see when trying to use the SDK (specifically, v2 SQS client) is:
Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1
It looks like the correct environment variables are in place when I check describe pod
:
Environment:
AWS_STS_REGIONAL_ENDPOINTS: regional
AWS_DEFAULT_REGION: us-east-1
AWS_REGION: us-east-1
AWS_ROLE_ARN: arn:aws:iam::**************:role/dev-node-api
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-snv7w (ro)
I have my ServiceAccount annotated like this:
apiVersion: v1
kind: ServiceAccount
metadata:
name: node-api-service-account
namespace: app
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::************:role/dev-node-api
and I attach the service account to the deployment like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: node-api-deploy
namespace: app
spec:
template:
spec:
serviceAccountName: node-api-service-account
...
I also set the trust relationship on the role to allow federation from the cluster's OIDC provider:
$ aws eks describe-cluster --name my-cluster --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5
> **************************************1F61
trust relationship in IAM role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::************:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/********************************1F61"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/********************************1F61:aud": "sts.amazonaws.com",
"oidc.eks.us-east-1.amazonaws.com/id/********************************1F61:sub": "system:serviceaccount:apis:node-api-service-account"
}
}
}
]
}
So from what I can tell, it looks like the role is being properly acquired but fore some reason the Javascript SDK is not picking up the credentials from the AWS_WEB_IDENTITY_TOKEN_FILE
. Are there any logs I can use to debug this?
答案1
得分: 0
问题出在信任关系条件中的命名空间错误。在此工作过程中,我们更改了集群的命名空间,但忘记更新信任关系中的命名空间。更新后重新启动部署导致SDK成功使用。
英文:
The problem was an incorrect namespace in the Trust Relationship condition. We changed the namespaces of the cluster during this work, and had forgotten to update the namespace in the trust relations. Updating that and restarting deployments resulted in successful SDK usage.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论