英文:
AWS MSK kafka Not authorized to access topics
问题
我的 Kafka 集群启用了 IAM 认证。我能成功地通过假定正确的 IAM 角色来生产和消费来自主题 test-topic2 的消息，如果策略如下所示；
但现在我想将策略缩小到特定的集群，所以我将其更改为以下内容；
在 Producer 端，我收到以下错误；
在 Consumer 端；
我在这里漏掉了什么？
英文:
My Kafka cluster is IAM auth enabled. I am successfully able to produce and consume messages from topic test-topic2 by assuming the correct IAM role if the policy is as follows;
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowKafkaTopicWrite",
"Effect": "Allow",
"Action": [
"kafka:*",
"kafka-cluster:*"
],
"Resource": "*"
}
]
}
But now I want to narrow down the policy to a specific cluster, so I change it to following;
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowKafkaTopicWrite",
"Effect": "Allow",
"Action": [
"kafka:*",
"kafka-cluster:*"
],
"Resource": [
"arn:aws:kafka:eu-west-1:123456789:cluster/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9",
"arn:aws:kafka:eu-west-1:123456789:cluster/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9/*",
"arn:aws:kafka:eu-west-1:123456789:cluster/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9/topic/test-topic2"
]
}
]
}
I get following error on Producer side;
[2023-06-15 15:28:42,476] ERROR Error when sending message to topic test-topic2 with key: null, value: 1 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test-topic2]
And on consumer;
[2023-06-15 15:28:00,208] WARN [Consumer clientId=console-consumer, groupId=console-consumer-46486] Error while fetching metadata with correlation id 3 : {test-topic2=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
[2023-06-15 15:28:00,210] ERROR [Consumer clientId=console-consumer, groupId=console-consumer-46486] Topic authorization failed for topics [test-topic2] (org.apache.kafka.clients.Metadata)
[2023-06-15 15:28:00,211] ERROR Error processing message, terminating consumer process: (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test-topic2]
What am I missing here ?
答案1
得分: 1
IAM策略中的主题ARN似乎不正确，根据文档 - https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html#msk-iam-resources
主题 - arn:aws:kafka:region:account-id:topic/cluster-name/cluster-uuid/topic-name
因此，在这种情况下，主题ARN应该是arn:aws:kafka:eu-west-1:123456789:topic/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9/test-topic2。
英文:
topic arn in your IAM policy seems incorrect, as per the documentation - https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html#msk-iam-resources
Topic - arn:aws:kafka:region:account-id:topic/cluster-name/cluster-uuid/topic-name
so the topic arn in this case should be arn:aws:kafka:eu-west-1:123456789:topic/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9/test-topic2
答案2
得分: 0
在手动测试策略后，结果发现我遗漏了对group和transactional-id的授权，而且主题授权也不正确（如@MrocKK所指出的）。以下是有效的最小策略（读取/写入test-topic2）：
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"kafka-cluster:DescribeTopicDynamicConfiguration",
"kafka-cluster:AlterGroup",
"kafka-cluster:DescribeCluster",
"kafka-cluster:ReadData",
"kafka-cluster:DescribeTopic",
"kafka-cluster:DescribeTransactionalId",
"kafka-cluster:DescribeGroup",
"kafka-cluster:DescribeClusterDynamicConfiguration",
"kafka-cluster:Connect"
],
"Resource": [
"arn:aws:kafka:eu-west-1:123456789:cluster/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9",
"arn:aws:kafka:eu-west-1:123456789:topic/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9/test-topic2",
"arn:aws:kafka:eu-west-1:123456789:group/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9/*",
"arn:aws:kafka:eu-west-1:123456789:transactional-id/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9/*"
]
}
要将其设置为只读，请从Action中简单地移除WriteDataIdempotently和WriteData。
更多详情请查看此链接：https://github.com/maxcotec/aws-IAM-auth-msk-python
英文:
After manually testing policies, turns out I was missing grants to group as well as transactional-id. And ofcourse the topic grant was incorrect (as pointed out by @MrocKK). Here is the minimal policy that works (read/Write on/to test-topic2)
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"kafka-cluster:DescribeTopicDynamicConfiguration",
"kafka-cluster:AlterGroup",
"kafka-cluster:WriteDataIdempotently",
"kafka-cluster:DescribeCluster",
"kafka-cluster:ReadData",
"kafka-cluster:DescribeTopic",
"kafka-cluster:DescribeTransactionalId",
"kafka-cluster:DescribeGroup",
"kafka-cluster:DescribeClusterDynamicConfiguration",
"kafka-cluster:Connect",
"kafka-cluster:WriteData"
],
"Resource": [
"arn:aws:kafka:eu-west-1:123456789:cluster/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9",
"arn:aws:kafka:eu-west-1:123456789:topic/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9/test-topic2",
"arn:aws:kafka:eu-west-1:123456789:group/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9/*",
"arn:aws:kafka:eu-west-1:123456789:transactional-id/test-cluster/9f4ea0a3-75bc-4ff9-a971-73efa2ef73c9-9/*"
]
}
And to make it read-only, simple remove WriteDataIdempotently and WriteData from Actions.
More details here: https://github.com/maxcotec/aws-IAM-auth-msk-python
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论