英文:

Why do I need to specify all the brokers in Kafka with Kraft in a 3 node broker/controller cluster?

问题

Deployed a 3 node kafka cluster in kubernetes using a statefulset using Kraft.

在Kubernetes中使用StatefulSet部署了一个包含3个Kafka节点的集群，使用了Kraft。

Within the cluster, the configuration is as follows (pseudo-code):

集群内的配置如下（伪代码）：

• controller quorum: 0@kafka-0,1@kafka-1,2@kafka-2

• listeners: broker://:9092,controller://:9093,LOCAL://localhost:9094

• advertised_listeners: broker://:9092,controller://:9093,LOCAL://localhost:9094

• 控制器仲裁：0@kafka-0,1@kafka-1,2@kafka-2

• 监听器：broker://:9092,controller://:9093,LOCAL://localhost:9094

• 广告监听器：broker://:9092,controller://:9093,LOCAL://localhost:9094

All nodes are broker/controller

所有节点都是broker/controller节点。

Within the kubernetes cluster, this kraft configuration is working flawlessly. The cluster stands up and services pointing to all three brokers are able to produce and consume messages as expected.

在Kubernetes集群内，这个Kraft配置运行良好。集群启动并且指向所有三个broker的服务能够按预期生产和消费消息。

However, to access the kafka cluster from outside Kubernetes, I need to port-forward to gain access (or otherwise ingress).

然而，要从Kubernetes外部访问Kafka集群，我需要使用端口转发来获得访问权限（或者使用ingress）。

Since Kraft utilizes all controllers/brokers as "leader" nodes as per the documentation, why am I unable to connect to a single node and get full access to the partitions.

由于Kraft根据文档将所有控制器/代理节点都用作“leader”节点，为什么我无法连接到单个节点并完全访问分区？

For example, if I port forward kafka-0 and connect to it, I have access to create and list topics.

例如，如果我将端口转发到kafka-0并连接到它，我可以创建和列出主题。

However, if I create a topic with 3 partitions, they might be configured as follows:

但是，如果我创建一个具有3个分区的主题，它们可能配置如下：

• partition 0 : leader 1 : replicas 3

• partition 1 : leader 2 : replicas 3

• partition 2 : leader 0 : replicas 3

• 分区 0：领导者 1：副本 3

• 分区 1：领导者 2：副本 3

• 分区 2：领导者 0：副本 3

If I port forward kafka-0 (which maps to leader 0 in the above example), I am able to push a message ONLY to partition 2. If I attempt to push a message to partition 0 or partition 1, I receive the following error NOT_ENOUGH_REPLICAS.

如果我将端口转发到kafka-0（在上面的示例中映射到领导者 0），我只能将消息推送到分区 2。如果我尝试将消息推送到分区 0 或分区 1，我会收到以下错误 NOT_ENOUGH_REPLICAS。

I would've expected the broker/controller node I port-forwarded to forward the produced message to the correct broker/controller that owned the partition, in a proxy-like pattern.

我本来期望将消息推送到了端口转发的broker/controller节点，然后由该节点将消息转发到拥有分区的正确的broker/controller节点，类似代理的模式。

However, this was the not the case.

然而，情况并非如此。

What actually happens when you specify a subset of the kafka nodes in your connection string (bootstrap-servers for example) under Kraft?

在Kraft下，在连接字符串（例如bootstrap-servers）中指定Kafka节点的子集时，实际发生了什么？

英文:

Deployed a 3 node kafka cluster in kubernetes using a statefulset using Kraft.

Within the cluster, the configuration is as follows (pseudo-code):

• controller quorum: 0@kafka-0,1@kafka-1,2@kafka-2
• listeners: broker://:9092,controller://:9093,LOCAL://localhost:9094
• advertised_listeners: broker://:9092,controller://:9093,LOCAL://localhost:9094

All nodes are broker/controller

Within the kubernetes cluster, this kraft configuration is working flawlessly. The cluster stands up and services pointing to all three brokers are able to produce and consume messages as expected.

However, to access the kafka cluster from outside Kubernetes, I need to port-forward to gain access (or otherwise ingress).

Since Kraft utilizes all controllers/brokers as "leader" nodes as per the documentation, why am I unable to connect to a single node and get full access to the partitions.

For example, if I port forward kafka-0 and connect to it, I have access to create and list topics.

However, if I create a topic with 3 partitions, they might be configured as follows:

• partition 0 : leader 1 : replicas 3
• partition 1 : leader 2 : replicas 3
• partition 2 : leader 0 : replicas 3

If I port forward kafka-0 (which maps to leader 0 in the above example), I am able to push a message ONLY to partition 2. If I attempt to push a message to partition 0 or partition 1, I receive the following error NOT_ENOUGH_REPLICAS.

I would've expected the broker/controller node I port-forwarded to forward the produced message to the correct broker/controller that owned the partition, in a proxy-like pattern.

However, this was the not the case.

What actually happens when you specify a subset of the kafka nodes in your connection string (bootstrap-servers for example) under Kraft?

答案1

得分: 1

这就是为什么它被称为 BOOTSTRAP_SERVERS_CONFIG，因为初始连接仅用于引导，它连接到其中一个指定的服务器；一旦连接建立，该代理返回每个主题/分区的领袖信息。

在需要时，客户端然后连接到特定分区的领袖。您必须能够访问所有代理实例。

Kraft与Zookeeper没有区别；它的工作方式相同。

英文:

That's why it's called BOOTSTRAP_SERVERS_CONFIG since the initial connection is for bootstrapping only, it is made to one of the servers specified there; once the connection is established, that broker returns information about the leader for each topic/partition.

When needed, the client then connects to the leader for a particular partition. You must have access to all broker instances.

Kraft Vs. Zookeeper makes no difference; it works the same.

答案2

得分: 0

我能够仅将消息推送到分区2，因为您仅转发了一个代理的端口。Kafka客户端需要写入领导者分区，因此是集群中的每个代理。一个端口无法代理整个集群。这是与引导协议无关的问题，但在通信到达一个代理时，您应该始终提供多个代理（或解析为多个的DNS名称，这是Ingress的作用，但根据k8s文档，Ingress仅用于HTTP流量）。

英文:

> I am able to push a message ONLY to partition 2

Because you've only forwarded ports of one broker.

Kafka clients are required to write to the leader partitions, so each broker in your cluster. One port will not proxy the entire cluster

This is a separate issue from bootstrap protocol, but you should always give more than one broker (or a DNS name that resolves to many, which is what an Ingress would be, but Ingress are only for HTTP traffic, according to k8s docs) in case networking errors when communicating to that one