无法将消息发送到在WSL 2上运行的Kafka主题(从Windows系统)

huangapple go评论98阅读模式
英文:

Unable to produce to Kafka topic that is running on WSL 2 from Windows

问题

我在Ubuntu WSL2上成功运行了最新版本的Kafka。我可以从我在WSL上运行的Ubuntu内部正常启动zookeeper、kafka服务器、创建主题、控制台生产和控制台消费。然而,当我进入我的Windows上的Intellij并创建一个简单的Java生产者时,它似乎无法连接到代理。

版本和主机名

    Java版本:1.8
    Kafka版本:2.6
    主机名(来自Ubuntu):KDAAPPDEV04
    主机名(来自PowerShell):KDAAPPDEV04
    java.net.InetAddress.getLocalHost().getHostName() = KDAAPPDEV04
    java.net.InetAddress.getLocalHost().getCanonicalHostName() = KDAAPPDEV04
    CMD中的netstat:
        TCP    [::1]:9092             [::]:0                 LISTENING

server.properties
我在另一个SO答案中找到了这些设置,但对我没有起作用。

advertised.listeners=PLAINTEXT://127.0.0.1:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT
listeners=PLAINTEXT://0.0.0.0:9092

然后尝试了以下设置(并重新启动了zookeeper和kafka)

advertised.listeners=PLAINTEXT://KDAAPPDEV04:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT
listeners=PLAINTEXT://0.0.0.0:9092

生产者

我使用三个不同的值运行了这个生产者:hostname、localhost和127.0.0.1,但它从未连接到代理。

    public class ProducerDemo {

    private static Logger logger = LoggerFactory.getLogger(ProducerDemo.class);

    public static void main(String[] args) throws UnknownHostException {

        System.out.println(InetAddress.getLocalHost().getHostName());
        System.out.println(InetAddress.getLocalHost().getCanonicalHostName());

        String bootstrapServers = "127.0.0.1:9092";
//        String bootstrapServers = "localhost:9092";
//        String bootstrapServers = "KDAAPPDEV04:9092";

        //创建生产者属性
        Properties properties = new Properties();
        properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

        //创建生产者
        KafkaProducer<String, String> producer = new KafkaProducer<String, String>(properties);

        //创建生产者记录
        ProducerRecord<String, String> record = new ProducerRecord<String, String>("first-topic", "hola mundo");

        //发送数据
        producer.send(record);

        //刷新 + 关闭
        producer.flush();
        producer.close();
    }
}

错误

[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka版本:2.6.0
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka提交ID:62abe01bee039651
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka启动时间:1601666175706
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] 无法建立与节点 -1 (KDAAPPDEV04/my-ipconfig-address-here:9092) 的连接。代理可能不可用。
英文:

I am running the latest Kafka on Ubuntu WSL2 successfully. I can start zookeeper, kafka server, create topics, console produce and console consume just fine from within the Ubuntu that I have running on the WSL. However, when I go into my Intellij on Windows and create a simple Java Producer it does not seem to be able to connect to the broker

Versions & Hostname

    Java version: 1.8
    Kafka Version: 2.6
    hostname (from Ubuntu): KDAAPPDEV04
    hostname (from Powershell): KDAAPPDEV04
    java.net.InetAddress.getLocalHost().getHostName() = KDAAPPDEV04
    java.net.InetAddress.getLocalHost().getCanonicalHostName() = KDAAPPDEV04
    netstat from CMD:
        TCP    [::1]:9092             [::]:0                 LISTENING

server.properties
I found this settings on another SO answer but these did not work for me.

advertised.listeners=PLAINTEXT://127.0.0.1:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT
listeners=PLAINTEXT://0.0.0.0:9092

then tried (and restarted zookeeper and kafka)

advertised.listeners=PLAINTEXT://KDAAPPDEV04:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT
listeners=PLAINTEXT://0.0.0.0:9092

Producer

I run this producer with three different values: hostname, localhost and 127.0.0.1 but it never connects to the broker

    public class ProducerDemo{

    private static Logger logger = LoggerFactory.getLogger(ProducerDemo.class);

    public static void main(String[] args) throws UnknownHostException{

        System.out.println(InetAddress.getLocalHost().getHostName());
        System.out.println(InetAddress.getLocalHost().getCanonicalHostName());

        String bootstrapServers = &quot;127.0.0.1:9092&quot;;
//        String bootstrapServers = &quot;localhost:9092&quot;;
//        String bootstrapServers = &quot;KDAAPPDEV04:9092&quot;;

        //create Producer properties
        Properties properties = new Properties();
        properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,bootstrapServers);
        properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class.getName());

        //create the producer
        KafkaProducer&lt;String,String&gt; producer = new KafkaProducer&lt;String, String&gt;(properties);

        //create a producer record
        ProducerRecord&lt;String,String&gt; record = new ProducerRecord&lt;String, String&gt;(&quot;first-topic&quot;,&quot;hola mundo&quot;);

        //send data
        producer.send(record);

        //flush + close
        producer.flush();
        producer.close();
    }
}

Error

[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 2.6.0
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: 62abe01bee039651
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1601666175706
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node -1 (KDAAPPDEV04/my-ipconfig-address-here:9092) could not be established. Broker may not be available.

答案1

得分: 26

遇到了相同的问题。根本原因似乎是WSL2在IPv6和本地主机方面存在问题(参见:https://github.com/microsoft/WSL/issues/4851)

我找到的唯一解决方法,不需要在每次重启时更改配置(如上面“172.*”的建议),是在Linux中运行的Kafka服务器配置和Windows中的Java客户端中都使用IPv6环回地址::1。

在server.properties中,我有以下配置:

listeners=PLAINTEXT://[::1]:9092

同样地,在我的Java客户端引导服务器配置中,我使用

"[::1]:9092"
英文:

Had this same issue. The root cause seems to be that WSL2 is broken with regards to IPv6 and localhost (See: https://github.com/microsoft/WSL/issues/4851)

The only fix I found that doesn't involve changing configs every time you reboot (per the "172.*" suggestion above) is to use the IPv6 loopback address ::1 in both the Kafka server config running in Linux and the Java client in Windows.

In server.properties I have this:

listeners=PLAINTEXT://[::1]:9092

And likewise in my Java client bootstrap server config I use

&quot;[::1]:9092&quot;

答案2

得分: 11

我遇到过你现在遇到的完全相同的问题,我按照以下方法解决了它:

  1. 在我的WSL2 Ubuntu shell中运行了以下命令:

    ip addr | grep "eth0"
    

    我记录了inet属性对应的IP地址,例如172.27.10.68。

  2. 在我的Kafka server.properties中,我将listeners属性的值替换为:

    listeners=PLAINTEXT://172.27.10.68:9092
    

    我将advertised.listeners属性注释掉了。但你也可以选择将问题中的IP分配给这个属性,然后将listeners属性设置为0.0.0.0。
    但我猜你是在使用Kafka进行测试/学习,所以我会保持简单。

  3. 我没有改变Zookeeper的默认IP:端口设置。

  4. 我正在使用Schema Registry,因此我将Kafka引导属性修改如下:

    kafkastore.bootstrap.servers=PLAINTEXT://172.27.10.68:9092
    

    我没有改变默认的模式注册表监听器listeners=http://0.0.0.0:8081。

  5. 我在我的IntelliJ Kafka Producer中使用了相同的IP(如上所列)。
    它愉快地连接到了我在WSL2中的Kafka broker。

有关WSL2网络的更多信息,请访问https://learn.microsoft.com/en-us/windows/wsl/compare-versions。

这种设置的唯一问题是,每次关闭或重新启动Windows机器,或者关闭Ubuntu终端时,eth0的IP地址都会更改。
这会导致需要重做步骤2、4和5。我确信有更好的方法,但我尝试过的所有方法都失败了,除了这个方法。

英文:

I had the exact problem you are having and I resolved it as follows:

  1. I ran the following command in my WSL2 Ubuntu shell:
    ip addr | grep "eth0"
    I made note of the ip address against the inet property, for example, 172.27.10.68
  2. In my Kafka server.properties I replaced the listeners property value as follows:
    listeners=PLAINTEXT://172.27.10.68:9092
    I commented out the advertised.listeners property. But you can alternatively assign
    the ip in question to this property, and have the listeners property set to 0.0.0.0.
    But I assume you are using the Kafka installation for testing/learning purposes,
    so I would keep it simple.
  3. I made no change to the Zookeeper's default ip:port
  4. I am using the Schema Registry, so I modified the Kafka bootstrap property as follows:
    kafkastore.bootstrap.servers=PLAINTEXT://172.27.10.68:9092
    I made no change to the default schema registry listener listeners=http://0.0.0.0:8081
  5. I used the same ip (as listed above) in my IntelliJ Kafka Producer.
    It then happily connected to my Kafka broker in WSL2.

More information on WSL2 networking can be found at https://learn.microsoft.com/en-us/windows/wsl/compare-versions .

The only problem with this setup is that every time you shutdown or restart your Windows machine, or close your Ubuntu terminal, the ip address for eth0 changes. And this results in redoing steps 2, 4 and 5. I am sure there is a better way, but everything I tried failed, except for this.

答案3

得分: 3

停止 Kafka 和 Zookeeper,然后

在 WSL2 上禁用 IPv6:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

启动 Kafka,然后你可以开始使用!

英文:

Stop Kafka and Zookeeper, then

Disable IPv6 on WSL2:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

Start Kafka, and you're good to go!

答案4

得分: 2

WSL2 在虚拟化平台上运行,你需要使用端口代理来连接运行在 WSL2 上的 Kafka Broker。

步骤1:使用以下命令检查你的 WSL2 IP,并复制 inet 的值。

$ ifconfig
inet 172.X.X.X

步骤2:以管理员权限打开命令提示符。

netsh interface portproxy add v4tov4 listenport=9092 listenaddress=0.0.0.0 connectport=9092 connectaddress=172.X.X.X

现在你应该能够成功连接了。

注意:每次重新启动机器,WSL2 的 IP 都会发生变化。

英文:

WSL2 runs on hypervisor and you need port proxy to connect Kafka Broker running on WSL2.

Step 1 . Check you WSL2 IP using following command and copy inet value

$ ifconfig
inet 172.X.X.X 

Step 2. Open cmd with Admin permsissions

   netsh interface portproxy add v4tov4 listenport=9092 listenaddress=0.0.0.0 connectport=9092 connectaddress=172.X.X.X

You should be able to connect now

Note : WSL2 IP changes everytime you restart machine

答案5

得分: 2

我能找到一个解决方法。感谢Goose的评论。

  1. 我在我的WSL2 Ubuntu shell中运行了以下命令:ip addr。
  2. 然后针对inet属性的global eth0,运行ip address命令。例如,inet 172.20.XXX.XXX/20 .... scope global eth0。
  3. 我在docker-compose.yml中用这个IP地址替换了所有的localhost。
  4. 我在springboot的yml或properties文件中用这个IP地址替换了localhost。
  5. 我的Kafka生产者和消费者能够从Windows连接到在Ubuntu - WSL 2中运行的Kafka。
英文:

I am able to find a work around . Thanks to Goose's comments

  1. I ran the following command in my WSL2 Ubuntu shell: ip addr
  2. Then ip address against the inet property global eth0 . for example, inet 172.20.XXX.XXX/20 .... scope global eth0
  3. I replaced all localhost with this IP address in the docker-compose.yml
  4. I replaced the localhost with this IP address in springboot yml or properties file.
  5. My Kafka producer and consumer able to connect to the Kafka running in Ubunti - WSL 2 from Windows

答案6

得分: 1

我在使用IntelliJ运行Kafka生产者,在Ubuntu终端上运行WSL2时遇到了这个问题。

首先,停止Kafka和Zookeeper。然后在WSL2上依次运行以下命令:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

之后,在kafka文件夹中,进入 config/server.properties 并编辑文件,添加以下行:

listeners=PLAINTEXT://localhost:9092

当这些命令成功执行后,重新启动Zookeeper和Kafka。

链接:https://www.conduktor.io/kafka/kafka-fundamentals

英文:

I got this problem when running a kafka producer in IntelliJ and a consumer in ubuntu terminal while on WSL2.

First, stop Kafka and Zookeeper. Then run these commands on WSL2, one by one:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

After that, in the kakfa folder, go to config/server.properties and edit the file to add the line:

listeners=PLAINTEXT://localhost:9092

When these commands have succeeded relaunch zookeeper and kafka.

https://www.conduktor.io/kafka/kafka-fundamentals

答案7

得分: 0

这不是最优解决方案,但是如果你在Ubuntu/WSL上运行生产者,你将能够进行连接。这意味着如果你在使用Windows IDE编写代码,切换到Ubuntu,使用命令行编译器并运行生产者。请参考这篇帖子:https://stackoverflow.com/questions/62511091/error-connecting-to-kafka-server-via-ide-in-wsl2

英文:

This is not the optimal solution, but you will be able to connect if you run your producer in Ubuntu/WSL. This means if you are using a Windows IDE, writing the code, switching to Ubuntu and using a command line compiler and running the producer. See this post https://stackoverflow.com/questions/62511091/error-connecting-to-kafka-server-via-ide-in-wsl2

答案8

得分: 0

  1. 在文件 etc/sysctl.conf 中添加以下内容:

    net.ipv6.conf.all.disable_ipv6=1
    net.ipv6.conf.default.disable_ipv6=1
    
  2. 在你的 server.properties 中将 listeners=PLAINTEXT://:9092 替换为 listeners=PLAINTEXT://localhost:9092

  3. 使用以下命令更新 sysctl 配置(每次重新启动机器时都需要运行此命令来更新配置):

    sudo sysctl -p
    
英文:
  1. Edit the file etc/sysctl.conf and add following lines in it.

    net.ipv6.conf.all.disable_ipv6=1

    net.ipv6.conf.default.disable_ipv6=1

  2. Replace listeners=PLAINTEXT://:9092 with listeners=PLAINTEXT://localhost:9092 in your server.properties.

  3. Update the sysctl config by using the following command. (Everytime you restart your machine this command needs to be run to update the configuration)

    sudo sysctl -p

答案9

得分: 0

Step 1
在WSL2上禁用IPv6:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

Step 2:
在Kafka文件夹的\config\server.properties
listeners=PLAINTEXT://:9092替换为listeners=PLAINTEXT://localhost:9092

Step 3:
在WLS2的Ubuntu中,

sudo sysctl -p

在进行了以上更改后,我在本地环境中使其正常工作。

英文:

Step 1
Disable IPv6 on WSL2:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

Step 2:

In \config\server.properties under Kafka folder
Replace listeners=PLAINTEXT://:9092 with listeners=PLAINTEXT://localhost:9092 in your server.properties.

Step 3:
In WLS2 Ubuntu,

sudo sysctl -p

After making above changes it worked for me in my local

huangapple
  • 本文由 发表于 2020年10月3日 03:45:36
  • 转载请务必保留本文链接:https://go.coder-hub.com/64177422.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定