英文:
Unable to produce to Kafka topic that is running on WSL 2 from Windows
问题
我在Ubuntu WSL2上成功运行了最新版本的Kafka。我可以从我在WSL上运行的Ubuntu内部正常启动zookeeper、kafka服务器、创建主题、控制台生产和控制台消费。然而,当我进入我的Windows上的Intellij并创建一个简单的Java生产者时,它似乎无法连接到代理。
版本和主机名
Java版本:1.8
Kafka版本:2.6
主机名(来自Ubuntu):KDAAPPDEV04
主机名(来自PowerShell):KDAAPPDEV04
java.net.InetAddress.getLocalHost().getHostName() = KDAAPPDEV04
java.net.InetAddress.getLocalHost().getCanonicalHostName() = KDAAPPDEV04
CMD中的netstat:
TCP [::1]:9092 [::]:0 LISTENING
server.properties
我在另一个SO答案中找到了这些设置,但对我没有起作用。
advertised.listeners=PLAINTEXT://127.0.0.1:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT
listeners=PLAINTEXT://0.0.0.0:9092
然后尝试了以下设置(并重新启动了zookeeper和kafka)
advertised.listeners=PLAINTEXT://KDAAPPDEV04:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT
listeners=PLAINTEXT://0.0.0.0:9092
生产者
我使用三个不同的值运行了这个生产者:hostname、localhost和127.0.0.1,但它从未连接到代理。
public class ProducerDemo {
private static Logger logger = LoggerFactory.getLogger(ProducerDemo.class);
public static void main(String[] args) throws UnknownHostException {
System.out.println(InetAddress.getLocalHost().getHostName());
System.out.println(InetAddress.getLocalHost().getCanonicalHostName());
String bootstrapServers = "127.0.0.1:9092";
// String bootstrapServers = "localhost:9092";
// String bootstrapServers = "KDAAPPDEV04:9092";
//创建生产者属性
Properties properties = new Properties();
properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
//创建生产者
KafkaProducer<String, String> producer = new KafkaProducer<String, String>(properties);
//创建生产者记录
ProducerRecord<String, String> record = new ProducerRecord<String, String>("first-topic", "hola mundo");
//发送数据
producer.send(record);
//刷新 + 关闭
producer.flush();
producer.close();
}
}
错误
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka版本:2.6.0
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka提交ID:62abe01bee039651
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka启动时间:1601666175706
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] 无法建立与节点 -1 (KDAAPPDEV04/my-ipconfig-address-here:9092) 的连接。代理可能不可用。
英文:
I am running the latest Kafka on Ubuntu WSL2 successfully. I can start zookeeper, kafka server, create topics, console produce and console consume just fine from within the Ubuntu that I have running on the WSL. However, when I go into my Intellij on Windows and create a simple Java Producer it does not seem to be able to connect to the broker
Versions & Hostname
Java version: 1.8
Kafka Version: 2.6
hostname (from Ubuntu): KDAAPPDEV04
hostname (from Powershell): KDAAPPDEV04
java.net.InetAddress.getLocalHost().getHostName() = KDAAPPDEV04
java.net.InetAddress.getLocalHost().getCanonicalHostName() = KDAAPPDEV04
netstat from CMD:
TCP [::1]:9092 [::]:0 LISTENING
server.properties
I found this settings on another SO answer but these did not work for me.
advertised.listeners=PLAINTEXT://127.0.0.1:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT
listeners=PLAINTEXT://0.0.0.0:9092
then tried (and restarted zookeeper and kafka)
advertised.listeners=PLAINTEXT://KDAAPPDEV04:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT
listeners=PLAINTEXT://0.0.0.0:9092
Producer
I run this producer with three different values: hostname, localhost and 127.0.0.1 but it never connects to the broker
public class ProducerDemo{
private static Logger logger = LoggerFactory.getLogger(ProducerDemo.class);
public static void main(String[] args) throws UnknownHostException{
System.out.println(InetAddress.getLocalHost().getHostName());
System.out.println(InetAddress.getLocalHost().getCanonicalHostName());
String bootstrapServers = "127.0.0.1:9092";
// String bootstrapServers = "localhost:9092";
// String bootstrapServers = "KDAAPPDEV04:9092";
//create Producer properties
Properties properties = new Properties();
properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,bootstrapServers);
properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class.getName());
//create the producer
KafkaProducer<String,String> producer = new KafkaProducer<String, String>(properties);
//create a producer record
ProducerRecord<String,String> record = new ProducerRecord<String, String>("first-topic","hola mundo");
//send data
producer.send(record);
//flush + close
producer.flush();
producer.close();
}
}
Error
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 2.6.0
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: 62abe01bee039651
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1601666175706
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node -1 (KDAAPPDEV04/my-ipconfig-address-here:9092) could not be established. Broker may not be available.
答案1
得分: 26
遇到了相同的问题。根本原因似乎是WSL2在IPv6和本地主机方面存在问题(参见:https://github.com/microsoft/WSL/issues/4851)
我找到的唯一解决方法,不需要在每次重启时更改配置(如上面“172.*”的建议),是在Linux中运行的Kafka服务器配置和Windows中的Java客户端中都使用IPv6环回地址::1。
在server.properties中,我有以下配置:
listeners=PLAINTEXT://[::1]:9092
同样地,在我的Java客户端引导服务器配置中,我使用
"[::1]:9092"
英文:
Had this same issue. The root cause seems to be that WSL2 is broken with regards to IPv6 and localhost (See: https://github.com/microsoft/WSL/issues/4851)
The only fix I found that doesn't involve changing configs every time you reboot (per the "172.*" suggestion above) is to use the IPv6 loopback address ::1 in both the Kafka server config running in Linux and the Java client in Windows.
In server.properties I have this:
listeners=PLAINTEXT://[::1]:9092
And likewise in my Java client bootstrap server config I use
"[::1]:9092"
答案2
得分: 11
我遇到过你现在遇到的完全相同的问题,我按照以下方法解决了它:
-
在我的WSL2 Ubuntu shell中运行了以下命令:
ip addr | grep "eth0"
我记录了inet属性对应的IP地址,例如172.27.10.68。
-
在我的Kafka server.properties中,我将listeners属性的值替换为:
listeners=PLAINTEXT://172.27.10.68:9092
我将advertised.listeners属性注释掉了。但你也可以选择将问题中的IP分配给这个属性,然后将listeners属性设置为0.0.0.0。
但我猜你是在使用Kafka进行测试/学习,所以我会保持简单。 -
我没有改变Zookeeper的默认IP:端口设置。
-
我正在使用Schema Registry,因此我将Kafka引导属性修改如下:
kafkastore.bootstrap.servers=PLAINTEXT://172.27.10.68:9092
我没有改变默认的模式注册表监听器listeners=http://0.0.0.0:8081。
-
我在我的IntelliJ Kafka Producer中使用了相同的IP(如上所列)。
它愉快地连接到了我在WSL2中的Kafka broker。
有关WSL2网络的更多信息,请访问https://learn.microsoft.com/en-us/windows/wsl/compare-versions。
这种设置的唯一问题是,每次关闭或重新启动Windows机器,或者关闭Ubuntu终端时,eth0的IP地址都会更改。
这会导致需要重做步骤2、4和5。我确信有更好的方法,但我尝试过的所有方法都失败了,除了这个方法。
英文:
I had the exact problem you are having and I resolved it as follows:
- I ran the following command in my WSL2 Ubuntu shell:
ip addr | grep "eth0"
I made note of the ip address against the inet property, for example, 172.27.10.68 - In my Kafka server.properties I replaced the listeners property value as follows:
listeners=PLAINTEXT://172.27.10.68:9092
I commented out the advertised.listeners property. But you can alternatively assign
the ip in question to this property, and have the listeners property set to 0.0.0.0.
But I assume you are using the Kafka installation for testing/learning purposes,
so I would keep it simple. - I made no change to the Zookeeper's default ip:port
- I am using the Schema Registry, so I modified the Kafka bootstrap property as follows:
kafkastore.bootstrap.servers=PLAINTEXT://172.27.10.68:9092
I made no change to the default schema registry listener listeners=http://0.0.0.0:8081 - I used the same ip (as listed above) in my IntelliJ Kafka Producer.
It then happily connected to my Kafka broker in WSL2.
More information on WSL2 networking can be found at https://learn.microsoft.com/en-us/windows/wsl/compare-versions .
The only problem with this setup is that every time you shutdown or restart your Windows machine, or close your Ubuntu terminal, the ip address for eth0 changes. And this results in redoing steps 2, 4 and 5. I am sure there is a better way, but everything I tried failed, except for this.
答案3
得分: 3
停止 Kafka 和 Zookeeper,然后
在 WSL2 上禁用 IPv6:
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
启动 Kafka,然后你可以开始使用!
英文:
Stop Kafka and Zookeeper, then
Disable IPv6 on WSL2:
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
Start Kafka, and you're good to go!
答案4
得分: 2
WSL2 在虚拟化平台上运行,你需要使用端口代理来连接运行在 WSL2 上的 Kafka Broker。
步骤1:使用以下命令检查你的 WSL2 IP,并复制 inet 的值。
$ ifconfig
inet 172.X.X.X
步骤2:以管理员权限打开命令提示符。
netsh interface portproxy add v4tov4 listenport=9092 listenaddress=0.0.0.0 connectport=9092 connectaddress=172.X.X.X
现在你应该能够成功连接了。
注意:每次重新启动机器,WSL2 的 IP 都会发生变化。
英文:
WSL2 runs on hypervisor and you need port proxy to connect Kafka Broker running on WSL2.
Step 1 . Check you WSL2 IP using following command and copy inet value
$ ifconfig
inet 172.X.X.X
Step 2. Open cmd with Admin permsissions
netsh interface portproxy add v4tov4 listenport=9092 listenaddress=0.0.0.0 connectport=9092 connectaddress=172.X.X.X
You should be able to connect now
Note : WSL2 IP changes everytime you restart machine
答案5
得分: 2
我能找到一个解决方法。感谢Goose的评论。
- 我在我的WSL2 Ubuntu shell中运行了以下命令:ip addr。
- 然后针对inet属性的global eth0,运行ip address命令。例如,inet 172.20.XXX.XXX/20 .... scope global eth0。
- 我在docker-compose.yml中用这个IP地址替换了所有的localhost。
- 我在springboot的yml或properties文件中用这个IP地址替换了localhost。
- 我的Kafka生产者和消费者能够从Windows连接到在Ubuntu - WSL 2中运行的Kafka。
英文:
I am able to find a work around . Thanks to Goose's comments
- I ran the following command in my WSL2 Ubuntu shell: ip addr
- Then ip address against the inet property global eth0 . for example, inet 172.20.XXX.XXX/20 .... scope global eth0
- I replaced all localhost with this IP address in the docker-compose.yml
- I replaced the localhost with this IP address in springboot yml or properties file.
- My Kafka producer and consumer able to connect to the Kafka running in Ubunti - WSL 2 from Windows
答案6
得分: 1
我在使用IntelliJ运行Kafka生产者,在Ubuntu终端上运行WSL2时遇到了这个问题。
首先,停止Kafka和Zookeeper。然后在WSL2上依次运行以下命令:
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
之后,在kafka文件夹中,进入 config/server.properties 并编辑文件,添加以下行:
listeners=PLAINTEXT://localhost:9092
当这些命令成功执行后,重新启动Zookeeper和Kafka。
链接:https://www.conduktor.io/kafka/kafka-fundamentals
英文:
I got this problem when running a kafka producer in IntelliJ and a consumer in ubuntu terminal while on WSL2.
First, stop Kafka and Zookeeper. Then run these commands on WSL2, one by one:
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
After that, in the kakfa folder, go to config/server.properties and edit the file to add the line:
listeners=PLAINTEXT://localhost:9092
When these commands have succeeded relaunch zookeeper and kafka.
答案7
得分: 0
这不是最优解决方案,但是如果你在Ubuntu/WSL上运行生产者,你将能够进行连接。这意味着如果你在使用Windows IDE编写代码,切换到Ubuntu,使用命令行编译器并运行生产者。请参考这篇帖子:https://stackoverflow.com/questions/62511091/error-connecting-to-kafka-server-via-ide-in-wsl2
英文:
This is not the optimal solution, but you will be able to connect if you run your producer in Ubuntu/WSL. This means if you are using a Windows IDE, writing the code, switching to Ubuntu and using a command line compiler and running the producer. See this post https://stackoverflow.com/questions/62511091/error-connecting-to-kafka-server-via-ide-in-wsl2
答案8
得分: 0
-
在文件 etc/sysctl.conf 中添加以下内容:
net.ipv6.conf.all.disable_ipv6=1 net.ipv6.conf.default.disable_ipv6=1
-
在你的
server.properties
中将listeners=PLAINTEXT://:9092
替换为listeners=PLAINTEXT://localhost:9092
。 -
使用以下命令更新 sysctl 配置(每次重新启动机器时都需要运行此命令来更新配置):
sudo sysctl -p
英文:
-
Edit the file etc/sysctl.conf and add following lines in it.
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
-
Replace
listeners=PLAINTEXT://:9092
withlisteners=PLAINTEXT://localhost:9092
in your server.properties. -
Update the sysctl config by using the following command. (Everytime you restart your machine this command needs to be run to update the configuration)
sudo sysctl -p
答案9
得分: 0
Step 1
在WSL2上禁用IPv6:
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
Step 2:
在Kafka文件夹的\config\server.properties
中
将listeners=PLAINTEXT://:9092
替换为listeners=PLAINTEXT://localhost:9092
。
Step 3:
在WLS2的Ubuntu中,
sudo sysctl -p
在进行了以上更改后,我在本地环境中使其正常工作。
英文:
Step 1
Disable IPv6 on WSL2:
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
Step 2:
In \config\server.properties
under Kafka folder
Replace listeners=PLAINTEXT://:9092 with listeners=PLAINTEXT://localhost:9092
in your server.properties
.
Step 3:
In WLS2 Ubuntu,
sudo sysctl -p
After making above changes it worked for me in my local
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论