ModuleNotFoundError: 找不到模块名为 ‘pyspark.streaming.kafka’

huangapple go评论67阅读模式
英文:

ModuleNotFoundError: No module named 'pyspark.streaming.kafka'

问题

SPARK_VERSION = '3.1.2'.

SCALA_VERSION = '2.12'.

import findspark

findspark.add_packages(['org.apache.spark:spark-sql-kafka-0-10_' + SCALA_VERSION + ':' + SPARK_VERSION])
findspark.init()

from pyspark import SparkContext, SparkConf
import sys
import time
from pyspark.context import SparkContext
from pyspark import SparkContext, SparkConf
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils

ModuleNotFoundError                       Traceback (most recent call last)

/tmp/ipykernel_567977/2450515063.py in <module>
      4 from pyspark import SparkContext, SparkConf
      5 from pyspark.streaming import StreamingContext
----> 6 from pyspark.streaming.kafka import KafkaUtils

ModuleNotFoundError: No module named 'pyspark.streaming.kafka'
英文:

SPARK_VERSION = '3.1.2'.

SCALA_VERSION = '2.12'.

import findspark

findspark.add_packages([&#39;org.apache.spark:spark-sql-kafka-0-10_&#39; + SCALA_VERSION + &#39;:&#39; + SPARK_VERSION ])
findspark.init()
 
from pyspark import SparkContext, SparkConf
import sys
import time
from pyspark.context import SparkContext
from pyspark import SparkContext, SparkConf
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils

ModuleNotFoundError                       Traceback (most recent call last)


/tmp/ipykernel_567977/2450515063.py in &lt;module&gt;
      4 from pyspark import SparkContext, SparkConf
      5 from pyspark.streaming import StreamingContext
----&gt; 6 from pyspark.streaming.kafka import KafkaUtils


ModuleNotFoundError: No module named &#39;pyspark.streaming.kafka&#39;

答案1

得分: 2

KafkaUtils 包属于传统的 Spark Streaming,而您正在尝试使用 Spark Structured Streaming 中的 Kafka 支持

实际上,在 2023 年使用传统的 Spark Streaming 是没有必要的。只需按照文档中的说明访问 Kafka 中的数据。

df = spark \
  .readStream \
  .format("kafka") \
  .option("kafka.bootstrap.servers", "host:port") \
  .option("subscribe", "my_topic") \
  .load()
英文:

The KafkaUtils package belongs to the legacy Spark Streaming, while you're trying Kafka support in Spark Structured Streaming.

Really, there should be no reason to use legacy Spark Streaming in 2023rd. Just access data in Kafka as described in the documentation.

df = spark \
  .readStream \
  .format(&quot;kafka&quot;) \
  .option(&quot;kafka.bootstrap.servers&quot;, &quot;host:port&quot;) \
  .option(&quot;subscribe&quot;, &quot;my_topic&quot;) \
  .load()

huangapple
  • 本文由 发表于 2023年2月23日 21:23:08
  • 转载请务必保留本文链接:https://go.coder-hub.com/75545444.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定