英文:
AttributeError: 'NoneType' object has no attribute 'randomSplit'
问题
我在尝试在pySpark中执行randomSplit时一直收到错误。
我已经添加了这些依赖项：
#步骤1：安装依赖!apt-get install openjdk-8-jdk-headless -qq > /dev/null!wget -q https://archive.apache.org/dist/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz!tar xf spark-3.3.0-bin-hadoop3.tgz!pip install -q findspark#步骤2：添加环境变量import osos.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"os.environ["SPARK_HOME"] = "spark-3.3.0-bin-hadoop3"#步骤3：初始化Pysparkimport findsparkfindspark.init()
创建了pySpark环境：
#创建Spark上下文from pyspark.sql import SparkSessionspark = SparkSession.builder.appName('lr_example').getOrCreate()
并添加了这些：
# 导入VectorAssembler和Vectorsfrom pyspark.ml.linalg import Vectorsfrom pyspark.ml.feature import VectorAssemblerfrom pyspark.ml.regression import LinearRegression
然而，每次我运行以下代码时：
final_df = output.select("features", "medv").show()train_data, test_data = final_df.randomSplit([0.7, 0.3])
我收到以下错误：
---------------------------------------------------------------------------AttributeError Traceback (most recent call last)<ipython-input-76-e27b8ca71b51> in <cell line: 1>()----> 1 train_data, test_data = final_df.randomSplit([0.7, 0.3])AttributeError: 'NoneType' object has no attribute 'randomSplit'
有什么想法吗？我搜索了需要导入的内容，似乎已经拥有一切，但它无法加载。GitHub文档链接
英文:
I keep receiving an error when trying to randomSplit in pySpark.
I've added these dependencies:
#Step 1: Install Dependencies!apt-get install openjdk-8-jdk-headless -qq > /dev/null!wget -q https://archive.apache.org/dist/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz!tar xf spark-3.3.0-bin-hadoop3.tgz!pip install -q findspark#Step 2: Add environment variablesimport osos.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"os.environ["SPARK_HOME"] = "spark-3.3.0-bin-hadoop3"#Step 3: Initialize Pysparkimport findsparkfindspark.init()
Created the pySpark environment:
#creating spark contextfrom pyspark.sql import SparkSessionspark = SparkSession.builder.appName('lr_example').getOrCreate()
and added these:
# Import VectorAssembler and Vectorsfrom pyspark.ml.linalg import Vectorsfrom pyspark.ml.feature import VectorAssemblerfrom pyspark.ml.regression import LinearRegression
However, every time I run this:
final_df = output.select("features", "medv").show()train_data, test_data = final_df.randomSplit([0.7, 0.3])
I get this:
---------------------------------------------------------------------------AttributeError Traceback (most recent call last)<ipython-input-76-e27b8ca71b51> in <cell line: 1>()----> 1 train_data, test_data = final_df.randomSplit([0.7, 0.3])AttributeError: 'NoneType' object has no attribute 'randomSplit'
Any ideas? I searched around for what needs to be imported and it seems I have everything but it won't load. Link to Github doc
答案1
得分: 1
final_df = output.select("features", "medv").show()
final_df = output.select("features", "medv") # create df
final_df.show() # print it
英文:
you left out the only important line
final_df = output.select("features", "medv").show()
show prints the results but returns None ... so you are setting final_df to none
instead
final_df = output.select("features", "medv") # create dffinal_df.show() # print it
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论