Pyspark 在数组<string> 上使用 DocumentAssembler。

huangapple go评论58阅读模式
英文:

Pyspark use DocumentAssembler on array<string>

问题

I am trying to use DocumentAssembler for an array of strings. The documentation says: "DocumentAssembler可以读取String列或Array[String]。" But when I do a simple example:

data = spark.createDataFrame([["Spark NLP is an open-source text processing library."]]).toDF("text")
documentAssembler = DocumentAssembler().setInputCol("text").setOutputCol("document")
result = documentAssembler.transform(data)

result.select("document").show(truncate=False)

I am getting an error:

AnalysisException: [CANNOT_UP_CAST_DATATYPE] Cannot up cast input from "ARRAY<STRING>" to "STRING".
The type path of the target object is:
- root class: "java.lang.String"
You can either add an explicit cast to the input data or choose a higher precision type of the field in the target object.
Maybe I don't understand something?
英文:

I am trying to use DocumentAssembler for array of strings. The documentation says: "The DocumentAssembler can read either a String column or an Array[String])".
But when I do a simple example:

data = spark.createDataFrame([[[&quot;Spark NLP is an open-source text processing library.&quot;]]]).toDF(&quot;text&quot;)
documentAssembler = DocumentAssembler().setInputCol(&quot;text&quot;).setOutputCol(&quot;document&quot;)
result = documentAssembler.transform(data)

result.select(&quot;document&quot;).show(truncate=False)

I am getting an error

AnalysisException: [CANNOT_UP_CAST_DATATYPE] Cannot up cast input from &quot;ARRAY&lt;STRING&gt;&quot; to &quot;STRING&quot;.
The type path of the target object is:
- root class: &quot;java.lang.String&quot;
You can either add an explicit cast to the input data or choose a higher precision type of the field in the target object

Maybe I don't understand something?

答案1

得分: 0

I think you just added an extra [] around the input

This is working:

data = spark.createDataFrame([["Spark NLP is an open-source text processing library."]]).toDF("text")
documentAssembler = DocumentAssembler().setInputCol("text").setOutputCol("document")
result = documentAssembler.transform(data)

result.select("document").show(truncate=False)
+----------------------------------------------------------------------------------------------+
|document                                                                                      |
+----------------------------------------------------------------------------------------------+
|[{document, 0, 51, Spark NLP is an open-source text processing library., {sentence -> 0}, []}]|
+----------------------------------------------------------------------------------------------+
英文:

I think you just added an extra [] around the input

This is working:

data = spark.createDataFrame([[&quot;Spark NLP is an open-source text processing library.&quot;]]).toDF(&quot;text&quot;)
documentAssembler = DocumentAssembler().setInputCol(&quot;text&quot;).setOutputCol(&quot;document&quot;)
result = documentAssembler.transform(data)

result.select(&quot;document&quot;).show(truncate=False)
+----------------------------------------------------------------------------------------------+
|document                                                                                      |
+----------------------------------------------------------------------------------------------+
|[{document, 0, 51, Spark NLP is an open-source text processing library., {sentence -&gt; 0}, []}]|
+----------------------------------------------------------------------------------------------+

huangapple
  • 本文由 发表于 2023年5月22日 18:55:48
  • 转载请务必保留本文链接:https://go.coder-hub.com/76305454.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定