将Spark Dataset列从UDT转换为Array

huangapple go评论70阅读模式
英文:

Convert a Spark Dataset column from a UDT to Array<String>

问题

I'm using the Spark OrientDB connector to retrieve some data that looks like the following:

character title
Tony Stark ["Iron Man"]
James Buchanan Barnes ["Captain America: The First Avenger","Captain America: The Winter Soldier","Captain America: Civil War","Avengers: Infinity War"]
Marcus Bledsoe ["Captain America: The Winter Soldier"]

The Dataframe returns this as [character: string, title: embeddedlist]. An EmbeddedList is a UDT defined here

I would like to treat the title as an Array<String> so that I can do the following:

val vertices = df
  .select(explode(concat(array('character), 'title)) as "x")
  .distinct.rdd.map(_.getAs[String](0))
  .zipWithIndex.map(_.swap)

I'm not sure how to cast/convert the EmbeddedList correctly. Running this as-is results in the error: cannot resolve 'concat(array(name), out)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [array<string>, array<string>]

Any help/pointers are appreciated.

Edit: The way I'm receiving the data is in this structure:

val df: DataFrame = Seq(
  "Tony Stark" -> EmbeddedList(Array("Iron Man")),
  "James Buchanan Barnes" -> EmbeddedList(Array("Captain America: The First Avenger", "Captain America: The Winter Soldier", "Captain America: Civil War", "Avengers: Infinity War")),
  "Marcus Bledsoe" -> EmbeddedList(Array("Captain America: The Winter Soldier"))
).toDF("character", "title")
英文:

I'm using the Spark OrientDB connector to retrieve some data that looks like the following:

character title
Tony Stark ["Iron Man"]
James Buchanan Barnes ["Captain America: The First Avenger","Captain America: The Winter Soldier","Captain America: Civil War","Avengers: Infinity War"]
Marcus Bledsoe ["Captain America: The Winter Soldier"]

The Dataframe returns this as [character: string, title: embeddedlist]. An EmbeddedList is a UDT defined here

I would like to treat the title as an Array&lt;String&gt; so that I can do the following:

    val vertices = df
      .select(explode(concat(array(&#39;character), &#39;title)) as &quot;x&quot;)
      .distinct.rdd.map(_.getAs[String](0))
      .zipWithIndex.map(_.swap)

I'm not sure how to cast/convert the EmbeddedList correctly. Running this as-is results in the error: cannot resolve &#39;concat(array(name), out)&#39; due to data type mismatch: input to function concat should have been string, binary or array, but it&#39;s [array&lt;string&gt;, array&lt;string&gt;]

Any help/pointers are appreciated.

Edit: The way I'm receiving the data is in this structure:

    val df: DataFrame = Seq(
      &quot;Tony Stark&quot; -&gt; EmbeddedList(Array(&quot;Iron Man&quot;)),
      &quot;James Buchanan Barnes&quot; -&gt; EmbeddedList(Array(&quot;Captain America: The First Avenger&quot;, &quot;Captain America: The Winter Soldier&quot;, &quot;Captain America: Civil War&quot;, &quot;Avengers: Infinity War&quot;)),
      &quot;Marcus Bledsoe&quot; -&gt; EmbeddedList(Array(&quot;Captain America: The Winter Soldier&quot;))
    ).toDF(&quot;character&quot;, &quot;title&quot;)

答案1

得分: 0

我能够通过将 EmbeddedList 转换为字符串,然后按逗号拆分来解决这个问题。我必须相信有一种更加优雅的方法来做这个,但至少目前这个方法有效。

val vertices = df
  .withColumn("title", col("title").cast("String"))
  .withColumn("title", split(col("title"), ", "))
  .select(explode(concat(array('character), 'title)) as "x")
  .distinct.rdd.map(_.getAs[String](0))
  .zipWithIndex.map(_.swap)
英文:

I was able to solve this by casting the EmbeddedList to a string and then splitting on the comma. I have to believe there's a more elegant way to do this but this works for now at least.

    val vertices = df
      .withColumn(&quot;title&quot;, col(&quot;title&quot;).cast(&quot;String&quot;))
      .withColumn(&quot;title&quot;, split(col(&quot;title&quot;), &quot;, &quot;))
      .select(explode(concat(array(&#39;character), &#39;title)) as &quot;x&quot;)
      .distinct.rdd.map(_.getAs[String](0))
      .zipWithIndex.map(_.swap)

huangapple
  • 本文由 发表于 2023年8月4日 01:18:58
  • 转载请务必保留本文链接:https://go.coder-hub.com/76830314.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定