英文:
Cast a column to binary datatype in spark
问题
我有一个Spark用例，我需要创建一个空列并将其转换为二进制数据类型。我尝试了下面的方法，但没有成功。当我将"Binary"替换为"integer"时，它可以工作。我还尝试了"BinaryType"和"Array[Byte]"，但肯定漏掉了一些东西。
val ip1 = sqlContext.read
.parquet("/home/hadoop/work/aa/bbb/ccc/data/today")
.toDF()
val ip2 = ip1
.withColumn("user_hll", lit("0"))
.select(col("start_timestamp_hr"), col("user_hll"))
.withColumn("hll", col("user_hll").cast("Binary"))
任何帮助将不胜感激。
英文:
I have a Spark use case where I have to create a null column and cast to a binary datatype. I tried the below but it is not working. When I replace Binary by integer, it works. I also tried BinaryType and Array[Byte]. Must be missing something here.
val ip1 = sqlContext.read
.parquet("/home/hadoop/work/aa/bbb/ccc/data/today")
.toDF();
val ip2 = ip1
.withColumn("user_hll",lit("0"))
.select(col("start_timestamp_hr"),col("user_hll"))
.withColumn("hll",col("user_hll").cast("Binary"))
Any help is appreciated
答案1
得分: 3
.withColumn("hll", lit(null).cast(org.apache.spark.sql.types.BinaryType))
英文:
If you want a null binary column, all you need is:
.withColumn("hll", lit(null).cast(org.apache.spark.sql.types.BinaryType))
答案2
得分: 1
不需要转换，保持原文如下：
instead of casting you could also use lit directly
.withColumn("hll",lit("0".getBytes)) // gives [30]
or
.withColumn("hll",lit(Array.empty[Bytes])) // gives []
or if you want null, you can do :
.withColumn("hll",typedLit(Option.empty[Array[Byte]])) // gives null
英文:
instead of casting you could also use lit directly
.withColumn("hll",lit("0".getBytes)) // gives [30]
or
.withColumn("hll",lit(Array.empty[Bytes])) // gives []
or if you want ǹull, you can do :
.withColumn("hll",typedLit(Option.empty[Array[Byte]])) // gives null
答案3
得分: 0
It worked. Simple miss from my side
.cast("Binary")
Any better way would be appreciated
英文:
It worked. Simple miss from my side
.cast("Binary")
Any better way would be appreciated
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论