英文:
convert dataframe to dictionary pyspark
问题
我有一个表格，如下所示
|item_name|item_value|timestamp |idx|description|
+---------+----------+----------------------------+---+-----------+
|A |0.25 |2023-03-01T17:20:00.000+0000|0 |unitA& |
|B |0.34 |2023-03-01T17:20:00.000+0000|0 |null |
|C |0.34 |2023-03-01T17:20:00.000+0000|0 |C& |
|D |0.45 |2023-03-01T17:20:00.000+0000|0 |unitD& |
|A |0.3 |2023-03-01T17:25:00.000+0000|1 |unitA& |
|B |0.54 |2023-03-01T17:25:00.000+0000|1 |null |
|C |0.3 |2023-03-01T17:25:00.000+0000|1 |C& |
|D |0.21 |2023-03-01T17:25:00.000+0000|1 |unitD& |
|A |0.54 |2023-03-01T17:30:00.000+0000|2 |unitA& |
我想将其转换为以下结构的字典
{'unitA&': {0: 0.25, 1: 0.3, 2: 0.54},
'C&': {0: 0.34, 1: 0.3},
'unitD&': {0: 0.45, 1: 0.21}}
我该如何使用Pyspark来实现这个目标？
非常感谢您的任何帮助！
英文:
I have a table like this below
|item_name|item_value|timestamp |idx|description|
+---------+----------+----------------------------+---+-----------+
|A |0.25 |2023-03-01T17:20:00.000+0000|0 |unitA& |
|B |0.34 |2023-03-01T17:20:00.000+0000|0 |null |
|C |0.34 |2023-03-01T17:20:00.000+0000|0 |C& |
|D |0.45 |2023-03-01T17:20:00.000+0000|0 |unitD& |
|A |0.3 |2023-03-01T17:25:00.000+0000|1 |unitA& |
|B |0.54 |2023-03-01T17:25:00.000+0000|1 |null |
|C |0.3 |2023-03-01T17:25:00.000+0000|1 |C& |
|D |0.21 |2023-03-01T17:25:00.000+0000|1 |unitD& |
|A |0.54 |2023-03-01T17:30:00.000+0000|2 |unitA& |
and I want to convert it to a dict in this structure
{'unitA&': {0: 0.25,1: 0.3,2:0.54},
'C&': {0: 0.34, 1: 0.3},
'unitD& ': {0: 0.45,1: 0.21}}
How can I do this with Pyspark?
Any help would be appreciated!
答案1
得分: 1
你可以使用内置的 Spark 函数 map_from_entries、collect_list 和 struct 来获取你需要的结果。
示例:
df.show(20,False)
#sampledata
#+---------+----------+----------------------------+---+-----------+
#|item_name|item_value|timestamp |idx|description|
#+---------+----------+----------------------------+---+-----------+
#|A |0.25 |2023-03-01T17:20:00.000+0000|0 |unitA& |
#|B |0.34 |2023-03-01T17:20:00.000+0000|0 |null |
#|C |0.34 |2023-03-01T17:20:00.000+0000|0 |C& |
#|D |0.45 |2023-03-01T17:20:00.000+0000|0 |unitD& |
#|A |0.3 |2023-03-01T17:25:00.000+0000|1 |unitA& |
#|B |0.54 |2023-03-01T17:25:00.000+0000|1 |null |
#|C |0.3 |2023-03-01T17:25:00.000+0000|1 |C& |
#|D |0.21 |2023-03-01T17:25:00.000+0000|1 |unitD& |
#|A |0.54 |2023-03-01T17:30:00.000+0000|2 |unitA& |
#+---------+----------+----------------------------+---+-----------+
df = df.filter(col("description").isNotNull()).groupBy("description").\
agg(map_from_entries(collect_list(struct(col("idx"),col("item_value")))).alias("temp_desc")).\
groupBy(lit(1)).\
agg(map_from_entries(collect_list(struct(col("description"),col("temp_desc")))).alias("a")).\
select("a")
req_dict = str(df.collect()[0][0])
print(req_dict)
结果:
{'unitD&': {0: 0.45, 1: 0.21}, 'C&': {0: 0.34, 1: 0.3}, 'unitA&': {0: 0.25, 1: 0.3, 2: 0.54}}
英文:
You can use map_from_entries,collect_list,struct inbuilt spark functions to get your required result.
Example:
df.show(20,False)
#sampledata
#+---------+----------+----------------------------+---+-----------+
#|item_name|item_value|timestamp |idx|description|
#+---------+----------+----------------------------+---+-----------+
#|A |0.25 |2023-03-01T17:20:00.000+0000|0 |unitA& |
#|B |0.34 |2023-03-01T17:20:00.000+0000|0 |null |
#|C |0.34 |2023-03-01T17:20:00.000+0000|0 |C& |
#|D |0.45 |2023-03-01T17:20:00.000+0000|0 |unitD& |
#|A |0.3 |2023-03-01T17:25:00.000+0000|1 |unitA& |
#|B |0.54 |2023-03-01T17:25:00.000+0000|1 |null |
#|C |0.3 |2023-03-01T17:25:00.000+0000|1 |C& |
#|D |0.21 |2023-03-01T17:25:00.000+0000|1 |unitD& |
#|A |0.54 |2023-03-01T17:30:00.000+0000|2 |unitA& |
#+---------+----------+----------------------------+---+-----------+
df = df.filter(col("description").isNotNull()).groupBy("description").\
agg(map_from_entries(collect_list(struct(col("idx"),col("item_value")))).alias("temp_desc")).\
groupBy(lit(1)).\
agg(map_from_entries(collect_list(struct(col("description"),col("temp_desc")))).alias("a")).\
select("a")
req_dict = str(df.collect()[0][0])
print(req_dict)
Result:
{'unitD&': {0: 0.45, 1: 0.21}, 'C&': {0: 0.34, 1: 0.3}, 'unitA&': {0: 0.25, 1: 0.3, 2: 0.54}}
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论