How to rename the columns inside nested column in pyspark

I have a column product inside which there is a nested column called Color. I want to remove the {} from the column color. <br>

I don't want to flatten the column and rename it. I directly want to rename the column or drop the column.

|-- product: struct (nullable = true) 
| |-- {Color}: string (nullable = true) 

I have tried dropping it but it doesn't work. I don't want to create a new struct as I have many more nested columns and they are too much.

|-- product: struct (nullable = true)
| |-- {Color}: string (nullable = true)#

Try with .withField to update the field name without flattening.

Then use .dropFields to drop nested columns from struct.

Example:

#sample json
json = '{"product":{"{Color}":"a"}}'
df = spark.read.json( sc.parallelize([json]))

#create Color column by using `.withField` and copy the `{Color}`data
#use .dropFields to drop struct columns
df1= df.withColumn("product", df['product'].withField('Color',col('product.`{Color}`'))).\
withColumn("product", col("product").dropFields("`{Color}`"))
df1.printSchema()
df1.show(10,False)

#root
# |-- product: struct (nullable = true)
# |    |-- Color: string (nullable = true)
#
#+-------+
#|product|
#+-------+
#|{a}      |
#+-------+

• You can use withColumn to rename the nested column. The following is a code that you can use. I have a dataframe with following schema (same as yours):

df.printSchema()

• Now using withColumn as shown in the below image, you can change the name of your nested column color:

from pyspark.sql.functions import col, struct

df.withColumn("product", struct(col("product.{Color}").alias("Color"))).printSchema()