pandas dataframe : how to update specific rows in hive table

I want to update a single column in a hive table. Here is how i am selecting the data

from datetime import timedelta, date
import pandas as pd
from pyspark import SparkContext
from pyspark.sql import SQLContext,HiveContext
from pyspark.sql import SparkSession
import os
sc = SparkContext.getOrCreate()
spark = SparkSession.builder.appName("job1").enableHiveSupport().getOrCreate()


test1=spark.sql("""
select * from myPartitionedTable
where part1='adqwf' and 
      part2='avgewg' and
      col2='filter_condition'
""")

import pandas as pd
import json
import urllib.parse
spark.conf.set("spark.sql.execution.arrow.enabled", "true")
pandasDF = test1.toPandas()
smvr={}

for index, row in pandasDF.iterrows():
    newpr = []
    pr = pandasDF.iloc[index]['col1']
    if pr:
        
        for p in pr:
            newp = p.asDict()
            newp['attribute1']=p.attribute1+'_suffix'
            newpr.append(newp)
        print(newpr[0])
        pandasDF.iloc[index]['attribute1'][0]=newpr[0]
        print(pandasDF.iloc[index]['attribute1'][0])
        
        break

so with the above snippet, i am able to see that

print(pandasDF.iloc[index]['attribute1'][0])

has the updated value. Now, how can i update the underlying hive table as well? I was going through some resources online and the following suggestions appeared

spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")
data.write.mode("overwrite").insertInto("partitioned_table")

this could work but note that in my select query i have

      col2='filter_condition'

so basically i dont want to overwrite the entire partition, but only the rows that i have selected.

Is that possible with the above approach?

I think I understand this right, but I am unable to test it. Let me know how this works.

You need to modify your code to update the pd.DataFrame with the new values, and then convert the updated pd.DataFrame back to a Spark DataFrame using the createDataFrame() method of the SparkSession.

from pyspark.sql.functions import col
from pyspark.sql.types import StructType, StructField, StringType

# Define the schema of your DataFrame
schema = StructType([
    StructField("col1", StringType()),
    StructField("attribute1", StringType()),
    StructField("products", StringType()),
    StructField("part1", StringType()),
    StructField("part2", StringType())
])

# Create a new DataFrame with the updated values
updated_df = spark.createDataFrame(
    pandasDF,
    schema=schema
).withColumn("products", col("products").cast("array<struct<attribute1:string,attribute2:string>>"))

# Enable dynamic partition overwrite mode
spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")

# Overwrite only the selected rows in your Hive table
updated_df.write.mode("overwrite").partitionBy("part1", "part2").format("hive").saveAsTable("myPartitionedTable")