Append Value in rows with existing value in Databricks

I am new to Databricks so bear with me if I sound stupid.
I have a requirement wherein I am doing validations to a data frame and I currently have defined functions for each validation like one for null-check one for Date_range, every time my function meets the validation rules it should mark 1 in **Validation_remark **column like below

Name	ID	Date_of Birth	position	Validation remarks
dam 	1	02-04-1992	Manager 	
dana		02-04-1992	Associate	1
rich	3	02-04-1992	VP	
danial	4	02-04-1992	CEO	
mathew		02-04-1910	Manager 	1

but the problem here here is i am not able to figure out what is the reason for the function to mark it 1 whether it's because ID col is empty or whether it's because Date_of_birth os > 100 years or may be both.

So I want to know if I can append the reason like below.

Name	ID	Date_of Birth	position	Validation remarks
dam 	1	02-04-1992	Manager 	
dana		02-04-1992	Associate	ID id null
rich	3	02-04-1992	VP	
danial	4	02-04-1992	CEO	
mathew		02-04-1910	Manager 	['ID is null', 'Date_of_Birth is > 100 years']

i.e if the row have blank ID then note that late if it also has Date_of Birth > 100 years append that value as well like above.

I just want to know how do I append the values to Validation remarks

You could use a custom udf function in PySpark to do this.

from pyspark.sql.functions import *
from pyspark.sql.types import ArrayType, StringType
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
# Create example DataFrame
df = spark.createDataFrame([("dam", "1", "02-04-1992", "Manager"),
("dana", "", "02-04-1992", "Associate"),
("rich", "3", "02-04-1992", "VP"),
("danial", "4", "02-04-1992", "CEO"),
("mathew", "", "02-04-1910", "Manager")],
["Name", "ID", "Date_of_birth", "position"])
df.show()
# +------+---+-------------+---------+
# |  Name| ID|Date_of_birth| position|
# +------+---+-------------+---------+
# |   dam|  1|   02-04-1992|  Manager|
# |  dana|   |   02-04-1992|Associate|
# |  rich|  3|   02-04-1992|       VP|
# |danial|  4|   02-04-1992|      CEO|
# |mathew|   |   02-04-1910|  Manager|
# +------+---+-------------+---------+
# Python func to check ID and date of birth, return list of remarks
def validate_row(row):
remarks = []
if row.ID == "":
remarks.append("ID is null")
if (row.Date_of_birth is not None and
int(row.Date_of_birth.split("-")[-1]) <= 2023 - 100):
remarks.append("Date_of_Birth is > 100 years") # I will check with current year 2023
return remarks
# Create an UDF func from Python func above
validate_udf = udf(validate_row, ArrayType(StringType()))
# Apply the func above to each row in DF
df = df.withColumn("Validation remarks", validate_udf(struct(df.columns)))
df.show()
# +------+---+-------------+---------+--------------------+
# |  Name| ID|Date_of_birth| position|  Validation remarks|
# +------+---+-------------+---------+--------------------+
# |   dam|  1|   02-04-1992|  Manager|                  []|
# |  dana|   |   02-04-1992|Associate|        [ID is null]|
# |  rich|  3|   02-04-1992|       VP|                  []|
# |danial|  4|   02-04-1992|      CEO|                  []|
# |mathew|   |   02-04-1910|  Manager|[ID is null, Date_of_Birth is > 100 years|
# +------+---+-------------+---------+--------------------+

Here are my 2 cents:

1. Created a dataframe as follows:

    +------+----+-------------+---------+
   |  Name|  ID|Date_of_Birth| position|
   +------+----+-------------+---------+
   |   dam|   1|   1992-04-02|  Manager|
   |  dana|null|   1992-04-02|Associate|
   |  rich|   3|   1992-04-02|       VP|
   |danial|   4|   1992-04-02|      CEO|
   |mathew|null|   1910-04-02|  Manager|
   +------+----+-------------+---------+
   

2. Put all your conditions in the list(cond_result_list):

    df_input = df_input.withColumn('Validation remarks',lit(('')))
   cond_result_list = [
   ["id is null",'Id is Null'],
   ["DATEDIFF(CURRENT_DATE(), Date_of_Birth) / 365.25 > 100",'Date_of_Birth is > 100 years']
   ]
   for i in range(len(cond_result_list)):
   df_input = df_input.withColumn('Validation remarks',
   concat(col('Validation remarks'), 
   expr("""case when {} then '{}{}' else '' end """.format(cond_result_list[i][0],'|',cond_result_list[i][1]))
   )
   )
   df_input= df_input.withColumn("Validation remarks", split(df_input["Validation remarks"], "\|"))
   df_input = df_input.withColumn("Validation remarks", array_remove(df_input["Validation remarks"], ""))
   

3. Print the dataframe:

   df_input.show(truncate=False)
   +------+----+-------------+---------+------------------------------------------+
   |Name  |ID  |Date_of_Birth|position |Validation remarks                        |
   +------+----+-------------+---------+------------------------------------------+
   |dam   |1   |1992-04-02   |Manager  |[]                                        |
   |dana  |null|1992-04-02   |Associate|[Id is Null]                              |
   |rich  |3   |1992-04-02   |VP       |[]                                        |
   |danial|4   |1992-04-02   |CEO      |[]                                        |
   |mathew|null|1910-04-02   |Manager  |[Id is Null, Date_of_Birth is > 100 years]|
   +------+----+-------------+---------+------------------------------------------+