Loading data in a dataframe - pyspark

I'm trying to load data into a dataframe from a partitionned structure.
I have a specific period selected in my example and then loop to have my data range.
But This is not very performant!

start_date = inputdate

end_date = inputend

df_union = None

for single_date in pd.date_range(start_date, end_date, freq='D'):
    yy = single_date.strftime("%y")
    mm = single_date.strftime("%m")
    dd = single_date.strftime("%d")

    df = spark.read.parquet(f"abfs://XXXXX.dfs.core.windows.net/coredb/user_action/{yy}/{mm}/{dd}/user_action.parquet") \
                    .select("user_action_id", "account_id", "inserted" , "partner_id", "status", "service_id") 
            
            
    if df_union is None:
        df_union = df
    else:
        df_union = df_union.unionAll(df)

Any help ?

You can directly use wildcard paths to read all files (with same schema) directly.

• I have a file called sample1.csv (3 rows) in my source directories structured as yyyy/MM/dd. I have 3 date folder structures i.e., 2023/02/25, 2023/02/26 and 2023/02/27. Using the following code I am able to read the data successfully:

df = spark.read.option("header",True).csv('abfss://data@datalk2702.dfs.core.windows.net/2023/*/*/sample1.csv')

display(df.limit(10))

• But this reads all files with name sample1.csv from everything inside 2023 folder. If you want to read only in between startdate and enddate, use the following code:

import pandas as pd
start_date = '2023-02-25'
end_date = '2023-02-27'

base_path = 'abfss://data@datalk2702.dfs.core.windows.net/'
paths = []  

for single_date in pd.date_range(start_date, end_date, freq='D'):

    paths.append(base_path + f'{single_date.strftime("%Y")}/{single_date.strftime("%m")}/{single_date.strftime("%d")}/sample1.csv')

#print(paths)

df = spark.read.option("header",True).csv(paths)
display(df.limit(10))