How can we merge column headers from multiple CSVs into one dataframe, and list source file names for each file in one column?

Here is the code that I am testing.

# import necessary libraries
import pandas as pd
import os
import glob
  
  
# use glob to get all the csv files 
# in the folder
path = 'C:\\Users\\'
csv_files = glob.glob(os.path.join(path, "*.csv"))
csv_files


df_headers = pd.DataFrame()


# loop over the list of csv files
for f in csv_files:
    #print(type(f))

    # read the csv file
    df = pd.read_csv(f, nrows=1)
    print(df.shape)
    
    
    df_headers = pd.concat([df_headers, df], axis=0)
    df_headers['file_name'] = f
      

df_headers.to_csv('C:\\Users\\ryans\\Desktop\\out.csv')

This almost works, but it always writes the last file to the column in df_headers['file_name'], so only the last file that the loop goes through, is actually listed in 'file_name'.

It is because whenever you assign a single value to a whole column, the same value gets repeated.

For example, if you have your csv files like -

csv_files = ['1.csv', '2.csv', '3.csv']

<b> Ist Iteration </b><br>

Your df will be like

|Some columns......|'filename'|<br>
|Some values.......|'1.csv'|<br>

<b> IInd Iteration </b><br>

Your df will be like

|Some columns......|'filename'|<br>
|Some values.......|'2.csv'|<br>
|Some values.......|'2.csv'|<br>

and so on.

When you are assigning a single value to a column, the same value gets assigned to all the values in the column. Simply put, a dataframe like-

|A|B|<br>
|1|2|<br>
|3|4|<br>

if you do,

df['B'] = 5
, all values in B will become 5, therefore your dataframe becomes-

|A|B|<br>
|1|5|<br>
|3|5|<br>

A solution to your case could be-

# import necessary libraries
import pandas as pd
import os
import glob
  
  
# use glob to get all the csv files 
# in the folder
path = 'C:\\Users\\'
csv_files = glob.glob(os.path.join(path, "*.csv"))
csv_files


df_headers = pd.DataFrame()


# loop over the list of csv files
for f in csv_files:
    #print(type(f))

    # read the csv file
    df = pd.read_csv(f, nrows=1)
    print(df.shape)
    
    
    df_headers = pd.concat([df_headers, df], axis=0)

df_headers['file_name'] = csv_files
      

df_headers.to_csv('C:\\Users\\ryans\\Desktop\\out.csv')