Compare Latest CSV with all CSV in directory and remove the matching from the latest and write new rows in new file with python

the code will not work properly e.g. when the name of files are something else.

for example when the file name is carre123.csv, it wont compare correctly. but when I changed the file name to test123.csv it works fine.

here is the code

import os
import pandas as pd

# Set the directory where the CSV files are stored
directory = '/PATH/csv-files'

# Get a list of all the CSV files in the directory
csv_files = [os.path.join(directory, f) for f in os.listdir(directory) if f.endswith('.csv')]
#print(csv_files)

# Sort the CSV files by modification time and select the last file as the latest file
latest_file = sorted(csv_files, key=os.path.getmtime)[-1]
#print(latest_file)

# Read the contents of the latest CSV file into a pandas DataFrame
latest_data = pd.read_csv(latest_file)
#print(latest_data)

# Iterate over all the previous CSV files
for csv_file in csv_files[:-1]:
    # Read the contents of the previous CSV file into a pandas DataFrame
    prev_data = pd.read_csv(csv_file)
    #print(prev_data)

    # Identify the rows in the latest CSV file that match the rows in the previous CSV file
    matches = latest_data.isin(prev_data.to_dict('list')).all(axis=1)
    print(matches)

    # Remove the matching rows from the latest CSV file
    latest_data = latest_data[~matches]

# Write the remaining rows in the latest CSV file to a new file
latest_data.to_csv('/NEWPATH/diff.csv', index=False)

when the file name is carre123.csv, it wont compare correctly. but when I changed the file name to test123.csv it works fine.

I think your code has a bug, which may be what is causing the problem. The for loop is over csv_files[:-1] which is not sorted by modification time, so depending on the file names this may cause the loop to include latest_file. Try storing the sorted list, sorted(csv_files, key=os.path.getmtime), then select the last one for latest_file and loop over the remaining files. Maybe there is something else wrong too, but based on the example you provided, this looks like the only issue I can see that is obviously a problem.