如何使用Python按时间顺序从一个文件中导入所有CSV文件？

huangapple
go评论108阅读模式
英文:

How to import all csv files from one file in chronological order with python?

问题

I have around 2000 CSV files in my folder. I want to read them in in their chronological order. They are named with numbers so it must be easy I thought.

我有大约2000个CSV文件在我的文件夹中。我想按照它们的时间顺序读取它们。它们以数字命名，所以我认为这应该很容易。

I am reading them in with this following code. I can imagine a very simple solution since there must be an easy parameter for that. But I havent found anything :(((

我正在使用以下代码读取它们。我可以想象有一个非常简单的解决方案，因为肯定有一个简单的参数可以做到这一点。但是我还没有找到任何信息：(((

  1. def csv_to_df():
  2.     dff_all_from_csv = []
  3.     
  4.     for root, dirs, files in os.walk("output/csv_files"):
  5.         for file in files:
  6.             df = pd.read_csv(os.path.join(root, file))
  7.             dff_all_from_csv.append(df)
  8.     return dff_all_from_csv

这是我的代码，我正在使用它来读取这些文件。我认为应该有一个简单的解决方案，因为肯定有一个易于使用的参数来实现这个目标。但是我还没有找到任何信息：(((

英文:

I have around 2000 CSV files in my folder. I want to read them in in their chronological order. They are named with numbers so it must be easy I thought.

如何使用Python按时间顺序从一个文件中导入所有CSV文件？

I am reading them in with this following code. I can imagine a very simple solution since there must be an easy parameter for that. But I havent found anything :(((

  1. def csv_to_df():
  2.     dff_all_from_csv = []
  3.     
  4.     for root, dirs, files in os.walk("output/csv_files"):
  5.         for file in files:
  6.             df = pd.read_csv(os.path.join(root, file))
  7.             dff_all_from_csv.append(df)
  8.     return dff_all_from_csv

答案1

得分: 2

你可以split filename，然后使用 stem/number 作为 sorting 的 key：

  1. def csv_to_df():
  2.     dff_all_from_csv = []
  3.     
  4.     for root, dirs, files in os.walk("output/csv_files"):
  5.         for file in sorted(files, key=lambda x: int(x.split(".")[0])): # <- line updated
  6.             df = pd.read_csv(os.path.join(root, file))
  7.             dff_all_from_csv.append(df)
  8.     return dff_all_from_csv

或者使用 natsorted 来自 [tag:natsort]：

  1. #pip install natsort
  2. from natsort import natsorted
  4. ...
  5. for root, dirs, files in os.walk("output/csv_files"):
  6.     for file in natsorted(files): # <- line updated
  7.         ...
英文:

You can split the filename and use the stem/number as a sorting key :

  1. def csv_to_df():
  2.     dff_all_from_csv = []
  3.     
  4.     for root, dirs, files in os.walk(&quot;output/csv_files&quot;):
  5.         for file in sorted(files, key=lambda x: int(x.split(&quot;.&quot;)[0])): # &lt;- line updated
  6.             df = pd.read_csv(os.path.join(root, file))
  7.             dff_all_from_csv.append(df)
  8.     return dff_all_from_csv

Or use natsorted from [tag:natsort] :

  1. #pip install natsort
  2. from natsort import natsorted
  4.     ...
  5.     for root, dirs, files in os.walk(&quot;output/csv_files&quot;):
  6.         for file in natsorted(files): # &lt;- line updated
  7.         ...

答案2

得分: 0

你可以尝试：

  1. column_df = pd.read_csv(r'1.csv')
  2. column_df.columns
  4. all_csv_df = pd.DataFrame(columns=column_df.columns)
  5. for i in range(1,5):
  6.     r = pd.read_csv(r''+str(i)+'.csv')
  7.     all_csv_df = all_csv_df.append(r)
  9.     
  10. all_csv_df
英文:

you can try:

  1. column_df = pd.read_csv(r&#39;1.csv&#39;)
  2. column_df.columns
  4. all_csv_df = pd.DataFrame(columns=column_df.columns)
  5. for i in range(1,5):
  6.     r = pd.read_csv(r&#39;&#39;+str(i)+&#39;.csv&#39;)
  7.     all_csv_df = all_csv_df.append(r)
  9.     
  10. all_csv_df

答案3

得分: 0

你可以使用 pathlib 和 lstat 属性来按创建时间 (st_ctime) 或修改时间 (st_mtime) 对文件进行排序：

  1. import pathlib
  3. DATA_DIR = 'output/csv_files'
  5. dff_all_from_csv = [pd.read_csv(f) for f in sorted(DATA_DIR.glob('*.csv'),
  6.                                                    key=lambda x: x.lstat().st_mtime)]
英文:

You can use pathlib and lstat attribute to sort your file by creation time (st_ctime) or modification time (st_mtime):

  1. import pathlib
  3. DATA_DIR = &#39;output/csv_files&#39;
  5. dff_all_from_csv = [pd.read_csv(f) for f in sorted(DATA_DIR.glob(&#39;*.csv&#39;),
  6.                                                    key=lambda x: x.lstat().st_mtime)]

答案4

得分: 0

您可以使用 os.path.getmtime() 获取csv文件的日期。您可以将创建日期添加到一个列表中，然后可以从排序后的列表中打开数据框架。

  1. import os
  2. import time
  3. import pandas as pd
  5. path_to_csv_files = "./csv_files/"
  7. # 存储每个csv文件的名称和最后修改日期的元组列表
  8. metadata = list()
  10. for _, _, files in os.walk("./csv_files"):
  11.    for name in files:
  12.       # 检索最后修改日期并将其格式化为可按数字排序的形式
  13.       creation_date = time.strftime("%Y%m%d%H%M%S", time.gmtime(os.path.getmtime(f"{path_to_csv_files}{name}")))
  14.       # 将其转换为整数，以便我们可以按日期对元数据进行排序
  15.       creation_date = int(creation_date)
  16.       metadata.append((name, creation_date))
  18. # 按日期对元数据进行排序
  19. metadata = sorted(
  20.     metadata, 
  21.     key=lambda x: x[1]
  22.     )
  24. # 按日期顺序放置的数据框架列表
  25. list_of_df_from_csv = list()
  27. for name, _ in metadata:
  28.    path_to_csv = path_to_csv_files + name
  29.    df = pd.read_csv(path_to_csv)
  30.    list_of_df_from_csv.append(df)
英文:

You can retrieve the date of a csv file using os.path.getmtime(). You can add the creation dates into a list that you can sort. Then you can open the dataframes from the sorted list.

  1. import os
  2. import time
  3. import pandas as pd
  5. path_to_csv_files = &quot;./csv_files/&quot;
  7. # list in which we&#39;ll store the name and the last modification date of each csv file
  8. metadata = list()
  10. for _, _, files in os.walk(&quot;./csv_files&quot;):
  11.    for name in files:
  12.       # retrieving the last modif date and formating it so it is is numerically sortable
  13.       creation_date = time.strftime(&quot;%Y%m%d%H%M%S&quot;,time.gmtime(os.path.getmtime(f&quot;{path_to_csv_files}{name}&quot;)))
  14.       # turing it into an int so we can sort the metadata per date
  15.       creation_date = int(creation_date)
  16.       metadata.append((name, creation_date))
  18. # sorting the metadata per date
  19. metadata = sorted(
  20.     metadata, 
  21.     key=lambda x: x[1]
  22.     )
  24. # list of dataframes placed in date order
  25. list_of_df_from_csv = list()
  27. for name, _ in metadata:
  28.    path_to_csv = path_to_csv_files+name
  29.    df = pd.read_csv(path_to_csv)
  30.    list_of_df_from_csv.append(df)

答案5

得分: 0

我尝试了类似这样的方法，它完美运行：

  1. import os
  2. import pandas as pd
  4. def csv_to_df():
  5.     
  6.     folder_path = "output/csv_files"
  7.     
  8.     files = [os.path.join(folder_path, f) for f in os.listdir(folder_path) if f.endswith('.csv')]
  9.     files = sorted(files, key=os.path.getmtime)
  10.     
  11.     dff_all_from_csv = []
  12.     for file in files:
  13.         df = pd.read_csv(file)
  14.         dff_all_from_csv.append(df)
  15.     
  16.     return dff_all_from_csv
英文:

I tried something like this and it works perfect:

  1. import os
  2. import pandas as pd
  4. def csv_to_df():
  5.     
  6.     folder_path = &quot;output/csv_files&quot;
  7.     
  8.     files = [os.path.join(folder_path, f) for f in os.listdir(folder_path) if f.endswith(&#39;.csv&#39;)]
  9.     files = sorted(files, key=os.path.getmtime)
  10.     
  11.     dff_all_from_csv = []
  12.     for file in files:
  13.         df = pd.read_csv(file)
  14.         dff_all_from_csv.append(df)
  15.     
  16.     return dff_all_from_csv

huangapple
  • 本文由 发表于 2023年5月10日 20:37:25
  • 转载请务必保留本文链接：https://go.coder-hub.com/76218516.html
  • chronological
  • csv
  • dataframe
  • pandas
  • python
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定