英文:
Polars read_csv ignore_errors what to do if you can't ignore them?
问题
使用 polars.read_csv 处理大型数据集时出现了字段分隔符问题，导致失败。ignore_errors 选项会跳过错误记录，但我不知道是跳过了一个还是成千上万个记录。是否有办法将错误记录导入到错误文件中，或者报告被忽略的行数？
我希望世界足够简单，以支持单字符列分隔符，但这还没有发生 - 为什么 pandas/pyarrow/polars 不支持多字符字段分隔符呢？
英文:
Using polars.read_csv on a large data set results in a failure because of a field delimiter issue. Ignore_errors skips the erroneous records, but I have no idea if one or thousands of records were ignored. Is there a way to pipe the bad records to a bad file or report the number of ignored rows?
I wish the world was simple enough for data to support single character column delimiters, but that hasn't happened yet - why doesn't pandas/pyarrow/polars support multi character field delimiters?
答案1
得分: 1
Polars库在使用ignore_errors参数时不提供将坏记录导向单独文件或报告被忽略行数的机制。您可以通过以下方式手动执行，但我不确定这是否符合您的要求：
import polars as pl# 定义CSV文件的路径csv_file = "path/to/your/file.csv"# 创建一个空的DataFrame以存储坏记录bad_records = pl.DataFrame()# 读取CSV文件并手动处理错误with open(csv_file, "r") as file:for line in file:try:# 将行解析为DataFramedf = pl.from_csv_string(line, delimiter=',')# 根据需要处理有效的DataFrame# ...except Exception:# 如果发生错误，将该行附加到bad_records DataFramebad_records = bad_records.append(pl.DataFrame())# 将坏记录写入单独的CSV文件bad_records.to_csv("path/to/bad_records.csv")# 获取被忽略行数ignored_rows = len(bad_records)# 打印被忽略行数print(f"被忽略的行数：{ignored_rows}")
关于您的第二个问题，您可以在读取CSV文件时使用pandas.read_csv()函数的"sep"参数来更改字段分隔符。"sep"参数允许您指定CSV文件中使用的分隔符字符或字符串。例如：
df = pd.read_csv(csv_file, sep=';') # 用您希望的分隔符替换';'
英文:
Polars library doesn't provide a mechanism to pipe the bad records to a separate file or report the number of ignored rows when using the ignore_errors parameter. You could do it manually in the following way but I don't know if it's what you want:
import polars as pl# Define the path to your CSV filecsv_file = "path/to/your/file.csv"# Create an empty DataFrame to store the bad recordsbad_records = pl.DataFrame()# Read the CSV file and handle errors manuallywith open(csv_file, "r") as file:for line in file:try:# Parse the line as a DataFramedf = pl.from_csv_string(line, delimiter=',')# Process the valid DataFrame as needed# ...except Exception:# If an error occurs, append the line to the bad_records DataFramebad_records = bad_records.append(pl.DataFrame())# Write the bad records to a separate CSV filebad_records.to_csv("path/to/bad_records.csv")# Get the count of ignored rowsignored_rows = len(bad_records)# Print the number of ignored rowsprint(f"Number of ignored rows: {ignored_rows}")
Regarding your second question., in Pandas you can change the field delimiters when reading a CSV file by specifying the "sep" parameter in the pandas.read_csv() function. The "sep" parameter allows you to specify the delimiter character or string used in the CSV file.
For example:
df = pd.read_csv(csv_file, sep=';') # Replace ';' with your desired delimiter
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论