在R中如何处理多个csv文件以识别空值？

huangapple
go评论102阅读模式
英文:

How to process multiple csv files for identifying null values in R?

问题

我有各种 .csv 文件。每个文件都有多列。我正在使用提供的 R 代码进行质量检查，针对特定列，查看有多少行具有有效值，有多少行为空值。该代码在单个 csv 文件上运行良好。但我想对所有 csv 文件运行此代码，并需要每个 csv 文件的输出。此外，我想要一个日志文件。请问有谁能帮我修改代码，使其能够处理各种 csv 文件。

  1. install.packages("readr") 
  2. library(readr)
  4. check_column <- function(df, column) {
  5.   valid_values <- !is.na(df[[column]])
  6.   num_valid <- sum(valid_values)
  7.   num_null <- nrow(df) - num_valid
  8.   return(c(num_valid, num_null))
  9. }
  11. # 读取 CSV 文件
  12. df <- read_csv("data.csv")
  14. for (column in names(df)) {
  15.   results <- check_column(df, column)
  16.   print(paste(column, ": ", results[1], " 有效, ", results[2], " 空值"))
  17. }

示例数据：（并非所有文件具有相同数量的列）

Csv1.csv

  1. D_T  Temp (°C)  Press (Pa)  ...
  2. 2021-03-01 00:00:00+00  28  1018  ...
  3. 2021-03-02 00:00:00+00  27  1017  ...
  4. 2021-03-03 00:00:00+00  28  1019  ...
  5. ..
  6. ..
  8. Csv2.csv

D_T Temp (°C) Vel (m/s) Press (Pa)...
2022-03-01 00:00:00+00 28 118 1018 ...
2022-03-02 00:00:00+00 27 117 1019 ...
2022-03-03 00:00:00+00 28 119 1018 ...
..
..

  2. <details>
  3. <summary>英文:</summary>
  5. I have various .csv files. Each file has multiple columns. I am using the given code in R to pursue a quality check that for a particular column, how many rows have valid values and how many are null. The code works well for a single csv file. But I want to run the code for all the csv files and need output for each csv file. Additionally, I want a log file. Could anyone please help me by modifying the code how it can be used to process various csv files.
  7.     install.packages(&quot;readr&quot;) 
  8.     library(readr)
  10.     check_column &lt;- function(df, column) {
  11.       valid_values &lt;- !is.na(df[[column]])
  12.       num_valid &lt;- sum(valid_values)
  13.       num_null &lt;- nrow(df) - num_valid
  14.       return(c(num_valid, num_null))
  15.     }
  17.     #Read the CSV file
  18.     df &lt;- read_csv(&quot;data.csv&quot;)
  19.     
  20.     for (column in names(df)) {
  21.       results &lt;- check_column(df, column)
  22.       print(paste(column, &quot;: &quot;, results[1], &quot; valid, &quot;, results[2], &quot; null&quot;))
  23.     }
  25. Sample data: (Not all files have same number of columns)
  27. Csv1.csv
  29.     D_T  Temp (&#176;C)  Press (Pa)  ...
  30.     2021-03-01 00:00:00+00  28  1018  ...
  31.     2021-03-02 00:00:00+00  27  1017  ...
  32.     2021-03-03 00:00:00+00  28  1019  ...
  33.     ..
  34.     .. 
  36. Csv2.csv
  38.     D_T  Temp (&#176;C)  Vel (m/s)  Press (Pa_...
  39.     2022-03-01 00:00:00+00  28  118  1018  ...
  40.     2022-03-02 00:00:00+00  27  117  1019  ...
  41.     2022-03-03 00:00:00+00  28  119  1018  ...
  42.     ..
  43.     .. 
  45. </details>
  48. # 答案1
  49. **得分**: 1
  51. 以下是翻译好的代码部分：
  53. ```R
  54. 如何像这样做呢？这将不会在一个变量中存储任何内容。如果您需要帮助，请告诉我。
  56. library(readr)
  58. for(files in list.files(pattern=".*csv$")) {
  59.     file <- read_csv(files)
  60.     out <- file(paste0(files, ".log"), open = "w")
  61.     sapply(colnames(file), function(x) {
  62.             cat(
  63.                     paste0(x, ":"),
  64.                     sum(!is.na(file[, x])),
  65.                     "valid,",
  66.                     sum(is.na(file[, x])),
  67.                     "null\n",
  68.                     file = out
  69.             )
  70.     })
  71.     close(out)
  72. }
  74. 要写入一个文件中：
  76. library(readr)
  78. out <- file("output.log", open = "w")
  79. for(files in list.files(pattern=".*csv$")) {
  80.         file <- read_csv(files)
  81.         cat(files, "\n", file = out)
  82.         sapply(colnames(file), function(x) {
  83.                 cat(
  84.                         paste0(x, ":"),
  85.                         sum(!is.na(file[, x])),
  86.                         "valid,",
  87.                         sum(is.na(file[, x])),
  88.                         "null\n",
  89.                         file = out
  90.                 )
  91.         })   
  92. }
  93. close(out)

希望这对您有帮助。

英文:

How about something like this? This will not store anything in a variable. Let me know if you need help with it.

  1. library(readr)    
  2.     
  3. for(files in list.files(pattern=&quot;.*csv$&quot;)) {
  4.     file &lt;- read_csv(files)
  5.     out &lt;- file(paste0(files, &quot;.log&quot;), open = &quot;w&quot;)
  6.     sapply(colnames(file), function(x) {
  7.             cat(
  8.                     paste0(x, &quot;:&quot;),
  9.                     sum(!is.na(file[, x])),
  10.                     &quot;valid,&quot;,
  11.                     sum(is.na(file[, x])),
  12.                     &quot;null\n&quot;,
  13.                     file = out
  14.             )
  15.     })
  16.     close(out)
  17. }

To write into one file only:

  1. library(readr)    
  3. out &lt;- file(&quot;output.log&quot;, open = &quot;w&quot;)
  4. for(files in list.files(pattern=&quot;.*csv$&quot;)) {
  5.         file &lt;- read_csv(files)
  6.         cat(files, &quot;\n&quot;, file = out)
  7.         sapply(colnames(file), function(x) {
  8.                 cat(
  9.                         paste0(x, &quot;:&quot;),
  10.                         sum(!is.na(file[, x])),
  11.                         &quot;valid,&quot;,
  12.                         sum(is.na(file[, x])),
  13.                         &quot;null\n&quot;,
  14.                         file = out
  15.                 )
  16.         })   
  17. }
  18. close(out)

huangapple
  • 本文由 发表于 2023年7月7日 06:21:47
  • 转载请务必保留本文链接：https://go.coder-hub.com/76632850.html
  • csv
  • data-quality
  • r
  • readr
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定