英文:
Detect rows where uniqueness is not given in polars
问题
I understand that you want to translate the provided code and text. Here is the translated content:
目前我有以下问题。我需要检查列值`ID`，`table`和`value_a`是否不唯一时是否存在侵权。
df = pl.DataFrame(
{
"ID": ["1", "1", "1", "1", "1"],
"column": ["foo", "foo", "bar", "ham", "egg"],
"table": ["A", "A", "C", "D", "E"],
"value_a": ["tree", tree, None, "bean", None,],
"value_b": ["Lorem", "Ipsum", "Dal", "Curry", "Dish",],
"mandatory": ["M", "M", "M", "CM", "M"]
}
)
print(df)
shape: (5, 6)
┌─────┬────────┬───────┬─────────┬─────────┬───────────┐
│ ID ┆ column ┆ table ┆ value_a ┆ value_b ┆ mandatory │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str ┆ str ┆ str ┆ str │
╞═════╪════════╪═══════╪═════════╪═════════╪═══════════╡
│ 1 ┆ foo ┆ A ┆ tree ┆ Lorem ┆ M │
│ 1 ┆ foo ┆ B ┆ tree ┆ Ipsum ┆ M │
│ 1 ┆ bar ┆ C ┆ null ┆ Dal ┆ M │
│ 1 ┆ ham ┆ D ┆ bean ┆ Curry ┆ M │
│ 1 ┆ egg ┆ E ┆ null ┆ Dish ┆ M │
└─────┴────────┴───────┴─────────┴─────────┴───────────┘
对于df，应创建侵权报告，其中包含以下专用输出：
shape: (2, 8)
┌───────┬─────┬────────┬───────┬─────────┬─────────┬───────────┬─────────────────────────┐
│ index ┆ ID ┆ column ┆ table ┆ value_a ┆ value_b ┆ mandatory ┆ warning │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ str ┆ str ┆ str ┆ str ┆ str ┆ str │
╞═══════╪═════╪════════╪═══════╪═════════╪═════════╪═══════════╪═════════════════════════╡
│ 0 ┆ 1 ┆ foo ┆ A ┆ tree ┆ Lorem ┆ M ┆ 行值不唯一 │
│ 1 ┆ 1 ┆ foo ┆ A ┆ tree ┆ Ipsum ┆ M ┆ 行值不唯一 │
└───────┴─────┴────────┴───────┴─────────┴─────────┴───────────┴─────────────────────────┘
报告应包含一个`index`和一个`warning`列。我使用以下代码行来识别行中是否有任何空值：
report = (df.with_row_count("index")
.filter(pl.any(pl.col("*").is_null()) & pl.col("mandatory").eq("M"))
.with_columns(pl.lit("检测到缺失值").alias("warning"))
)
如何调整此代码以便在一方面检测缺失值，另一方面识别不唯一的行？也许我可以创建两个报告，然后使用`.vstack()`将两个报告组合成最终报告。您会如何解决这个问题？
英文:
Currently I have the following problem. I have to check if there is an infringement if the column values ID, table and value_a are not unique.
df = pl.DataFrame(
{
"ID": ["1", "1", "1", "1", "1"],
"column": ["foo", "foo", "bar", "ham", "egg"],
"table": ["A", "A", "C", "D", "E"],
"value_a": ["tree", tree, None, "bean", None,],
"value_b": ["Lorem", "Ipsum", "Dal", "Curry", "Dish",],
"mandatory": ["M", "M", "M", "CM", "M"]
}
)
print(df)
shape: (5, 6)
┌─────┬────────┬───────┬─────────┬─────────┬───────────┐
│ ID ┆ column ┆ table ┆ value_a ┆ value_b ┆ mandatory │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str ┆ str ┆ str ┆ str │
╞═════╪════════╪═══════╪═════════╪═════════╪═══════════╡
│ 1 ┆ foo ┆ A ┆ tree ┆ Lorem ┆ M │
│ 1 ┆ foo ┆ B ┆ tree ┆ Ipsum ┆ M │
│ 1 ┆ bar ┆ C ┆ null ┆ Dal ┆ M │
│ 1 ┆ ham ┆ D ┆ bean ┆ Curry ┆ M │
│ 1 ┆ egg ┆ E ┆ null ┆ Dish ┆ M │
└─────┴────────┴───────┴─────────┴─────────┴───────────┘
In the case of df a infringement report should be created with the following dedicated output:
shape: (2, 8)
┌───────┬─────┬────────┬───────┬─────────┬─────────┬───────────┬─────────────────────────┐
│ index ┆ ID ┆ column ┆ table ┆ value_a ┆ value_b ┆ mandatory ┆ warning │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ str ┆ str ┆ str ┆ str ┆ str ┆ str │
╞═══════╪═════╪════════╪═══════╪═════════╪═════════╪═══════════╪═════════════════════════╡
│ 0 ┆ 1 ┆ foo ┆ A ┆ tree ┆ Lorem ┆ M ┆ Row value is not unique │
│ 1 ┆ 1 ┆ foo ┆ A ┆ tree ┆ Ipsum ┆ M ┆ Row value is not unique │
└───────┴─────┴────────┴───────┴─────────┴─────────┴───────────┴─────────────────────────┘
The report should contain an index and a warning column. I used this line of code to identify if there are any null values in a row:
report = (df.with_row_count("index")
.filter(pl.any(pl.col("*").is_null()) & pl.col("mandatory").eq("M"))
.with_columns(pl.lit("Missing value detected").alias("warning"))
)
How do I need to adapt this code so on the one hand I detect missing values and on the other hand I identify ununique rows. Maybe I create two reports and use .vstack() to combine both reports to a final one. How would you solve it?
答案1
得分: 3
你可以创建一个 struct 并使用 .is_duplicated
df.with_columns(
warning = pl.struct(["ID", "table", "value_a"]).is_duplicated()
)
shape: (5, 7)
┌─────┬────────┬───────┬─────────┬─────────┬───────────┬─────────┐
│ ID | column | table | value_a | value_b | mandatory | warning │
│ --- | --- | --- | --- | --- | --- | --- │
╞═════╪════════╪═══════╪═════════╪═════════╪═══════════╪═════════╡
│ 1 | foo | A | tree | Lorem | M | true │
│ 1 | foo | A | tree | Ipsum | M | true │
│ 1 | bar | C | null | Dal | M | false │
│ 1 | ham | D | bean | Curry | CM | false │
│ 1 | egg | E | null | Dish | M | false │
└─────┴────────┴───────┴─────────┴─────────┴───────────┴─────────┘
英文:
You can create a struct and use .is_duplicated
df.with_columns(
warning = pl.struct(["ID", "table", "value_a"]).is_duplicated()
)
shape: (5, 7)
┌─────┬────────┬───────┬─────────┬─────────┬───────────┬─────────┐
│ ID | column | table | value_a | value_b | mandatory | warning │
│ --- | --- | --- | --- | --- | --- | --- │
│ str | str | str | str | str | str | bool │
╞═════╪════════╪═══════╪═════════╪═════════╪═══════════╪═════════╡
│ 1 | foo | A | tree | Lorem | M | true │
│ 1 | foo | A | tree | Ipsum | M | true │
│ 1 | bar | C | null | Dal | M | false │
│ 1 | ham | D | bean | Curry | CM | false │
│ 1 | egg | E | null | Dish | M | false │
└─────┴────────┴───────┴─────────┴─────────┴───────────┴─────────┘
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论