Detect missing values in polars based on conditions

I am currently identifying rows with missing values and filtering them into a report. I am adding an index and a "warning" column.

report = (df.with_row_count("index")
                .filter(pl.any(pl.col("*").is_null()) & pl.col("mandatory").eq("M"))
                .with_columns(pl.lit("Missing value detected").alias("warning"))
         )

The current requirement is that if the value "M" occurs in column "mandatory" and a missing value occurs in the same row, this row is added to the report. Now there is a new additional requirement. I will explain it on the basis of the example dataset. It can be that in the column "column" can be the same values. In this case "foo". According to the first rule, row 2 should be written into the report. However, an exception condition should now prevent this. If already one row has no zeros in the column "column", all other equal values are not considered in the further consideration. In this case, row 1 fulfills this condition, so row two should not be considered.

df = pl.DataFrame(
    {
        "ID": ["1", "1", "1", "1", "1"],
        "column": ["foo", "foo", "bar", "ham", "egg"],
        "table": ["A", "B", "C", "D", "E"],
        "value_a": ["tree", None, None, "bean", None,],
        "value_b": ["Lorem", "Ipsum", "Dal", "Curry", "Dish",],
        "mandatory": ["M", "M", "M", "CM", "M"],
    }
)
print(df)

shape: (5, 6)
┌─────┬────────┬───────┬─────────┬─────────┬───────────┐
│ ID  ┆ column ┆ table ┆ value_a ┆ value_b ┆ mandatory │
│ --- ┆ ---    ┆ ---   ┆ ---     ┆ ---     ┆ ---       │
│ str ┆ str    ┆ str   ┆ str     ┆ str     ┆ str       │
╞═════╪════════╪═══════╪═════════╪═════════╪═══════════╡
│ 1   ┆ foo    ┆ A     ┆ tree    ┆ Lorem   ┆ M         │
│ 1   ┆ foo    ┆ B     ┆ null    ┆ Ipsum   ┆ M         │
│ 1   ┆ bar    ┆ C     ┆ null    ┆ Dal     ┆ M         │
│ 1   ┆ ham    ┆ D     ┆ bean    ┆ Curry   ┆ CM        │
│ 1   ┆ egg    ┆ E     ┆ null    ┆ Dish    ┆ M         │
└─────┴────────┴───────┴─────────┴─────────┴───────────┘

The expected report should look like this

shape: (2, 8)
┌───────┬─────┬────────┬───────┬─────────┬─────────┬───────────┬────────────────────────┐
│ Index ┆ ID  ┆ column ┆ table ┆ value_a ┆ value_b ┆ mandatory ┆ warning                │
│ ---   ┆ --- ┆ ---    ┆ ---   ┆ ---     ┆ ---     ┆ ---       ┆ ---                    │
│ i64   ┆ str ┆ str    ┆ str   ┆ f64     ┆ str     ┆ str       ┆ str                    │
╞═══════╪═════╪════════╪═══════╪═════════╪═════════╪═══════════╪════════════════════════╡
│ 1     ┆ 1   ┆ bar    ┆ C     ┆ null    ┆ Dal     ┆ M         ┆ Missing value detected │
│ 2     ┆ 1   ┆ egg    ┆ E     ┆ null    ┆ Dish    ┆ M         ┆ Missing value detected │
└───────┴─────┴────────┴───────┴─────────┴─────────┴───────────┴────────────────────────┘

How can I implement the additional condition to identfy the rows with missing values correctly?

Here's one way to define the additional rule which could be combined with the others:

df.with_columns(~pl.all(pl.col("*").is_not_null()).any().over("column"))

shape: (5, 7)
┌─────┬────────┬───────┬─────────┬─────────┬───────────┬───────┐
│ ID  ┆ column ┆ table ┆ value_a ┆ value_b ┆ mandatory ┆ all   │
│ --- ┆ ---    ┆ ---   ┆ ---     ┆ ---     ┆ ---       ┆ ---   │
│ str ┆ str    ┆ str   ┆ str     ┆ str     ┆ str       ┆ bool  │
╞═════╪════════╪═══════╪═════════╪═════════╪═══════════╪═══════╡
│ 1   ┆ foo    ┆ A     ┆ tree    ┆ Lorem   ┆ M         ┆ false │
│ 1   ┆ foo    ┆ B     ┆ null    ┆ Ipsum   ┆ M         ┆ false │
│ 1   ┆ bar    ┆ C     ┆ null    ┆ Dal     ┆ M         ┆ true  │
│ 1   ┆ ham    ┆ D     ┆ bean    ┆ Curry   ┆ CM        ┆ false │
│ 1   ┆ egg    ┆ E     ┆ null    ┆ Dish    ┆ M         ┆ true  │
└─────┴────────┴───────┴─────────┴─────────┴───────────┴───────┘

I suggest you to create sub-table that will check your condition and then you can join it with main df.

df.join(
    other = df.groupby("column").agg(
        null_cond = pl.any(pl.all().is_null()).all()
    ),
    on="column", how="left"
).filter((pl.col("mandatory") == "M") & pl.col("null_cond"))

And I forgot about window functions! @jqurious has suggested much better & elegant solution:

df.filter(
    (pl.col("mandatory") == "M") & 
    pl.any(pl.all().is_null()).all().over("column")
)

┌─────┬────────┬───────┬─────────┬─────────┬───────────┐
│ ID  ┆ column ┆ table ┆ value_a ┆ value_b ┆ mandatory │
│ --- ┆ ---    ┆ ---   ┆ ---     ┆ ---     ┆ ---       │
│ str ┆ str    ┆ str   ┆ str     ┆ str     ┆ str       │
╞═════╪════════╪═══════╪═════════╪═════════╪═══════════╡
│ 1   ┆ bar    ┆ C     ┆ null    ┆ Dal     ┆ M         │
│ 1   ┆ egg    ┆ E     ┆ null    ┆ Dish    ┆ M         │
└─────┴────────┴───────┴─────────┴─────────┴───────────┘