英文:
How to filter DataFrame by any string column matching any regex pattern in a list?
问题
以下是翻译好的部分：
- String columns can be selected with:
df.select(pl.col(pl.Utf8))
- and a Dataframe's rows can be filtered with a regex pattern for a single column, with something like:
df.filter(pl.col("feature").str.contains("dangerous"))
- How can a DataFrame be filtered with a list of regex patterns that could appear in any string column? I.e., if any string in a row matches any regex pattern, then keep that entire row, discard the rest.
EDIT 1
Here's a generated df and patterns to test functionality and performance.
import randomfrom faker import Fakerimport polars as plrandom.seed(42)Faker.seed(42)faker = Faker()df_len = 10000df = pl.DataFrame([pl.Series("a", [random.randint(0, 511) for _ in range(df_len)]).cast(pl.Binary),pl.Series("b", [random.randint(0, 1) for _ in range(df_len)]).cast(pl.Boolean),pl.Series("c", faker.sentences(df_len), pl.Utf8),pl.Series("d", [random.randint(0, 255) for _ in range(df_len)], pl.UInt8),pl.Series("e", faker.words(df_len), pl.Utf8),pl.Series("f",[random.randint(0, 255) * random.TWOPI for _ in range(df_len)],pl.Float32,),pl.Series("g", faker.words(df_len), pl.Utf8),])patterns = [r"(?i)dangerous", r"always", r"(?i)prevent"]
print(df) yields:
shape: (10000, 7)┌───────────────┬───────┬─────────────────────────────────────┬─────┬───────────┬────────────┬──────────┐│ a ┆ b ┆ c ┆ d ┆ e ┆ f ┆ g ││ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ││ binary ┆ bool ┆ str ┆ u8 ┆ str ┆ f32 ┆ str │╞═══════════════╪═══════╪═════════════════════════════════════╪═════╪═══════════╪════════════╪══════════╡│ [binary data] ┆ false ┆ Agent every development say. ┆ 164 ┆ let ┆ 980.17688 ┆ yard ││ [binary data] ┆ true ┆ Beautiful instead ahead despite ... ┆ 210 ┆ reach ┆ 458.672516 ┆ son ││ [binary data] ┆ false ┆ Information last everything than... ┆ 230 ┆ arm ┆ 50.265484 ┆ standard ││ [binary data] ┆ false ┆ Choice whatever from behavior be... ┆ 29 ┆ operation ┆ 929.911438 ┆ final ││ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... ││ [binary data] ┆ true ┆ Building sign recently avoid upo... ┆ 132 ┆ practice ┆ 282.743347 ┆ big ││ [binary data] ┆ false ┆ Paper will board. ┆ 72 ┆ similar ┆ 376.991119 ┆ just ││ [binary data] ┆ true ┆ Technology money worker spring m... ┆ 140 ┆ sign ┆ 94.24778 ┆ audience ││ [binary data] ┆ false ┆ A third traditional ago. ┆ 40 ┆ available ┆ 615.752136 ┆ always │└───────────────┴───────┴─────────────────────────────────────┴─────┴───────────┴────────────┴──────────┘
EDIT 2
Using @jqurious's answer (the fastest so far), the correct output of df.filter(pl.any(pl.col(pl.Utf8).str.contains(regex))) is:
shape: (146, 7)┌───────────────┬───────┬─────────────────────────────────────┬─────┬───────────┬─────────────┬──────────┐│ a ┆ b ┆ c ┆ d ┆ e ┆ f ┆ g ││ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ││ binary ┆ bool ┆ str ┆ u8 ┆ str ┆ f32 ┆ str │╞═══════════════╪═══════╪═════════════════════════════════════╪═════╪═══════════╪═════════════╪══════════╡│ [binary data] ┆ true ┆ During prevent accept seem show ... ┆ 137 ┆ various ┆ 471.238892 ┆ customer ││ [binary data] ┆ true ┆ Ball always it focus economy bef... ┆ 179 ┆ key ┆ 471.238892 ┆ guy ││ [binary data] ┆ false ┆ Admit attack energy always. ┆ 175 ┆ purpose ┆ 1281.769775 ┆ wonder ││ [binary data] ┆ false ┆ Beyond prevent entire staff. ┆ 242 ┆ hair ┆ 904.778687 ┆ around ││ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... ││ [binary data] ┆ true ┆ Your sure piece simple always so... ┆ 247 ┆ recently ┆ 1055.575073 ┆ laugh ││ [binary data] ┆ false ┆ Difference all machine let charg... ┆ 178 ┆ former ┆ 1061.858276 ┆ always ││ [binary data] ┆ true ┆ Morning carry event tell prevent... ┆ 3 ┆ entire ┆ 1432.566284 ┆ hit │<details><summary>英文:</summary>String columns can be selected with:```lang-pythondf.select(pl.col(pl.Utf8))
and a Dataframe's rows can be filtered with a regex pattern for a single column, with something like:
df.filter(pl.col("feature").str.contains("dangerous"))
How can a DataFrame be filtered with a list of regex patterns that could appear in any string column? I.e., if any string in a row matches any regex pattern, then keep that entire row, discard the rest.
EDIT 1
Here's a generated df and patterns to test functionality and performance.
import randomfrom faker import Fakerimport polars as plrandom.seed(42)Faker.seed(42)faker = Faker()df_len = 10000df = pl.DataFrame([pl.Series("a", [random.randint(0, 511) for _ in range(df_len)]).cast(pl.Binary),pl.Series("b", [random.randint(0, 1) for _ in range(df_len)]).cast(pl.Boolean),pl.Series("c", faker.sentences(df_len), pl.Utf8),pl.Series("d", [random.randint(0, 255) for _ in range(df_len)], pl.UInt8),pl.Series("e", faker.words(df_len), pl.Utf8),pl.Series("f",[random.randint(0, 255) * random.TWOPI for _ in range(df_len)],pl.Float32,),pl.Series("g", faker.words(df_len), pl.Utf8),])patterns = [r"(?i)dangerous", r"always", r"(?i)prevent"]
print(df) yields:
shape: (10000, 7)┌───────────────┬───────┬─────────────────────────────────────┬─────┬───────────┬────────────┬──────────┐│ a ┆ b ┆ c ┆ d ┆ e ┆ f ┆ g ││ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ││ binary ┆ bool ┆ str ┆ u8 ┆ str ┆ f32 ┆ str │╞═══════════════╪═══════╪═════════════════════════════════════╪═════╪═══════════╪════════════╪══════════╡│ [binary data] ┆ false ┆ Agent every development say. ┆ 164 ┆ let ┆ 980.17688 ┆ yard ││ [binary data] ┆ true ┆ Beautiful instead ahead despite ... ┆ 210 ┆ reach ┆ 458.672516 ┆ son ││ [binary data] ┆ false ┆ Information last everything than... ┆ 230 ┆ arm ┆ 50.265484 ┆ standard ││ [binary data] ┆ false ┆ Choice whatever from behavior be... ┆ 29 ┆ operation ┆ 929.911438 ┆ final ││ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... ││ [binary data] ┆ true ┆ Building sign recently avoid upo... ┆ 132 ┆ practice ┆ 282.743347 ┆ big ││ [binary data] ┆ false ┆ Paper will board. ┆ 72 ┆ similar ┆ 376.991119 ┆ just ││ [binary data] ┆ true ┆ Technology money worker spring m... ┆ 140 ┆ sign ┆ 94.24778 ┆ audience ││ [binary data] ┆ false ┆ A third traditional ago. ┆ 40 ┆ available ┆ 615.752136 ┆ always │└───────────────┴───────┴─────────────────────────────────────┴─────┴───────────┴────────────┴──────────┘
EDIT 2
Using @jqurious's answer (the fastest so far), the correct output of df.filter(pl.any(pl.col(pl.Utf8).str.contains(regex))) is:
shape: (146, 7)┌───────────────┬───────┬─────────────────────────────────────┬─────┬───────────┬─────────────┬──────────┐│ a ┆ b ┆ c ┆ d ┆ e ┆ f ┆ g ││ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ││ binary ┆ bool ┆ str ┆ u8 ┆ str ┆ f32 ┆ str │╞═══════════════╪═══════╪═════════════════════════════════════╪═════╪═══════════╪═════════════╪══════════╡│ [binary data] ┆ true ┆ During prevent accept seem show ... ┆ 137 ┆ various ┆ 471.238892 ┆ customer ││ [binary data] ┆ true ┆ Ball always it focus economy bef... ┆ 179 ┆ key ┆ 471.238892 ┆ guy ││ [binary data] ┆ false ┆ Admit attack energy always. ┆ 175 ┆ purpose ┆ 1281.769775 ┆ wonder ││ [binary data] ┆ false ┆ Beyond prevent entire staff. ┆ 242 ┆ hair ┆ 904.778687 ┆ around ││ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... ││ [binary data] ┆ true ┆ Your sure piece simple always so... ┆ 247 ┆ recently ┆ 1055.575073 ┆ laugh ││ [binary data] ┆ false ┆ Difference all machine let charg... ┆ 178 ┆ former ┆ 1061.858276 ┆ always ││ [binary data] ┆ true ┆ Morning carry event tell prevent... ┆ 3 ┆ entire ┆ 1432.566284 ┆ hit ││ [binary data] ┆ false ┆ A third traditional ago. ┆ 40 ┆ available ┆ 615.752136 ┆ always │└───────────────┴───────┴─────────────────────────────────────┴─────┴───────────┴─────────────┴──────────┘
答案1
得分: 3
可以将列表转换为单个正则表达式。
df.filter(pl.any(pl.col(pl.Utf8).str.contains(regex)))
英文:
You can turn the list into a single regex.
regex = "|".join(f"(?:{pattern})" for pattern insorted(patterns, key=len, reverse=True))
df.filter(pl.any(pl.col(pl.Utf8).str.contains(regex)))
答案2
得分: 1
为了检查每个字符串列的正则表达式模式，您可以使用.fold()方法：
df = pl.DataFrame({"a": ["foo", "fo", "foa"],"b": ["foa", "fo", "foo"]})df.filter(pl.fold(acc=pl.lit(False),f=lambda acc, col: acc | col.str.contains("foo"),exprs=pl.col(pl.Utf8)))
另一种方法是将所有字符串列连接成一个单一列，然后应用正则表达式：
df.filter(pl.concat_str(pl.col(pl.Utf8)).str.contains("foo"))
英文:
To check regex patterns for each string column, you can use .fold() method
df = pl.DataFrame({"a": ["foo", "fo", "foa"],"b": ["foa", "fo", "foo"]})df.filter(pl.fold(acc=pl.lit(False),f=lambda acc, col: acc | col.str.contains("foo"),exprs=pl.col(pl.Utf8)))
┌─────┬─────┐│ a ┆ b ││ --- ┆ --- ││ str ┆ str │╞═════╪═════╡│ foo ┆ foa ││ foa ┆ foo │└─────┴─────┘
Another way to do it - concat all string columns into a single one and then apply regex expressions:
df.filter(pl.concat_str(pl.col(pl.Utf8)).str.contains("foo"))
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论