Offset when filtering timezone aware datetimes in python polars

I have a Dataframe with timezone aware datetimes in the column "datetime". The original timezone is UTC.

from zoneinfo import ZoneInfo
(
    pl.DataFrame(
        {
            "datetime": [
                "[01/Aug/2023:00:00:02 +0200]",
                "[01/Aug/2023:02:00:02 +0200]",
                "[03/Aug/2023:01:00:02 +0200]",
            ]
        }
    )
    .with_columns(pl.col("datetime").str.to_datetime("[%d/%b/%Y:%H:%M:%S %z]"))
    .with_columns(pl.col("datetime").dt.convert_time_zone("Europe/Berlin"))
    .filter(
        pl.col("datetime")
        .cast(pl.Date)
        .is_between(
            datetime(2023, 8, 1, tzinfo=ZoneInfo("Europe/Berlin")),
            datetime(2023, 8, 3, tzinfo=ZoneInfo("Europe/Berlin")),
        )
    )
)

The conversion into CEST ("Europe/Berlin") works. However, when I filter for datetime there is a 2 hour offset.

datetime
datetime[μs, Europe/Berlin]
2023-08-01 02:00:02 CEST
2023-08-03 01:00:02 CEST

The first row from the original dataset is not in the list but it should. The third row is in the list but it should not.

This looks like the difference between UTC and CEST. The result is the same if the python datetime object is naive (e. g. just datetime(2023,8,1)).

How do I get polars to take the timezone into account when filtering.

Since you correctly localize your datetimes to a time zone, your filter should reflect that, i.e. also use the aware datetime, without casting back to a naive date with .cast(pl.Date):

from zoneinfo import ZoneInfo
import polars as pl

df = (
    pl.DataFrame(
        {
            "datetime": [
                "[01/Aug/2023:00:00:02 +0200]",
                "[01/Aug/2023:02:00:02 +0200]",
                "[03/Aug/2023:01:00:02 +0200]",
            ]
        }
    )
    .with_columns(pl.col("datetime").str.to_datetime("[%d/%b/%Y:%H:%M:%S %z]"))
    .with_columns(pl.col("datetime").dt.convert_time_zone("Europe/Berlin"))
)

print(
    df.filter(
        pl.col("datetime")
        .is_between(
            datetime(2023, 8, 1, tzinfo=ZoneInfo("Europe/Berlin")),
            datetime(2023, 8, 3, tzinfo=ZoneInfo("Europe/Berlin")),
        )
    )
)
┌─────────────────────────────┐
│ datetime                    │
│ ---                         │
│ datetime[μs, Europe/Berlin] │
╞═════════════════════════════╡
│ 2023-08-01 00:00:02 CEST    │
│ 2023-08-01 02:00:02 CEST    │
└─────────────────────────────┘

To observe the effect of casting back to pl.Date (or pl.Datetime, for better illustration), run for example

df = (
    pl.DataFrame(
        {
            "datetime": [
                "[01/Aug/2023:00:00:02 +0200]",
                "[01/Aug/2023:02:00:02 +0200]",
                "[03/Aug/2023:01:00:02 +0200]",
            ]
        }
    )
    .with_columns(pl.col("datetime").str.to_datetime("[%d/%b/%Y:%H:%M:%S %z]"))
    .with_columns(pl.col("datetime").dt.convert_time_zone("Europe/Berlin"))
    .with_columns(pl.col("datetime").cast(pl.Datetime))
)

print(df["datetime"])
[
	2023-07-31 22:00:02
	2023-08-01 00:00:02
	2023-08-02 23:00:02
]

Naive datetime in polars resembles UTC.