Pandas/Dask read_parquet columns case insensitive

Can i have a columns argument on pd.read_parquet() that filters columns, but is case insensitive, I have files with the same columns, but some are camel case, some are all capital, some are lowercase, it is a mess, and i can't read all columns and filter afterwards, and sometimes I have to read directly to pandas.

I know read_csv has a usecols argument that can be callable, so when the files are csvs I can do this: pd.read_csv(filepath, usecols=lambda col: col.lower() in cols)

But read_parquet columns argument can't be callable, how can I do something similar?

This is a workaround only, but what one can do is use dask to lazy-load the parquet, inspect the column list, pick the ones of interest and do the actual load (or continue in the lazy fashion).

Here's the rough pseudocode:

from dask.dataframe import read_parquet

ddf = read_parquet("some_parquet")

# select columns
cols_of_interest = [c for c in ddf.columns if c.lower() in cols]

# continue with the dask.dataframe
ddf = read_parquet("some_parquet", columns= cols_of_interest)

# or convert to pandas, if necessary
df = ddf.compute()

You can use pyarrow:

import pyarrow.parquet as pq

metadata = pq.read_metadata('data.parquet')

cols = ['col1', 'col3']
cols = [c for c in metadata.schema.names if c.lower() in cols]

df = pd.read_parquet('data.parquet', columns=cols)
df.columns = df.columns.str.lower()

Output:

>>> metadata.schema.names
['COL1', 'col2', 'Col3']

>>> df
        col1      col3
0   9.451444  8.799611
1   3.805668  9.194838
2   1.643645  5.300303
3   4.782400  0.301559
4   8.264088  9.652009
..       ...       ...
95  0.248484  2.904245
96  3.572653  6.826785
97  3.063543  8.223073
98  2.060533  9.996808
99  5.724856  3.476133

[100 rows x 2 columns]