QST: What is the canonical way to convert a column of type string[pyarrow] to boolean within a pandas dataframe?

I'm wanting to convert string data which is indeed boolean (or null) e.g. values are y / n / NA, or true / false / NA, or even a mix of these.

When using pandas with numpy backend as default, conversion from string data to boolean works smoothly:

import pandas as pd

df = pd.DataFrame({"col1": ["true", None, "false"]})
assert df.dtypes["col1"] == "object", df.dtypes["col1"]
# convert to boolean
df["col1"] = df["col1"].replace({'true': True, 'false': False}).astype(bool)
assert df.dtypes["col1"] == bool, df.dtypes["col1"]

However, when using the pyarrow backend (in my use case, I was actually using pd.read_parquet with dtype_backend - but I set the type explicitly in the example below):

df_pyarrow = pd.DataFrame(
    {"col1": ["true", None, "false"]}, dtype="string[pyarrow]"
)
assert df_pyarrow.dtypes["col1"] == "string", df_pyarrow.dtypes["col1"]
df_pyarrow["col1"] = (
    df_pyarrow["col1"]
    .replace({'true': True, 'false': False})  # fails at this step
    .astype(bool)
)
assert df_pyarrow.dtypes["col1"] == "bool[pyarrow]", df_pyarrow.dtypes["col1"]

but this fails at the .replace() because pyarrow complains, rightly!, that True and False are not valid values for a string[pyarrow]: TypeError: Scalar must be NA or str.

I have found that this method works:

df_pyarrow["col1"] = df_pyarrow["col1"] == "true"

assert df_pyarrow.dtypes["col1"] == "bool[pyarrow]", df_pyarrow.dtypes["col1"]

However:

• df_pyarrow.info() still says col1 is string[pyarrow]
• this method isn't as flexible: what if there were multiple values for True/False

What is the canonical way to convert a column of type string[pyarrow] to boolean within a pandas dataframe?

You should use map instead of replace

df_pyarrow["col1"] = (
    df_pyarrow["col1"]
    .map({'true': True, 'false': False})
    .astype("bool[pyarrow]")
)

It works with numpy because an array of string in numpy is actually and array of object. So you can replace strings with booleans, they are still objects.

With pandas and the pyarrow backend, it is more strict about type.

If I use:

df = (pd.read_parquet('data.parquet', dtype_backend='pyarrow')
        .astype({'col1': 'bool[pyarrow]'}))

I get:

>>> df
    col1
0   True
1   <NA>
2  False

>>> df.dtypes
col1    bool[pyarrow]
dtype: object

>>> df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3 entries, 0 to 2
Data columns (total 1 columns):
 #   Column  Non-Null Count  Dtype        
---  ------  --------------  -----        
 0   col1    2 non-null      bool[pyarrow]
dtypes: bool[pyarrow](1)
memory usage: 130.0 bytes

Minimal Reproducible Example:

df = pd.DataFrame({'col1': ['true', None, 'false']})
df.to_parquet('data.parquet', engine='pyarrow')