Axial inconsistency of pandas.diff

Consider the dataframe:

df = pd.DataFrame({'col': [True, False]})

The following code works:

df['col'].diff()

The result is:

0     NaN
1    True
Name: col, dtype: object

However, the code:

df.T.diff(axis=1)

gives the error:

numpy boolean subtract, the `-` operator, is not supported, use the bitwise_xor, the `^` operator, or the logical_xor function instead.

Is that a bug?

It seems like this behaviour is intentional as per GH15856. Arithmetic operations (+, -, *, /, etc.) between boolean arrays in NumPy are not (or not anymore?) supported.

With diff on axis=1, pandas tries to compute the difference between consecutive elements along the columns axis (which happens to hold booleans here because of the transposition) and since NumPy is run under the hood to compute that, a TypeError is raised.

print(df.T)

        0      1
col  True  False

np.array(False) - np.array(True)

TypeError: numpy boolean subtract, the `-` operator, is not supported, use the bitwise_xor, the `^` operator, or the logical_xor function instead.

This can be counterintuitive, because the same operation when using Python boolean, succeeds :

False - True
# return -1

But @seberg explains why :

> This is a pretty old deprecation, although I do seem to remember some
> discussion about only deprecating the unary operator -False not True -
> True. Note that Python booleans are different from NumPy ones, they
> are practically integers. NumPy booleans behave much less like
> integers, if you add two booleans you get a boolean again, etc.

The behavior you're seeing would appear to be at odds with the docs which clearly state:

> For boolean dtypes, this uses operator.xor() rather than operator.sub(). The result is calculated according to current dtype in DataFrame, however dtype of the result is always float64.

Also interesting is the following test:

df = pd.DataFrame({'col': [True, False], 'col2': [True, False]})

print("","df:",sep='\n')
print(df,df.dtypes,sep='\n')

print("","diff of df:",sep='\n')
res = df.diff()
print(res,res.dtypes,sep='\n')

print("","diff of df['col']:",sep='\n')
res = df['col'].diff()
print(res,res.dtypes,sep='\n')

print("","df.T:",sep='\n')
res = df.T
print(res,res.dtypes,sep='\n')

print("","diff(axis=0) of df.T:",sep='\n')
res = df.T.diff(axis=0)
print(res,res.dtypes,sep='\n')

print("","df.T.astype(object):",sep='\n')
res = df.T.astype(object)
print(res,res.dtypes,sep='\n')

print("","diff(axis=1) of df.T.astype(object):",sep='\n')
res = df.T.astype(object).diff(axis=1)
print(res,res.dtypes,sep='\n')

try:
    print("","diff(axis=1) of df.T:",sep='\n')
    res = df.T.diff(axis=1)
    print(res,res.dtypes,sep='\n')
except TypeError:
    print('got TypeError')

Output:

df:
     col   col2
0   True   True
1  False  False
col     bool
col2    bool
dtype: object

diff of df:
    col  col2
0   NaN   NaN
1  True  True
col     object
col2    object
dtype: object

diff of df['col']:
0     NaN
1    True
Name: col, dtype: object
object

df.T:
         0      1
col   True  False
col2  True  False
0    bool
1    bool
dtype: object

diff(axis=0) of df.T:
          0      1
col     NaN    NaN
col2  False  False
0    object
1    object
dtype: object

df.T.astype(object):
         0      1
col   True  False
col2  True  False
0    object
1    object
dtype: object

diff(axis=1) of df.T.astype(object):
        0   1
col   NaN  -1
col2  NaN  -1
0    object
1    object
dtype: object

diff(axis=1) of df.T:
got TypeError

If we change the column types to object using astype() before the call to diff(axis=1), no error is raised and the result appears to cast the boolean values to int prior to performing the diff using integer subtraction.

However, as OP points out, this same operation without astype(object) raises the TypeError TypeError: numpy boolean subtract, the -operator, is not supported, use the bitwise_xor, the^ operator, or the logical_xor function instead., despite the claim in the diff() docs that For boolean dtypes, this uses operator.xor() rather than operator.sub().