“pandas.diff”的”axial inconsistency”

huangapple
go评论108阅读模式
英文:

Axial inconsistency of pandas.diff

问题

The following code works:

这段代码可以正常工作：

  1. df['col'].diff()

The result is:

结果如下：

  1. 0     NaN
  2. 1    True
  3. Name: col, dtype: object

However, the code:

然而，下面的代码：

  1. df.T.diff(axis=1)

gives the error:

会产生错误：

  1. numpy boolean subtract, the `-` operator, is not supported, use the bitwise_xor, the `^` operator, or the logical_xor function instead.

Is that a bug?

这是一个错误吗？

英文:

Consider the dataframe:

  1. df = pd.DataFrame({'col': [True, False]})

The following code works:

  1. df['col'].diff()

The result is:

  1. 0     NaN
  2. 1    True
  3. Name: col, dtype: object

However, the code:

  1. df.T.diff(axis=1)

gives the error:

  1. numpy boolean subtract, the `-` operator, is not supported, use the bitwise_xor, the `^` operator, or the logical_xor function instead.

Is that a bug?

答案1

得分: 1

这似乎是有意为之，根据 GH15856。在NumPy中，布尔数组之间的算术运算 (+, -, *, /, 等) 不再被支持。

在 axis=1 上使用 diff 时，pandas 试图计算沿着列轴的连续元素之间的差异（因为在这里由于转置而包含布尔值），由于底层运行了NumPy来计算，因此会引发 TypeError。

  1. print(df.T)
  3.         0      1
  4. col  True  False
  6. np.array(False) - np.array(True)
  8. TypeError: numpy boolean subtract, the `-` operator, is not supported, use the bitwise_xor, the `^` operator, or the logical_xor function instead.

这可能令人困惑，因为使用Python布尔值进行相同操作成功：

  1. False - True
  2. # 返回 -1

但是 @seberg 解释了原因：

> 这是一个相当古老的弃用，虽然我似乎记得有一些关于只弃用 -False 而不是 True - True 的讨论。请注意，Python布尔值与NumPy布尔值不同，它们在实际上是整数。NumPy布尔值的行为更不像整数，如果将两个布尔值相加，你会再次得到一个布尔值，等等。

英文:

It seems like this behaviour is intentional as per GH15856. Arithmetic operations (+, -, *, /, etc.) between boolean arrays in NumPy are not (or not anymore?) supported.

With diff on axis=1, pandas tries to compute the difference between consecutive elements along the columns axis (which happens to hold booleans here because of the transposition) and since NumPy is run under the hood to compute that, a TypeError is raised.

  1. print(df.T)
  3.         0      1
  4. col  True  False
  6. np.array(False) - np.array(True)
  8. TypeError: numpy boolean subtract, the `-` operator, is not supported, use the bitwise_xor, the `^` operator, or the logical_xor function instead.

This can be counterintuitive, because the same operation when using Python boolean, succeeds :

  1. False - True
  2. # return -1

But @seberg explains why :

> This is a pretty old deprecation, although I do seem to remember some
> discussion about only deprecating the unary operator -False not True -
> True. Note that Python booleans are different from NumPy ones, they
> are practically integers. NumPy booleans behave much less like
> integers, if you add two booleans you get a boolean again, etc.

答案2

得分: 1

以下是您要翻译的内容：

"你正在看到的行为似乎与文档中明确说明的不符：

对于布尔数据类型，这里使用的是 operator.xor() 而不是 operator.sub()。结果根据 DataFrame 中的当前数据类型计算，但结果的数据类型始终为 float64。

还有一个有趣的测试：

  1. df = pd.DataFrame({'col': [True, False], 'col2': [True, False]})
  3. print("", "df:", sep='\n')
  4. print(df, df.dtypes, sep='\n')
  6. print("", "df 的差异:", sep='\n')
  7. res = df.diff()
  8. print(res, res.dtypes, sep='\n')
  10. print("", "df['col'] 的差异:", sep='\n')
  11. res = df['col'].diff()
  12. print(res, res.dtypes, sep='\n')
  14. print("", "df.T:", sep='\n')
  15. res = df.T
  16. print(res, res.dtypes, sep='\n')
  18. print("", "df.T 的差异（axis=0）:", sep='\n')
  19. res = df.T.diff(axis=0)
  20. print(res, res.dtypes, sep='\n')
  22. print("", "df.T 转换为 object 数据类型:", sep='\n')
  23. res = df.T.astype(object)
  24. print(res, res.dtypes, sep='\n')
  26. print("", "df.T 转换为 object 数据类型后的差异（axis=1）:", sep='\n')
  27. res = df.T.astype(object).diff(axis=1)
  28. print(res, res.dtypes, sep='\n')
  30. try:
  31.     print("", "df.T 的差异（axis=1）:", sep='\n')
  32.     res = df.T.diff(axis=1)
  33.     print(res, res.dtypes, sep='\n')
  34. except TypeError:
  35.     print('得到 TypeError')

输出：

  1. df:
  2.      col   col2
  3. 0   True   True
  4. 1  False  False
  5. col     bool
  6. col2    bool
  7. dtype: object
  9. df 的差异:
  10.     col  col2
  11. 0   NaN   NaN
  12. 1  True  True
  13. col     object
  14. col2    object
  15. dtype: object
  17. df['col'] 的差异:
  18. 0     NaN
  19. 1    True
  20. Name: col, dtype: object
  21. object
  23. df.T:
  24.          0      1
  25. col   True  False
  26. col2  True  False
  27. 0    bool
  28. 1    bool
  29. dtype: object
  31. df.T 的差异（axis=0）:
  32.           0      1
  33. col     NaN    NaN
  34. col2  False  False
  35. 0    object
  36. 1    object
  37. dtype: object
  39. df.T 转换为 object 数据类型:
  40.          0      1
  41. col   True  False
  42. col2  True  False
  43. 0    object
  44. 1    object
  45. dtype: object
  47. df.T 转换为 object 数据类型后的差异（axis=1）:
  48.         0   1
  49. col   NaN  -1
  50. col2  NaN  -1
  51. 0    object
  52. 1    object
  53. dtype: object
  55. df.T 的差异（axis=1）:
  56. 得到 TypeError

如果我们在调用 diff(axis=1) 之前将列的数据类型更改为 object 类型，将不会引发错误，并且结果似乎会将布尔值转换为整数，然后执行整数减法。然而，正如 OP 指出的，没有 使用 astype(object) 进行的相同操作会引发 TypeError：“TypeError: numpy boolean subtract, the - operator, is not supported, use the bitwise_xor, the ^ operator, or the logical_xor function instead.”，尽管 diff() 文档中声称“对于布尔数据类型，这里使用的是 operator.xor() 而不是 operator.sub()”。"

英文:

The behavior you're seeing would appear to be at odds with the docs which clearly state:

> For boolean dtypes, this uses operator.xor() rather than operator.sub(). The result is calculated according to current dtype in DataFrame, however dtype of the result is always float64.

Also interesting is the following test:

  1. df = pd.DataFrame({'col': [True, False], 'col2': [True, False]})
  3. print("","df:",sep='\n')
  4. print(df,df.dtypes,sep='\n')
  6. print("","diff of df:",sep='\n')
  7. res = df.diff()
  8. print(res,res.dtypes,sep='\n')
  10. print("","diff of df['col']:",sep='\n')
  11. res = df['col'].diff()
  12. print(res,res.dtypes,sep='\n')
  14. print("","df.T:",sep='\n')
  15. res = df.T
  16. print(res,res.dtypes,sep='\n')
  18. print("","diff(axis=0) of df.T:",sep='\n')
  19. res = df.T.diff(axis=0)
  20. print(res,res.dtypes,sep='\n')
  22. print("","df.T.astype(object):",sep='\n')
  23. res = df.T.astype(object)
  24. print(res,res.dtypes,sep='\n')
  26. print("","diff(axis=1) of df.T.astype(object):",sep='\n')
  27. res = df.T.astype(object).diff(axis=1)
  28. print(res,res.dtypes,sep='\n')
  30. try:
  31.     print("","diff(axis=1) of df.T:",sep='\n')
  32.     res = df.T.diff(axis=1)
  33.     print(res,res.dtypes,sep='\n')
  34. except TypeError:
  35.     print('got TypeError')

Output:

  1. df:
  2.      col   col2
  3. 0   True   True
  4. 1  False  False
  5. col     bool
  6. col2    bool
  7. dtype: object
  9. diff of df:
  10.     col  col2
  11. 0   NaN   NaN
  12. 1  True  True
  13. col     object
  14. col2    object
  15. dtype: object
  17. diff of df['col']:
  18. 0     NaN
  19. 1    True
  20. Name: col, dtype: object
  21. object
  23. df.T:
  24.          0      1
  25. col   True  False
  26. col2  True  False
  27. 0    bool
  28. 1    bool
  29. dtype: object
  31. diff(axis=0) of df.T:
  32.           0      1
  33. col     NaN    NaN
  34. col2  False  False
  35. 0    object
  36. 1    object
  37. dtype: object
  39. df.T.astype(object):
  40.          0      1
  41. col   True  False
  42. col2  True  False
  43. 0    object
  44. 1    object
  45. dtype: object
  47. diff(axis=1) of df.T.astype(object):
  48.         0   1
  49. col   NaN  -1
  50. col2  NaN  -1
  51. 0    object
  52. 1    object
  53. dtype: object
  55. diff(axis=1) of df.T:
  56. got TypeError

If we change the column types to object using astype() before the call to diff(axis=1), no error is raised and the result appears to cast the boolean values to int prior to performing the diff using integer subtraction.

However, as OP points out, this same operation without astype(object) raises the TypeError TypeError: numpy boolean subtract, the -operator, is not supported, use the bitwise_xor, the^ operator, or the logical_xor function instead., despite the claim in the diff() docs that For boolean dtypes, this uses operator.xor() rather than operator.sub().

huangapple
  • 本文由 发表于 2023年4月11日 02:07:46
  • 转载请务必保留本文链接：https://go.coder-hub.com/75979562.html
  • pandas
  • python
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定