使用pyarrow字符串与pandas的map或apply函数。

huangapple go评论63阅读模式
英文:

Using pyarrow strings with pandas map or apply

问题

I trying to create a new DataFrame column based on a transformation from another.
我试图基于另一个DataFrame列的转换来创建一个新的列。

I'd like to have the return dtype as pyarrow[string] and not object.
我希望返回的数据类型是pyarrow[string]而不是object

But this create an intermediate series with object dtype.
但是这会创建一个中间的object数据类型的Series。

Is there a way to directly create a pd.Series with string[pyarrow] type from s?
是否有一种方法可以直接从s创建一个带有string[pyarrow]类型的pd.Series?

英文:

I trying to create a new DataFrame column based on a transformation from another.
I'd like to have the return dtype as pyarrow[string] and not object.

I can do

In [2]: s = pd.Series([1, 2, 1, 2])
In [3]: names = {1: 'Creative', 2: 'VeriFone'}
In [4]: s.map(names)
Out[4]: 
0    Creative
1    VeriFone
2    Creative
3    VeriFone
dtype: object
In [5]: s.map(names).astype('string[pyarrow]')
Out[5]: 
0    Creative
1    VeriFone
2    Creative
3    VeriFone
dtype: string

But this create an intermediate series with object dtype.

I also tried:

In [6]: arr = pa.array(['', 'Creative', 'VeriFone'])
In [7]: s.apply(arr.__getitem__)
Out[7]: 
0    Creative
1    VeriFone
2    Creative
3    VeriFone
dtype: object

But the return dtype is object.

Is there a way to directly create a pd.Series with string[pyarrow] type from s?

答案1

得分: 0

当调用pd.Series.mappd.Series.apply时,Python代码会被执行,因此无论如何都会有一个中间的Python对象。

如果您正在寻求性能,最好使用矢量化操作。例如,您可以使用join:

s = pd.Series([1, 2, 1, 2])
names = pd.Series({1: 'Creative', 2: 'VeriFone'}, dtype='string[pyarrow]', name='name')
s.to_frame().join(names)['name']
英文:

When calling pd.Series.map or pd.Series.apply python code gets executed so there will be an intermediate python object no matter what.

If you're looking for performances it's best to use vectorized operations. For example you can use join:

s = pd.Series([1, 2, 1, 2])
names = pd.Series({1: 'Creative', 2: 'VeriFone'}, dtype='string[pyarrow]', name='name')
s.to_frame().join(names)['name']

huangapple
  • 本文由 发表于 2023年5月31日 22:50:28
  • 转载请务必保留本文链接:https://go.coder-hub.com/76374776.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定