transform with groupby: want to print each column name in each group in the function feed to transform, but something went strange

transform with groupby: want to print each column name in each group in the function feed to transform, but something went strange, because there is an extra output with '600010.SH', which is the first group key.

my data here:test_data.csv

my code is:

import pandas as pd

i = 1
def MAD_single(grp):
    global i
    print(i, grp.name)
    print(grp.head(10))
    i += 1
    # computing here...

df = pd.read_csv('data_input/test_data.csv')
temp = df.groupby('code')[['high', 'low']].transform(MAD_single)

the output is:

1 high
0    1.99
1    1.94
Name: high, dtype: float64
2 low
0    1.92
1    1.91
Name: low, dtype: float64
3 600010.SH
   high   low
0  1.99  1.92
1  1.94  1.91
4 high
2    4.41
3    4.38
Name: high, dtype: float64
5 low
2    4.34
3    4.35
Name: low, dtype: float64

I am wondering:

1. why the following output will be printed?
2. why the second group key is not printed as 600010.SH?
   Thanks!

3 600010.SH
   high   low
0  1.99  1.92
1  1.94  1.91

This is a great question but to understand the result you get, you need to understand how Pandas is going to handle the transformation.

Here, I will only deal with the case where you transform:

1. a DataFrame (not a Series)
2. with a custom user function (not a string, like 'mean' or 'sum')
3. with the default engine (not numba or cython)

The implementation of this case can be found in the _transform_general method. All other cases are almost the same but not quite.

When trying to transform a DataFrame, there are 2 main steps:

1. Apply your function on the first group
2. Apply your function on other groups

During the first step, Pandas try to find the best path to apply the transformation.

Pandas defines two lambda functions in _define_paths method

• fast_path: the function will be applied on the whole data frame at once
• slow_path: the function will be applied on each column (or index) once at a time

fast_path is something like func(group) and slow_path is group.apply(lambda x: func(x)).
Maybe, you understand now why these functions are named fast and slow.

Now they are set, Pandas can call them on the first group. This is what the _choose_path method does.
And the trick is here. First, Pandas apply (safely) the slow_path function on the first group then it try to apply the fast_path function on the same group.
If the last function succeeded and some checks passed, Pandas will use the fast_path with the remaining groups to speed up the process.

It's important for large transformation with many groups. Suppose you have a dataframe with 100 groups and 30 columns to transform.
For the slow path, your function will be called 3001 (100+1+29*100) times against only 130 (30+1+99) times for the fast path.

However, keep in mind, using fast or slow path only depends on what the custom function returns. I slightly modified your custom function for better understanding and demonstration purpose:

i = 0
def MAD_single(grp, slow_strategy=True):
    global i
    print(f'{i}: "{grp.name}" is a {grp.__class__.__name__}')
    i += 1
    return 5 if slow_strategy else grp * 5

g = df.groupby('code')[['high', 'low']]

First run, we force slow_path by returning a scalar value:

>>> g.transform(MAD_single, slow_strategy=True)
0: "high" is a Series  # 1st group, slow_path
1: "low" is a Series  # 1st group, slow_path
2: "600010.SH" is a DataFrame  # 1st group, fast_path
3: "high" is a Series  # 2nd group, slow path
4: "low" is a Series  # 2nd group, slow path

The fast_path function was called only once because the result didn't pass the checks.

Now, we force fast_path by returning a same dimension array:

>>> g.transform(MAD_single, slow_strategy=False)
0: "high" is a Series  # 1st group, slow_path
1: "low" is a Series  # 1st group, slow_path
2: "600010.SH" is a DataFrame  # 1st group, fast_path
3: "600028.SH" is a DataFrame  # 2nd group, fast_path

As the result pass the checks, the fast_path function will be used now for remaining groups.

In conclusion, the transformation process seems to follow these steps:

• 1 - Apply the transformation on each column of the first group (slow path)
• 2 - Try to apply the transformation on all columns of the first group (fast path)
• 3 - Check the results of 2: fast path (4a) if checks are right else slow path (4b) as fallback
• 4a - Apply the transformation on each column of the remaining groups (slow path)
• 4b - Apply the transormation on all columns of the remaining groups (fast path)

Disclaimer: I'm not a Pandas developer so maybe some information as a result of my understanding is wrong but the obtained results and code analysis seem to confirm this strategy.