Python ColumnTransformer SettingWithCopyWarning

I'm receiving a SettingWithCopyWarning when applying transformations to a DataFrame using a scikit-learn ColumnTransformer, and I'm not sure why that is.

This is my code.

import pandas as pd
import numpy as np
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import FunctionTransformer

import warnings
warnings.filterwarnings("always")

def filling_nan(frame):
    """Fills columns that have null values with zeros."""
    frame.fillna(0, inplace=True)
    return frame

def as_is(frame):
    """Returns the DataFrame as it is."""
    return frame

np.random.seed(1337)
df = pd.DataFrame(data=np.random.random(size=(5,5)), columns=list('ABCDE'))
df = df.applymap(lambda x: np.nan if x<0.15 else x) # Set a few numbers in the dataframe to NaN.

print(df)

This is what the original DataFrame looks like...

          A         B         C         D         E
0  0.262025  0.158684  0.278127  0.459317  0.321001
1  0.518393  0.261943  0.976085  0.732815       NaN
2  0.386275  0.628501       NaN  0.983549  0.443225
3  0.789558  0.794119  0.361262  0.416104  0.584258
4  0.760172  0.187808  0.288167  0.670219  0.499648

Then I create the steps in the ColumnTransformer and I specify the index of the column rather than the column name.

step_filling_nans = ('filling_nans', FunctionTransformer(filling_nan, validate=False), [2, 4])
step_as_is = ('as_is', FunctionTransformer(as_is, validate=False), [0, 1, 3])

Then I create the ColumnTransformer...

trans = ColumnTransformer(
    transformers=[
        step_filling_nans
        , step_as_is # I could pass 'passthrough' to the remainder keyword instead of doing this step.
    ], remainder='drop')

Finally, I print the result of the applying the ColumnTransformer to my DataFrame.

print(trans.fit_transform(df))

This is the output of the transformations. The ColumnTransformer returns a numpy array as expected (with columns 'C' and 'E' first and second respectively), but I don't understand why I'm receiving the SettingWithCopy warning.

[[0.27812652 0.32100054 0.26202468 0.15868397 0.45931689]
 [0.97608528 0.         0.51839282 0.26194293 0.73281455]
 [0.         0.44322487 0.38627507 0.62850118 0.98354861]
 [0.36126157 0.58425813 0.78955834 0.79411858 0.41610394]
 [0.28816715 0.49964826 0.76017177 0.18780841 0.67021886]]

/bigdisk0/users/belladam/.conda/envs/day_zero_retention/lib/python3.6/site-packages/pandas/core/frame.py:3787: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  downcast=downcast, **kwargs)

I have managed to fix it by changing the filling_nan() function slightly, but I don't understand why that fixes it.

def filling_nan(frame):
    """Fills columns that have null values with zeros."""
    frame = frame.fillna(0)
    return frame

I've been unable to reproduce the result outside of using a ColumnTransformer so was wondering if it was something to do with that?

I believe ColumnTransformer behaves this way because it is capable of running in parallel different transformations related to different columns. You can take a look at the sklearn code itself here, at line 448. If you want to use parallelization then it is not really safe to work on the same object (same memory location).

By avoiding using inplace, you are actually working on copy of the original object, and this settles the problem:

def filling_nan(frame):
    """Fills columns that have null values with zeros."""
    frame = frame.fillna(0)
    return frame

def filling_nan_inplace(frame):
    """Fills columns that have null values with zeros."""
    frame.fillna(0, inplace=True)
    return frame

print(id(df))
print(id(filling_nan_inplace(df)))
print(id(filling_nan(df)))

output:

2088604584760
2088604584760
2088604583304