ModuleNotFoundError: No module named 'pandas.core.indexes.numeric' using Metaflow

I used Metaflow to load a Dataframe. It was successfully unpickled from the artifact store, but when I try to view its index using df.index, I get an error that says ModuleNotFoundError: No module named 'pandas.core.indexes.numeric'. Why?

I've looked at other answers with similar error messages here and here, which say that this is caused by trying to unpickle a dataframe with older versions of Pandas. However, my error is slightly different, and it is not fixed by upgrading Pandas (pip install pandas -U).

This issue is caused by the new Pandas 2.0.0 release breaking backwards compatibility with Pandas 1.x, although I don't see this documented in the release notes. The solution is to downgrade pandas to the 1.x series: pip install "pandas<2.0.0"

Try using the pandas.read_pickle() method to load the file instead of the pickle module:

import pandas as pd

df = pd.read_pickle("file.pkl")

The pandas method should provide compatibility to read older files, and "is only guaranteed to be backwards compatible to pandas 0.20.3 provided the object was serialized with to_pickle." My tests with pandas-1.x show it can also read some files written from the pickle module too.

Try using pd.compat.pickle_compat.load() as that was only solution in my case:

import pandas as pd

df = pd.compat.pickle_compat.load('file.pkl') 

So i dont know why this works but joblib.load was failing to read the pickle with the same error "module named 'pandas.core.indexes.numeric'" then i installed prefect and simple_salesforce and some how it now works... not sure why but i think worth mentioning