NameError: name 'pd' is not defined when loading a pickled file - but pandas IS defined

I'm trying to model a simple pipeline using Apache Airflow. I'm running an instance locally using docker. Some of the stuff that needs to be done is loading a pickled sklearn model and transforming a pandas dataframe. When i load that model and i try to use it i get the most simple error.

NameError: name 'pd' is not defined

So, first thing i do is go to the top to import pandas... but pandas is there.

I transcribe here my script and my environment relevant files.

A simplified version of my airflow task:

import dill
import pandas as pd

model_file = 'models/the_model.pkl'


def task_run_model(**context):

    # Load the pre-trained models from the .pkl files
    with open(model_file, 'rb') as f:
        model = dill.load(f)

    # Test models
    file_name = "train_set.csv"
    time_series_df = pd.read_csv(file_name)
    train_features_df = model.transform(time_series_df)

    return train_features_df

The error with the stack trace:

[2023-05-14, 19:21:10 UTC] {taskinstance.py:1847} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 181, in execute
    return_value = self.execute_callable()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 198, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/opt/airflow/dags/my_tasks/transformation.py", line 15, in task_run_model
    train_features_df = model.transform(time_series_df)
  File "/Users/<USER_NAME>/Repos/algorithms/Projects/xxxxxxxxx/model.py", line 19, in transform
NameError: name 'pd' is not defined
[2023-05-14, 19:21:10 UTC] {taskinstance.py:1368} INFO - Marking task as FAILED. dag_id=feature_creation, task_id=create_features, execution_date=20230514T192058, start_date=20230514T192110, end_date=20230514T192110
[2023-05-14, 19:21:10 UTC] {standard_task_runner.py:104} ERROR - Failed to execute job 4 for task create_features (name 'pd' is not defined; 248)
[2023-05-14, 19:21:10 UTC] {local_task_job_runner.py:232} INFO - Task exited with return code 1
[2023-05-14, 19:21:10 UTC] {taskinstance.py:2674} INFO - 0 downstream tasks scheduled from follow-on schedule check

of course i don't have access to /Users/<USER_NAME>/Repos/algorithms/Projects/xxxxxxxxx/model.py, i just have the .pkl file that was given to me

The only data that i have about that environment is that it was built with this dependencies:

pandas           : 1.3.5
numpy            : 1.21.6
dateutil         : 2.8.2
scipy            : 1.10.1

This is my environment:

docker-compose.yml

---
version: '3.4'

x-common:
  &common
  build:
    context: .
    dockerfile: Dockerfile
  user: "${AIRFLOW_UID}:0"
  env_file: 
    - .env
  volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./plugins:/opt/airflow/plugins
    - ./models:/opt/airflow/models
    - ./tests:/opt/airflow/tests
    - /var/run/docker.sock:/var/run/docker.sock

x-depends-on:
  &depends-on
  depends_on:
    postgres:
      condition: service_healthy
    airflow-init:
      condition: service_completed_successfully

services:
  postgres:
    image: postgres:13
    container_name: postgres
    ports:
      - "5434:5432"
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "airflow"]
      interval: 5s
      retries: 5
    env_file:
      - .env

  scheduler:
    <<: *common
    <<: *depends-on
    container_name: pipeline-scheduler
    command: scheduler
    restart: on-failure
    ports:
      - "8793:8793"

  webserver:
    <<: *common
    <<: *depends-on
    container_name: pipeline-webserver
    restart: always
    command: webserver
    ports:
      - "8080:8080"
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
      interval: 30s
      timeout: 30s
      retries: 5
  
  airflow-init:
    <<: *common
    container_name: pipeline-init
    entrypoint: /bin/bash
    command:
      - -c
      - |
        mkdir -p /sources/logs /sources/dags /sources/plugins /sources/models
        chown -R "${AIRFLOW_UID}:0" /sources/{logs,dags,plugins,models}
        exec /entrypoint airflow version

Dockerfile

FROM apache/airflow:latest-python3.8
USER root
RUN apt-get update && \
    apt-get clean && \
    apt-get install vim-tiny -y && \
    apt-get autoremove -yqq --purge && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*
USER airflow
ENV PYTHONPATH "${PYTHONPATH}:${AIRFLOW_HOME}"
COPY requirements.txt /tmp/requirements.txt
RUN pip install -r /tmp/requirements.txt

requirements.yml

pip==22.3.1
scikit-learn==1.1.3
numpy==1.21.6
scipy==1.10.1
pandas==1.3.5
dill==0.3.6
python-dateutil==2.8.2

.env

# Meta-Database
POSTGRES_USER=airflow
POSTGRES_PASSWORD=airflow
POSTGRES_DB=airflow

# Airflow Core
AIRFLOW__CORE__FERNET_KEY=UKMzEm3yIuFYEq1y3-2FxPNWSVwRASpahmQ9kQfEr8E=
AIRFLOW__CORE__EXECUTOR=LocalExecutor
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION=True
AIRFLOW__CORE__LOAD_EXAMPLES=False
AIRFLOW_UID=0

# Backend DB
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__DATABASE__LOAD_DEFAULT_CONNECTIONS=False

# Airflow Init
_AIRFLOW_DB_UPGRADE=True
_AIRFLOW_WWW_USER_CREATE=True
_AIRFLOW_WWW_USER_USERNAME=airflow
_AIRFLOW_WWW_USER_PASSWORD=airflow

To build this I'm using

docker compose up -d

The pickle DOES run locally if i run it in PyCharm

import pandas as pd
import dill

model_file = 'models/the_model.pkl'

train_dataset_file = 'datasets/train.csv'
test_dataset_file = 'datasets/test.csv'

# Load the pre-trained model from the .pkl files
with open(model_file, 'rb') as f:
    model = dill.load(f)

# Load the datasets
train_df = pd.read_csv(train_dataset_file)
test_df = pd.read_csv(test_dataset_file)

# Test model
train_features_df: pd.DataFrame = model.transform(train_df)
test_features_df: pd.DataFrame = model.transform(test_df)


print(train_features_df, test_features_df)

Locally i'm using python 3.8, pandas==1.5.3 & dill==0.3.6
(yes, the first thing i tried was upgrading pandas to 1.5.3 in the requirements.txt, but same same.

https://stackoverflow.com/a/65318623/10972050 <- answer

Basically

import pandas as pd
import __main__
__main__.pd = pd

with open('pandize.pkl', 'rb') as f:
    p = dill.load(f)

p(1)