S3FS is not recognised in lambda when using torch

I have a Kedro Pipeline Node that on AWS Lambda that accesses s3. It runs if I'm not using torch but fails with Install s3fs to access S3 if I add torch as a dependency.

I have a Kedro Pipeline I want to deploy on AWS Step Functions.
My requirements look like this:

Python3.9

Pillow==9.5.0
aws_lambda_powertools==2.15.0
fsspec==2023.5.0
kedro==0.18.8
numpy==1.24.3
pandas==2.0.1
pydantic==1.10.7
pytest==7.3.1
rasterio==1.3.6
rawpy==0.18.1
s3fs==2023.5.0

The lambda access some data on s3.
With this setup everything runs fine.

However if I add torch,

torch==2.0.1+cpu -f https://download.pytorch.org/whl/torch_stable.html
torchvision==0.15.2+cpu -f https://download.pytorch.org/whl/torch_stable.html

I get the following error:

{
  "errorMessage": "\nInstall s3fs to access S3.\nFailed to instantiate DataSet 'companies' of type 'kedro.extras.datasets.pandas.csv_dataset.CSVDataSet'.",
  "errorType": "DataSetError",
  "requestId": "3da771f3-af50-49a9-98de-0a6d924018f2",
  "stackTrace": [
    "  File \"/home/app/lambda_handler.py\", line 18, in handler\n    session.run(node_names=[node_to_run])\n",
    "  File \"/home/app/kedro/framework/session/session.py\", line 413, in run\n    catalog = context._get_catalog(\n",
    "  File \"/home/app/kedro/framework/context/context.py\", line 287, in _get_catalog\n    catalog = settings.DATA_CATALOG_CLASS.from_config(\n",
    "  File \"/home/app/kedro/io/data_catalog.py\", line 277, in from_config\n    data_sets[ds_name] = AbstractDataSet.from_config(\n",
    "  File \"/home/app/kedro/io/core.py\", line 162, in from_config\n    raise DataSetError(\n"
  ]
}

This error is also just appearing in the lambda. if I install all those requirements locally on my linux it runs fine.

I was able to solve this issue. Torch did install a version of urllib3 that was incompatible with s3fs. So what I had to do was install torch with "urllib3<2" and it worked.