Force Dataflow workers to use Python 3?

I have a simple batch Apache Beam Pipeline. When run locally - DirectRunner works fine, but with DataflowRunner it fails to install 1 dependency from requirements.txt. The reason is that the specific package is for Python 3, and the workers are (apparently) running the pipeline with Python 2.

The pipeline is done and working fine locally (DirectRunner) with Python 3.7.6. I'm using the latest Apache Beam SDK (apache-beam==2.16.0 in my requirements.txt).

One of the modules required by my pipeline is:
from lbcapi3 import api

So my requirements.txt sent to GCP has a line with:
lbcapi3==1.0.0

That module (lbcapi3) is in PyPI, but it's only targeted for Python 3.x. When I run the pipeline in Dataflow I get:

ERROR: Could not find a version that satisfies the requirement lbcapi3==1.0.0 (from -r requirements.txt (line 27)) (from versions: none)\r\nERROR: No matching distribution found for lbcapi3==1.0.0 (from -r requirements.txt (line 27))\r\n'

That makes me think that the Dataflow worker is running the pipeline with Python 2.x to install the dependencies in requirements.txt.

Is there a way to specify the Python version to use by a Goggle Dataflow pipeline (the workers)?

I tried by adding this as the first line of the my file api-etl.py, but didn't work:

#!/usr/bin/env python3

Thanks!

Follow the instructions in the quickstart to get up and running with your pipeline. When installing the Apache Beam SDK, make sure to install version 2.16 (since this is the first version that officially supports Python 3). Please, check your version.

You can use the Apache Beam SDK with Python versions 3.5, 3.6, or 3.7 if you are keen to migrate from Python 2.x environments.

For more information, refer to this documentation. Also, take a look for preinstalled dependencies.

Edited, after providing additional information:

I have reproduced problem on Dataflow. I see two solutions.

1. You can use --extra_package option, which would allow staging local packages in an accessible way. Instead of listing local package in the requirements.txt, create a tarball of the local package (e.g. my_package.tar.gz) and use --extra_package option to stage them.

Clone the repository from Github:

$ git clone https://github.com/6ones/lbcapi3.git
$ cd lbcapi3/

Build the tarball with the following command:

$ python setup.py sdist

The last few lines will look like this:

Writing lbcapi3-1.0.0/setup.cfg
creating dist
Creating tar archive
removing 'lbcapi3-1.0.0' (and everything under it)

Then, run your pipeline with the following command-line option:

 --extra_package /path/to/package/package-name

In my case:

--extra_package /home/user/dataflow-prediction-example/lbcapi3/dist/lbcapi3-1.0.0.tar.gz

Make sure, that all of required options are provided in the command (job_name, project, runner, staging_location, temp_location):

python prediction/run.py --runner DataflowRunner --project $PROJECT --staging_location $BUCKET/staging --temp_location $BUCKET/temp --job_name $PROJECT-prediction-cs --setup_file prediction/setup.py --model $BUCKET/model --source cs --input $BUCKET/input/images.txt --output $BUCKET/output/predict --extra_package /home/user/dataflow-prediction-example/lbcapi3/dist/lbcapi3-1.0.0.tar.gz

The error you faced, would disappear.

2. Second solution - posting the additional libraries that your app is using in setup.py file, refer to the documentation.

Create a setup.py file for your project:

 import setuptools

 setuptools.setup(
    name='PACKAGE-NAME',
    version='PACKAGE-VERSION',
    install_requires=[],
    packages=setuptools.find_packages(),
 )

You can get rid of the requirements.txt file and instead, add all packages contained in requirements.txt to the install_requires field of the setup call.

The simple answer is when deploying your pipeline, you need to make sure that your local environment is on python 3.5 3.6 or 3.7. If it is, then the Dataflow worker will have the same version once your job is launched.