Selenium script with Firefox fails if executed by cron, but runs correctly if executed manually

I configured a Python script (called test_fire7.py), which uses Selenium and Firefox to be executed by cron in a Python virtual environment called e1. The Python script runs correctly, if executed manually from the console. It runs correctly both with the virtual environment activated as well without activating the environment explicitly, but using a command like this: /root/pye/e1/bin/python3 /root/pye/test_fire7.py

Here is the entry in crontab:

30 20 * * * cd /root/pye && /root/pye/e1/bin/python3 /root/pye/test_fire7.py command arg 2>&1 | logger -t mycmd

And this is the header of test_fire7.py (I omit the rest of the script, because it runs correctly by itself):

from selenium import webdriver
from selenium.webdriver import FirefoxOptions
from selenium.webdriver.common.by import By

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

opts = FirefoxOptions()
opts.add_argument("--headless")

driver = webdriver.Firefox(options=opts)

And here is the error, after cron runs the script, captured in syslog:

root@wi-master:~/pye# grep 'mycmd' /var/log/syslog
May 27 20:30:01 wi-master mycmd: Traceback (most recent call last):
May 27 20:30:01 wi-master mycmd:   File "/root/pye/test_fire7.py", line 15, in <module>
May 27 20:30:01 wi-master mycmd:     driver = webdriver.Firefox(options=opts)
May 27 20:30:01 wi-master mycmd:   File "/root/pye/e1/lib/python3.8/site-packages/selenium/webdriver/firefox/webdriver.py", line 201, in __init__
May 27 20:30:01 wi-master mycmd:     super().__init__(command_executor=executor, options=options, keep_alive=True)
May 27 20:30:01 wi-master mycmd:   File "/root/pye/e1/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 286, in __init__
May 27 20:30:01 wi-master mycmd:     self.start_session(capabilities, browser_profile)
May 27 20:30:01 wi-master mycmd:   File "/root/pye/e1/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 378, in start_session
May 27 20:30:01 wi-master mycmd:     response = self.execute(Command.NEW_SESSION, parameters)
May 27 20:30:01 wi-master mycmd:   File "/root/pye/e1/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 440, in execute
May 27 20:30:01 wi-master mycmd:     self.error_handler.check_response(response)
May 27 20:30:01 wi-master mycmd:   File "/root/pye/e1/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 245, in check_response
May 27 20:30:01 wi-master mycmd:     raise exception_class(message, screen, stacktrace)
May 27 20:30:01 wi-master mycmd: selenium.common.exceptions.SessionNotCreatedException: Message: Expected browser binary location, but unable to find binary in default location, no 'moz:firefoxOptions.binary' capability provided, and no binary flag set on the command line

Other scripts run correctly using exactly the same kind of entries in crontab. So I assume the problem is with Selenium in some way.

I have Firefox installed:

root@wi-master:~/pye# firefox -v
Mozilla Firefox 113.0.2

And also geckodriver:

root@wi-master:~/pye# geckodriver --version
geckodriver 0.33.0 ( 2023-05-22)

Again, the script passes without errors, if executed manually. The error message in syslog given above suggests, that Firefox is not installed. But this is not the case!

Update:
I tried adding the executable path of geckodriver when calling the webdriver, but with no success. I'm not sure if I locate the executable path correctly

root@wi-master:~# which geckodriver
/snap/bin/geckodriver

I also passed the Firefox binary explicitely like this, but also without success.

from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
driver=webdriver.Firefox(firefox_binary=FirefoxBinary(getoutput("find /snap/firefox -name firefox").split("\n")[-1]), options=opts)

With this the Selenium error changes to

selenium.common.exceptions.WebDriverException: Message: Service /snap/firefox/2667/usr/lib/firefox/firefox unexpectedly exited. Status code was: 255

It looks like the working directory of your script is different from when you start it manually. One solution could be to set the working directory from the script, relative to the script itself.

Say you have the script in some folder, and when you start it manually, you start it from that folder. If you add this to the script:

import os
import pathlib

os.chdir(pathlib.Path(__file__).parent)

This changes the working directory to the folder the script is in.

Of course, if you manually started the script from some other folder (like your home directory, profile directory, etc.) you could try and change to that instead.

However, keep in mind that the cron process may not be allowed to access folders that you do have access to in your interactive session. If you set the correct working directory before execution, make sure the process is allowed to access (and execute) the binary there.

I finally got a solution. It happend through very much trial and error after all suggestions I found on the internet didn't work for me.

For completeness, here is my system:

• Ubuntu 20.04.6 LTS
• Python 3.8.10
• selenium 4.9.1
• Mozilla Firefox 113.0.2
• geckodriver 0.33.0

I use virtual environments and run the scripts from root.

First, I observed where Firefox and geckodriver reside:

root@wikijs-master:~# which firefox
/snap/bin/firefox
root@wikijs-master:~# which geckodriver
/snap/bin/geckodriver

Based on that I added export PATH=$PATH:/snap/bin/; to the crontab entry from above:

* * * * *  export PATH=$PATH:/snap/bin/; cd /root/pye && /root/pye/e1/bin/python3 /root/pye/test_fire7.py command arg 2>&1 | logger -t mycmd

And here is a sample python script that cron can execute!

from selenium import webdriver
from selenium.webdriver import FirefoxOptions

opts = FirefoxOptions()
opts.add_argument("--headless")

driver=webdriver.Firefox(options=opts)
driver.get("https://google.com")
print(driver.current_url)

I want to note that other solutions like adding executable paths, absolute paths or binary locations within the webdriver attributes, changing the working directory or modifying the permissions for the cron user didn't work for me.