Selenium/chrome: Why do I first need to manually run chrome before I can succeed to do the same using selenium?

I'm using selenium/chrome (crostini/Linux) to check prices from www.amtrak.com/tickets/departure.html. I was seeing different results from a browser started manually (website application worked) vs a browser started by selenium (website application failed with an "unknown error"). I initially thought I had a coding issue, but eventually discovered I could replicate the failure by calling chrome manually using the same options as was used by selenium.

I also discovered I could prevent the failure if I set my selenium initiated browser to run in a static user-data-dir but first pre-conditioned this area by manually running chrome (without any options other than --user-data-dir and --profile-directory) in the same area.

Replication steps for my test case:

0. Manually start chrome, go to https://www.amtrak.com/tickets/departure.html
1. Click "New Search"
2. From: "PDX"
3. To: "JAX"
4. Depart Date: 4/2/23
5. ...defaults for everything else...
6. click "FIND TRAINS"

Image of input entry form (sorry, not enough points to embed the image)

I'm expecting something like this picture where search results are shown in a green outline.

When I try to automate the above in (python) selenium, I find that I get an "unknown error" if I just instantiate a brand new browser instance using the following (and then walk through the form manually as above)

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
chrome_options.add_experimental_option("detach", True)
driver = webdriver.Chrome(options=chrome_options)
driver.get("https://amtrak.com/tickets/departure.html")

Under the hood, I can see from chrome://version that this method calls chrome with its own random brand new user data area (Example: chrome --user-data-dir=/tmp/.com.google.Chrome.VEejr7 ...). I've done enough debugging to find out that the specific error occurs when javascript attempts to parse a non-existent json that the code expected to find in the sessionStorage area.

However, I've found that I can make the automated solution work if I:

1. Start chrome manually using: "chrome --user-data-dir=/home/robinson/.config/google-chrome --profile-directory=selenium", enter "guest mode", and then fill out the form and click "FIND TRAINS"
2. Start chrome using selenium with these same switches:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
chrome_options.add_experimental_option("detach", True)
chrome_options.add_argument(**"user-data-dir=/home/robinson/.config/google-chrome"**);
chrome_options.add_argument(**'--profile-directory=selenium'**)
driver = webdriver.Chrome(options=chrome_options)
driver.get("https://amtrak.com/tickets/departure.html")

What I'm stumped on is -- WHY? Why can't I successfully run from a virgin user data area that has never been touched by a previous browser session? Is there some switch I can call selenium with that ensures I can do this successfully from a new user data area?

Note that I've played with many options... below are a few that I've already tried:

chrome_options.add_argument('--incognito')
chrome_options.add_argument("--enable-javascript")
chrome_options.add_argument("--enable-file-cookies")
chrome_options.add_experimental_option("excludeSwitches", ['enable-automation'])

Thanks,
David Robinson

OK, I think this is a bug in either chrome or webdriver (or selenium is incorrectly using options when it calls chrome). If I use selenium to only open a (detached) browser to a website, these are the options it uses to call chrome:

> /usr/bin/google-chrome --allow-pre-commit-input --disable-background-networking --disable-client-side-phishing-detection --disable-default-apps --disable-hang-monitor --disable-popup-blocking --disable-prompt-on-repost --disable-sync --enable-automation --enable-blink-features=ShadowDOMV0 --enable-logging --log-level=0 --no-first-run --no-service-autorun --password-store=basic --remote-debugging-port=0 --test-type=webdriver --use-mock-keychain --user-data-dir=/tmp/.com.google.Chrome.edqG7B --flag-switches-begin --flag-switches-end data:,

It turns out that if both "--enable-automation" and "--remote-debugging-port=0" are used at the same time when calling chrome, then it creates problems for the javascript application that is running at amtrak.com (specifically it seems to prevent the application from writing json to the sessionStorage area).

The workaround that I've implemented is to add the following lines to my selenium code:

   options.add_experimental_option("excludeSwitches",["enable-automation"])
   options.add_argument("--remote-debugging-port=9999")

The first line prevents selenium from adding "--enable-automation" to the commandline when it calls chrome (Note that this also turns off the "Chrome is being controlled by automated test software" banner). The second line sets the remote debugging port to anything other than 0. (If anyone knows how to completely eliminate this option from being passed onto chrome, I'm all ears.)

The combination of both actions sets up the selenium-initiated browser to correctly run the application that amtrak.com currently has in place... without any need to run in a static area that has been pre-conditioned with a manual browser run.

(This also eliminates the need to use the --user-data-dir and --profile-directory options)