Selenium Python Chrome: Extremely slow. Are cookies the problem?

I read that Selenium Chrome can run faster if you use implicit waits, headless, ID and CSS selectors etc. Before implementing those changes, I want to know whether cookies or caching could be slowing me down.

Does Selenium store cookies and cache like a normal browser or does it reload all assets everytime it navigates to a new page on a website?

If yes, then this would slow down the process of scraping websites with millions of identical profile pages, where the scripts and images are similar for each profile.

Is yes, is there a way to avoid this problem? Interested in using cookies and cache during a session and then destroying after the browser is closed.

Edit, more details:

sel_options = {'proxy': {'https': pString}}
prefs = {'download.default_directory' : dFolder}
options.add_experimental_option('prefs', prefs)
blocker = os.path.join( os.getcwd(), "extension_iijehicfndmapfeoplkdpinnaicikehn")
options.add_argument(f"--load-extension={blocker}")
wS = "--window-size="+s1+","+s2
options.add_argument(wS)
if headless == "yes": options.add_argument("--headless");
driver = uc.Chrome(seleniumwire_options=sel_options, options=options, use_subprocess=True, version_main=109)
stealth(driver, languages=["en-US", "en"], vendor="Google Inc.", platform="Win32",  webgl_vendor="Intel Inc.", renderer="Intel Iris OpenGL Engine", fix_hairline=True)
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": agent})
navigate("https://linkedin.com")

I don't think my proxy or extension is the culprit, because I have a similar automation app running with no speed issue.

By default, Selenium WebDriver may not enable caching or store cookies like a normal browser and yes i do think that is always interesting to use session caching instead of re-downloading the same files after each scraping cron job run.

Cache (Selenium/Chrome) with a static browser profile:

You can create a Chrome profile with caching enabled and then use that profile with Selenium. This way, the browser will utilize the same profile directory to store all your files during all your scraping runs and you may not lose files in between runs.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--user-data-dir=/path/to/profile/folder")
options.add_argument("--disk-cache-dir=/path/to/cache/folder")
driver = webdriver.Chrome(options=options)

Replace /path/to/profile/folder with the desired folder where you want to store the profile data, and /path/to/cache/folder with the folder where you want to store the cache. This way, the browser will utilize the cache during subsequent page loads within the same session.

It is also good to know that the python selenium library provides methods to manage cookies. You can use driver.manage().getCookies() to retrieve cookies and driver.manage().addCookie(cookie) to add a cookie to the current session.

By default, the chromedriver being used by Selenium interacts with the cookies provided by the website linkedIn.com which you are scraping. These cookies will only be maintained by the chromedriver until you close the driver session. When you launch a new instance of chromedriver the old cookies do not exist. LinkedIn.com will provide a new batch of cookies to use in this with new chromedriver session.

In my opinion the cookies and caching from an active session aren’t causing your performance issues when scraping linkedIn.com.

You stated in your question that you were scraping millions of identical profile pages on linkedIn.com. Scraping this many profiles is very time consuming even for an automated Selenium session. The mostly likely culprit of your performance issue is rate-limiting being imposed by linkedIn.com.

I would assume that you are receiving HTTP 429 Too Many Requests response status code when your scraping activities become slow. You would need to do some debugging to determine this.

Here is some code that you can modified to get the HTTP status code:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

chrome_options = Options()
chrome_options.add_argument("--disable-infobars")
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-popup-blocking")
chrome_options.add_argument('--ignore-ssl-errors')
chrome_options.add_argument('--ignore-certificate-errors')

# disable the banner "Chrome is being controlled by automated test software"
chrome_options.add_experimental_option("useAutomationExtension", False)
chrome_options.add_experimental_option("excludeSwitches", ['enable-automation'])

capabilities = DesiredCapabilities().CHROME
capabilities['goog:loggingPrefs'] = {'performance': 'ALL'}

driver = webdriver.Chrome(executable_path='/usr/local/bin/chromedriver', options=chrome_options, desired_capabilities=capabilities)

driver.get(f'https://www.linkedin.com')

perfLog = driver.get_log('performance')
for logIndex in range(0, len(perfLog)):  # Parse the Chrome Performance logs
    logMessage = json.loads(perfLog[logIndex]["message"])["message"]
    if logMessage["method"] == "Network.responseReceived":  # Filter out HTTP responses
        print(logMessage["params"]["response"]) 

I noted that when I was checking the status code for my connection that linkedIn.com also using cloudflare for protection. This is going to create a major issue when scraping millions of identical profile pages on linkedIn.com. You will likely get blocked or your proxy addresses will get blacklisted.

Your current code will require redesigning to handle rate-limiting, status codes of 429 and any cloudflare hurdles.