I am trying to webscrape Income Statement data from https://finance.yahoo.com/quote/AAPL/financials?p=AAPL

I am trying to simulate a click on Operating Expense to expand the row to see the Research and Development values like such:

This is my code so far:

def clickOperatingExpense(url):
        options = Options()
        options.add_argument('--headless')
        driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
        driver.get(url)
        # This is where I encounter a selenium.common.exceptions.ElementClickInterceptedException
        WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="Col1-1-Financials-Proxy"]/section/div[4]/div[1]/div[1]/div[2]/div[4]/div[1]/div[1]/div[1]/button'))).click()

Just for some more information, the full error is:

selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element <button aria-label="Operating Expense" class="P(0) M(0) Va(m) Bd(0) Fz(s) Mend(2px) tgglBtn">...</button> is not clickable at point (39, 592). Other element would receive the click: <div tabindex="1" class="lightbox-wrapper Ta(c) Pos(f) T(0) Start(0) H(100%) W(100%) Bgc($modalBackground) Ovy(a) Z(50) Op(1)">...</div>

The issue looks like a div tag would receive the click but I'm not sure why. How can I make the <button> tag receive the click instead?

To click and expand the element with text as Operating Expense you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:

• Using CSS_SELECTOR:

  driver.get("https://finance.yahoo.com/quote/AAPL/financials?p=AAPL")
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[aria-label='Close']"))).click()
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[title='Operating Expense'] > button[aria-label='Operating Expense']"))).click()
  

• Using XPATH:

  driver.get("https://finance.yahoo.com/quote/AAPL/financials?p=AAPL")
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@aria-label='Close']"))).click()
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@title='Operating Expense']/button[@aria-label='Operating Expense']"))).click()
  

• Browser Snapshot:

• Note: You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

Try using Java script to perform click() as below:

driver.execute_script("arguments[0].click();", element)

Reference: https://stackoverflow.com/a/75645682/7598774