ElementClickInterceptedException. I am getting this error while scraping data for airline reviews on the website "Airline quality" using Selenium

huangapple go评论70阅读模式
英文:

ElementClickInterceptedException. I am getting this error while scraping data for airline reviews on the website "Airline quality" using Selenium

问题

Website Url : 英国航空公司评论

rating_title = []
for i in range(35):
    print("抓取 ", i+1)
    titles = browser.find_elements_by_xpath("//h2[@class='text_header']")
    for title in titles:
        rating_title.append(title.text)
    next_button = browser.find_element_by_partial_link_text(">>")
    next_button.click()
    sleep(5)

笔记本单元格

我写了这段代码来获取评论,一开始它运行得很好,但后来我不知道发生了什么。 它开始出现这个错误(ElementClickInterceptedException)。
每当你到达第二页时,会弹出一个广告弹窗,这可能是这个错误的原因吗?(我很可能在这方面弄错了)

附言:编辑:

rating_title = []
for i in range(35):
    print("抓取 ", i+1)
    titles = browser.find_elements_by_xpath("//h2[@class='text_header']")
    for title in titles:
        rating_title.append(title.text)
    next_button = WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, ">>")))
    next_button.click()

我将代码更新为这样,但仍然出现相同的错误。

英文:

Website Url : British Airways Reviews

rating_title = []
for i in range(35):
    print("Scraping ", i+1)
    titles = browser.find_elements_by_xpath("//h2[@class='text_header']")
    for title in titles:
        rating_title.append(title.text)
    next_button = browser.find_element_by_partial_link_text(">>")
    next_button.click()
    sleep(5)

Notebook Cell

I wrote this code to get reviews, firstly it was running fine but then I don't know what happened. It started giving this error(ElementClickInterceptedException).
Whenever you get to the second page there is an ad pop-up, could that be the reason for this error?(Most Probably I'm wrong about this)

P.S. Edit:

rating_title = []
for i in range(35):
    print("Scraping ", i+1)
    titles = browser.find_elements_by_xpath("//h2[@class='text_header']")
    for title in titles:
        rating_title.append(title.text)
    next_button = WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, ">>")))
    next_button.click()

I updated the code to this, still getting the same error.

答案1

得分: 0

ElementClickInterceptedException的根本原因: 下一页按钮(>>)被下面的广告弹窗遮挡。

解决方案: 您需要向页面的极底部滚动,下一页按钮(>>)将清晰可见,不会被遮挡。

以下是完整可工作的代码:

import time
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
import selenium.webdriver.support.expected_conditions as EC
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.maximize_window()
driver.get("https://www.airlinequality.com/airline-reviews/british-airways/?sortby=post_date%3ADesc&pagesize=100")
wait = WebDriverWait(driver, 30)
rating_title = []
for i in range(35):
    print("正在爬取 ", i+1)
    titles = wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//h2[@class='text_header']")))
    for title in titles:
        rating_title.append(title.text)
    # 下面的代码将页面滚动到底部,以避免由于广告弹窗而出现ElementClickIntercepted异常
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    next_button = wait.until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, ">>")))
    next_button.click()
    wait.until(EC.frame_to_be_available_and_switch_to_it((By.ID, "aswift_7")))
    # 下面的if-else条件用于处理点击“下一页”按钮“>>”后出现的间歇性广告弹窗
    if (len(driver.find_elements(By.XPATH, "//iframe[@title='Advertisement']")) >= 1):
        wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//iframe[@title='Advertisement']")))
        wait.until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Close']"))).click()
        time.sleep(1)
    else:
        time.sleep(1)

    driver.switch_to.default_content()

控制台输出:

正在爬取  1
正在爬取  2
正在爬取  3
正在爬取  4
正在爬取  5
正在爬取  6
正在爬取  7
正在爬取  8
正在爬取  9
正在爬取  10
正在爬取  11
正在爬取  12
正在爬取  13
正在爬取  14
正在爬取  15
正在爬取  16
正在爬取  17
正在爬取  18
正在爬取  19
正在爬取  20
正在爬取  21
正在爬取  22
正在爬取  23
正在爬取  24
正在爬取  25
正在爬取  26
正在爬取  27
正在爬取  28
正在爬取  29
正在爬取  30
正在爬取  31
正在爬取  32
正在爬取  33
正在爬取  34
正在爬取  35

请注意,这是代码的中文翻译部分。

英文:

Root cause for ElementClickInterceptedException: Next button(>>) was overlapped by the below ad pop-up.

ElementClickInterceptedException. I am getting this error while scraping data for airline reviews on the website "Airline quality" using Selenium

Soution: You need to scroll down to the extreme bottom of the page, Next button(>>) will be visible clearly and doesn't get overlapped.

Here is the full working code:

import time
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
import selenium.webdriver.support.expected_conditions as EC
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.maximize_window()
driver.get("https://www.airlinequality.com/airline-reviews/british-airways/?sortby=post_date%3ADesc&pagesize=100")
wait = WebDriverWait(driver, 30)
rating_title = []
for i in range(35):
    print("Scraping ", i+1)
    titles = wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//h2[@class='text_header']")))
    for title in titles:
        rating_title.append(title.text)
    # Below line scrolls down to the bottom of the page, this line is to avoid ElementClickIntercepted exception due to the ad popup
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    next_button = wait.until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, ">>")))
    next_button.click()
    wait.until(EC.frame_to_be_available_and_switch_to_it((By.ID, "aswift_7")))
    # below if-else condition is to handle the intermittent ad popup which appears after clicking Next_button ">>"
    if (len(driver.find_elements(By.XPATH, "//iframe[@title='Advertisement']")) >= 1):
        wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//iframe[@title='Advertisement']")))
        wait.until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Close']"))).click()
        time.sleep(1)
    else:
        time.sleep(1)

    driver.switch_to.default_content()

Console Output:

Scraping  1
Scraping  2
Scraping  3
Scraping  4
Scraping  5
Scraping  6
Scraping  7
Scraping  8
Scraping  9
Scraping  10
Scraping  11
Scraping  12
Scraping  13
Scraping  14
Scraping  15
Scraping  16
Scraping  17
Scraping  18
Scraping  19
Scraping  20
Scraping  21
Scraping  22
Scraping  23
Scraping  24
Scraping  25
Scraping  26
Scraping  27
Scraping  28
Scraping  29
Scraping  30
Scraping  31
Scraping  32
Scraping  33
Scraping  34
Scraping  35

Process finished with exit code 0

huangapple
  • 本文由 发表于 2023年6月25日 17:47:43
  • 转载请务必保留本文链接:https://go.coder-hub.com/76549787.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定