英文:
ElementClickInterceptedException. I am getting this error while scraping data for airline reviews on the website "Airline quality" using Selenium
问题
Website Url : 英国航空公司评论
rating_title = []
for i in range(35):
print("抓取 ", i+1)
titles = browser.find_elements_by_xpath("//h2[@class='text_header']")
for title in titles:
rating_title.append(title.text)
next_button = browser.find_element_by_partial_link_text(">>")
next_button.click()
sleep(5)
我写了这段代码来获取评论,一开始它运行得很好,但后来我不知道发生了什么。 它开始出现这个错误(ElementClickInterceptedException)。
每当你到达第二页时,会弹出一个广告弹窗,这可能是这个错误的原因吗?(我很可能在这方面弄错了)
附言:编辑:
rating_title = []
for i in range(35):
print("抓取 ", i+1)
titles = browser.find_elements_by_xpath("//h2[@class='text_header']")
for title in titles:
rating_title.append(title.text)
next_button = WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, ">>")))
next_button.click()
我将代码更新为这样,但仍然出现相同的错误。
英文:
Website Url : British Airways Reviews
rating_title = []
for i in range(35):
print("Scraping ", i+1)
titles = browser.find_elements_by_xpath("//h2[@class='text_header']")
for title in titles:
rating_title.append(title.text)
next_button = browser.find_element_by_partial_link_text(">>")
next_button.click()
sleep(5)
I wrote this code to get reviews, firstly it was running fine but then I don't know what happened. It started giving this error(ElementClickInterceptedException).
Whenever you get to the second page there is an ad pop-up, could that be the reason for this error?(Most Probably I'm wrong about this)
P.S. Edit:
rating_title = []
for i in range(35):
print("Scraping ", i+1)
titles = browser.find_elements_by_xpath("//h2[@class='text_header']")
for title in titles:
rating_title.append(title.text)
next_button = WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, ">>")))
next_button.click()
I updated the code to this, still getting the same error.
答案1
得分: 0
ElementClickInterceptedException
的根本原因: 下一页按钮(>>
)被下面的广告弹窗遮挡。
解决方案: 您需要向页面的极底部滚动,下一页按钮(>>
)将清晰可见,不会被遮挡。
以下是完整可工作的代码:
import time
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
import selenium.webdriver.support.expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.maximize_window()
driver.get("https://www.airlinequality.com/airline-reviews/british-airways/?sortby=post_date%3ADesc&pagesize=100")
wait = WebDriverWait(driver, 30)
rating_title = []
for i in range(35):
print("正在爬取 ", i+1)
titles = wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//h2[@class='text_header']")))
for title in titles:
rating_title.append(title.text)
# 下面的代码将页面滚动到底部,以避免由于广告弹窗而出现ElementClickIntercepted异常
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
next_button = wait.until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, ">>")))
next_button.click()
wait.until(EC.frame_to_be_available_and_switch_to_it((By.ID, "aswift_7")))
# 下面的if-else条件用于处理点击“下一页”按钮“>>”后出现的间歇性广告弹窗
if (len(driver.find_elements(By.XPATH, "//iframe[@title='Advertisement']")) >= 1):
wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//iframe[@title='Advertisement']")))
wait.until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Close']"))).click()
time.sleep(1)
else:
time.sleep(1)
driver.switch_to.default_content()
控制台输出:
正在爬取 1
正在爬取 2
正在爬取 3
正在爬取 4
正在爬取 5
正在爬取 6
正在爬取 7
正在爬取 8
正在爬取 9
正在爬取 10
正在爬取 11
正在爬取 12
正在爬取 13
正在爬取 14
正在爬取 15
正在爬取 16
正在爬取 17
正在爬取 18
正在爬取 19
正在爬取 20
正在爬取 21
正在爬取 22
正在爬取 23
正在爬取 24
正在爬取 25
正在爬取 26
正在爬取 27
正在爬取 28
正在爬取 29
正在爬取 30
正在爬取 31
正在爬取 32
正在爬取 33
正在爬取 34
正在爬取 35
请注意,这是代码的中文翻译部分。
英文:
Root cause for ElementClickInterceptedException
: Next button(>>
) was overlapped by the below ad pop-up.
Soution: You need to scroll down to the extreme bottom of the page, Next button(>>
) will be visible clearly and doesn't get overlapped.
Here is the full working code:
import time
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
import selenium.webdriver.support.expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.maximize_window()
driver.get("https://www.airlinequality.com/airline-reviews/british-airways/?sortby=post_date%3ADesc&pagesize=100")
wait = WebDriverWait(driver, 30)
rating_title = []
for i in range(35):
print("Scraping ", i+1)
titles = wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//h2[@class='text_header']")))
for title in titles:
rating_title.append(title.text)
# Below line scrolls down to the bottom of the page, this line is to avoid ElementClickIntercepted exception due to the ad popup
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
next_button = wait.until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, ">>")))
next_button.click()
wait.until(EC.frame_to_be_available_and_switch_to_it((By.ID, "aswift_7")))
# below if-else condition is to handle the intermittent ad popup which appears after clicking Next_button ">>"
if (len(driver.find_elements(By.XPATH, "//iframe[@title='Advertisement']")) >= 1):
wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//iframe[@title='Advertisement']")))
wait.until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Close']"))).click()
time.sleep(1)
else:
time.sleep(1)
driver.switch_to.default_content()
Console Output:
Scraping 1
Scraping 2
Scraping 3
Scraping 4
Scraping 5
Scraping 6
Scraping 7
Scraping 8
Scraping 9
Scraping 10
Scraping 11
Scraping 12
Scraping 13
Scraping 14
Scraping 15
Scraping 16
Scraping 17
Scraping 18
Scraping 19
Scraping 20
Scraping 21
Scraping 22
Scraping 23
Scraping 24
Scraping 25
Scraping 26
Scraping 27
Scraping 28
Scraping 29
Scraping 30
Scraping 31
Scraping 32
Scraping 33
Scraping 34
Scraping 35
Process finished with exit code 0
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论