英文:
Best way to go back 10 pages with selenium
问题
我正在尝试爬取多个用户资料。
我的当前代码是：
all_users = driver.find_elements(By.XPATH,
for pages in range(2, 14):
for user in all_users:
profile_links = user.get_attribute('href')
names_searched.append(profile_links)
go_profile(names_searched, counter, driver)
driver.back()
scroll(driver)
nxt_button = wait.until(EC.element_to_be_clickable((By.XPATH, "//*[contains(@class, 'artdeco-pagination__button--next')]").click()))
counter += 1
我不确定下一页按钮是否正常工作。到目前为止，它无法找到下一页按钮。如果我能得到在LinkedIn搜索结果中找到下一页按钮的帮助，那将很有帮助。
我的最大问题是，我需要返回到搜索结果页面的原始URL，然后再返回10页。例如，如果我在搜索结果的第2页，我想遍历该页上的所有可用资料，然后返回到第2页，然后点击下一页按钮转到第3页。在第3页，我想爬取所有资料，然后返回到第3页，然后点击下一页按钮，以此类推。
每页大约有10个资料，这就是为什么我想要返回10页，而不只是执行10次driver.back()的原因。
英文:
I'm trying to crawl several profiles.
My current code is:
all_users = driver.find_elements(By.XPATH,
for pages in range(2,14):
for user in all_users:
profile_links = user.get_attribute('href')
names_searched.append(profile_links)
go_profile(names_searched, counter, driver)
driver.back()
scroll(driver)
nxt_button = wait.until(EC.element_to_be_clickable((By.XPATH, "//*[contains(@class, \'artdeco-pagination__button--next\')]").click()))
counter += 1
I'm not 100% sure that the next page button works properly for now. So far it has not been able to find the next button. If I can get help with finding the next button from linkedin search results that would be helpful.
My biggest issue is I need to go back 10 pages to the original url to the search result page where it started off.
For example, if I was on page 2 of the search result, I want to go through all the profiles available on that page then go back to page 2 then click next button to go to page 3. On page 3 I want to crawl all profiles, go back to page 3 than click next button for 4 and so on.
Each page has about 10 profiles so that is why I want to go back 10 pages.
Is there a better way to go back 10 pages than just doing driver.back() 10 times?
答案1
得分: 2
另一种方法是，而不是多次点击返回到之前的多个页面，你可以直接使用URL来实现。
如果你在人们搜索的第一页：
https://www.linkedin.com/search/results/people/?keywords=your_query
如果你在人们搜索的第二页：
https://www.linkedin.com/search/results/people/?keywords=your_query&page=2
如果你在人们搜索的第十页：
https://www.linkedin.com/search/results/people/?keywords=your_query&page=10
正如你可能注意到的，URL包含了关于你所在页面的信息。假设你在第10页，想要返回到第1页，只需直接使用第1页的URL来返回。
希望这对你有用。
英文:
Another way around, instead of clicking multiple times to go back to a number of pages back, you can directly do it by using the URL.
if you are on the first page of people's search
https://www.linkedin.com/search/results/people/?keywords=your_query
if you are on the 2nd page of people's search
https://www.linkedin.com/search/results/people/?keywords=your_query&page=2
if you are on the 10th page of people's search
https://www.linkedin.com/search/results/people/?keywords=your_query&page=10
As you might notice, the URL contains information about the page you are on. And let's say you're on page 10 and want to return back to the 1st page, simply use the URL of page 1 to return back to it directly.
I hope it's useful.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论