2023年7月6日 13:56:14go评论73阅读模式

英文:

not able to get a link while web scraping

问题

我想使用Python进行网页抓取，在选择“T20I”时进行。为此，我需要在请求和BeautifulSoup中放入一个特定的链接。

每当我打开https://www.espncricinfo.com/cricketers/team/india-6 这个链接时，我会看到一个页面上选择了“INTL”。

选择“INTL”的图片:
无法在网络抓取中获取链接。

但是当我选择“T20I”时，我会看到一个不同的页面，但链接相似 https://www.espncricinfo.com/cricketers/team/india-6

选择“T20I”的图片:
无法在网络抓取中获取链接。

那么在这种情况下，我应该怎么做才能获取数据呢？当选择“T20I”时，我如何获取数据？

英文:

I want to do web scraping using Python, on a page when 'T20I' is selected. For that, I need to put a specific link in requests and beautifulsoup.

whenever I open https://www.espncricinfo.com/cricketers/team/india-6 this link, I get a page with "INTL" selected.

Image with "INTL" selected:
无法在网络抓取中获取链接。

But when I select "T20I", I get a different page but with similar link https://www.espncricinfo.com/cricketers/team/india-6

image with "T20I" selected:
无法在网络抓取中获取链接。

Then what should I do in this situation to retrieve the data? How will I get data when "T20I" is selected?

答案1

得分: 1

我建议使用Selenium。以下是一个可以正常工作的示例：

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException

chrome_options = Options()
chrome_options.add_argument("--start-maximized")
browser = webdriver.Chrome(r"YOUR chromedriver.exe的路径", options=chrome_options)

browser.get("https://www.espncricinfo.com/cricketers/team/india-6")
# 通过XPATH查找要点击的元素
button = browser.find_element(By.XPATH, '//*[@id="main-container"]/div[5]/div[1]/div[2]/div[2]/div/div/div[2]/div[2]/div[1]/div/div[5]/a/span/span')
# 点击按钮 "T20I"
button.click()

尽情享用！

英文:

I recommend to use Selenium
Here is the example which can work just fine

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException

chrome_options = Options()
chrome_options.add_argument(&quot;--start-maximized&quot;);
browser =webdriver.Chrome(r&quot;YOUR chromedriver.exe&quot;, options=chrome_options)


browser.get(&quot;https://www.espncricinfo.com/cricketers/team/india-6&quot;)
#find element you want to click by XPATH
button = browser.find_element(By.XPATH, &#39;//*[@id=&quot;main-container&quot;]/div[5]/div[1]/div[2]/div[2]/div/div/div[2]/div[2]/div[1]/div/div[5]/a/span/span&#39;)
#click the button &quot;T20I&quot;
button.click()

Enjoy it!

答案2

得分: 0

数据通过JavaScript呈现。有一个API来获取数据。当有API可用时，请不要使用Selenium。

import requests
import pandas as pd

url = 'https://hs-consumer-api.espncricinfo.com/v1/pages/player/search'
payload = {
    'mode': 'BOTH',
    'page': '1',
    'records': '40',
    'filterActive': 'true',
    'filterTeamId': '6',
    'filterClassId': '3',
    'filterFormatLevel': 'ALL',
    'sort': 'ALPHA_ASC'
}

jsonData = requests.get(url, params=payload).json()

df = pd.DataFrame(jsonData['results'])

# 输出：前5行，共31行
print(df.head().to_string())

输出：前5行，共31行

请注意，输出中的一些内容包含HTML编码，因此可能需要进一步处理以进行显示。

英文:

Data is rendered through javascript. There is an api that pulls that. Don't use Selenium when there is an api.

import requests
import pandas as pd

url = &#39;https://hs-consumer-api.espncricinfo.com/v1/pages/player/search&#39;
payload = {
    &#39;mode&#39;: &#39;BOTH&#39;,
    &#39;page&#39;: &#39;1&#39;,
    &#39;records&#39;: &#39;40&#39;,
    &#39;filterActive&#39;: &#39;true&#39;,
    &#39;filterTeamId&#39;: &#39;6&#39;,
    &#39;filterClassId&#39;: &#39;3&#39;,
    &#39;filterFormatLevel&#39;: &#39;ALL&#39;,
    &#39;sort&#39;: &#39;ALPHA_ASC&#39;}

jsonData = requests.get(url, params=payload).json()

df = pd.DataFrame(jsonData[&#39;results&#39;])

Output: 1st 5 rows of 31 rows

print(df.head().to_string())
id  objectId            name             longName  mobileName       indexName     battingName    fieldingName                 slug                              imageUrl                              dateOfBirth dateOfDeath gender battingStyles bowlingStyles longBattingStyles        longBowlingStyles                                                                                                                                                                                                                                                                                                                      image  countryTeamId playerRoleTypeIds          playingRoles                                                                                                                                                                                                                                                                                                                                                                                                                                            headshotImage
0  101430   1125976  Arshdeep Singh       Arshdeep Singh              Arshdeep Singh  Arshdeep Singh  Arshdeep Singh       arshdeep-singh  /db/PICTURES/CMS/356700/356795.1.png    {&#39;year&#39;: 1999, &#39;month&#39;: 2, &#39;date&#39;: 5}        None      M         [lhb]         [lmf]   [left-hand bat]   [left-arm medium-fast]  {&#39;id&#39;: 356795, &#39;objectId&#39;: 1365005, &#39;slug&#39;: &#39;arshdeep-singh-player-portrait&#39;, &#39;url&#39;: &#39;/db/PICTURES/CMS/356700/356795.1.png&#39;, &#39;width&#39;: 160, &#39;height&#39;: 213, &#39;caption&#39;: &#39;Arshdeep Singh player portrait&#39;, &#39;longCaption&#39;: &#39;Arshdeep Singh player portrait&#39;, &#39;credit&#39;: &#39;Getty Images&#39;, &#39;photographer&#39;: None, &#39;peerUrls&#39;: None}              6               [4]              [bowler]        {&#39;id&#39;: 322178, &#39;objectId&#39;: 1264653, &#39;slug&#39;: &#39;arshdeep-singh-player-page-headshot-cutout-2021&#39;, &#39;url&#39;: &#39;/db/PICTURES/CMS/322100/322178.png&#39;, &#39;width&#39;: 600, &#39;height&#39;: 436, &#39;caption&#39;: &#39;Arshdeep Singh player page headshot cutout, 2021&#39;, &#39;longCaption&#39;: &#39;Arshdeep Singh player page headshot cutout, 2021&#39;, &#39;credit&#39;: None, &#39;photographer&#39;: None, &#39;peerUrls&#39;: {&#39;FILM&#39;: None, &#39;WIDE&#39;: None, &#39;SQUARE&#39;: &#39;/db/PICTURES/CMS/322100/322178.square.png&#39;}}
1   12894     26421        R Ashwin  Ravichandran Ashwin      Ashwin       Ashwin, R        R Ashwin          Ashwin  ravichandran-ashwin    /db/PICTURES/CMS/302300/302395.jpg   {&#39;year&#39;: 1986, &#39;month&#39;: 9, &#39;date&#39;: 17}        None      M         [rhb]          [ob]  [right-hand bat]     [right-arm offbreak]                               {&#39;id&#39;: 302395, &#39;objectId&#39;: 1220592, &#39;slug&#39;: &#39;r-ashwin-portrait&#39;, &#39;url&#39;: &#39;/db/PICTURES/CMS/302300/302395.jpg&#39;, &#39;width&#39;: 160, &#39;height&#39;: 200, &#39;caption&#39;: &#39;R Ashwin portrait&#39;, &#39;longCaption&#39;: &#39;R Ashwin portrait, April 2020&#39;, &#39;credit&#39;: &#39;Getty Images&#39;, &#39;photographer&#39;: None, &#39;peerUrls&#39;: None}              6              [11]  [bowling allrounder]                                                                                                                                           {&#39;id&#39;: 316521, &#39;objectId&#39;: 1251150, &#39;slug&#39;: &#39;r-ashwin-headshot&#39;, &#39;url&#39;: &#39;/db/PICTURES/CMS/316500/316521.png&#39;, &#39;width&#39;: 600, &#39;height&#39;: 436, &#39;caption&#39;: &#39;R Ashwin headshot&#39;, &#39;longCaption&#39;: &#39;R Ashwin headshot&#39;, &#39;credit&#39;: None, &#39;photographer&#39;: None, &#39;peerUrls&#39;: {&#39;FILM&#39;: None, &#39;WIDE&#39;: None, &#39;SQUARE&#39;: None}}
2   73507    694211      Avesh Khan           Avesh Khan  Avesh Khan      Avesh Khan      Avesh Khan      Avesh Khan           avesh-khan  /db/PICTURES/CMS/200000/200065.1.jpg  {&#39;year&#39;: 1996, &#39;month&#39;: 12, &#39;date&#39;: 13}        None      M         [rhb]         [rfm]  [right-hand bat]  [right-arm fast-medium]                             {&#39;id&#39;: 200065, &#39;objectId&#39;: 807641, &#39;slug&#39;: &#39;avesh-khan-portrait&#39;, &#39;url&#39;: &#39;/db/PICTURES/CMS/200000/200065.1.jpg&#39;, &#39;width&#39;: 160, &#39;height&#39;: 200, &#39;caption&#39;: &#39;Avesh Khan portrait&#39;, &#39;longCaption&#39;: &#39;Avesh Khan portrait, November 2014&#39;, &#39;credit&#39;: &#39;MPCA&#39;, &#39;photographer&#39;: None, &#39;peerUrls&#39;: None}              6               [4]              [bowler]                    {&#39;id&#39;: 322244, &#39;objectId&#39;: 1264747, &#39;slug&#39;: &#39;avesh-khan-player-page-headshot-cutout-2021&#39;, &#39;url&#39;: &#39;/db/PICTURES/CMS/322200/322244.png&#39;, &#39;width&#39;: 600, &#39;height&#39;: 436, &#39;caption&#39;: &#39;Avesh Khan player page headshot cutout, 2021&#39;, &#39;longCaption&#39;: &#39;Avesh Khan player page headshot cutout, 2021&#39;, &#39;credit&#39;: None, &#39;photographer&#39;: None, &#39;peerUrls&#39;: {&#39;FILM&#39;: None, &#39;WIDE&#39;: None, &#39;SQUARE&#39;: &#39;/db/PICTURES/CMS/322200/322244.square.png&#39;}}
3   70640    625383       JJ Bumrah       Jasprit Bumrah      Bumrah      Bumrah, JJ       JJ Bumrah          Bumrah       jasprit-bumrah  /db/PICTURES/CMS/356800/356849.1.png   {&#39;year&#39;: 1993, &#39;month&#39;: 12, &#39;date&#39;: 6}        None      M         [rhb]          [rf]  [right-hand bat]         [right-arm fast]                          {&#39;id&#39;: 356849, &#39;objectId&#39;: 1365132, &#39;slug&#39;: &#39;bumrah-player-portrait&#39;, &#39;url&#39;: &#39;/db/PICTURES/CMS/356800/356849.1.png&#39;, &#39;width&#39;: 160, &#39;height&#39;: 206, &#39;caption&#39;: &#39;Bumrah player portrait&#39;, &#39;longCaption&#39;: &#39;Bumrah player portrait&#39;, &#39;credit&#39;: &#39;Getty Images&#39;, &#39;photographer&#39;: None, &#39;peerUrls&#39;: None}              6               [4]              [bowler]        {&#39;id&#39;: 319940, &#39;objectId&#39;: 1260219, &#39;slug&#39;: &#39;jasprit-bumrah-player-page-headshot-cutout-2021&#39;, &#39;url&#39;: &#39;/db/PICTURES/CMS/319900/319940.png&#39;, &#39;width&#39;: 600, &#39;height&#39;: 436, &#39;caption&#39;: &#39;Jasprit Bumrah player page headshot cutout, 2021&#39;, &#39;longCaption&#39;: &#39;Jasprit Bumrah player page headshot cutout, 2021&#39;, &#39;credit&#39;: None, &#39;photographer&#39;: None, &#39;peerUrls&#39;: {&#39;FILM&#39;: None, &#39;WIDE&#39;: None, &#39;SQUARE&#39;: &#39;/db/PICTURES/CMS/319900/319940.square.png&#39;}}
4   61325    430246       YS Chahal     Yuzvendra Chahal      Chahal      Chahal, YS       YS Chahal          Chahal     yuzvendra-chahal    /db/PICTURES/CMS/312100/312155.png   {&#39;year&#39;: 1990, &#39;month&#39;: 7, &#39;date&#39;: 23}        None      M         [rhb]         [lbg]  [right-hand bat]        [legbreak googly]    {&#39;id&#39;: 312155, &#39;objectId&#39;: 1239214, &#39;slug&#39;: &#39;yuzvendra-chahal-portrait&#39;, &#39;url&#39;: &#39;/db/PICTURES/CMS/312100/312155.png&#39;, &#39;width&#39;: 160, &#39;height&#39;: 200, &#39;caption&#39;: &#39;Yuzvendra Chahal portrait&#39;, &#39;longCaption&#39;: &#39;Yuzvendra Chahal portrait, November 2020&#39;, &#39;credit&#39;: &#39;Getty Images&#39;, &#39;photographer&#39;: None, &#39;peerUrls&#39;: None}              6               [4]              [bowler]  {&#39;id&#39;: 319955, &#39;objectId&#39;: 1260243, &#39;slug&#39;: &#39;yuzvendra-chahal-player-page-headshot-cutout-2021&#39;, &#39;url&#39;: &#39;/db/PICTURES/CMS/319900/319955.png&#39;, &#39;width&#39;: 600, &#39;height&#39;: 436, &#39;caption&#39;: &#39;Yuzvendra Chahal player page headshot cutout, 2021&#39;, &#39;longCaption&#39;: &#39;Yuzvendra Chahal player page headshot cutout, 2021&#39;, &#39;credit&#39;: None, &#39;photographer&#39;: None, &#39;peerUrls&#39;: {&#39;FILM&#39;: None, &#39;WIDE&#39;: None, &#39;SQUARE&#39;: &#39;/db/PICTURES/CMS/319900/319955.square.png&#39;}}

通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。

无法在网络抓取中获取链接。

问题

答案1

答案2

在pandas中格式化一个包含两行的表格。

如何处理包含进度条的日志输出？

Python ColumnTransformer 设置 SettingWithCopyWarning

如何更新字段以设置默认值？

What's the correct way to type hint an empty list as a literal in python?

如何在Highcharts Gantt中更改本地化的星期名称

如何在同一个流中使用多个过滤器和映射函数？

如何使用Map/Set来将代码优化到O(n)？

.NET MAUI Android在GitHub Actions上构建失败，错误代码为1。

如何在Playwright视觉比较中屏蔽多个定位器？

在C++中，可以使用可变模板参数来检索类型的内部类型。

selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: stale element not found

Creating and opening a URL to log in to Website via Basic Auth with Robot Framework/Selenium (Python)

AG Grid 在上下文菜单中以大文本形式打开

发表评论