英文:
How to scrape data from new (2023) PGA Tour website in Python
问题
PGA巡回赛更新了他们的网站（截止到2023年2月7日），完全破坏了我用于数据抓取的方式。以前它有一个“隐藏”的URL，可以通过查看开发者工具中的网络选项卡来发现。然后我可以使用Python中的Requests库来获取数据表格的内容。
关于以前的工作原理，可以参考我的先前帖子的回应：https://stackoverflow.com/questions/70141129/what-to-do-when-python-requests-get-gets-a-browser-error-from-the-website。
现在似乎所有的数据都被隐藏，无法像以前那样通过URL访问。我希望有更熟练于网络抓取技巧的人可以指导我如何做到之前的链接所做的事情：
- 对于任何锦标赛，能够从任何年份/赛季获取锦标赛历史数据（新网站的示例：https://www.pgatour.com/tournaments/2023/fortinet-championship/R2023464/past-results）。
- 对于任何统计数据，能够从任何年份/赛季获取统计数据（新网站的示例：https://www.pgatour.com/stats/detail/02674）。
初始尝试显示可以从当前页面获取表格（但不包括以前的年份），而且提取的一些数据不是文本，而是格式化代码。
import requests
import pandas as pd
tournament_url = 'https://www.pgatour.com/tournaments/2023/fortinet-championship/R2023464/past-results'
headers = {'User-Agent': 'Mozilla/5.0'}
t = pd.read_html(requests.get(tournament_url, headers=headers).text)[0]
t
编辑：我从下面的回应中看到这是使用GraphQL。我发现如果你点击Network选项卡中的graphql行，然后查看Payload选项卡，你会看到这些变量：
{
"tournamentPastResultsId": "R2023464",
"year": 2022
}
这些似乎提供了有关问题的锦标赛ID和年份，理论上你可以简单地更新这些值并进行查询，选择任何锦标赛，任何年份。将这些集成到抓取中会模仿之前的做法。但我不确定如何做到这一点，我将进一步研究Selenium。希望它能够以某种方式传递这些变量。
编辑 2：
对于统计数据（例如https://www.pgatour.com/stats/detail/02567），我能够修改代码以获取适当的表格（参见下文）。
以下是供参考的代码（感谢@Jurakin！）：
import pandas as pd
from numpy import NaN
import requests
# 在请求头中似乎需要一个常量令牌（'x-api-key'）
X_API_KEY = "da2-gsrx5bibzbb4njvhl7t37wqyl4"
YEAR = 2022 # 统计季节
STAT_ID = "02567" # 统计ID
# 准备负载
payload = {
"operationName": "StatDetails",
"variables": {
"tourCode": "R",
"statId": STAT_ID,
"year": YEAR,
"eventQuery": None
},
"query": "query StatDetails($tourCode: TourCode!, $statId: String!, $year: Int, $eventQuery: StatDetailEventQuery) {\n statDetails(\n tourCode: $tourCode\n statId: $statId\n year: $year\n eventQuery: $eventQuery\n ) {\n tourCode\n year\n displaySeason\n statId\n statType\n tournamentPills {\n tournamentId\n displayName\n }\n yearPills {\n year\n displaySeason\n }\n statTitle\n statDescription\n tourAvg\n lastProcessed\n statHeaders\n statCategories {\n category\n displayName\n subCategories {\n displayName\n stats {\n statId\n statTitle\n }\n }\n }\n rows {\n ... on StatDetailsPlayer {\n __typename\n playerId\n playerName\n country\n countryFlag\n rank\n rankDiff\n rankChangeTendency\n stats {\n statName\n statValue\n color\n }\n }\n ... on StatDetailTourAvg {\n __typename\n displayName\n value\n }\n }\n }\n}"
}
# 发送请求
page = requests.post("https://orchestrator.pgatour.com/graphql", json=payload, headers={"x-api-key": X_API_KEY})
# 检查状态码
page.raise_for_status()
# 获取数据
data = page.json()["data"]["statDetails"]["rows"]
# 格式化为网页中的表格
table = map(lambda item: {
"rank": item["rank"],
"player": item["playerName"],
"average": item["stats"][0]["statValue"],
}, data)
# 转换为数据框
s = pd.DataFrame(table)
s
编辑 3 - 后续问题：
上面的回应适用于统计ID中包含5个字符的统计数据。但对于统计ID中只有3个字符的统计数据（例如https://www.pgatour.com/stats/detail/156），它可以正确提取数据，但在表格映射部分失败，尽管我可以看到它们的响应格式是相同的，所以我不明白为什么这不起作用，而其他的却起作用。
import pandas as pd
from numpy import NaN
import requests
# 在请求头中似乎需要一个常量令牌（'x-api-key'）
X_API_KEY = "da2-gsrx5bibzbb4njvhl7t37wqyl4"
YEAR = 2022 # 统计季节
# STAT_ID = "02567" # 统计ID SGOTT
STAT_ID = "156" # 平鸟平均值 -
<details>
<summary>英文:</summary>
The PGA tour updated their website (as of Feb 7, 2023) that completely broke the way I was scraping it for data. It used to have a "hidden" URL that you could uncover by looking at the Network tab in Developer tools. Then I could use that "hidden" URL with Requests in Python to pull the data tables.
For background on how it used to work, see the response from this previous post of mine: https://stackoverflow.com/questions/70141129/what-to-do-when-python-requests-get-gets-a-browser-error-from-the-website.
Now it seems like all the data is obscured away from accessing it via a URL like before. I'm hoping someone more fluent in web-scraping tricks can point me in the right direction to do what that previous link did:
1. For any tournament, be able to pull tournament history from any year/season. (Example from the new site: https://www.pgatour.com/tournaments/2023/fortinet-championship/R2023464/past-results)
2. For any statistic, be able to pull stats from any year/season. (Example from the new site: https://www.pgatour.com/stats/detail/02674)
Initial try shows ability to pull the table off the current page (but not previous years) and some of the data that is pulled is not text, but rather formatting code.
import requests
import pandas as pd
tournament_url = 'https://www.pgatour.com/tournaments/2023/fortinet-championship/R2023464/past-results'
headers = {'User-Agent': 'Mozilla/5.0'}
t = pd.read_html(requests.get(tournament_url, headers=headers).text)[0]
t
**EDIT**: I see from a response below that this is using GraphQL. I discovered that if you click on the `graphql` line in the Network tab and then look at the Payload tab, you'll see these variables: `{
"tournamentPastResultsId": "R2023464",
"year": 2022
}`.
These seem to give the tournament ID and year in question so that in theory you can simply update these values in a query and pick any tournament, any year. Integrating these into the scraping would mimic how it was done prior. I'm not sure how to do that though. I'll do some more research on Selenium. Hopefully it is able to pass through these variables somehow.
**EDIT 2**:
The answer was given below for how to do this for the tournament data. For Stats data (e.g. https://www.pgatour.com/stats/detail/02567), I was able to modify the code to get the appropriate table (see below).
Posted below for reference (thanks to @Jurakin!)
import pandas as pd
from numpy import NaN
import requests
# in the requests header seems to be a constant token ('x-api-key') that is needed
X_API_KEY = "da2-gsrx5bibzbb4njvhl7t37wqyl4"
YEAR = 2022 # Stats Season
STAT_ID = "02567" # Stat ID
# prepare the payload
payload = {
"operationName": "StatDetails",
"variables": {
"tourCode": "R",
"statId": STAT_ID,
"year": YEAR,
"eventQuery": None
},
"query": "query StatDetails($tourCode: TourCode!, $statId: String!, $year: Int, $eventQuery: StatDetailEventQuery) {\n statDetails(\n tourCode: $tourCode\n statId: $statId\n year: $year\n eventQuery: $eventQuery\n ) {\n tourCode\n year\n displaySeason\n statId\n statType\n tournamentPills {\n tournamentId\n displayName\n }\n yearPills {\n year\n displaySeason\n }\n statTitle\n statDescription\n tourAvg\n lastProcessed\n statHeaders\n statCategories {\n category\n displayName\n subCategories {\n displayName\n stats {\n statId\n statTitle\n }\n }\n }\n rows {\n ... on StatDetailsPlayer {\n __typename\n playerId\n playerName\n country\n countryFlag\n rank\n rankDiff\n rankChangeTendency\n stats {\n statName\n statValue\n color\n }\n }\n ... on StatDetailTourAvg {\n __typename\n displayName\n value\n }\n }\n }\n}"
}
# post the request
page = requests.post("https://orchestrator.pgatour.com/graphql", json=payload, headers={"x-api-key": X_API_KEY})
# check for status code
page.raise_for_status()
# get the data
data = page.json()["data"]["statDetails"]["rows"]
# print(data)
# format to a table that is in the webpage
table = map(lambda item: {
"rank": item["rank"],
"player": item["playerName"],
"average": item["stats"][0]["statValue"],
}, data)
# convert the dataframe
s = pd.DataFrame(table)
s
**EDIT 3 - FOLLOW UP QUESTION**:
The answer above for stats work for stats with 5 characters in the Stat ID. But there are others with 3 characters (e.g. https://www.pgatour.com/stats/detail/156) that do grab the data correctly, but fail in the table mapping portion despite what I can tell are identical Response formats, so I am at a loss why this does not work and the other does.
import pandas as pd
from numpy import NaN
import requests
# in the requests header seems to be a constant token ('x-api-key') that is needed
X_API_KEY = "da2-gsrx5bibzbb4njvhl7t37wqyl4"
YEAR = 2022 # Stats Season
# STAT_ID = "02567" # Stat ID SGOTT
STAT_ID = "156" # Birdie Average - doesn't work for stats that only have three numbers and I can't figure out why
# prepare the payload
payload = {
"operationName": "StatDetails",
"variables": {
"tourCode": "R",
"statId": STAT_ID,
"year": YEAR,
"eventQuery": None
},
"query": "query StatDetails($tourCode: TourCode!, $statId: String!, $year: Int, $eventQuery: StatDetailEventQuery) {\n statDetails(\n tourCode: $tourCode\n statId: $statId\n year: $year\n eventQuery: $eventQuery\n ) {\n tourCode\n year\n displaySeason\n statId\n statType\n tournamentPills {\n tournamentId\n displayName\n }\n yearPills {\n year\n displaySeason\n }\n statTitle\n statDescription\n tourAvg\n lastProcessed\n statHeaders\n statCategories {\n category\n displayName\n subCategories {\n displayName\n stats {\n statId\n statTitle\n }\n }\n }\n rows {\n ... on StatDetailsPlayer {\n __typename\n playerId\n playerName\n country\n countryFlag\n rank\n rankDiff\n rankChangeTendency\n stats {\n statName\n statValue\n color\n }\n }\n ... on StatDetailTourAvg {\n __typename\n displayName\n value\n }\n }\n }\n}"
}
# post the request
page = requests.post("https://orchestrator.pgatour.com/graphql", json=payload, headers={"x-api-key": X_API_KEY})
# check for status code
page.raise_for_status()
# get the data
data = page.json()["data"]["statDetails"]["rows"]
print(data)
# format to a table that is in the webpage
table = map(lambda item: {
"RANK": item["rank"],
"PLAYER": item["playerName"],
"AVERAGE": item["stats"][0]["statValue"],
}, data)
# convert the dataframe
s = pd.DataFrame(table)
s
</details>
# 答案1
**得分**: 2
I will provide translations for the code portions you provided:
```python
from selenium import webdriver
from selenium.webdriver.common.by import By
import pandas as pd
driver = webdriver.Chrome()
# load page
driver.get("https://www.pgatour.com/tournaments/2023/fortinet-championship/R2023464/past-results")
# get table
table = driver.find_element(By.CSS_SELECTOR, "table.chakra-table")
assert table, "table not found"
# remove empty rows
driver.execute_script("""arguments[0].querySelectorAll("td.css-1au52ex").forEach((e) => e.parentElement.remove())""", table)
# get html of the table
table_html = table.get_attribute("outerHTML")
# quit selenium
driver.quit()
df = pd.read_html(table_html)[0]
print(df)
Outputs:
Pos Player R1 R2 R3 R4 To Par FedExCup Pts Official Money
0 1 Max Homa -7 -5 E -4 -16 500.0 $1,440,000
1 2 Danny Willett -4 -8 E -3 -15 300.0 $872,000
2 3 Taylor Montgomery -4 -1 E -8 -13 190.0 $552,000
3 T4 Justin Lower -9 -1 -3 +1 -12 122.5 $360,000
4 T4 Byeong Hun An -6 -4 -1 -1 -12 122.5 $360,000
.. ... ... .. ... ... ... ... ... ...
151 CUT Doc Redman +2 +6 NaN NaN +8 0.0 $0
152 CUT Kyle Stanley +6 +2 NaN NaN +8 0.0 $0
153 CUT Jim Herman -1 +10 NaN NaN +9 0.0 $0
154 CUT Taylor Lowe +9 +8 NaN NaN +17 0.0 $0
155 W/D Brandon Matthews - NaN NaN NaN E 0.0 $0
[156 rows x 9 columns]
Let me know if you need a translation for the second part of your code as well.
英文:
As you can see in devtools, the page uses graphql. graphql is a bit complicated for me and would take a long time to deobfuscate and understand the code, so I used selenium4 to run the javascript and build the table.
from selenium import webdriver
from selenium.webdriver.common.by import By
import pandas as pd
driver = webdriver.Chrome()
# load page
driver.get("https://www.pgatour.com/tournaments/2023/fortinet-championship/R2023464/past-results")
# get table
table = driver.find_element(By.CSS_SELECTOR, "table.chakra-table")
assert table, "table not found"
# remove empty rows
driver.execute_script("""arguments[0].querySelectorAll("td.css-1au52ex").forEach((e) => e.parentElement.remove())""", table)
# get html of the table
table_html = table.get_attribute("outerHTML")
# quit selenium
driver.quit()
df = pd.read_html(table_html)[0]
print(df)
Outputs:
Pos Player R1 R2 R3 R4 To Par FedExCup Pts Official Money
0 1 Max Homa -7 -5 E -4 -16 500.0 $1,440,000
1 2 Danny Willett -4 -8 E -3 -15 300.0 $872,000
2 3 Taylor Montgomery -4 -1 E -8 -13 190.0 $552,000
3 T4 Justin Lower -9 -1 -3 +1 -12 122.5 $360,000
4 T4 Byeong Hun An -6 -4 -1 -1 -12 122.5 $360,000
.. ... ... .. ... ... ... ... ... ...
151 CUT Doc Redman +2 +6 NaN NaN +8 0.0 $0
152 CUT Kyle Stanley +6 +2 NaN NaN +8 0.0 $0
153 CUT Jim Herman -1 +10 NaN NaN +9 0.0 $0
154 CUT Taylor Lowe +9 +8 NaN NaN +17 0.0 $0
155 W/D Brandon Matthews - NaN NaN NaN E 0.0 $0
[156 rows x 9 columns]
EDIT:
I created script that uses graphql api to fetch the data as you told me in the comments.
import pandas as pd
from numpy import NaN
import requests
# in the requests header seems to be a constant token
X_API_KEY = "da2-gsrx5bibzbb4njvhl7t37wqyl4"
YEAR = 2023
PAST_RESULTS_ID = "R2023464"
# prepare the payload
payload = {
"operationName": "TournamentPastResults",
"variables": {
"tournamentPastResultsId": PAST_RESULTS_ID,
"year": YEAR
},
"query": "query TournamentPastResults($tournamentPastResultsId: ID!, $year: Int) {\n tournamentPastResults(id: $tournamentPastResultsId, year: $year) {\n id\n players {\n id\n position\n player {\n id\n firstName\n lastName\n shortName\n displayName\n abbreviations\n abbreviationsAccessibilityText\n amateur\n country\n countryFlag\n lineColor\n }\n rounds {\n score\n parRelativeScore\n }\n additionalData\n total\n parRelativeScore\n }\n rounds\n additionalDataHeaders\n availableSeasons {\n year\n displaySeason\n }\n winner {\n id\n firstName\n lastName\n totalStrokes\n totalScore\n countryFlag\n countryName\n purse\n points\n }\n }\n}"
}
# post the request
page = requests.post("https://orchestrator.pgatour.com/graphql", json=payload, headers={"x-api-key": X_API_KEY})
# check for status code
page.raise_for_status()
# get the data
data = page.json()["data"]["tournamentPastResults"]["players"]
# format to a table that is in the webpage
table = map(lambda item: {
"pos": item["position"],
"player": item["player"]["displayName"],
"r1": item["rounds"][0]["parRelativeScore"] if len(item["rounds"]) > 0 else NaN,
"r2": item["rounds"][1]["parRelativeScore"] if len(item["rounds"]) > 1 else NaN,
"r3": item["rounds"][2]["parRelativeScore"] if len(item["rounds"]) > 2 else NaN,
"r4": item["rounds"][3]["parRelativeScore"] if len(item["rounds"]) > 3 else NaN,
"to par": item["parRelativeScore"],
"fedexcup pts": item["additionalData"][0],
"official money": item["additionalData"][1],
}, data)
# convert the dataframe
df = pd.DataFrame(table)
print(df)
EDIT 3:
The code raises a KeyError: 'rank' error because the item does not have a rank attribute. I used the following code to get an invalid item:
# get the data
data = page.json()["data"]["statDetails"]["rows"]
for item in data:
if "rank" not in item:
print(item)
# Outputs:
# {"__typename": "StatDetailTourAvg", "displayName": "Tour Average", "value": "3.64"},
As you can see, his __typename is different from all the others. I found two solutions:
Solution A
Filter out items that's __typename is not equal to StatDetailsPlayer:
...
# get the data
data = page.json()["data"]["statDetails"]["rows"]
# print(data)
# filter out items, thats __typename is not "StatDetailsPlayer" like
# {"__typename": "StatDetailTourAvg", "displayName": "Tour Average", "value": "3.64"}
data = filter(lambda item: item.get("__typename", NaN) == "StatDetailsPlayer", data)
# format to a table that is in the webpage
table = map(lambda item: {
"RANK": item["rank"],
"PLAYER": item["playerName"],
"AVERAGE": item["stats"][0]["statValue"],
}, data)
# convert the dataframe
s = pd.DataFrame(table)
print(s)
Solution B
Attempts to retrieve attributes from the object if possible, otherwise returns NaN.
...
def get(obj: object, keys: list, default=NaN):
"""
obj = {"a": {"b": {"c": [0, 1, 2, 3]}}}
keys = ["a", "b", "c", 0]
# returns 0
out = get(obj, keys, default=NaN)
# return NaN
out = get(obj, ["a", "c"])
"""
for key in keys:
try:
obj = obj[key]
except KeyError:
return default
return obj
# format to a table that is in the webpage
table = map(lambda item: {
"RANK": item.get("rank", NaN), # NaN is default (using buit-in function)
"PLAYER": item.get("playerName", NaN),
"AVERAGE": get(item, ["stats", 0, "statValue"] default=NaN), # my function (built-in function does not support multiple keys)
}, data)
# convert the dataframe
s = pd.DataFrame(table)
print(s)
Difference
Solution A does not contain invalid row, Solution B does.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论