英文:
scrape website through it's api using request
问题
我想从这个网站上抓取实时比赛数据:https://egamersworld.com/matches
我尝试使用这个API:https://api.egamersworld.com/matches?lang=en
但没有返回任何数据:
import requests
session = requests.Session()
url = "https://api.egamersworld.com/matches?lang=en"
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0', "referer": "https://egamersworld.com/"}
r = session.get(url, timeout=30, headers=headers)
print(r.status_code) #200
r.json() #{"list":[]}
r.status_code
返回 200,但 r.json()
什么都没有返回。
我该如何使用这个API获取数据?
英文:
I want to scrape live matches data from this website: https://egamersworld.com/matches
I tried using the api: https://api.egamersworld.com/matches?lang=en
but ruturn nothing:
import requests
session = requests.Session()
url = "https://api.egamersworld.com/matches?lang=en"
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0', "referer": "https://egamersworld.com/"}
r = session.get(url, timeout=30, headers=headers)
print(r.status_code) #200
r.json() #{"list":[]}
r.status_code
return 200, but the r.json()
return nothing.
How I can get the data using this api ?
答案1
得分: 2
这是您提供的代码和结果:
我无法看到从该页面访问API端点,但是有几个不同的端点,这是从其中一个获取数据的一种方法:
import requests
from bs4 import BeautifulSoup as bs
import pandas as pd
headers = {
'x-customheader': '435b8c763ab097da9947a385cfe127b0',
'origin': 'https://egamersworld.com',
'referer': 'https://egamersworld.com/',
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'
}
url = 'https://api.egamersworld.com/live_events?lang=en'
r = requests.get(url, headers=headers)
df = pd.json_normalize(r.json())
print(df)
终端中的结果:
_id tags title game slug prize start_date finish_date prize_num image is_live
0 Nkqg6l-SR [{'title': 'LPL', 'slug': 'lpl'}] LPL Summer 2023 lol lpl-2023-summer-Nkqg6l-SR $597,366 2023-05-28T23:00:00.000Z 2023-08-07T23:00:00.000Z 597366 /uploads/tournaments/lpl-summer-20231685005813... True
1 4kxt3zcXVa [{'title': 'ESL', 'slug': 'esl'}, {'title': 'I... IEM Dallas 2023 counterstrike iem-dallas-2023-4kxt3zcXVa $250,000 2023-05-29T00:00:00.000Z 2023-06-04T00:00:00.000Z 250000 /uploads/tournaments/intel-extreme-masters-dal... True
2 NyVSQFNp3 [{'title': 'DPC', 'slug': 'dpc'}] DPC 2023 Tour 3: CN Division I (Upper) dota2 dpc-2022-2023-tour-3-season-3-china-division-i... $205,000 2023-05-16T00:00:00.000Z 2023-06-04T00:00:00.000Z 205000 /uploads/tournaments/dpc-2022-2023-summer-tour... True
3 V1u4WtNT2 [{'title': 'DPC', 'slug': 'dpc'}] DPC 2023 Tour 3: NA Division I (Upper) dota2 dpc-2022-2023-tour-3-season-3-na-division-i-up... $205,000 2023-05-15T00:00:00.000Z 2023-06-04T00:00:00.000Z 205000 /uploads/tournaments/dpc-2022-2023-summer-tour... True
4 VJvpbYN6h [{'title': 'DPC', 'slug': 'dpc'}] DPC 2023 Tour 3: SA Division I (Upper) dota2 dpc-2022-2023-tour-3-season-3-sa-division-i-up... $205,000 2023-05-15T00:00:00.000Z 2023-06-04T00:00:00.000Z 205000 /uploads/tournaments/dpc-2022-2023-summer-tour... True
5 4ymHMF4Tn [{'title': 'DPC', 'slug': 'dpc'}] DPC 2023 Tour 3: WEU Division I (Upper) dota2 dpc-2022-2023-tour-3-season-3-weu-division-i-u... $205,000 2023-05-15T00:00:00.000Z 2023-06-04T00:00:00.000Z 205000 /uploads/tournaments/dpc-2022-2023-summer-tour... True
6 N1hQNtNp2 [{'title': 'DPC', 'slug': 'dpc'}] DPC 2023 Tour 3: SEA Division I (Upper) dota2 dpc-2022-2023-tour-3-season-3-sea-division-i-u... $205,000 2023-05-15T00:00:00.000Z 2023-06-04T00:00:00.000Z 205000 /uploads/tournaments/dpc-2022-2023-summer-tour... True
7 4JtpGFET2 [{'title': 'DPC', 'slug': 'dpc'}] DPC 2023 Tour 3: EEU Division I (Upper) dota2 dpc-2022-2023-tour-3-season-3-eeu-division-i-u... $205,000 2023-05-14T00:00:00.000Z 2023-06-04T00:00:00.000Z 205000 /uploads/tournaments/dpc-2022-2023-summer-tour... True
8 41twsuwVC [{'title': 'Champion of Champions Tour', 'slug... CCT 2023 Online Finals 1 counterstrike cct-2023-online-finals-1-41twsuwVC $200,000 2023-05-23T00:00:00.000Z 2023-05-30T00:00:00.000Z 200000 /uploads/tournaments/cct-2023-online-finals-11... True
9 N174ae0_0 [] Europe League 2023 - Stage 2 rainbowsix europe-league-2023-stage-2-N174ae0_0 $191,049 2023-05-01T23:00:00.000Z 2023-05-30T23:00:00.000Z 191049 /uploads/tournaments
<details>
<summary>英文:</summary>
I could not see that API endpoint being accessed from that page, however, there are a couple different ones, and here is one way of getting the data from one of them:
import requests
from bs4 import BeautifulSoup as bs
import pandas as pd
headers = {
'x-customheader': '435b8c763ab097da9947a385cfe127b0',
'origin': 'https://egamersworld.com',
'referer': 'https://egamersworld.com/',
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'
}
url = 'https://api.egamersworld.com/live_events?lang=en'
r = requests.get(url, headers=headers)
df = pd.json_normalize(r.json())
print(df)
Result in terminal:
_id tags title game slug prize start_date finish_date prize_num image is_live
0 Nkqg6l-SR [{'title': 'LPL', 'slug': 'lpl'}] LPL Summer 2023 lol lpl-2023-summer-Nkqg6l-SR $597,366 2023-05-28T23:00:00.000Z 2023-08-07T23:00:00.000Z 597366 /uploads/tournaments/lpl-summer-20231685005813... True
1 4kxt3zcXVa [{'title': 'ESL', 'slug': 'esl'}, {'title': 'I... IEM Dallas 2023 counterstrike iem-dallas-2023-4kxt3zcXVa $250,000 2023-05-29T00:00:00.000Z 2023-06-04T00:00:00.000Z 250000 /uploads/tournaments/intel-extreme-masters-dal... True
2 NyVSQFNp3 [{'title': 'DPC', 'slug': 'dpc'}] DPC 2023 Tour 3: CN Division I (Upper) dota2 dpc-2022-2023-tour-3-season-3-china-division-i... $205,000 2023-05-16T00:00:00.000Z 2023-06-04T00:00:00.000Z 205000 /uploads/tournaments/dpc-2022-2023-summer-tour... True
3 V1u4WtNT2 [{'title': 'DPC', 'slug': 'dpc'}] DPC 2023 Tour 3: NA Division I (Upper) dota2 dpc-2022-2023-tour-3-season-3-na-division-i-up... $205,000 2023-05-15T00:00:00.000Z 2023-06-04T00:00:00.000Z 205000 /uploads/tournaments/dpc-2022-2023-summer-tour... True
4 VJvpbYN6h [{'title': 'DPC', 'slug': 'dpc'}] DPC 2023 Tour 3: SA Division I (Upper) dota2 dpc-2022-2023-tour-3-season-3-sa-division-i-up... $205,000 2023-05-15T00:00:00.000Z 2023-06-04T00:00:00.000Z 205000 /uploads/tournaments/dpc-2022-2023-summer-tour... True
5 4ymHMF4Tn [{'title': 'DPC', 'slug': 'dpc'}] DPC 2023 Tour 3: WEU Division I (Upper) dota2 dpc-2022-2023-tour-3-season-3-weu-division-i-u... $205,000 2023-05-15T00:00:00.000Z 2023-06-04T00:00:00.000Z 205000 /uploads/tournaments/dpc-2022-2023-summer-tour... True
6 N1hQNtNp2 [{'title': 'DPC', 'slug': 'dpc'}] DPC 2023 Tour 3: SEA Division I (Upper) dota2 dpc-2022-2023-tour-3-season-3-sea-division-i-u... $205,000 2023-05-15T00:00:00.000Z 2023-06-04T00:00:00.000Z 205000 /uploads/tournaments/dpc-2022-2023-summer-tour... True
7 4JtpGFET2 [{'title': 'DPC', 'slug': 'dpc'}] DPC 2023 Tour 3: EEU Division I (Upper) dota2 dpc-2022-2023-tour-3-season-3-eeu-division-i-u... $205,000 2023-05-14T00:00:00.000Z 2023-06-04T00:00:00.000Z 205000 /uploads/tournaments/dpc-2022-2023-summer-tour... True
8 41twsuwVC [{'title': 'Champion of Champions Tour', 'slug... CCT 2023 Online Finals 1 counterstrike cct-2023-online-finals-1-41twsuwVC $200,000 2023-05-23T00:00:00.000Z 2023-05-30T00:00:00.000Z 200000 /uploads/tournaments/cct-2023-online-finals-11... True
9 N174ae0_0 [] Europe League 2023 - Stage 2 rainbowsix europe-league-2023-stage-2-N174ae0_0 $191,049 2023-05-01T23:00:00.000Z 2023-05-30T23:00:00.000Z 191049 /uploads/tournaments/europe-league-2023-stage-... True
Page is also being hydrated with info from a couple of websockets connections. Here is one way of getting that streaming information:
import websocket
def on_message(ws, message):
print(message)
def on_close(ws, close_status_code,close_msg):
print("closed connection")
socket = 'wss://ws.egamersworld.com/socket.io/?EIO=3&transport=websocket'
ws = websocket.WebSocketApp(socket,on_message= on_message, on_close=on_close)
ws.run_forever()
Response in terminal:
0{"sid":"fVEBxmYMAFO-RAPoCQtq","upgrades":[],"pingInterval":25000,"pingTimeout":20000}
40
42["cs",{"NkKs8f2rA":{"mapStat":{"homeTeam":[{"steamId":"1:1:73849600","dbId":19996,"name":"Jelo","score":17,"deaths":9,"assists":0,"alive":false,"money":56500,"damagePrRound":802,"hp":0,"kevlar":false,"helmet":false,"nick":"Jelo","hasDefusekit":false,"advancedStats":{"kast":1,"entryKills":0,"entryDeaths":0,"multiKillRounds":1,"oneOnXWins":0,"flashAssists":0}},{"steamId":"1:1:571082935","dbId":22703,"name":"moonwalk","score":8,"deaths":10,"assists":0,"alive":false,"money":43200,"damagePrRound":369,"hp":0,"kevlar":false,"helmet":false,"nick":"moonwalk","hasDefusekit":false,"advancedStats":{"kast":1,"entryKills":0,"entryDeaths":0,"multiKillRounds":1,"oneOnXWins":0,"flashAssists":.....
</details>
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论