英文:
Web Scraping Got Empty Array Values
问题
你好，我正在尝试抓取一个股票网站的信息，以获取股票按部门分类的信息，网站链接如下：
在终端中，我运行了以下命令：
scrapy shell "https://nepsealpha.com/"
response.xpath("//table[@id='fixTable']//tbody//tr")
但是我得到的输出是一个空列表 []。
我觉得内容是通过 JavaScript 渲染的。有没有办法在不使用 Selenium 的情况下实现？
英文:
Hello All I was trying to scrape a stock website to get stock sector wise info in this website.
I just hit scrapy shell in terminal if the data is acheivable for this table
In the terminal this was my command
after I ran scrapy shell "https://nepsealpha.com/"
response.xpath("//table[@id='fixTable']//tbody//tr")
but the output I get is empty list = []
I feel content is being rendered with javascript
anyway I can do without use of selenium?
答案1
得分: 1
以下是翻译后的内容，不包括代码部分：
"The data you're after comes from an API endpoint."
你需要的数据来自一个API端点。
"You can get it and then massage it back to the form of a table or use only parts of it."
你可以获取它，然后将其转换成表格形式或仅使用其中的部分。
"Here's how:"
以下是如何操作的方法：
"Output:"
输出：
英文:
The data you're after comes from an API endpoint.
You can get it and then massage it back to the form of a table or use only parts of it.
Here's how:
import requests
import pandas as pd
api_endpoint = "https://nepsealpha.com/api/smx9841/dashboard_board"
payload = {
"_token": "K5fwARzoE7j49mIE5hdeZUqeoYgQGUXnsUeS7SG1"
}
headers = {
"Accept": "application/json",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
"X-Requested-With": "XMLHttpRequest",
}
response = requests.request("POST", api_endpoint, headers=headers, data=payload)
data = response.json()["home_table"]
df = pd.json_normalize(data)
print(df)
Output:
id index_name ... indexvalue.percent_change indexvalue.turn_over_value
0 41538 NEPSE ... 0.74 1204409411.62
1 41541 BANKING ... 0.51 181973963.3
2 41548 TRADING ... 0.75 121077893
3 41550 HOTELS ... 1.05 9852264.4
4 41547 DEVBANK ... 1.16 52379584.2
5 41543 HYDROPOWER ... 1.67 317296705.3
6 41546 FINANCE ... 1.18 39390111.3
7 41542 NONLIFEINSU ... 0.97 42936919.2
8 41544 MANUFACTURE ... -1.06 126684476.5
9 41549 OTHERS ... 0.71 21071125.9
10 41540 MICROFINANCE ... 0.5 171446649.5
11 41545 LIFEINSU ... 0.46 52938619.8
12 41551 INVESTMENT ... 1.11 44639401.2
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论