英文:
Web Scraping Got Empty Array Values
问题
你好，我正在尝试抓取一个股票网站的信息，以获取股票按部门分类的信息，网站链接如下：
在终端中，我运行了以下命令：
scrapy shell "https://nepsealpha.com/"response.xpath("//table[@id='fixTable']//tbody//tr")
但是我得到的输出是一个空列表 []。
我觉得内容是通过 JavaScript 渲染的。有没有办法在不使用 Selenium 的情况下实现？
英文:
Hello All I was trying to scrape a stock website to get stock sector wise info in this website.
I just hit scrapy shell in terminal if the data is acheivable for this table
In the terminal this was my command
after I ran scrapy shell "https://nepsealpha.com/"
response.xpath("//table[@id='fixTable']//tbody//tr")
but the output I get is empty list = []
I feel content is being rendered with javascript
anyway I can do without use of selenium?
答案1
得分: 1
以下是翻译后的内容，不包括代码部分：
"The data you're after comes from an API endpoint."
你需要的数据来自一个API端点。
"You can get it and then massage it back to the form of a table or use only parts of it."
你可以获取它，然后将其转换成表格形式或仅使用其中的部分。
"Here's how:"
以下是如何操作的方法：
"Output:"
输出：
英文:
The data you're after comes from an API endpoint.
You can get it and then massage it back to the form of a table or use only parts of it.
Here's how:
import requestsimport pandas as pdapi_endpoint = "https://nepsealpha.com/api/smx9841/dashboard_board"payload = {"_token": "K5fwARzoE7j49mIE5hdeZUqeoYgQGUXnsUeS7SG1"}headers = {"Accept": "application/json","User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36","X-Requested-With": "XMLHttpRequest",}response = requests.request("POST", api_endpoint, headers=headers, data=payload)data = response.json()["home_table"]df = pd.json_normalize(data)print(df)
Output:
id index_name ... indexvalue.percent_change indexvalue.turn_over_value0 41538 NEPSE ... 0.74 1204409411.621 41541 BANKING ... 0.51 181973963.32 41548 TRADING ... 0.75 1210778933 41550 HOTELS ... 1.05 9852264.44 41547 DEVBANK ... 1.16 52379584.25 41543 HYDROPOWER ... 1.67 317296705.36 41546 FINANCE ... 1.18 39390111.37 41542 NONLIFEINSU ... 0.97 42936919.28 41544 MANUFACTURE ... -1.06 126684476.59 41549 OTHERS ... 0.71 21071125.910 41540 MICROFINANCE ... 0.5 171446649.511 41545 LIFEINSU ... 0.46 52938619.812 41551 INVESTMENT ... 1.11 44639401.2
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论