Beautiful Soup没有找到文件中存在的HTML。

huangapple go评论76阅读模式
英文:

Beautiful Soup not finding HTML that exists with file

问题

我正在查看Netflix的招聘页面: https://jobs.netflix.com/search

url = "https://jobs.netflix.com/search"

response=requests.get(url)
response = response.content
soup= BeautifulSoup(response, 'html.parser')

jobList = soup.find_all("div", {"class" : "css-160787d exb5qdx0"})

最后的方法返回了None,我不明白为什么。我已经确认文件已正确读入,因此不明白原因。

我已尝试使用find和find_all,但都没有成功。我也研究了使用selenium,但网站似乎不是动态的。

英文:

I'm looking at the Netflix jobs page: https://jobs.netflix.com/search

url = "https://jobs.netflix.com/search"

response=requests.get(url)
response = response.content
soup= BeautifulSoup(response, 'html.parser')

jobList = soup.find_all("div", {"class" : "css-160787d exb5qdx0"})

The last method is returning None and I don't understand why. I am reading the file in properly because I have checked.

I have tried both the find and find all with no success. I looked into using selenium but the website doesn't seem dynamic.

答案1

得分: 1

这个页面是通过一个API端点进行数据填充,通过XHR调用。以下是获取该信息的一种方式:

import requests
import pandas as pd

headers= {
    'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36'
}
url = 'https://jobs.netflix.com/api/search'
r = requests.get(url, headers=headers)
df = pd.json_normalize(r.json(), record_path=['records', ['postings']])
print(df)

终端中的结果:

     	search_text 	text 	lever_id 	state 	updated_at 	created_at 	team 	slug 	location 	external_id 	... 	_index 	_type 	_score 	_version 	_explanation 	sort 	id 	subteam 	organization 	alternate_locations
0 	Amsterdam, Netherlands \r\n \r\n Creative Pro... 	Senior Program Manager - Creative Production I... 	c09326a8-eabe-49bc-844c-28fa3a3d6361 	Public 	2023-07-10T13:00:10+00:00 	2023-07-10T09:38:57Z 	[Creative Production] 	senior-program-manager-creative-production-inn... 	Amsterdam, Netherlands 	284698346 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	64abd1f9b5811b900a89a18a 	NaN 	NaN 	NaN
1 	Manila, Philippines \r\n \r\n Netflix Technol... 	Netflix IT Support Specialist 	835f66aa-e7fc-46dc-a03d-5b39abf74afb 	Public 	2023-07-10T13:00:10+00:00 	2023-07-10T08:26:19Z 	[Netflix Technology Services] 	netflix-it-support-specialist 	Manila, Philippines 	284690884 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	64abc190aabfb410a82540a4 	[Netflix Technology Services] 	NaN 	NaN
2 	Tokyo, Japan \r\n Corporate Functions \r\n Tal... 	HR Associate - Japan 	6bdf2520-bbb7-4e40-9f0e-c562fc6391e1 	Public 	2023-07-10T13:00:10+00:00 	2023-07-10T05:43:57Z 	[Talent and Recruiting] 	hr-associate-japan 	Tokyo, Japan 	284673328 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	64ab9ae37a9df82a8330ae9d 	[Human Resources] 	[Corporate Functions] 	NaN
3 	Remote, California \r\n Corporate Functions \r... 	People Scientist 	1ba707a0-8bb2-436d-921a-1a985dd7fd5b 	Public 	2023-07-10T13:00:10+00:00 	2023-07-07T17:59:45Z 	[Talent and Recruiting] 	people-scientist 	Remote, California 	284310527 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	64a87cdaaabfb4e0b925418d 	[Human Resources] 	[Corporate Functions] 	NaN
4 	Remote, United States \r\n \r\n Core Engineer... 	Software Engineer (L5) - Ads Revenue Infrastru... 	0e733d7e-3a0a-4dd5-ab5e-51c970b484e3 	Public 	2023-07-10T13:00:10+00:00 	2023-07-07T15:50:51Z 	[Core Engineering] 	software-engineer-l5-ads-revenue-infrastructure 	Remote, United States 	284278602 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	64a8368db5811b243289a8dc 	[Content Engineering Core] 	NaN 	NaN
5 	Remote, United States \r\n Games \r\n Boss Fig... 	Director of Game Development 	46b233f1-8fff-4052-ac70-94f1e5ff6372 	Public 	2023-07-10T13:00:10+00:00 	2023-07-03T23:29:19Z 	[Boss Fight Entertainment] 	director-of-game-development 	Remote, United States 	283904007 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	64a950f83d6b6272441b736d 	NaN 	[Games] 	NaN
6 	Remote, United States \r\n Corporate Functions... 	Executive Coach 	c1a8ea67-fb0a-498d-8296-5929f1a2555a 	Public 	2023-07-10T13:00:10+00:00 	2023-06-30T21:04:51Z 	[Talent and Recruiting] 	executive-coach 	Remote, United States 	283427947 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	649f44af3d6b62a03c7c6f7c 	[Human Resources] 	[Corporate Functions] 	NaN
7 	Los Gatos, California \r\n Corporate Functions... 	Business Process Analyst, Finance 	f1078eb0-bf61-4f03-82f2-2b7c1c786830 	Public 	2023-07-10T13:00:10+00:00 	2023-06-30T14:32:35Z 	[Finance] 	business-process-analyst-finance 	Los Gatos, California 	283385129 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	649ee81ab5811b9d
<details>
<summary>英文:</summary>
That page is being hydrated from an API endpoint, via an XHR call. Here is one way of obtaining that information:
import requests
import pandas as pd
headers= {
&#39;User-Agent&#39;:&#39;Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36&#39;
}
url = &#39;https://jobs.netflix.com/api/search&#39;
r = requests.get(url, headers=headers)
df = pd.json_normalize(r.json(), record_path=[&#39;records&#39;, [&#39;postings&#39;]])
print(df)
Result in terminal:
search_text 	text 	lever_id 	state 	updated_at 	created_at 	team 	slug 	location 	external_id 	... 	_index 	_type 	_score 	_version 	_explanation 	sort 	id 	subteam 	organization 	alternate_locations
0 	Amsterdam, Netherlands \r\n \r\n Creative Pro... 	Senior Program Manager - Creative Production I... 	c09326a8-eabe-49bc-844c-28fa3a3d6361 	Public 	2023-07-10T13:00:10+00:00 	2023-07-10T09:38:57Z 	[Creative Production] 	senior-program-manager-creative-production-inn... 	Amsterdam, Netherlands 	284698346 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	64abd1f9b5811b900a89a18a 	NaN 	NaN 	NaN
1 	Manila, Philippines \r\n \r\n Netflix Technol... 	Netflix IT Support Specialist 	835f66aa-e7fc-46dc-a03d-5b39abf74afb 	Public 	2023-07-10T13:00:10+00:00 	2023-07-10T08:26:19Z 	[Netflix Technology Services] 	netflix-it-support-specialist 	Manila, Philippines 	284690884 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	64abc190aabfb410a82540a4 	[Netflix Technology Services] 	NaN 	NaN
2 	Tokyo, Japan \r\n Corporate Functions \r\n Tal... 	HR Associate - Japan 	6bdf2520-bbb7-4e40-9f0e-c562fc6391e1 	Public 	2023-07-10T13:00:10+00:00 	2023-07-10T05:43:57Z 	[Talent and Recruiting] 	hr-associate-japan 	Tokyo, Japan 	284673328 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	64ab9ae37a9df82a8330ae9d 	[Human Resources] 	[Corporate Functions] 	NaN
3 	Remote, California \r\n Corporate Functions \r... 	People Scientist 	1ba707a0-8bb2-436d-921a-1a985dd7fd5b 	Public 	2023-07-10T13:00:10+00:00 	2023-07-07T17:59:45Z 	[Talent and Recruiting] 	people-scientist 	Remote, California 	284310527 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	64a87cdaaabfb4e0b925418d 	[Human Resources] 	[Corporate Functions] 	NaN
4 	Remote, United States \r\n \r\n Core Engineer... 	Software Engineer (L5) - Ads Revenue Infrastru... 	0e733d7e-3a0a-4dd5-ab5e-51c970b484e3 	Public 	2023-07-10T13:00:10+00:00 	2023-07-07T15:50:51Z 	[Core Engineering] 	software-engineer-l5-ads-revenue-infrastructure 	Remote, United States 	284278602 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	64a8368db5811b243289a8dc 	[Content Engineering Core] 	NaN 	NaN
5 	Remote, United States \r\n Games \r\n Boss Fig... 	Director of Game Development 	46b233f1-8fff-4052-ac70-94f1e5ff6372 	Public 	2023-07-10T13:00:10+00:00 	2023-07-03T23:29:19Z 	[Boss Fight Entertainment] 	director-of-game-development 	Remote, United States 	283904007 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	64a950f83d6b6272441b736d 	NaN 	[Games] 	NaN
6 	Remote, United States \r\n Corporate Functions... 	Executive Coach 	c1a8ea67-fb0a-498d-8296-5929f1a2555a 	Public 	2023-07-10T13:00:10+00:00 	2023-06-30T21:04:51Z 	[Talent and Recruiting] 	executive-coach 	Remote, United States 	283427947 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	649f44af3d6b62a03c7c6f7c 	[Human Resources] 	[Corporate Functions] 	NaN
7 	Los Gatos, California \r\n Corporate Functions... 	Business Process Analyst, Finance 	f1078eb0-bf61-4f03-82f2-2b7c1c786830 	Public 	2023-07-10T13:00:10+00:00 	2023-06-30T14:32:35Z 	[Finance] 	business-process-analyst-finance 	Los Gatos, California 	283385129 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	649ee81ab5811b9d27624638 	[Finance] 	[Corporate Functions] 	[Los Angeles, California]
8 	Seoul, South Korea \r\n Corporate Functions \r... 	Senior Manager - Country Lead, Production Fina... 	8e35e3d6-f762-4b53-99e8-6cc62f1875db 	Public 	2023-07-10T13:00:14+00:00 	2023-06-30T05:25:07Z 	[Finance] 	senior-manager-country-lead-production-finance... 	Seoul, South Korea 	283325929 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	649e6863aabfb4704f86b47f 	[Finance] 	[Corporate Functions] 	NaN
9 	Singapore, Singapore \r\n Marketing and Public... 	Manager, Ads B2B Marketing - APAC 	f3275e7e-38af-4bb6-84b6-a99094738986 	Public 	2023-07-10T13:00:10+00:00 	2023-06-29T05:21:12Z 	[Partner Marketing] 	manager-ads-b2b-marketing-apac 	Singapore, Singapore 	283167491 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	649d15b5aabfb4551aed6809 	NaN 	[Marketing and Publicity] 	NaN
10 	Mumbai, India \r\n Content \r\n \r\n \r\n Ma... 	Manager, Series - India 	9a630609-3041-4698-a95d-543efe80a93c 	Public 	2023-07-10T13:00:10+00:00 	2023-06-29T02:52:34Z 	NaN 	manager-india-series-and-films 	Mumbai, India 	283151020 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	64a502957a9df822a932aa71 	NaN 	[Content] 	NaN
11 	Remote, United States \r\n \r\n Data Science ... 	Data Science and Analytics Manager - Streaming... 	ecf66443-f3cf-4ded-ab7b-c3ed84a2b348 	Public 	2023-07-10T13:00:17+00:00 	2023-06-29T00:32:39Z 	[Data Science and Engineering] 	data-science-and-analytics-manager-streaming-p... 	Remote, United States 	283135676 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	649cd1beb5811bc57c7eeed6 	[Data Science and Engineering] 	NaN 	NaN
12 	Remote, United States \r\n \r\n Creative Prod... 	Regional Planning Lead - UCAN 	4e8eb40e-c4fb-4775-b9f1-620c5fcc3f87 	Public 	2023-07-10T13:00:10+00:00 	2023-06-28T22:42:58Z 	[Creative Production] 	regional-planning-lead-ucan 	Remote, United States 	283123645 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	649cb7f23d6b62cec27a7039 	[Product Creative Core] 	NaN 	NaN
13 	Amsterdam, Netherlands \r\n \r\n \r\n \r\n ... 	Manager, Finance Strategy &amp; Analysis (FS&amp;A) , ... 	4c920cf4-1bd5-4630-9833-c16c7c3cac12 	Public 	2023-07-10T13:00:17+00:00 	2023-06-28T08:44:17Z 	NaN 	manager-revenue-analytics 	Amsterdam, Netherlands 	283031908 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	64a587cf7a9df89cbdc9f25d 	NaN 	NaN 	NaN
14 	Los Angeles, California \r\n Marketing and Pub... 	AV Producer - UCAN 	2c76fb58-8c56-442e-98be-028889094b0c 	Public 	2023-07-10T13:00:10+00:00 	2023-06-27T23:13:25Z 	[Creative Marketing Production] 	av-producer-ucan 	Los Angeles, California 	282969550 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	649b6d7aaabfb4675b853b6d 	[Marketing Production] 	[Marketing and Publicity] 	NaN
15 	Manila, Philippines \r\n Corporate Functions \... 	AP Invoice Processing Specialist, Manila (Japa... 	bd698f4b-aac7-46d7-9087-053f077e05d7 	Public 	2023-07-10T13:00:10+00:00 	2023-06-27T02:45:47Z 	[Finance] 	ap-invoice-processing-specialist-manila-japane... 	Manila, Philippines 	282835505 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	649a4e5d3d6b62657a2b93c7 	[Finance] 	[Corporate Functions] 	NaN
16 	Manila, Philippines \r\n Corporate Functions \... 	Accounts Payable Invoice Processing Specialist... 	e0048b97-2c8b-4ae3-8fe8-a88e8c17b210 	Public 	2023-07-10T13:00:10+00:00 	2023-06-27T01:41:20Z 	[Finance] 	ap-invoice-processing-specialist-manila 	Manila, Philippines 	282828368 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	649a3f22b5811b71eb80f9c4 	[Finance] 	[Corporate Functions] 	NaN
17 	Dallas, Texas \r\n Games \r\n Boss Fight Enter... 	IT Support Specialist - Boss Fight Entertainment 	2d3c4f18-4f81-46ea-a8e8-b402865c47ab 	Public 	2023-07-10T13:00:10+00:00 	2023-06-26T23:39:17Z 	[Boss Fight Entertainment] 	it-support-specialist-boss-fight 	Dallas, Texas 	282814656 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	649b052eaabfb4f2708532c6 	NaN 	[Games] 	NaN
18 	Los Gatos, California \r\n Games \r\n Netflix ... 	Player Support Lead, Netflix Games Studio 	3a3aa925-861a-4567-a367-8c9ce9edf2bc 	Public 	2023-07-10T13:00:10+00:00 	2023-06-26T16:26:37Z 	[Netflix Games Studio] 	player-support-lead-netflix-games-studio 	Los Gatos, California 	282767770 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	6499bd167a9df88c1302b79a 	NaN 	[Games] 	NaN
19 	Warsaw, Poland \r\n Corporate Functions \r\n T... 	HR Business Partner, CEE 	4abd34d0-c9a3-4469-8533-f757c2b65063 	Public 	2023-07-10T13:00:17+00:00 	2023-06-23T16:13:14Z 	[Talent and Recruiting] 	hr-business-partner-cee 	Warsaw, Poland 	282717815 	... 	5e7cf522815a780d9124028c 	_doc 	None 	None 	None 	None 	649951413d6b626dab2b8129 	[Human Resources] 	[Corporate Functions] 	NaN
20 rows &#215; 22 columns
</details>

huangapple
  • 本文由 发表于 2023年7月10日 20:59:08
  • 转载请务必保留本文链接:https://go.coder-hub.com/76653989.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定