英文:
Beautiful Soup not finding HTML that exists with file
问题
我正在查看Netflix的招聘页面: https://jobs.netflix.com/search
url = "https://jobs.netflix.com/search"
response=requests.get(url)
response = response.content
soup= BeautifulSoup(response, 'html.parser')
jobList = soup.find_all("div", {"class" : "css-160787d exb5qdx0"})
最后的方法返回了None，我不明白为什么。我已经确认文件已正确读入，因此不明白原因。
我已尝试使用find和find_all，但都没有成功。我也研究了使用selenium，但网站似乎不是动态的。
英文:
I'm looking at the Netflix jobs page: https://jobs.netflix.com/search
url = "https://jobs.netflix.com/search"
response=requests.get(url)
response = response.content
soup= BeautifulSoup(response, 'html.parser')
jobList = soup.find_all("div", {"class" : "css-160787d exb5qdx0"})
The last method is returning None and I don't understand why. I am reading the file in properly because I have checked.
I have tried both the find and find all with no success. I looked into using selenium but the website doesn't seem dynamic.
答案1
得分: 1
这个页面是通过一个API端点进行数据填充，通过XHR调用。以下是获取该信息的一种方式：
import requests
import pandas as pd
headers= {
'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36'
}
url = 'https://jobs.netflix.com/api/search'
r = requests.get(url, headers=headers)
df = pd.json_normalize(r.json(), record_path=['records', ['postings']])
print(df)
终端中的结果：
search_text text lever_id state updated_at created_at team slug location external_id ... _index _type _score _version _explanation sort id subteam organization alternate_locations
0 Amsterdam, Netherlands \r\n \r\n Creative Pro... Senior Program Manager - Creative Production I... c09326a8-eabe-49bc-844c-28fa3a3d6361 Public 2023-07-10T13:00:10+00:00 2023-07-10T09:38:57Z [Creative Production] senior-program-manager-creative-production-inn... Amsterdam, Netherlands 284698346 ... 5e7cf522815a780d9124028c _doc None None None None 64abd1f9b5811b900a89a18a NaN NaN NaN
1 Manila, Philippines \r\n \r\n Netflix Technol... Netflix IT Support Specialist 835f66aa-e7fc-46dc-a03d-5b39abf74afb Public 2023-07-10T13:00:10+00:00 2023-07-10T08:26:19Z [Netflix Technology Services] netflix-it-support-specialist Manila, Philippines 284690884 ... 5e7cf522815a780d9124028c _doc None None None None 64abc190aabfb410a82540a4 [Netflix Technology Services] NaN NaN
2 Tokyo, Japan \r\n Corporate Functions \r\n Tal... HR Associate - Japan 6bdf2520-bbb7-4e40-9f0e-c562fc6391e1 Public 2023-07-10T13:00:10+00:00 2023-07-10T05:43:57Z [Talent and Recruiting] hr-associate-japan Tokyo, Japan 284673328 ... 5e7cf522815a780d9124028c _doc None None None None 64ab9ae37a9df82a8330ae9d [Human Resources] [Corporate Functions] NaN
3 Remote, California \r\n Corporate Functions \r... People Scientist 1ba707a0-8bb2-436d-921a-1a985dd7fd5b Public 2023-07-10T13:00:10+00:00 2023-07-07T17:59:45Z [Talent and Recruiting] people-scientist Remote, California 284310527 ... 5e7cf522815a780d9124028c _doc None None None None 64a87cdaaabfb4e0b925418d [Human Resources] [Corporate Functions] NaN
4 Remote, United States \r\n \r\n Core Engineer... Software Engineer (L5) - Ads Revenue Infrastru... 0e733d7e-3a0a-4dd5-ab5e-51c970b484e3 Public 2023-07-10T13:00:10+00:00 2023-07-07T15:50:51Z [Core Engineering] software-engineer-l5-ads-revenue-infrastructure Remote, United States 284278602 ... 5e7cf522815a780d9124028c _doc None None None None 64a8368db5811b243289a8dc [Content Engineering Core] NaN NaN
5 Remote, United States \r\n Games \r\n Boss Fig... Director of Game Development 46b233f1-8fff-4052-ac70-94f1e5ff6372 Public 2023-07-10T13:00:10+00:00 2023-07-03T23:29:19Z [Boss Fight Entertainment] director-of-game-development Remote, United States 283904007 ... 5e7cf522815a780d9124028c _doc None None None None 64a950f83d6b6272441b736d NaN [Games] NaN
6 Remote, United States \r\n Corporate Functions... Executive Coach c1a8ea67-fb0a-498d-8296-5929f1a2555a Public 2023-07-10T13:00:10+00:00 2023-06-30T21:04:51Z [Talent and Recruiting] executive-coach Remote, United States 283427947 ... 5e7cf522815a780d9124028c _doc None None None None 649f44af3d6b62a03c7c6f7c [Human Resources] [Corporate Functions] NaN
7 Los Gatos, California \r\n Corporate Functions... Business Process Analyst, Finance f1078eb0-bf61-4f03-82f2-2b7c1c786830 Public 2023-07-10T13:00:10+00:00 2023-06-30T14:32:35Z [Finance] business-process-analyst-finance Los Gatos, California 283385129 ... 5e7cf522815a780d9124028c _doc None None None None 649ee81ab5811b9d
<details>
<summary>英文:</summary>
That page is being hydrated from an API endpoint, via an XHR call. Here is one way of obtaining that information:
import requests
import pandas as pd
headers= {
'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36'
}
url = 'https://jobs.netflix.com/api/search'
r = requests.get(url, headers=headers)
df = pd.json_normalize(r.json(), record_path=['records', ['postings']])
print(df)
Result in terminal:
search_text text lever_id state updated_at created_at team slug location external_id ... _index _type _score _version _explanation sort id subteam organization alternate_locations
0 Amsterdam, Netherlands \r\n \r\n Creative Pro... Senior Program Manager - Creative Production I... c09326a8-eabe-49bc-844c-28fa3a3d6361 Public 2023-07-10T13:00:10+00:00 2023-07-10T09:38:57Z [Creative Production] senior-program-manager-creative-production-inn... Amsterdam, Netherlands 284698346 ... 5e7cf522815a780d9124028c _doc None None None None 64abd1f9b5811b900a89a18a NaN NaN NaN
1 Manila, Philippines \r\n \r\n Netflix Technol... Netflix IT Support Specialist 835f66aa-e7fc-46dc-a03d-5b39abf74afb Public 2023-07-10T13:00:10+00:00 2023-07-10T08:26:19Z [Netflix Technology Services] netflix-it-support-specialist Manila, Philippines 284690884 ... 5e7cf522815a780d9124028c _doc None None None None 64abc190aabfb410a82540a4 [Netflix Technology Services] NaN NaN
2 Tokyo, Japan \r\n Corporate Functions \r\n Tal... HR Associate - Japan 6bdf2520-bbb7-4e40-9f0e-c562fc6391e1 Public 2023-07-10T13:00:10+00:00 2023-07-10T05:43:57Z [Talent and Recruiting] hr-associate-japan Tokyo, Japan 284673328 ... 5e7cf522815a780d9124028c _doc None None None None 64ab9ae37a9df82a8330ae9d [Human Resources] [Corporate Functions] NaN
3 Remote, California \r\n Corporate Functions \r... People Scientist 1ba707a0-8bb2-436d-921a-1a985dd7fd5b Public 2023-07-10T13:00:10+00:00 2023-07-07T17:59:45Z [Talent and Recruiting] people-scientist Remote, California 284310527 ... 5e7cf522815a780d9124028c _doc None None None None 64a87cdaaabfb4e0b925418d [Human Resources] [Corporate Functions] NaN
4 Remote, United States \r\n \r\n Core Engineer... Software Engineer (L5) - Ads Revenue Infrastru... 0e733d7e-3a0a-4dd5-ab5e-51c970b484e3 Public 2023-07-10T13:00:10+00:00 2023-07-07T15:50:51Z [Core Engineering] software-engineer-l5-ads-revenue-infrastructure Remote, United States 284278602 ... 5e7cf522815a780d9124028c _doc None None None None 64a8368db5811b243289a8dc [Content Engineering Core] NaN NaN
5 Remote, United States \r\n Games \r\n Boss Fig... Director of Game Development 46b233f1-8fff-4052-ac70-94f1e5ff6372 Public 2023-07-10T13:00:10+00:00 2023-07-03T23:29:19Z [Boss Fight Entertainment] director-of-game-development Remote, United States 283904007 ... 5e7cf522815a780d9124028c _doc None None None None 64a950f83d6b6272441b736d NaN [Games] NaN
6 Remote, United States \r\n Corporate Functions... Executive Coach c1a8ea67-fb0a-498d-8296-5929f1a2555a Public 2023-07-10T13:00:10+00:00 2023-06-30T21:04:51Z [Talent and Recruiting] executive-coach Remote, United States 283427947 ... 5e7cf522815a780d9124028c _doc None None None None 649f44af3d6b62a03c7c6f7c [Human Resources] [Corporate Functions] NaN
7 Los Gatos, California \r\n Corporate Functions... Business Process Analyst, Finance f1078eb0-bf61-4f03-82f2-2b7c1c786830 Public 2023-07-10T13:00:10+00:00 2023-06-30T14:32:35Z [Finance] business-process-analyst-finance Los Gatos, California 283385129 ... 5e7cf522815a780d9124028c _doc None None None None 649ee81ab5811b9d27624638 [Finance] [Corporate Functions] [Los Angeles, California]
8 Seoul, South Korea \r\n Corporate Functions \r... Senior Manager - Country Lead, Production Fina... 8e35e3d6-f762-4b53-99e8-6cc62f1875db Public 2023-07-10T13:00:14+00:00 2023-06-30T05:25:07Z [Finance] senior-manager-country-lead-production-finance... Seoul, South Korea 283325929 ... 5e7cf522815a780d9124028c _doc None None None None 649e6863aabfb4704f86b47f [Finance] [Corporate Functions] NaN
9 Singapore, Singapore \r\n Marketing and Public... Manager, Ads B2B Marketing - APAC f3275e7e-38af-4bb6-84b6-a99094738986 Public 2023-07-10T13:00:10+00:00 2023-06-29T05:21:12Z [Partner Marketing] manager-ads-b2b-marketing-apac Singapore, Singapore 283167491 ... 5e7cf522815a780d9124028c _doc None None None None 649d15b5aabfb4551aed6809 NaN [Marketing and Publicity] NaN
10 Mumbai, India \r\n Content \r\n \r\n \r\n Ma... Manager, Series - India 9a630609-3041-4698-a95d-543efe80a93c Public 2023-07-10T13:00:10+00:00 2023-06-29T02:52:34Z NaN manager-india-series-and-films Mumbai, India 283151020 ... 5e7cf522815a780d9124028c _doc None None None None 64a502957a9df822a932aa71 NaN [Content] NaN
11 Remote, United States \r\n \r\n Data Science ... Data Science and Analytics Manager - Streaming... ecf66443-f3cf-4ded-ab7b-c3ed84a2b348 Public 2023-07-10T13:00:17+00:00 2023-06-29T00:32:39Z [Data Science and Engineering] data-science-and-analytics-manager-streaming-p... Remote, United States 283135676 ... 5e7cf522815a780d9124028c _doc None None None None 649cd1beb5811bc57c7eeed6 [Data Science and Engineering] NaN NaN
12 Remote, United States \r\n \r\n Creative Prod... Regional Planning Lead - UCAN 4e8eb40e-c4fb-4775-b9f1-620c5fcc3f87 Public 2023-07-10T13:00:10+00:00 2023-06-28T22:42:58Z [Creative Production] regional-planning-lead-ucan Remote, United States 283123645 ... 5e7cf522815a780d9124028c _doc None None None None 649cb7f23d6b62cec27a7039 [Product Creative Core] NaN NaN
13 Amsterdam, Netherlands \r\n \r\n \r\n \r\n ... Manager, Finance Strategy & Analysis (FS&A) , ... 4c920cf4-1bd5-4630-9833-c16c7c3cac12 Public 2023-07-10T13:00:17+00:00 2023-06-28T08:44:17Z NaN manager-revenue-analytics Amsterdam, Netherlands 283031908 ... 5e7cf522815a780d9124028c _doc None None None None 64a587cf7a9df89cbdc9f25d NaN NaN NaN
14 Los Angeles, California \r\n Marketing and Pub... AV Producer - UCAN 2c76fb58-8c56-442e-98be-028889094b0c Public 2023-07-10T13:00:10+00:00 2023-06-27T23:13:25Z [Creative Marketing Production] av-producer-ucan Los Angeles, California 282969550 ... 5e7cf522815a780d9124028c _doc None None None None 649b6d7aaabfb4675b853b6d [Marketing Production] [Marketing and Publicity] NaN
15 Manila, Philippines \r\n Corporate Functions \... AP Invoice Processing Specialist, Manila (Japa... bd698f4b-aac7-46d7-9087-053f077e05d7 Public 2023-07-10T13:00:10+00:00 2023-06-27T02:45:47Z [Finance] ap-invoice-processing-specialist-manila-japane... Manila, Philippines 282835505 ... 5e7cf522815a780d9124028c _doc None None None None 649a4e5d3d6b62657a2b93c7 [Finance] [Corporate Functions] NaN
16 Manila, Philippines \r\n Corporate Functions \... Accounts Payable Invoice Processing Specialist... e0048b97-2c8b-4ae3-8fe8-a88e8c17b210 Public 2023-07-10T13:00:10+00:00 2023-06-27T01:41:20Z [Finance] ap-invoice-processing-specialist-manila Manila, Philippines 282828368 ... 5e7cf522815a780d9124028c _doc None None None None 649a3f22b5811b71eb80f9c4 [Finance] [Corporate Functions] NaN
17 Dallas, Texas \r\n Games \r\n Boss Fight Enter... IT Support Specialist - Boss Fight Entertainment 2d3c4f18-4f81-46ea-a8e8-b402865c47ab Public 2023-07-10T13:00:10+00:00 2023-06-26T23:39:17Z [Boss Fight Entertainment] it-support-specialist-boss-fight Dallas, Texas 282814656 ... 5e7cf522815a780d9124028c _doc None None None None 649b052eaabfb4f2708532c6 NaN [Games] NaN
18 Los Gatos, California \r\n Games \r\n Netflix ... Player Support Lead, Netflix Games Studio 3a3aa925-861a-4567-a367-8c9ce9edf2bc Public 2023-07-10T13:00:10+00:00 2023-06-26T16:26:37Z [Netflix Games Studio] player-support-lead-netflix-games-studio Los Gatos, California 282767770 ... 5e7cf522815a780d9124028c _doc None None None None 6499bd167a9df88c1302b79a NaN [Games] NaN
19 Warsaw, Poland \r\n Corporate Functions \r\n T... HR Business Partner, CEE 4abd34d0-c9a3-4469-8533-f757c2b65063 Public 2023-07-10T13:00:17+00:00 2023-06-23T16:13:14Z [Talent and Recruiting] hr-business-partner-cee Warsaw, Poland 282717815 ... 5e7cf522815a780d9124028c _doc None None None None 649951413d6b626dab2b8129 [Human Resources] [Corporate Functions] NaN
20 rows × 22 columns
</details>
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论