如何使用BeautifulSoup从PubChem提取’Odor’信息

huangapple go评论104阅读模式
英文:

How to extract 'Odor' information from PubChem using BeautifulSoup

问题

我写了以下Python代码,从PubChem中提取特定分子的“odor”信息;在这种情况下,分子是辛醛(CID=31289)。这个分子的网页链接是:https://pubchem.ncbi.nlm.nih.gov/compound/31289#section=Odor

  1. import requests
  2. from bs4 import BeautifulSoup
  3. url = 'https://pubchem.ncbi.nlm.nih.gov/compound/31289#section=Odor'
  4. page = requests.get(url)
  5. soup = BeautifulSoup(page.content, 'html.parser')
  6. odor_section = soup.find('section', {'id': 'Odor'})
  7. odor_info = odor_section.find('div', {'class': 'section-content'})
  8. print(odor_info.text.strip())

我得到以下错误信息:
AttributeError: 'NoneType' object has no attribute 'find'
似乎BeautifulSoup没有提取整个页面的信息。

我期望的输出是:
橙花玫瑰香味,花香,蜡质,绿色

英文:

I wrote the following Python code extract 'odor' information from PubChem for a particular molecule; in this case molecule nonanal (CID=31289) The webpage for this molecule is: https://pubchem.ncbi.nlm.nih.gov/compound/31289#section=Odor

  1. import requests
  2. from bs4 import BeautifulSoup
  3. url = 'https://pubchem.ncbi.nlm.nih.gov/compound/31289#section=Odor'
  4. page = requests.get(url)
  5. soup = BeautifulSoup(page.content, 'html.parser')
  6. odor_section = soup.find('section', {'id': 'Odor'})
  7. odor_info = odor_section.find('div', {'class': 'section-content'})
  8. print(odor_info.text.strip())

I get the following error.
AttributeError: 'NoneType' object has no attribute 'find'
It seems that not the whole page information is extracted by BeautifulSoup.

I expect the following output:
Orange-rose odor, Floral, waxy, green

答案1

得分: 2

以下是翻译好的内容,代码部分不翻译:

"The page in question makes an AJAX request to load its data. We can see this in a web browser by looking at the Network tab of the dev tools (F12 in many browsers):

如何使用BeautifulSoup从PubChem提取’Odor’信息

如何使用BeautifulSoup从PubChem提取’Odor’信息

That is to say, the data simply isn't there when the initial page loads - so it isn't found by BeautifulSoup.

To solve the problem:

  • use Selenium, which can actually run the JavaScript code and thus populate the page with the desired data; or

  • simply query the API according to the request seen when loading the page in the browser. Thus:

  1. PubChem_Nonanal_CID = 31289
  2. compound_data_url = 'https://pubchem.ncbi.nlm.nih.gov/rest/pug_view/data/compound/{}/JSON/'
  3. compound_info = requests.get(compound_data_url.format(PubChem_Nonanal_CID))
  4. print(compound_info.json())

Parsing the JSON Reply

Parsing it proves a bit of a challenge, as it is comprised of many lists.
If the order of properties isn't guaranteed, you could opt for a solution like this:

  1. for section in compound_info.json()['Record']['Section']:
  2. if section['TOCHeading'] == "Chemical and Physical Properties":
  3. for sub_section in section['Section']:
  4. if sub_section['TOCHeading'] == 'Experimental Properties':
  5. for sub_sub_section in sub_section['Section']:
  6. if sub_sub_section['TOCHeading'] == "Odor":
  7. print(sub_sub_section['Information'][0]['Value']['StringWithMarkup'][0]['String'])
  8. break

Otherwise, follow the schema from a JSON-parsing website like jsonformatter.com

如何使用BeautifulSoup从PubChem提取’Odor’信息

  1. odor = compound_info.json()['Record']['Section'][3]['Section'][1]['Section'][2]['Information'][0]['Value']['StringWithMarkup'][0]['String']
英文:

The page in question makes an AJAX request to load its data. We can see this in a web browser by looking at the Network tab of the dev tools (F12 in many browsers):

如何使用BeautifulSoup从PubChem提取’Odor’信息

如何使用BeautifulSoup从PubChem提取’Odor’信息

That is to say, the data simply isn't there when the initial page loads - so it isn't found by BeautifulSoup.

To solve the problem:

  • use Selenium, which can actually run the JavaScript code and thus populate the page with the desired data; or

  • simply query the API according to the request seen when loading the page in the browser. Thus:

  1. PubChem_Nonanal_CID=31289
  2. coumpund_data_url = 'https://pubchem.ncbi.nlm.nih.gov/rest/pug_view/data/compound/{}/JSON/'
  3. compound_info = requests.get(coumpund_data_url.format(PubChem_Nonanal_CID))
  4. print (compund_info.json())

Parsing the JSON Reply

Parsing it proves a bit of a challenge, as it is comprised of many lists.
If the order of properties isn't guaranteed, you could opt for a solution like this:

  1. for section in compund_info.json()['Record']['Section']:
  2. if section['TOCHeading']=="Chemical and Physical Properties":
  3. for sub_section in section['Section']:
  4. if sub_section['TOCHeading'] == 'Experimental Properties':
  5. for sub_sub_section in sub_section['Section']:
  6. if sub_sub_section['TOCHeading']=="Odor":
  7. print(sub_sub_section['Information'][0]['Value']['StringWithMarkup'][0]['String'])
  8. break

Otherwise, follow the schema from a JSON-parsing website like jsonformatter.com

如何使用BeautifulSoup从PubChem提取’Odor’信息

  1. # object►Record►Section►3►Section►1►Section►2►Information►0►Value►StringWithMarkup►0►String`
  2. odor = compund_info.json()['Record']['Section'][3]['Section'][1]['Section'][2]['Information'][0]['Value']['StringWithMarkup'][0]['String']

huangapple
  • 本文由 发表于 2023年2月18日 21:26:29
  • 转载请务必保留本文链接:https://go.coder-hub.com/75493652.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定