英文:
Combine multiple BeautifulSoup calls
问题
我想遍历一个网页。
我使用soup来查找/选择html中的标签。
目前,我有两个分开的语句。但我想在一个语句中完成,这样我就不必两次遍历同一个页面。
我的代码如下:
headers = ({'User-Agent':
'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36'})
sapo="https://casa.sapo.pt/comprar-apartamentos/ofertas-recentes/distrito.lisboa/?pn=1"
soup = BeautifulSoup(response.text, 'html.parser')
data1 = [json.loads(x.string) for x in soup.find_all("script", type="application/ld+json")]
data2 = soup.select('div.property')
del data1[:2]
页面上有25个属性。data1返回27个结果,而前两个结果只是开头,所以我删除它们。所以我有25个带有10个"列"的结果。
现在我想将data2作为第11列。
我应该如何实现这个?
英文:
I want to iterate over a webpage.
I use soup to find/select the tags in the html.
For now, I have the two separated statements. But I'd like to have it done in one statement so I dont have to iterate over the same page twice.
My code is the following:
headers = ({'User-Agent':
'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36'})
sapo="https://casa.sapo.pt/comprar-apartamentos/ofertas-recentes/distrito.lisboa/?pn=1"
soup = BeautifulSoup(response.text, 'html.parser')
data1 = [json.loads(x.string) for x in soup.find_all("script", type="application/ld+json")]
data2= soup.select('div.property')
del data1[:2]
There are 25 properties on the page. data1 returns 27 results, whereas the first 2 results are just overhead, so I delete them. So I have 25 results with 10 "columns".
Now I'd like to have the data2 as an 11th column.
How could I achieve this?
答案1
得分: 1
以下是翻译好的部分:
"我不确定为什么你喜欢获取整个HTML元素,但我们可以继续。更改选择元素的策略并从容器开始:"
"根据您的评论提取href:"
"数据 = []"
"对于soup.select('div.property')中的每个元素:"
"d = {'link': 'https://casa.sapo.pt' + e.a.get('href')}"
"d.update(json.loads(e.script.string))"
"data.append(d)"
"pd.DataFrame(data)"
英文:
I am not sure why you like to get the whole HTML element, but here we go. Change your strategy selecting elements and start withe the containers:
data = []
for e in soup.select('div.property'):
d = {'html':e}
d.update(json.loads(e.script.string))
data.append(d)
pd.DataFrame(data)
EDIT
Based on your comment extract the href via
d = {'link':'https://casa.sapo.pt'+e.a.get('href')}
data = []
for e in soup.select('div.property'):
d = {'link':'https://casa.sapo.pt'+e.a.get('href')}
d.update(json.loads(e.script.string))
data.append(d)
pd.DataFrame(data)
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论