合并多个BeautifulSoup调用

huangapple go评论46阅读模式
英文:

Combine multiple BeautifulSoup calls

问题

我想遍历一个网页。
我使用soup来查找/选择html中的标签。
目前,我有两个分开的语句。但我想在一个语句中完成,这样我就不必两次遍历同一个页面。
我的代码如下:

headers = ({'User-Agent':
        'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36'})

sapo="https://casa.sapo.pt/comprar-apartamentos/ofertas-recentes/distrito.lisboa/?pn=1"
soup = BeautifulSoup(response.text, 'html.parser')

data1 = [json.loads(x.string) for x in soup.find_all("script", type="application/ld+json")]
data2 = soup.select('div.property')
del data1[:2]

页面上有25个属性。data1返回27个结果,而前两个结果只是开头,所以我删除它们。所以我有25个带有10个"列"的结果。
现在我想将data2作为第11列。

我应该如何实现这个?

英文:

I want to iterate over a webpage.
I use soup to find/select the tags in the html.
For now, I have the two separated statements. But I'd like to have it done in one statement so I dont have to iterate over the same page twice.
My code is the following:

headers = ({'User-Agent':
        'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36'})

sapo="https://casa.sapo.pt/comprar-apartamentos/ofertas-recentes/distrito.lisboa/?pn=1"
soup = BeautifulSoup(response.text, 'html.parser')

data1 = [json.loads(x.string) for x in soup.find_all("script", type="application/ld+json")]
data2= soup.select('div.property')
del  data1[:2]

There are 25 properties on the page. data1 returns 27 results, whereas the first 2 results are just overhead, so I delete them. So I have 25 results with 10 "columns".
Now I'd like to have the data2 as an 11th column.

How could I achieve this?

答案1

得分: 1

以下是翻译好的部分:

"我不确定为什么你喜欢获取整个HTML元素,但我们可以继续。更改选择元素的策略并从容器开始:"

"根据您的评论提取href:"

"数据 = []"

"对于soup.select('div.property')中的每个元素:"

"d = {'link': 'https://casa.sapo.pt' + e.a.get('href')}"

"d.update(json.loads(e.script.string))"

"data.append(d)"

"pd.DataFrame(data)"

英文:

I am not sure why you like to get the whole HTML element, but here we go. Change your strategy selecting elements and start withe the containers:

data = []

for e in soup.select('div.property'):
    d = {'html':e}
    d.update(json.loads(e.script.string))
    data.append(d)

pd.DataFrame(data)

EDIT

Based on your comment extract the href via

d = {'link':'https://casa.sapo.pt'+e.a.get('href')}

data = []

for e in soup.select('div.property'):
    d = {'link':'https://casa.sapo.pt'+e.a.get('href')}
    d.update(json.loads(e.script.string))
    data.append(d)

pd.DataFrame(data)

huangapple
  • 本文由 发表于 2023年2月6日 21:26:54
  • 转载请务必保留本文链接:https://go.coder-hub.com/75361909.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定