问题

我遇到了从页面中提取某些元素的问题：
https://tuning-tec.com/mercedes_w164_ml_mklasa_0507_black_led_seq_lpmed0-5789i

代码：

import requests
from bs4 import BeautifulSoup

URL = "https://tuning-tec.com/mercedes_w164_ml_mklasa_0507_black_led_seq_lpmed0-5789i"
page = requests.get(URL)
soup = BeautifulSoup(page.content, 'html.parser')

title = soup.find(class_="product_cart_title").text
price = soup.find(class_="icon_main_block_price_a")
number = soup.find(class_="product_cart_info").findAll('tr')[1].findAll('td')[1]
description = soup.find(id="tab_a")

print(description)

问题是当我想要获取tab_a时出现问题，

而在

<div align="left" class="product_cart_info" id="charlong_id">
</div>

中是空的。我该如何获取它？
我认为这可能与JavaScript有关。也许在页面加载时存在一些延迟？

import requests
from bs4 import BeautifulSoup


URL="https://tuning-tec.com/mercedes_w164_ml_mklasa_0507_black_led_seq_lpmed0-5789i"
page = requests.get(URL)
soup = BeautifulSoup(page.content, 'html.parser')

title=soup.find(class_="product_cart_title").text
price=soup.find(class_="icon_main_block_price_a")
number=soup.find(class_="product_cart_info").findAll('tr')[1].findAll('td')[1]
description=soup.find(id="tab_a")


print(description)

<div align="left" class="product_cart_info" id="charlong_id">
</div>

答案1

得分: 2

如评论中所述，信息是通过JavaScript加载的，因此BeautifulSoup无法看到它。但是，如果您查看Chrome/Firefox网络选项卡，您可以看到页面发出请求的位置：

import re
import requests
from bs4 import BeautifulSoup

url = 'https://tuning-tec.com/mercedes_w164_ml_mklasa_0507_black_led_seq_lpmed0-5789i'
ajax_url = 'https://tuning-tec.com/_template/_show_normal/_show_charlong.php?itemId={}'

soup = BeautifulSoup(requests.get(url).content, 'html.parser')

print(soup.select_one('.product_cart_title').get_text(strip=True))
print(soup.select_one('.icon_main_block_price_a').get_text(strip=True))
print(soup.select_one('td:contains("Symbol") ~ td').get_text(strip=True))

item_id = re.findall(r"ajax_update_stat\('(\d+)'\)", soup.text)[0]
soup2 = BeautifulSoup(requests.get(ajax_url.format(item_id)).content, 'html.parser')

print()

# just print some info:
for tr in soup2.select('tr'):
    print(re.sub(r' {2,}', ' ', tr.select_one('td').get_text(strip=True, separator=' ')))

输出：

MERCEDES W164 ML M-KLASA 05-07 BLACK LED SEQ
1788.62 PLN
LPMED0

PL
Opis
Lampy soczewkowe ze światłem pozycyjnym LED. Z dynamicznym kierunkowskazem. 100% nowe, w komplecie (lewa i prawa). Homologacja: norma E13 - dopuszczone do ruchu.
Szczegóły
Światła pozycyjne: DIODY Kierunkowskaz: DIODY Światła mijania: H9 w zestawie Światła drogowe: H1 w zestawie Regulacja: elektryczna (silniczek znajduje się w komplecie).
LED TUBE LIGHT Dynamic Turn Signal >>

import re
import requests
from bs4 import BeautifulSoup

url = 'https://tuning-tec.com/mercedes_w164_ml_mklasa_0507_black_led_seq_lpmed0-5789i'
ajax_url = 'https://tuning-tec.com/_template/_show_normal/_show_charlong.php?itemId={}'

soup = BeautifulSoup(requests.get(url).content, 'html.parser')

print(soup.select_one('.product_cart_title').get_text(strip=True))
print(soup.select_one('.icon_main_block_price_a').get_text(strip=True))
print(soup.select_one('td:contains("Symbol") ~ td').get_text(strip=True))

item_id = re.findall(r"ajax_update_stat\('(\d+)'\)", soup.text)[0]
soup2 = BeautifulSoup(requests.get(ajax_url.format(item_id)).content, 'html.parser')

print()

# just print some info:
for tr in soup2.select('tr'):
    print(re.sub(r' {2,}', ' ', tr.select_one('td').get_text(strip=True, separator=' ')))

MERCEDES W164 ML M-KLASA 05-07 BLACK LED SEQ
1788.62 PLN
LPMED0

PL
Opis
Lampy 
 soczewkowe ze światłem 
 pozycyjnym LED. Z dynamicznym 
 kierunkowskazem. 100% nowe, w komplecie 
 (lewa i prawa). Homologacja: norma E13 - 
 dopuszczone do ruchu.
Szczegóły
Światła pozycyjne: DIODY Kierunkowskaz: DIODY Światła 
 mijania: H9 w 
 zestawie Światła 
 drogowe: H1 w 
 zestawie Regulacja: elektryczna (silniczek znajduje się w 
 komplecie).
LED TUBE LIGHT Dynamic Turn Signal >>

答案2

得分: 0

A little change in the description, I don't know if it's working, have a look at the following code:

import re
import requests
from bs4 import BeautifulSoup

url = 'https://tuning-tec.com/mercedes_w164_ml_mklasa_0507_black_led_seq_lpmed0-5789i'
ajax_url = 'https://tuning-tec.com/_template/_show_normal/_show_charlong.php?itemId={}'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')


def unwrapElements(soup, elementsToFind):
    elements = soup.find_all(elementsToFind)
    for element in elements:
        element.unwrap()

print(soup.select_one('.product_cart_title').get_text(strip=True))
print(soup.select_one('.icon_main_block_price_a').get_text(strip=True))
print(soup.select_one('td:contains("Symbol") ~ td').get_text(strip=True))

item_id = re.findall(r"ajax_update_stat\('(\d+)'\)", soup.text)[0]
soup2 = BeautifulSoup(requests.get(ajax_url.format(item_id)).content, 'html.parser')
description=soup2.findAll('tr')[2].findAll('td')[1]
description.append(soup2.findAll('tr')[4].findAll('td')[1])

unwrapElements(description, "td")
unwrapElements(description, "font")
unwrapElements(description, "span")

print(description)

I need just these elements of description in the English language. It will be OK?

And anyway, thanks for the help!!

Only one thing, I don't know why it didn't remove all <td>.

<details>
<summary>英文:</summary>

A little change in the description, I don&#39;t know if it&#39;s working, have a look on the following code:

    import re
    import requests
    from bs4 import BeautifulSoup
    
    url = &#39;https://tuning-tec.com/mercedes_w164_ml_mklasa_0507_black_led_seq_lpmed0-5789i&#39;
    ajax_url = &#39;https://tuning-tec.com/_template/_show_normal/_show_charlong.php?itemId={}&#39;
    soup = BeautifulSoup(requests.get(url).content, &#39;html.parser&#39;)
    
    
    def unwrapElements(soup, elementsToFind):
        elements = soup.find_all(elementsToFind)
        for element in elements:
            element.unwrap()
    
    print(soup.select_one(&#39;.product_cart_title&#39;).get_text(strip=True))
    print(soup.select_one(&#39;.icon_main_block_price_a&#39;).get_text(strip=True))
    print(soup.select_one(&#39;td:contains(&quot;Symbol&quot;) ~ td&#39;).get_text(strip=True))
    
    item_id = re.findall(r&quot;ajax_update_stat\(&#39;(\d+)&#39;\)&quot;, soup.text)[0]
    soup2 = BeautifulSoup(requests.get(ajax_url.format(item_id)).content, &#39;html.parser&#39;)
    description=soup2.findAll(&#39;tr&#39;)[2].findAll(&#39;td&#39;)[1]
    description.append(soup2.findAll(&#39;tr&#39;)[4].findAll(&#39;td&#39;)[1])
    
    unwrapElements(description, &quot;td&quot;)
    unwrapElements(description, &quot;font&quot;)
    unwrapElements(description, &quot;span&quot;)
    
    
    print(description)

I need just these elements of description in English language. It will be OK?

And anyway thanks for help !!

Only one thing i don&#39;t know why he didn&#39;t remove all &lt;td&gt;

</details>