问题

我是新手，尝试爬取这个表格的16k行数据，https://www.levantineceramics.org/vessels，但是表格行位于tbody内，标准的网页爬取方法，使用pandas和beautiful soup都不起作用，因为它们显示为空白（空数据框）或（['

']）。

我尝试查看了关于pandas、beautiful soup和selenium的网页爬取教程，但没有成功。是否可能爬取这个表格，如果可以的话，您能指导我正确的方向吗？

这是我的代码结果：
['

']

from bs4 import BeautifulSoup as bs
import requests

url = 'https://www.levantineceramics.org/vessels'
page = requests.get(url)
data = bs(page.text, "html.parser")
table = data.body.findAll('tbody')
print(table)

(Note: I've removed the HTML tags in the code for better readability.)

from bs4 import BeautifulSoup as bs
import requests

url = 'https://www.levantineceramics.org/vessels'
page = requests.get(url)
data = bs(page.text, "html.parser")
table = data.body.findAll('tbody')
print(table)

答案1

得分: 2

以下是您提供的代码的翻译部分：

# 通过JavaScript从外部URL加载数据，因此BeautifulSoup无法看到它。 您可以尝试通过requests从其Ajax API获取数据：

import pandas as pd
import requests

api_url = "https://www.levantineceramics.org/vessels/datatable.json"

params = {
    "sEcho": "2",
    "iColumns": "12",
    "sColumns": ",,,,,,,,,,,",
    "iDisplayStart": "0",
    "iDisplayLength": "100",
}

data = requests.get(api_url, params=params).json()

# 创建<table>并通过pandas解析它

columns = [
    "ID",
    "Vessel registration number",
    "Vessel photos",
    "Vessel drawings",
    "Shape",
    "Functional category",
    "Date BCE/CE",
    "Period",
    "Site Name",
    "Country/region",
    "Contributors",
    "Action",
]

table = ["<tr>" + "\n".join(f"<th>{cell}</th>" for cell in columns) + "</tr>"]
for row in data["aaData"]:
    table.append("<tr>" + "\n".join(f"<td>{cell}</td>" for cell in row) + "</tr>")

df = pd.read_html("<table>" + "\n".join(table) + "</table>")[0]
print(df.head(10).to_markdown(index=False))

输出：

（注意：由于代码中包含HTML标签，因此将其保留在翻译中以确保代码的完整性。）

<details>
<summary>英文:</summary>

The data is loaded from external URL via Javascript, so BeautifulSoup doesn&#39;t see it. You can try to get data from their Ajax API via requests:

```py
import pandas as pd
import requests

api_url = &quot;https://www.levantineceramics.org/vessels/datatable.json&quot;

params = {
    &quot;sEcho&quot;: &quot;2&quot;,
    &quot;iColumns&quot;: &quot;12&quot;,
    &quot;sColumns&quot;: &quot;,,,,,,,,,,,&quot;,
    &quot;iDisplayStart&quot;: &quot;0&quot;,
    &quot;iDisplayLength&quot;: &quot;100&quot;,
}

data = requests.get(api_url, params=params).json()

# create &lt;table&gt; and parse it through pandas

columns = [
    &quot;ID&quot;,
    &quot;Vessel registration number&quot;,
    &quot;Vessel photos&quot;,
    &quot;Vessel drawings&quot;,
    &quot;Shape&quot;,
    &quot;Functional category&quot;,
    &quot;Date BCE/CE&quot;,
    &quot;Period&quot;,
    &quot;Site Name&quot;,
    &quot;Country/region&quot;,
    &quot;Contributors&quot;,
    &quot;Action&quot;,
]

table = [&quot;&lt;tr&gt;&quot; + &quot;\n&quot;.join(f&quot;&lt;th&gt;{cell}&lt;/th&gt;&quot; for cell in columns) + &quot;&lt;/tr&gt;&quot;]
for row in data[&quot;aaData&quot;]:
    table.append(&quot;&lt;tr&gt;&quot; + &quot;\n&quot;.join(f&quot;&lt;td&gt;{cell}&lt;/td&gt;&quot; for cell in row) + &quot;&lt;/tr&gt;&quot;)


df = pd.read_html(&quot;&lt;table&gt;&quot; + &quot;\n&quot;.join(table) + &quot;&lt;/table&gt;&quot;)[0]
print(df.head(10).to_markdown(index=False))

Prints: