如何使用Python3从网站提取表格

huangapple
go评论83阅读模式
英文:

How to extract table from website using python3

问题

I want to get/export table from https://www.ethernodes.org/nodes
to a txt file to access with bashscript.

OpenAI help me with this Python3 code but it get nothing

  1. import requests
  2. from bs4 import BeautifulSoup
  4. url = 'https://www.ethernodes.org/nodes?page=8'
  5. response = requests.get(url)
  6. soup = BeautifulSoup(response.text, 'html.parser')
  8. host_ips = []
  9. node_list = soup.find('ul', class_='nodes-list')
  10. if node_list is not None:
  11.     for li in node_list.find_all('li'):
  12.         host_ip = li.find('div', class_='node-host').text.strip()
  13.         host_ips.append(host_ip)
  15. print(host_ips)
英文:

I want to get/export table from https://www.ethernodes.org/nodes
to a txt file to access with bashscript.

OpenAI help me with this Python3 code but it get nothing

  1. import requests
  2. from bs4 import BeautifulSoup
  4. url = 'https://www.ethernodes.org/nodes?page=8'
  5. response = requests.get(url)
  6. soup = BeautifulSoup(response.text, 'html.parser')
  8. host_ips = []
  9. node_list = soup.find('ul', class_='nodes-list')
  10. if node_list is not None:
  11.     for li in node_list.find_all('li'):
  12.         host_ip = li.find('div', class_='node-host').text.strip()
  13.         host_ips.append(host_ip)
  15. print(host_ips)

答案1

得分: 1

以下是您可以获取数据并将其转储到.csv文件的代码部分：

  1. import time
  2. import pandas as pd
  3. import requests
  5. url = "https://www.ethernodes.org/data?"
  7. payload = {
  8.     "draw": "2",
  9.     "columns[0][data]": "id",
  10.     "columns[0][name]": "",
  11.     "columns[0][searchable]": "true",
  12.     "columns[0][orderable]": "true",
  13.     "columns[0][search][value]": "",
  14.     "columns[0][search][regex]": "false",
  15.     "columns[1][data]": "host",
  16.     "columns[1][name]": "",
  17.     "columns[1][searchable]": "true",
  18.     "columns[1][orderable]": "true",
  19.     "columns[1][search][value]": "",
  20.     "columns[1][search][regex]": "false",
  21.     "columns[2][data]": "isp",
  22.     "columns[2][name]": "",
  23.     "columns[2][searchable]": "true",
  24.     "columns[2][orderable]": "true",
  25.     "columns[2][search][value]": "",
  26.     "columns[2][search][regex]": "false",
  27.     "columns[3][data]": "country",
  28.     "columns[3][name]": "",
  29.     "columns[3][searchable]": "true",
  30.     "columns[3][orderable]": "true",
  31.     "columns[3][search][value]": "",
  32.     "columns[3][search][regex]": "false",
  33.     "columns[4][data]": "client",
  34.     "columns[4][name]": "",
  35.     "columns[4][searchable]": "true",
  36.     "columns[4][orderable]": "true",
  37.     "columns[4][search][value]": "",
  38.     "columns[4][search][regex]": "false",
  39.     "columns[5][data]": "clientVersion",
  40.     "columns[5][name]": "",
  41.     "columns[5][searchable]": "true",
  42.     "columns[5][orderable]": "true",
  43.     "columns[5][search][value]": "",
  44.     "columns[5][search][regex]": "false",
  45.     "columns[6][data]": "os",
  46.     "columns[6][name]": "",
  47.     "columns[6][searchable]": "true",
  48.     "columns[6][orderable]": "true",
  49.     "columns[6][search][value]": "",
  50.     "columns[6][search][regex]": "false",
  51.     "columns[7][data]": "lastUpdate",
  52.     "columns[7][name]": "",
  53.     "columns[7][searchable]": "true",
  54.     "columns[7][orderable]": "true",
  55.     "columns[7][search][value]": "",
  56.     "columns[7][search][regex]": "false",
  57.     "columns[8][data]": "inSync",
  58.     "columns[8][name]": "",
  59.     "columns[8][searchable]": "true",
  60.     "columns[8][orderable]": "true",
  61.     "columns[8][search][value]": "",
  62.     "columns[8][search][regex]": "false",
  63.     "order[0][column]": "0",
  64.     "order[0][dir]": "asc",
  65.     "start": "0",
  66.     "length": "100",
  67.     "search[value]": "",
  68.     "search[regex]": "false",
  69.     "_": time.time()
  70. }
  72. headers = {
  73.     "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 Edg/112.0.1722.48",
  74.     "Accept": "application/json, text/javascript, */*; q=0.01",
  75.     "X-Requested-With": "XMLHttpRequest",
  76. }
  78. data = requests.get(url, headers=headers, params=payload).json()["data"]
  79. df = pd.DataFrame(data)
  80. df.to_csv("nodes.csv", index=False)

输出:

如何使用Python3从网站提取表格

如果您只需要主机IP，可以添加以下内容：

  1. hosts = df["host"].values
  2. with open("hosts.txt", "w") as f:
  3.     f.write("\n".join(hosts))

然后您可以执行：

  1. $ cat hosts.txt
英文:

Here's how you can get the data and dump it to a .csv file.

  1. import time
  3. import pandas as pd
  4. import requests
  6. url = "https://www.ethernodes.org/data?"
  8. payload = {
  9.     "draw": "2",
  10.     "columns[0][data]": "id",
  11.     "columns[0][name]": "",
  12.     "columns[0][searchable]": "true",
  13.     "columns[0][orderable]": "true",
  14.     "columns[0][search][value]": "",
  15.     "columns[0][search][regex]": "false",
  16.     "columns[1][data]": "host",
  17.     "columns[1][name]": "",
  18.     "columns[1][searchable]": "true",
  19.     "columns[1][orderable]": "true",
  20.     "columns[1][search][value]": "",
  21.     "columns[1][search][regex]": "false",
  22.     "columns[2][data]": "isp",
  23.     "columns[2][name]": "",
  24.     "columns[2][searchable]": "true",
  25.     "columns[2][orderable]": "true",
  26.     "columns[2][search][value]": "",
  27.     "columns[2][search][regex]": "false",
  28.     "columns[3][data]": "country",
  29.     "columns[3][name]": "",
  30.     "columns[3][searchable]": "true",
  31.     "columns[3][orderable]": "true",
  32.     "columns[3][search][value]": "",
  33.     "columns[3][search][regex]": "false",
  34.     "columns[4][data]": "client",
  35.     "columns[4][name]": "",
  36.     "columns[4][searchable]": "true",
  37.     "columns[4][orderable]": "true",
  38.     "columns[4][search][value]": "",
  39.     "columns[4][search][regex]": "false",
  40.     "columns[5][data]": "clientVersion",
  41.     "columns[5][name]": "",
  42.     "columns[5][searchable]": "true",
  43.     "columns[5][orderable]": "true",
  44.     "columns[5][search][value]": "",
  45.     "columns[5][search][regex]": "false",
  46.     "columns[6][data]": "os",
  47.     "columns[6][name]": "",
  48.     "columns[6][searchable]": "true",
  49.     "columns[6][orderable]": "true",
  50.     "columns[6][search][value]": "",
  51.     "columns[6][search][regex]": "false",
  52.     "columns[7][data]": "lastUpdate",
  53.     "columns[7][name]": "",
  54.     "columns[7][searchable]": "true",
  55.     "columns[7][orderable]": "true",
  56.     "columns[7][search][value]": "",
  57.     "columns[7][search][regex]": "false",
  58.     "columns[8][data]": "inSync",
  59.     "columns[8][name]": "",
  60.     "columns[8][searchable]": "true",
  61.     "columns[8][orderable]": "true",
  62.     "columns[8][search][value]": "",
  63.     "columns[8][search][regex]": "false",
  64.     "order[0][column]": "0",
  65.     "order[0][dir]": "asc",
  66.     "start": "0",
  67.     "length": "100",
  68.     "search[value]": "",
  69.     "search[regex]": "false",
  70.     "_": time.time()
  71. }
  73. headers = {
  74.     "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 Edg/112.0.1722.48",
  75.     "Accept": "application/json, text/javascript, */*; q=0.01",
  76.     "X-Requested-With": "XMLHttpRequest",
  77. }
  79. data = requests.get(url, headers=headers, params=payload).json()["data"]
  80. df = pd.DataFrame(data)
  81. df.to_csv("nodes.csv", index=False)

Output:

如何使用Python3从网站提取表格

And if all you need is the hosts IPs add this:

  1. hosts = df["host"].values
  2. with open("hosts.txt", "w") as f:
  3.     f.write("\n".join(hosts))

Then you can

  1. $ cat hosts.txt

huangapple
  • 本文由 发表于 2023年4月19日 16:59:04
  • 转载请务必保留本文链接：https://go.coder-hub.com/76052595.html
  • beautifulsoup
  • python-3.x
  • web-scraping
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定