2017年3月26日 23:07:38go评论89阅读模式

英文:

Scrape HTML drop-down lists with Go?

问题

我正在使用golang.org/x/net/html包来从HTML页面中提取数据，目前一切都很顺利。然而，我不知道如何从下拉列表中提取数据，就像这样：

<!DOCTYPE html>
<html>
<body>

<select name="car" size="1" id="car">
  <option value="volvo">Volvo</option>
  <option value="saab">Saab</option>
  <option value="vw">VW</option>
  <option value="audi" selected>Audi</option>
</select>

<select name="animal" size="1" id="animal">
  <option value="dog">Dog</option>
  <option value="cat" selected>Cat</option>
  <option value="badger">Badger</option>
  <option value="mouse">Mouse</option>
</select>

我想提取预选的选项，所以结果应该是这样的：

car = audi
animal = cat

我该如何实现这个目标？如果golang.org/x/net/html无法满足我的需求，我还能做些什么来提取数据？

英文:

I'm using the package golang.org/x/net/html to scrape data out of HTML pages and this has been working fine so far. However, I don't know how to extract data from a drop-down list like this:

&lt;!DOCTYPE html&gt;
&lt;html&gt;
&lt;body&gt;

&lt;select name=&quot;car&quot; size=&quot;1&quot; id=&quot;car&quot;&gt;
  &lt;option value=&quot;volvo&quot;&gt;Volvo&lt;/option&gt;
  &lt;option value=&quot;saab&quot;&gt;Saab&lt;/option&gt;
  &lt;option value=&quot;vw&quot;&gt;VW&lt;/option&gt;
  &lt;option value=&quot;audi&quot; selected&gt;Audi&lt;/option&gt;
&lt;/select&gt;

&lt;select name=&quot;animal&quot; size=&quot;1&quot; id=&quot;animal&quot;&gt;
  &lt;option value=&quot;dog&quot;&gt;Dog&lt;/option&gt;
  &lt;option value=&quot;cat&quot; selected&gt;Cat&lt;/option&gt;
  &lt;option value=&quot;badger&quot;&gt;Badger&lt;/option&gt;
  &lt;option value=&quot;mouse&quot;&gt;Mouse&lt;/option&gt;
&lt;/select&gt;

</body>
</html>

I want to extract the pre-selected options, so the result becomes this:

car = audi
animal = cat

How can I accomplish this? In case golang.org/x/net/html is not capable of doing what I want, what else can I do to extract the data?

答案1

得分: 0

你可以使用"net/html"包来完成这个任务：

package main

import (
	"fmt"
	"golang.org/x/net/html"
	"strings"
)

func main() {
	s := "html"

	result := make(map[string]string)
	d := html.NewTokenizer(strings.NewReader(s))
	currID := ""
	for {
		tokenType := d.Next()
		if tokenType == html.ErrorToken {
			break
		}

		token := d.Token()
		switch tokenType {
		case html.StartTagToken:
			if token.Data == "select" {
				for _, a := range token.Attr {
					if a.Key == "id" {
						currID = a.Val
					}
				}
			}
			if token.Data == "option" {
				isSelected := false
				for _, a := range token.Attr {
					if a.Key == "selected" {
						isSelected = true
					}
				}
				if isSelected {
					for _, a := range token.Attr {
						if a.Key == "value" {
							result[currID] = a.Val
						}
					}
				}
			}
		}
	}

	fmt.Printf("%v\n", result)
}

P.S. 这段代码可以进行改进。

英文:

You absolutely can do it with "net/html":

package main

import (
	&quot;fmt&quot;
	&quot;golang.org/x/net/html&quot;
	&quot;strings&quot;
)

func main() {
	s := &quot;html&quot;

	result := make(map[string]string)
	d := html.NewTokenizer(strings.NewReader(s))
	currID := &quot;&quot;
	for {
		tokenType := d.Next()
		if tokenType == html.ErrorToken {
			break
		}

		token := d.Token()
		switch tokenType {
		case html.StartTagToken:
			if token.Data == &quot;select&quot; {
				for _, a := range token.Attr {
					if a.Key == &quot;id&quot; {
						currID = a.Val
					}
				}
			}
			if token.Data == &quot;option&quot; {
				isSelected := false
				for _, a := range token.Attr {
					if a.Key == &quot;selected&quot; {
						isSelected = true
					}
				}
				if isSelected {
					for _, a := range token.Attr {
						if a.Key == &quot;value&quot; {
							result[currID] = a.Val
						}
					}
				}
			}
		}
	}

	fmt.Printf(&quot;%v\n&quot;, result)
}

P.S. this code can be improved.

答案2

得分: 0

也许可以使用gokogiri来进行XPath选择器的操作：

car, _ := doc.Search("//select[@id='car']/option[@selected]/text()")
animal, _ := doc.Search("//select[@id='animal']/option[@selected]/text()")

英文:

Maybe use gokogiri for xpath selectors:

car, _ := doc.Search(&quot;//select[@id=&#39;car&#39;]/option[@selected]/text()&quot;)
animal, _ := doc.Search(&quot;//select[@id=&#39;animal&#39;]/option[@selected]/text()&quot;)

通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。

使用Go语言如何抓取HTML下拉列表？

问题

答案1

答案2

从shell脚本中运行Golang脚本。

Go：二进制编码的机制

Getting "rpc error: code = Unavailable desc = error reading from server: EOF" when trying to create a new etcdv3 client

你可以使用Google Go语言创建自己的颜色数组。

What's the correct way to type hint an empty list as a literal in python?

如何在Highcharts Gantt中更改本地化的星期名称

如何在同一个流中使用多个过滤器和映射函数？

如何使用Map/Set来将代码优化到O(n)？

.NET MAUI Android在GitHub Actions上构建失败，错误代码为1。

如何在Playwright视觉比较中屏蔽多个定位器？

在C++中，可以使用可变模板参数来检索类型的内部类型。

selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: stale element not found

Creating and opening a URL to log in to Website via Basic Auth with Robot Framework/Selenium (Python)

AG Grid 在上下文菜单中以大文本形式打开

发表评论