2022年11月8日 09:41:28go评论83阅读模式

英文:

Go colly returning a string instead of a slice of strings

问题

我正在尝试使用以下代码来爬取网页https://www.brasiltronic.com.br/pesquisa?pg=1&t=Fone%20de%20ouvido：

scraper.go


package scraper

import (
	"fmt"
	"strings"

	"github.com/gocolly/colly"
	"github.com/gocolly/colly/extensions"
)

type Product struct {
	name      string
	fullPrice string
	url       string
}

func Scraper(url string) []Product {

	products := make([]Product, 0)
	c := colly.NewCollector()
	colly.AllowedDomains("www.brasiltronic.com.br")
	c.OnHTML("ul.row", func(e *colly.HTMLElement) {
		name := e.ChildText("div > div.information > h3.name.no-medium.no-tablet")
		fullPrice := e.ChildText("strong.sale-price > span:nth-child(1)")
		replacer := strings.NewReplacer("R$", "", ",", ".")
		fullPrice = replacer.Replace(fullPrice)
		url := e.ChildAttr("div > div.information > a", "href")
		products = append(products, Product{name: name, fullPrice: fullPrice, url: url})
	})
	c.OnError(func(r *colly.Response, err error) {
		fmt.Println("Request URL:", r.Request.URL, "failed with response:", r.Request, "\nError:", err)
	})

	// 使用随机的 User-Agent 发送请求
	extensions.RandomUserAgent(c)

	c.Visit(url)

	return products
}

main.go

package main

import (
	"fmt"

	"github.com/Antonio-Costa00/Go-Price-Monitor/scraper"
)

func main() {

	url := "https://www.brasiltronic.com.br/pesquisa?pg=1&amp;t=Fone%20e%20ouvido"
	products := scraper.Scraper(url)
	fmt.Println(products)
}

这是输出结果：

[{Fone de Ouvido Profissional AKG K92 com fio - Preto e DouradoMicrofone de lapela JBL com fone de ouvido CSLM
20Fone de Ouvido Samson SR350 Over-ear Estéreo PretoFone de Ouvido Sennheiser CX100 BrancoFone de Ouvido Audio
-Technica ATH-M20xBT sem Fio com Bluetooth PretoFone de Ouvido Sennheiser HD100 com fio (Preto)Fone de Ouvido 
Sennheiser HD400S com fio (Preto)Fone de Ouvido Audio-Technica ATH-M40x Profissional para Monitoração com fio 
- PretoFone de Ouvido Audio-Technica ATH-M20x Profissional para Monitoração com fio - PretoFone de Ouvido Audi
o-Technica ATH-M30x Profissional para Monitoração com fio - PretoFone de Ouvido Audio-Technica ATH-AVC400 extr
a-auricualres SonicPro com fio - PretoFone de Ouvido Audio-Technica ATH-M50x Profissional para Monitoração com
 fio - PretoToca Discos Audio-Technica AT-LP60XHP-GM Automático Belt-Drive com Fone de Ouvido ATH-250AV ...Fon
e de Ouvido Sem Fio Sennheiser RS2000 - PretoFone de Ouvido Profissional AKG K361-BT Dobrável - PretoKit Micro
fone Samson C01U Pro PodCasting Pack SAC01UPROPKFone De Ouvido Profissional AKG K371-BT com Bluetooth - PretoH
eadset Audio-Technica ATH-101USB Single-Ear com fio USB - PretoFone de Ouvido Audio-Technica ATH-R70x Profissi
onal de referência abertos com fio - PretoKit Microfone Estudio Zoom ZUM-2 PMP com Headphone e tripé de mesaHe
adset Audio-Technica ATH-102USB Dual-Ear com fio USB - PretoFones de ouvido de monitoramento Sennheiser IE 40 
PRO Clear intra-auricularHeadset Gamer Audio-Technica ATH-G1WL Premium para Jogos Wireless - PretoMicrofone Sa
mson Q9U Cardióide XLR/USB  359.10 94.50 159.30 159.30 699.30 269.10 519.30 879.30 419.40 619.20 373.50 1.219.
50 1.399.50 1.479.60 769.50 1.599.30 1.009.80 219.60 2.559.60 1.129.50 224.10 759.60 1.649.70 1.619.10 /fone-d
e-ouvido-profissional-akg-k92-com-fio-preto-e-dourado-p1331225}]

我想要得到一个字符串切片，但实际上我得到了一个包含所有元素的切片。

有人可以帮我确定发生了什么吗？提前谢谢。

英文:

I am trying to scrape the webpage https://www.brasiltronic.com.br/pesquisa?pg=1&t=Fone%20de%20ouvido

with the following code:

scraper.go


package scraper

import (
	&quot;fmt&quot;
	&quot;strings&quot;

	&quot;github.com/gocolly/colly&quot;
	&quot;github.com/gocolly/colly/extensions&quot;
)

type Product struct {
	name      string
	fullPrice string
	url       string
}

func Scraper(url string) []Product {

	products := make([]Product, 0)
	c := colly.NewCollector()
	colly.AllowedDomains(&quot;www.brasiltronic.com.br&quot;)
	c.OnHTML(&quot;ul.row&quot;, func(e *colly.HTMLElement) {
		name := e.ChildText(&quot;div &gt; div.information &gt; h3.name.no-medium.no-tablet&quot;)
		fullPrice := e.ChildText(&quot;strong.sale-price &gt; span:nth-child(1)&quot;)
		replacer := strings.NewReplacer(&quot;R$&quot;, &quot;&quot;, &quot;,&quot;, &quot;.&quot;)
		fullPrice = replacer.Replace(fullPrice)
		url := e.ChildAttr(&quot;div &gt; div.information &gt; a&quot;, &quot;href&quot;)
		products = append(products, Product{name: name, fullPrice: fullPrice, url: url})
	})
	c.OnError(func(r *colly.Response, err error) {
		fmt.Println(&quot;Request URL:&quot;, r.Request.URL, &quot;failed with response:&quot;, r.Request, &quot;\nError:&quot;, err)
	})

	// Uses a random User-Agent in each request
	extensions.RandomUserAgent(c)

	c.Visit(url)

	return products
}

main.go

package main

import (
	&quot;fmt&quot;

	&quot;github.com/Antonio-Costa00/Go-Price-Monitor/scraper&quot;
)

func main() {

	url := &quot;https://www.brasiltronic.com.br/pesquisa?pg=1&amp;t=Fone%20e%20ouvido&quot;
	products := scraper.Scraper(url)
	fmt.Println(products)
}

This is the output:

[{Fone de Ouvido Profissional AKG K92 com fio - Preto e DouradoMicrofone de lapela JBL com fone de ouvido CSLM
20Fone de Ouvido Samson SR350 Over-ear Est&#233;reo PretoFone de Ouvido Sennheiser CX100 BrancoFone de Ouvido Audio
-Technica ATH-M20xBT sem Fio com Bluetooth PretoFone de Ouvido Sennheiser HD100 com fio (Preto)Fone de Ouvido 
Sennheiser HD400S com fio (Preto)Fone de Ouvido Audio-Technica ATH-M40x Profissional para Monitora&#231;&#227;o com fio 
- PretoFone de Ouvido Audio-Technica ATH-M20x Profissional para Monitora&#231;&#227;o com fio - PretoFone de Ouvido Audi
o-Technica ATH-M30x Profissional para Monitora&#231;&#227;o com fio - PretoFone de Ouvido Audio-Technica ATH-AVC400 extr
a-auricualres SonicPro com fio - PretoFone de Ouvido Audio-Technica ATH-M50x Profissional para Monitora&#231;&#227;o com
fio - PretoToca Discos Audio-Technica AT-LP60XHP-GM Autom&#225;tico Belt-Drive com Fone de Ouvido ATH-250AV ...Fon
e de Ouvido Sem Fio Sennheiser RS2000 - PretoFone de Ouvido Profissional AKG K361-BT Dobr&#225;vel - PretoKit Micro
fone Samson C01U Pro PodCasting Pack SAC01UPROPKFone De Ouvido Profissional AKG K371-BT com Bluetooth - PretoH
eadset Audio-Technica ATH-101USB Single-Ear com fio USB - PretoFone de Ouvido Audio-Technica ATH-R70x Profissi
onal de refer&#234;ncia abertos com fio - PretoKit Microfone Estudio Zoom ZUM-2 PMP com Headphone e trip&#233; de mesaHe
adset Audio-Technica ATH-102USB Dual-Ear com fio USB - PretoFones de ouvido de monitoramento Sennheiser IE 40 
PRO Clear intra-auricularHeadset Gamer Audio-Technica ATH-G1WL Premium para Jogos Wireless - PretoMicrofone Sa
mson Q9U Cardi&#243;ide XLR/USB  359.10 94.50 159.30 159.30 699.30 269.10 519.30 879.30 419.40 619.20 373.50 1.219.
50 1.399.50 1.479.60 769.50 1.599.30 1.009.80 219.60 2.559.60 1.129.50 224.10 759.60 1.649.70 1.619.10 /fone-d
e-ouvido-profissional-akg-k92-com-fio-preto-e-dourado-p1331225}]

I want to get a slice of strings, but instead, I am getting an element slice with all elements.

Can someone help me identify what's happening? Thanks in advance.

答案1

得分: 0

我找到了答案。我只是从colly中添加了一个foreach方法。

package scraper

import (
	"fmt"
	"strings"

	"github.com/gocolly/colly"
	"github.com/gocolly/colly/extensions"
)

type Product struct {
	name      string
	fullPrice string
	url       string
}

func Scraper(url string) []Product {

	products := make([]Product, 0)
	c := colly.NewCollector()
	colly.AllowedDomains("www.brasiltronic.com.br")
	c.OnHTML("ul.row", func(e *colly.HTMLElement) {
		e.ForEach("div.inner", func(i int, h *colly.HTMLElement) {
			name := h.ChildText("div > div.information > h3.name.no-medium.no-tablet")
			fullPrice := h.ChildText("strong.sale-price > span:nth-child(1)")
			replacer := strings.NewReplacer("R$", "", ",", ".")
			fullPrice = replacer.Replace(fullPrice)
			url = h.ChildAttr("div > div.information > a", "href")
			products = append(products, Product{name: name, fullPrice: fullPrice, url: url})
		})

	})
	c.OnError(func(r *colly.Response, err error) {
		fmt.Println("Request URL:", r.Request.URL, "failed with response:", r.Request, "\nError:", err)
	})

	// 使用随机的User-Agent进行请求
	extensions.RandomUserAgent(c)

	c.Visit(url)

	return products
}

输出结果：

[{Fone de Ouvido Profissional AKG K92 com fio - Preto e Dourado  359.10 /fone-de-ouvido-profissional-akg-k92-com-fio-preto-e-dourado-p1331225} {Microfone de lapela JBL com fone de ouvido CSLM20  94.50 /microfone-de-lapela-jbl-com-fone-de-ouvido-cslm20-p1331251} {Fone de Ouvido Samson SR350 Over-ear Estéreo Preto  159.30 /fone-de-ouvido-over-ear-samson-sr350-estereo-preto-p1331708} {Fone de Ouvido Sennheiser CX100 Branco  159.30 /fone-de-ouvido-in-ear-intra-auricular-sennheiser-cx100-branco-p1330663} {Fone de Ouvido Audio-Technica ATH-M20xBT sem Fio com Bluetooth Preto  699.30 /fone-de-ouvido-sem-fio-audio-technica-ath-m20xbt-com-bluetooth-preto-p1331720} {Fone de Ouvido Sennheiser HD100 com fio (Preto)  269.10 /fone-de-ouvido-sennheiser-hd100-preto-headphone-stereo-on-ear-com-fio-p1330153} {Fone de Ouvido Sennheiser HD400S com fio (Preto)  519.30 /fone-de-ouvido-sennheiser-hd400s-com-fio-cor-preto-p1330112} {Fone de Ouvido Audio-Technica ATH-M40x Profissional para Monitoração com fio - Preto  879.30 /fone-de-ouvido-audio-technica-ath-m40x-profissional-para-monitoracao-com-fio-preto-p1330999} {Fone de Ouvido Audio-Technica ATH-M20x Profissional para Monitoração com fio - Preto  419.40 /fone-de-ouvido-audio-technica-ath-m20x-profissional-para-monitoracao-com-fio-preto-p1330997} {Fone de Ouvido Audio-Technica ATH-M30x Profissional para Monitoração com fio - Preto  619.20 /fone-de-ouvido-audio-technica-ath-m30x-profissional-para-monitoracao-com-fio-preto-p1330998} {Fone de Ouvido Audio-Technica ATH-AVC400 extra-auricualres SonicPro com fio - Preto  373.50 /fone-de-ouvido-audio-technica-ath-avc400-extra-auricualres-sonicpro-com-fio-preto-p1330994} {Fone de Ouvido Audio-Technica ATH-M50x Profissional para Monitoração com fio - Preto  1.219.50 /fone-de-ouvido-audio-technica-ath-m50x-profissional-para-monitoracao-com-fio-preto-p1331000} {Toca Discos Audio-Technica AT-LP60XHP-GM Automático Belt-Drive com Fone de Ouvido ATH-250AV ...  1.399.50 /toca-discos-automatico-audio-technica-at-lp60xhp-belt-drive-com-fone-de-ouvido-ath-250av-grafite-p1331501} {Fone de Ouvido Sem Fio Sennheiser RS2000 - Preto  1.479.60 /fone-de-ouvido-sem-fio-sennheiser-rs2000-preto-p1331279} {Fone de Ouvido Profissional AKG K361-BT Dobrável - Preto  769.50 /fone-de-ouvido-profissional-akg-k361-bt-dobravel-preto-p1331228} {Kit Microfone Samson C01U Pro PodCasting Pack SAC01UPROPK  1.599.30 /kit-microfone-podcasting-samson-c01u-pro-pack-sac01upropk-p1331661} {Fone De Ouvido Profissional AKG K371-BT com Bluetooth - Preto  1.009.80 /fone-de-ouvido-profissional-akg-k371-bt-com-bluetooth-preto-p1331231} {Headset Audio-Technica ATH-101USB Single-Ear com fio USB - Preto  219.60 /headset-audio-technica-ath-101usb-single-ear-com-fio-usb-preto-p1331030} {Fone de Ouvido Audio-Technica ATH-R70x Profissional de referência abertos com fio - Preto  2.559.60 /fone-de-ouvido-audio-technica-ath-r70x-profissional-de-referencia-abertos-com-fio-preto-p1330975} {Kit Microfone Estúdio Zoom ZUM-2 PMP com Headphone e tripé de mesa  1.129.50 /kit-microfone-estudio-zoom-zum-2-pmp-com-headphone-e-tripe-de-mesa-p1331536} {Headset Audio-Technica ATH-102USB Dual-Ear com fio USB - Preto  224.10 /headset-audio-technica-ath-102usb-dual-ear-com-fio-usb-preto-p1331028} {Fones de ouvido de monitoramento Sennheiser IE 40 PRO Clear intra-auricular  759.60 /fone-de-ouvido-de-monitoramento-sennheiser-ie-40-pro-clear-intra-auricular-p1330695} {Headset Gamer Audio-Technica ATH-G1WL Premium para Jogos Wireless - Preto  1.649.70 /headset-gamer-audio-technica-ath-g1wl-premium-para-jogos-wireless-preto-p1330996} {Microfone Samson Q9U Cardióide XLR/USB  1.619.10 /microfone-samson-q9u-cardioide-xlr-usb-p1331659}]

英文:

I found the answer. I just add an foreach method from colly

package scraper

import (
	&quot;fmt&quot;
	&quot;strings&quot;

	&quot;github.com/gocolly/colly&quot;
	&quot;github.com/gocolly/colly/extensions&quot;
)

type Product struct {
	name      string
	fullPrice string
	url       string
}

package scraper

import (
	&quot;fmt&quot;
	&quot;strings&quot;

	&quot;github.com/gocolly/colly&quot;
	&quot;github.com/gocolly/colly/extensions&quot;
)

type Product struct {
	name      string
	fullPrice string
	url       string
}

func Scraper(url string) []Product {

	products := make([]Product, 0)
	c := colly.NewCollector()
	colly.AllowedDomains(&quot;www.brasiltronic.com.br&quot;)
	c.OnHTML(&quot;ul.row&quot;, func(e *colly.HTMLElement) {
		e.ForEach(&quot;div.inner&quot;, func(i int, h *colly.HTMLElement) {
			name := h.ChildText(&quot;div &gt; div.information &gt; h3.name.no-medium.no-tablet&quot;)
			fullPrice := h.ChildText(&quot;strong.sale-price &gt; span:nth-child(1)&quot;)
			replacer := strings.NewReplacer(&quot;R$&quot;, &quot;&quot;, &quot;,&quot;, &quot;.&quot;)
			fullPrice = replacer.Replace(fullPrice)
			url = h.ChildAttr(&quot;div &gt; div.information &gt; a&quot;, &quot;href&quot;)
			products = append(products, Product{name: name, fullPrice: fullPrice, url: url})
		})

	})
	c.OnError(func(r *colly.Response, err error) {
		fmt.Println(&quot;Request URL:&quot;, r.Request.URL, &quot;failed with response:&quot;, r.Request, &quot;\nError:&quot;, err)
	})

	// Uses a random User-Agent in each request
	extensions.RandomUserAgent(c)

	c.Visit(url)

	return products
}

Output:

[{Fone de Ouvido Profissional AKG K92 com fio - Preto e Dourado  359.10 /fone-de-ouvido-profissional-akg-k92-com-fio-preto-e-dourado-p1331225} {Microfone de lapela JBL com fone de ouvido CSLM20  94.50 /microfone-de-lapela-jbl-com-fone-de-ouvido-cslm20-p1331251} {Fone de Ouvido Samson SR350 Over-ear Est&#233;reo Preto  159.30 /fone-de-ouvido-over-ear-samson-sr350-estereo-preto-p1331708} {Fone de Ouvido Sennheiser CX100 Branco  159.30 /fone-de-ouvido-in-ear-int
ra-auricular-sennheiser-cx100-branco-p1330663} {Fone de Ouvido Audio-Technica ATH-M20xBT sem Fio com Bluetooth Preto  699.30 /fone-de-ouvido-sem-fio-audio-technica-ath-m20xbt-com-bluetooth-preto-p1331720} {Fone de Ouvido Sennheis
er HD100 com fio (Preto)  269.10 /fone-de-ouvido-sennheiser-hd100-preto-headphone-stereo-on-ear-com-fio-p1330153} {Fone de Ouvido Sennheiser HD400S com fio (Preto)  519.30 /fone-de-ouvido-sennheiser-hd400s-com-fio-cor-preto-p1330
112} {Fone de Ouvido Audio-Technica ATH-M40x Profissional para Monitora&#231;&#227;o com fio - Preto  879.30 /fone-de-ouvido-audio-technica-ath-m40x-profissional-para-monitoracao-com-fio-preto-p1330999} {Fone de Ouvido Audio-Technica ATH-M
20x Profissional para Monitora&#231;&#227;o com fio - Preto  419.40 /fone-de-ouvido-audio-technica-ath-m20x-profissional-para-monitoracao-com-fio-preto-p1330997} {Fone de Ouvido Audio-Technica ATH-M30x Profissional para Monitora&#231;&#227;o com fio
- Preto  619.20 /fone-de-ouvido-audio-technica-ath-m30x-profissional-para-monitoracao-com-fio-preto-p1330998} {Fone de Ouvido Audio-Technica ATH-AVC400 extra-auricualres SonicPro com fio - Preto  373.50 /fone-de-ouvido-audio-tec
hnica-ath-avc400-extra-auricualres-sonicpro-com-fio-preto-p1330994} {Fone de Ouvido Audio-Technica ATH-M50x Profissional para Monitora&#231;&#227;o com fio - Preto  1.219.50 /fone-de-ouvido-audio-technica-ath-m50x-profissional-para-monitor
acao-com-fio-preto-p1331000} {Toca Discos Audio-Technica AT-LP60XHP-GM Autom&#225;tico Belt-Drive com Fone de Ouvido ATH-250AV ...  1.399.50 /toca-discos-automatico-audio-technica-at-lp60xhp-belt-drive-com-fone-de-ouvido-ath-250av-gra
fite-p1331501} {Fone de Ouvido Sem Fio Sennheiser RS2000 - Preto  1.479.60 /fone-de-ouvido-sem-fio-sennheiser-rs2000-preto-p1331279} {Fone de Ouvido Profissional AKG K361-BT Dobr&#225;vel - Preto  769.50 /fone-de-ouvido-profissional-a
kg-k361-bt-dobravel-preto-p1331228} {Kit Microfone Samson C01U Pro PodCasting Pack SAC01UPROPK  1.599.30 /kit-microfone-podcasting-samson-c01u-pro-pack-sac01upropk-p1331661} {Fone De Ouvido Profissional AKG K371-BT com Bluetooth
- Preto  1.009.80 /fone-de-ouvido-profissional-akg-k371-bt-com-bluetooth-preto-p1331231} {Headset Audio-Technica ATH-101USB Single-Ear com fio USB - Preto  219.60 /headset-audio-technica-ath-101usb-single-ear-com-fio-usb-preto-p1
331030} {Fone de Ouvido Audio-Technica ATH-R70x Profissional de refer&#234;ncia abertos com fio - Preto  2.559.60 /fone-de-ouvido-audio-technica-ath-r70x-profissional-de-referencia-abertos-com-fio-preto-p1330975} {Kit Microfone Estudi
o Zoom ZUM-2 PMP com Headphone e trip&#233; de mesa  1.129.50 /kit-microfone-estudio-zoom-zum-2-pmp-com-headphone-e-tripe-de-mesa-p1331536} {Headset Audio-Technica ATH-102USB Dual-Ear com fio USB - Preto  224.10 /headset-audio-technic
a-ath-102usb-dual-ear-com-fio-usb-preto-p1331028} {Fones de ouvido de monitoramento Sennheiser IE 40 PRO Clear intra-auricular  759.60 /fone-de-ouvido-de-monitoramento-sennheiser-ie-40-pro-clear-intra-auricular-p1330695} {Headset
Gamer Audio-Technica ATH-G1WL Premium para Jogos Wireless - Preto  1.649.70 /headset-gamer-audio-technica-ath-g1wl-premium-para-jogos-wireless-preto-p1330996} {Microfone Samson Q9U Cardi&#243;ide XLR/USB  1.619.10 /microfone-samson-q
9u-cardioide-xlr-usb-p1331659}]

通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。

Go colly返回一个字符串而不是一个字符串切片。

问题

答案1

Golang html/templates：使用自定义分隔符解析文件

在gorp中使用Select时遇到的问题。

将对象传递给其他包中的结构体。

在Golang中，我如何定义一个特定类型的映射作为唯一类型？

What's the correct way to type hint an empty list as a literal in python?

如何在Highcharts Gantt中更改本地化的星期名称

如何在同一个流中使用多个过滤器和映射函数？

如何使用Map/Set来将代码优化到O(n)？

.NET MAUI Android在GitHub Actions上构建失败，错误代码为1。

如何在Playwright视觉比较中屏蔽多个定位器？

在C++中，可以使用可变模板参数来检索类型的内部类型。

selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: stale element not found

Creating and opening a URL to log in to Website via Basic Auth with Robot Framework/Selenium (Python)

AG Grid 在上下文菜单中以大文本形式打开

发表评论