英文:
Process concurrent HTTP requests in Golang
问题
我正在尝试处理一个包含200个URL的文件，并使用每个URL进行HTTP请求。每次最多同时处理10个URL（代码应该在10个URL完成处理之前阻塞）。我尝试用Go解决这个问题，但是我一直得到整个文件都被处理，并创建了200个并发连接。
你应该怎么做呢？
英文:
I'm trying to process a file which contains 200 URLs and use each URL to make an HTTP request. I need to process 10 URLs concurrently maximum each time (code should block until 10 URLs finish processing). Tried to solve it in go but I keep getting the whole file processed with 200 concurrent connection created.
for scanner.Scan() { // loop through each url in the file
// send each url to golang HTTPrequest
go HTTPrequest(scanner.Text(), channel, &wg)
}
fmt.Println(<-channel)
wg.Wait()
What should i do?
答案1
得分: 9
一个包含10个goroutine的池子，从一个通道中读取数据，应该能满足你的需求。
work := make(chan string)
// 获取原始的200个URL
var urlsToProcess []string = seedUrls()
// 启动一个包含10个goroutine的池子，并从work通道中读取URL
for i := 0; i <= 10; i++ {
go func(w chan string) {
url := <-w
}(work)
}
// 将URL写入work通道，阻塞直到一个worker goroutine能够开始工作
for _, url := range urlsToProcess {
work <- url
}
清理和请求结果留给你自己来完成。Go通道会阻塞，直到其中一个worker例程能够读取数据。
英文:
A pool of 10 go routines reading from a channel should fulfill your requirements.
work := make(chan string)
// get original 200 urls
var urlsToProcess []string = seedUrls()
// startup pool of 10 go routines and read urls from work channel
for i := 0; i<=10; i++ {
go func(w chan string) {
url := <-w
}(work)
}
// write urls to the work channel, blocking until a worker goroutine
// is able to start work
for _, url := range urlsToProcess {
work <- url
}
Cleanup and request results are left as an exercise for you. Go channels is will block until one of the worker routines is able to read.
答案2
得分: 2
像这样的代码
longTimeAct := func(index int, w chan struct{}, wg *sync.WaitGroup) {
defer wg.Done()
time.Sleep(1 * time.Second)
println(index)
<-w
}
wg := new(sync.WaitGroup)
ws := make(chan struct{}, 10)
for i := 0; i < 100; i++ {
ws <- struct{}{}
wg.Add(1)
go longTimeAct(i, ws, wg)
}
wg.Wait()
英文:
code like this
longTimeAct := func(index int, w chan struct{}, wg *sync.WaitGroup) {
defer wg.Done()
time.Sleep(1 * time.Second)
println(index)
<-w
}
wg := new(sync.WaitGroup)
ws := make(chan struct{}, 10)
for i := 0; i < 100; i++ {
ws <- struct{}{}
wg.Add(1)
go longTimeAct(i, ws, wg)
}
wg.Wait()
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论