英文:
Parallelism From Slices
问题
我想根据线程数并行运行它,但结果并不如我所期望的那样。我不知道如何使其高效和快速。
我最终得到了这段代码。
package main
import (
"fmt"
"io/ioutil"
"net/http"
"os"
"runtime"
"strconv"
"strings"
"sync"
"time"
)
func main() {
start := time.Now()
target := os.Args[1]
thread, _ := strconv.Atoi(os.Args[3])
file, err := ioutil.ReadFile(os.Args[2])
if err != nil {
fmt.Println("错误:请仔细检查文件" + os.Args[2] + "是否存在!")
os.Exit(0)
}
wordlist := strings.Split(string(file), "\n")
var wg sync.WaitGroup
runtime.GOMAXPROCS(runtime.NumCPU())
jobs := make(chan string)
for i := 0; i < thread; i++ {
wg.Add(1)
defer wg.Done()
for _, word := range wordlist {
go func(word string) {
jobs <- word
}(word)
}
}
go func() {
for job := range jobs {
code := visit(target + job)
fmt.Println(target + job + " - " + strconv.Itoa(code))
}
}()
wg.Wait()
elapsed := time.Since(start)
fmt.Printf("计时器:%s\n", elapsed)
}
func visit(url string) int {
data, err := http.Get(url)
if err != nil {
panic(err)
}
return data.StatusCode
}
希望能得到帮助。谢谢。
更新
这是我当前的结果:
$ go run test.go http://localhost/ word.txt 2
http://localhost/1 - 404
http://localhost/1 - 404
http://localhost/7 - 404
http://localhost/8 - 404
http://localhost/9 - 404
http://localhost/0 - 404
http://localhost/ - 200
http://localhost/3 - 404
http://localhost/2 - 404
http://localhost/4 - 404
http://localhost/6 - 404
http://localhost/2 - 404
http://localhost/3 - 404
http://localhost/4 - 404
http://localhost/5 - 404
http://localhost/9 - 404
http://localhost/7 - 404
http://localhost/8 - 404
http://localhost/0 - 404
http://localhost/5 - 404
http://localhost/ - 200
http://localhost/6 - 404
英文:
I want to make it run parallel based on number of thread. But the result was not as i expected. I dont know how to make it efficient and fast.
I ended up with this code.
package main
import (
"fmt"
"io/ioutil"
"net/http"
"os"
"runtime"
"strconv"
"strings"
"sync"
"time"
)
func main() {
start := time.Now()
target := os.Args[1]
thread, _ := strconv.Atoi(os.Args[3])
file, err := ioutil.ReadFile(os.Args[2])
if err != nil {
fmt.Println("Error: Please double check if the file " + os.Args[2] + " is exist!")
os.Exit(0)
}
wordlist := strings.Split(string(file), "\n")
var wg sync.WaitGroup
runtime.GOMAXPROCS(runtime.NumCPU())
jobs := make(chan string)
for i := 0; i < thread; i++ {
wg.Add(1)
defer wg.Done()
for _, word := range wordlist {
go func(word string) {
jobs <- word
}(word)
}
}
go func() {
for job := range jobs {
code := visit(target + job)
fmt.Println(target + job + " - " + strconv.Itoa(code))
}
}()
wg.Wait()
elapsed := time.Since(start)
fmt.Printf("Timer: %s\n", elapsed)
}
func visit(url string) int {
data, err := http.Get(url)
if err != nil {
panic(err)
}
return data.StatusCode
}
Any help would be appreciated. Thank you.
Update
This is my current results :
$ go run test.go http://localhost/ word.txt 2
http://localhost/1 - 404
http://localhost/1 - 404
http://localhost/7 - 404
http://localhost/8 - 404
http://localhost/9 - 404
http://localhost/0 - 404
http://localhost/ - 200
http://localhost/3 - 404
http://localhost/2 - 404
http://localhost/4 - 404
http://localhost/6 - 404
http://localhost/2 - 404
http://localhost/3 - 404
http://localhost/4 - 404
http://localhost/5 - 404
http://localhost/9 - 404
http://localhost/7 - 404
http://localhost/8 - 404
http://localhost/0 - 404
http://localhost/5 - 404
http://localhost/ - 200
http://localhost/6 - 404
答案1
得分: 1
你没有正确使用waitgroup。在main
的循环中的defer
语句从未被调用,因为main
函数从不返回,结果导致wg.Wait()
调用永远不会解除阻塞。
将defer
语句放在发送消息的goroutine中:
// ...
for i := 0; i < thread; i++ {
wg.Add(1)
for _, word := range wordlist {
go func(word string) {
defer wg.Done()
jobs <- word
}(word)
}
}
// ...
还要关闭jobs
通道:
// ...
wg.Wait()
close(jobs)
// ...
英文:
You are not using the waitgroup correctly. The defer
's in main
's for loop are never called since main
never returns and as a result, wg.Wait()
call never unblocks.
Put defer
calls in the goroutine sending the message:
// ...
for i := 0; i < thread; i++ {
wg.Add(1)
for _, word := range wordlist {
go func(word string) {
defer wg.Done()
jobs <- word
}(word)
}
}
// ...
Also close the jobs
channel:
// ...
wg.Wait()
close(jobs)
// ...
答案2
得分: 0
你应该为每个新的任务创建一个新的 goroutine,而不是为每个新添加的任务创建一个新的 goroutine。
在这里,你创建了大量的 goroutine 来将一个单词推送到通道中。对于这样一个简单的操作来说,这是过度设计了。
for _, word := range wordlist {
go func(word string) {
jobs <- word
}(word)
}
但是获取 URL 的任务更加耗费资源,但你只为此任务启动了一个 goroutine。
go func() {
for job := range jobs {
code := visit(target + job)
fmt.Println(target + job + " - " + strconv.Itoa(code))
}
}()
正如 @abhink 提到的,你没有正确使用 WaitGroup
。在创建每个 goroutine 之前使用 wg.Add(1)
,在每个 goroutine 结束时使用 wg.Done()
。当然,使用 wg.Wait()
来等待所有正在运行的 goroutine 完成。示例:
var wg sync.WaitGroup
wg.Add(1)
go func(param) {
defer wg.Done()
//...
}(url)
//...
wg.Wait()
英文:
You should create new goroutine per new job, not per new job added
Here you're creating a log of goroutines to push only 1 word to channel. This is an overkill for such a simple operation.
for _, word := range wordlist {
go func(word string) {
jobs <- word
}(word)
}
But job with fetching url is more heavier, but start only 1 goroutine for this.
go func() {
for job := range jobs {
code := visit(target + job)
fmt.Println(target + job + " - " + strconv.Itoa(code))
}
}()
As @abhink has mentioned you're not using WaitGroup
correctly. Do wg.Add(1)
before creating each goroutine and do wg.Done()
at the end of each goroutine. And of course wg.Wait()
to wait all running goroutines to finish. Example:
var wg sync.WaitGroup
wg.Add(1)
go func(param){
defer wg.Done()
//...
}(url)
//...
wg.Wait()
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论