从切片中进行并行处理

huangapple go评论64阅读模式
英文:

Parallelism From Slices

问题

我想根据线程数并行运行它,但结果并不如我所期望的那样。我不知道如何使其高效和快速。

我最终得到了这段代码。

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
    "os"
    "runtime"
    "strconv"
    "strings"
    "sync"
    "time"
)

func main() {
    start := time.Now()
    target := os.Args[1]
    thread, _ := strconv.Atoi(os.Args[3])
    file, err := ioutil.ReadFile(os.Args[2])
    if err != nil {
        fmt.Println("错误:请仔细检查文件" + os.Args[2] + "是否存在!")
        os.Exit(0)
    }
    wordlist := strings.Split(string(file), "\n")

    var wg sync.WaitGroup
    runtime.GOMAXPROCS(runtime.NumCPU())
    jobs := make(chan string)
    for i := 0; i < thread; i++ {
        wg.Add(1)
        defer wg.Done()
        for _, word := range wordlist {
            go func(word string) {
                jobs <- word
            }(word)
        }
    }

    go func() {
        for job := range jobs {
            code := visit(target + job)
            fmt.Println(target + job + " - " + strconv.Itoa(code))
        }
    }()
    wg.Wait()

    elapsed := time.Since(start)
    fmt.Printf("计时器:%s\n", elapsed)
}

func visit(url string) int {
    data, err := http.Get(url)
    if err != nil {
        panic(err)
    }
    return data.StatusCode
}

希望能得到帮助。谢谢。

更新
这是我当前的结果:

$ go run test.go http://localhost/ word.txt 2
http://localhost/1 - 404
http://localhost/1 - 404
http://localhost/7 - 404
http://localhost/8 - 404
http://localhost/9 - 404
http://localhost/0 - 404
http://localhost/ - 200
http://localhost/3 - 404
http://localhost/2 - 404
http://localhost/4 - 404
http://localhost/6 - 404
http://localhost/2 - 404
http://localhost/3 - 404
http://localhost/4 - 404
http://localhost/5 - 404
http://localhost/9 - 404
http://localhost/7 - 404
http://localhost/8 - 404
http://localhost/0 - 404
http://localhost/5 - 404
http://localhost/ - 200
http://localhost/6 - 404
英文:

I want to make it run parallel based on number of thread. But the result was not as i expected. I dont know how to make it efficient and fast.

I ended up with this code.

package main
import (
&quot;fmt&quot;
&quot;io/ioutil&quot;
&quot;net/http&quot;
&quot;os&quot;
&quot;runtime&quot;
&quot;strconv&quot;
&quot;strings&quot;
&quot;sync&quot;
&quot;time&quot;
)
func main() {
start := time.Now()
target := os.Args[1]
thread, _ := strconv.Atoi(os.Args[3])
file, err := ioutil.ReadFile(os.Args[2])
if err != nil {
fmt.Println(&quot;Error: Please double check if the file &quot; + os.Args[2] + &quot; is exist!&quot;)
os.Exit(0)
}
wordlist := strings.Split(string(file), &quot;\n&quot;)
var wg sync.WaitGroup
runtime.GOMAXPROCS(runtime.NumCPU())
jobs := make(chan string)
for i := 0; i &lt; thread; i++ {
wg.Add(1)
defer wg.Done()
for _, word := range wordlist {
go func(word string) {
jobs &lt;- word
}(word)
}
}
go func() {
for job := range jobs {
code := visit(target + job)
fmt.Println(target + job + &quot; - &quot; + strconv.Itoa(code))
}
}()
wg.Wait()
elapsed := time.Since(start)
fmt.Printf(&quot;Timer: %s\n&quot;, elapsed)
}
func visit(url string) int {
data, err := http.Get(url)
if err != nil {
panic(err)
}
return data.StatusCode
}

Any help would be appreciated. Thank you.

Update
This is my current results :

$ go run test.go http://localhost/ word.txt 2
http://localhost/1 - 404
http://localhost/1 - 404
http://localhost/7 - 404
http://localhost/8 - 404
http://localhost/9 - 404
http://localhost/0 - 404
http://localhost/ - 200
http://localhost/3 - 404
http://localhost/2 - 404
http://localhost/4 - 404
http://localhost/6 - 404
http://localhost/2 - 404
http://localhost/3 - 404
http://localhost/4 - 404
http://localhost/5 - 404
http://localhost/9 - 404
http://localhost/7 - 404
http://localhost/8 - 404
http://localhost/0 - 404
http://localhost/5 - 404
http://localhost/ - 200
http://localhost/6 - 404

答案1

得分: 1

你没有正确使用waitgroup。在main的循环中的defer语句从未被调用,因为main函数从不返回,结果导致wg.Wait()调用永远不会解除阻塞。

defer语句放在发送消息的goroutine中:

// ...
for i := 0; i < thread; i++ {
wg.Add(1)
for _, word := range wordlist {
go func(word string) {
defer wg.Done()
jobs <- word
}(word)
}
}
// ...

还要关闭jobs通道:

// ...
wg.Wait()
close(jobs)
// ...
英文:

You are not using the waitgroup correctly. The defer's in main's for loop are never called since main never returns and as a result, wg.Wait() call never unblocks.

Put defer calls in the goroutine sending the message:

// ...
for i := 0; i &lt; thread; i++ {
wg.Add(1)
for _, word := range wordlist {
go func(word string) {
defer wg.Done()
jobs &lt;- word
}(word)
}
}
// ...

Also close the jobs channel:

// ...
wg.Wait()
close(jobs)
// ...

答案2

得分: 0

你应该为每个新的任务创建一个新的 goroutine,而不是为每个新添加的任务创建一个新的 goroutine。

在这里,你创建了大量的 goroutine 来将一个单词推送到通道中。对于这样一个简单的操作来说,这是过度设计了。

for _, word := range wordlist {
    go func(word string) {
        jobs <- word
    }(word)
}

但是获取 URL 的任务更加耗费资源,但你只为此任务启动了一个 goroutine。

go func() {
    for job := range jobs {
        code := visit(target + job)
        fmt.Println(target + job + " - " + strconv.Itoa(code))
    }
}()

正如 @abhink 提到的,你没有正确使用 WaitGroup。在创建每个 goroutine 之前使用 wg.Add(1),在每个 goroutine 结束时使用 wg.Done()。当然,使用 wg.Wait() 来等待所有正在运行的 goroutine 完成。示例:

var wg sync.WaitGroup

wg.Add(1)
go func(param) {
    defer wg.Done()
    //...
}(url)

//...

wg.Wait()
英文:

You should create new goroutine per new job, not per new job added

Here you're creating a log of goroutines to push only 1 word to channel. This is an overkill for such a simple operation.

    for _, word := range wordlist {
go func(word string) {
jobs &lt;- word
}(word)
}

But job with fetching url is more heavier, but start only 1 goroutine for this.

go func() {
for job := range jobs {
code := visit(target + job)
fmt.Println(target + job + &quot; - &quot; + strconv.Itoa(code))
}
}()

As @abhink has mentioned you're not using WaitGroup correctly. Do wg.Add(1) before creating each goroutine and do wg.Done() at the end of each goroutine. And of course wg.Wait() to wait all running goroutines to finish. Example:

var wg sync.WaitGroup
wg.Add(1)   
go func(param){
defer wg.Done()
//...
}(url)
//...
wg.Wait()

huangapple
  • 本文由 发表于 2017年6月17日 14:33:34
  • 转载请务必保留本文链接:https://go.coder-hub.com/44601563.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定