Golang download multiple files in parallel using goroutines

huangapple go评论119阅读模式
英文:

Golang download multiple files in parallel using goroutines

问题

使用goroutines并行下载和保存文件是可能的。在你的代码中,当你在download_file函数前面加上go命令时,它不起作用的原因是你没有等待goroutine完成。你可以使用sync.WaitGroup来等待所有的goroutine完成。下面是修改后的代码示例:

  1. package main
  2. import (
  3. "encoding/json"
  4. "fmt"
  5. "io"
  6. "io/ioutil"
  7. "net/http"
  8. "net/url"
  9. "os"
  10. "path/filepath"
  11. "sync"
  12. )
  13. const app_key string = "<app_key>"
  14. const app_secret string = "<app_secret>"
  15. var code string
  16. type TokenResponse struct {
  17. AccessToken string `json:"access_token"`
  18. }
  19. type File struct {
  20. Path string
  21. }
  22. type FileListResponse struct {
  23. FileList []File `json:"contents"`
  24. }
  25. func download_file(file File, token TokenResponse, wg *sync.WaitGroup) {
  26. defer wg.Done()
  27. download_file := fmt.Sprintf("https://api-content.dropbox.com/1/files/dropbox/%s?access_token=%s", file.Path, token.AccessToken)
  28. resp, _ := http.Get(download_file)
  29. defer resp.Body.Close()
  30. filename := filepath.Base(file.Path)
  31. out, err := os.Create(filename)
  32. if err != nil {
  33. panic(err)
  34. }
  35. defer out.Close()
  36. io.Copy(out, resp.Body)
  37. }
  38. func main() {
  39. authorize_url := fmt.Sprintf("https://www.dropbox.com/1/oauth2/authorize?response_type=code&client_id=%s", app_key)
  40. // Get code
  41. fmt.Printf("1. Go to: %s\n", authorize_url)
  42. fmt.Println("2. Click 'Allow' (you might have to log in first)")
  43. fmt.Println("3. Copy the authorization code.")
  44. fmt.Printf("Enter the authorization code here: ")
  45. fmt.Scanf("%s", &code)
  46. // End get code
  47. // Get access token
  48. data := url.Values{}
  49. data.Add("code", code)
  50. data.Add("grant_type", "authorization_code")
  51. data.Add("client_id", app_key)
  52. data.Add("client_secret", app_secret)
  53. resp, _ := http.PostForm("https://api.dropbox.com/1/oauth2/token", data)
  54. defer resp.Body.Close()
  55. contents, _ := ioutil.ReadAll(resp.Body)
  56. var tr TokenResponse
  57. json.Unmarshal(contents, &tr)
  58. // End get access token
  59. // Get file list
  60. file_list_url := fmt.Sprintf("https://api.dropbox.com/1/metadata/dropbox/Camera Uploads?access_token=%s", tr.AccessToken)
  61. resp2, _ := http.Get(file_list_url)
  62. defer resp2.Body.Close()
  63. contents2, _ := ioutil.ReadAll(resp2.Body)
  64. var flr FileListResponse
  65. json.Unmarshal(contents2, &flr)
  66. // End get file list
  67. var wg sync.WaitGroup
  68. for i, file := range flr.FileList {
  69. wg.Add(1)
  70. go download_file(file, tr, &wg)
  71. if i >= 2 {
  72. break
  73. }
  74. }
  75. wg.Wait()
  76. }

在修改后的代码中,我们使用了sync.WaitGroup来等待所有的goroutine完成。每个goroutine在完成下载和保存文件后,调用wg.Done()来通知WaitGroup完成一个goroutine的执行。在主函数中,我们使用wg.Wait()来等待所有的goroutine完成。这样就可以实现并行下载和保存文件的功能。

英文:

Is it possible to download and save files in parallel using goroutines?

Below is my code which downloads files from my dropbox:

  1. package main
  2. import (
  3. &quot;encoding/json&quot;
  4. &quot;fmt&quot;
  5. &quot;io&quot;
  6. &quot;io/ioutil&quot;
  7. &quot;net/http&quot;
  8. &quot;net/url&quot;
  9. &quot;os&quot;
  10. &quot;path/filepath&quot;
  11. )
  12. const app_key string = &quot;&lt;app_key&gt;&quot;
  13. const app_secret string = &quot;&lt;app_secret&gt;&quot;
  14. var code string
  15. type TokenResponse struct {
  16. AccessToken string `json:&quot;access_token&quot;`
  17. }
  18. type File struct {
  19. Path string
  20. }
  21. type FileListResponse struct {
  22. FileList []File `json:&quot;contents&quot;`
  23. }
  24. func download_file(file File, token TokenResponse) {
  25. download_file := fmt.Sprintf(&quot;https://api-content.dropbox.com/1/files/dropbox/%s?access_token=%s&quot;, file.Path, token.AccessToken)
  26. resp, _ := http.Get(download_file)
  27. defer resp.Body.Close()
  28. filename := filepath.Base(file.Path)
  29. out, err := os.Create(filename)
  30. if err != nil {
  31. panic(err)
  32. }
  33. defer out.Close()
  34. io.Copy(out, resp.Body)
  35. }
  36. func main() {
  37. authorize_url := fmt.Sprintf(&quot;https://www.dropbox.com/1/oauth2/authorize?response_type=code&amp;client_id=%s&quot;, app_key)
  38. // Get code
  39. fmt.Printf(&quot;1. Go to: %s\n&quot;, authorize_url)
  40. fmt.Println(&quot;2. Click &#39;Allow&#39; (you might have to log in first)&quot;)
  41. fmt.Println(&quot;3. Copy the authorization code.&quot;)
  42. fmt.Printf(&quot;Enter the authorization code here: &quot;)
  43. fmt.Scanf(&quot;%s&quot;, &amp;code)
  44. // End get code
  45. // Get access token
  46. data := url.Values{}
  47. data.Add(&quot;code&quot;, code)
  48. data.Add(&quot;grant_type&quot;, &quot;authorization_code&quot;)
  49. data.Add(&quot;client_id&quot;, app_key)
  50. data.Add(&quot;client_secret&quot;, app_secret)
  51. resp, _ := http.PostForm(&quot;https://api.dropbox.com/1/oauth2/token&quot;, data)
  52. defer resp.Body.Close()
  53. contents, _ := ioutil.ReadAll(resp.Body)
  54. var tr TokenResponse
  55. json.Unmarshal(contents, &amp;tr)
  56. // End get access token
  57. // Get file list
  58. file_list_url := fmt.Sprintf(&quot;https://api.dropbox.com/1/metadata/dropbox/Camera Uploads?access_token=%s&quot;, tr.AccessToken)
  59. resp2, _ := http.Get(file_list_url)
  60. defer resp2.Body.Close()
  61. contents2, _ := ioutil.ReadAll(resp2.Body)
  62. var flr FileListResponse
  63. json.Unmarshal(contents2, &amp;flr)
  64. // End get file list
  65. for i, file := range flr.FileList {
  66. download_file(file, tr)
  67. if i &gt;= 2 {
  68. break
  69. }
  70. }
  71. }

It doesn't work when I prefix the download_file function with the go command.

  1. go download_file(file, tr)

答案1

得分: 13

这是因为你的主 goroutine 正在退出。你需要添加一个 WaitGroup 来等待所有的 goroutine 退出。例如,

  1. var wg sync.WaitGroup
  2. for i, file := range flr.FileList {
  3. wg.Add(1)
  4. go download_file(file, tr, wg)
  5. if i >= 2 {
  6. break
  7. }
  8. }
  9. wg.Wait()
  10. ...
  11. func download_file(file File, token TokenResponse, wg sync.WaitGroup) {
  12. ...
  13. wg.Done()
  14. }
英文:

That's because your main goroutine is exiting. You need to add a WaitGroup to wait until all the goroutines exit. For example,

  1. var wg sync.WaitGroup
  2. for i, file := range flr.FileList {
  3. wg.Add(1)
  4. go download_file(file, tr, wg)
  5. if i &gt;= 2 {
  6. break
  7. }
  8. }
  9. wg.Wait()
  10. ...
  11. func download_file(file File, token TokenResponse, wg sync.WaitGroup) {
  12. ...
  13. wg.Done()
  14. }

答案2

得分: 0

这个项目可能会帮助你和其他想要在Go语言中实现并发的人:

这个想法是在Go语言中创建一个集群来执行并行任务。https://github.com/waqar-alamgir/mini-go-cluster

可以对源代码进行调整以实现并发下载文件。

英文:

This project might help you and others that are looking into achieving concurrency in Go:

Idea is to create a cluster in Go to execute parallel jobs. https://github.com/waqar-alamgir/mini-go-cluster

Source can be tweaked to download files concurrently.

huangapple
  • 本文由 发表于 2014年5月13日 23:10:16
  • 转载请务必保留本文链接:https://go.coder-hub.com/23635070.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定