Requesting multiple URLs in Go

huangapple
go评论112阅读模式
英文:

Requesting multiple URLs in Go

问题

我有以下的Go程序：https://play.golang.org/p/-TUtJ7DIhi

  1. package main
  3. import (
  4. 	"encoding/json"
  5. 	"fmt"
  6. 	"io/ioutil"
  7. 	"net/http"
  8. 	"strconv"
  9. )
  11. func main() {
  12. 	body, err := get("https://hacker-news.firebaseio.com/v0/topstories.json")
  14. 	if err != nil {
  15. 		panic(err)
  16. 	}
  18. 	var ids [500]int
  19. 	if err = json.Unmarshal(body, &ids); err != nil {
  20. 		panic(err)
  21. 	}
  23. 	var contents []byte
  24. 	for _, value := range ids[0:10] {
  25. 		body, err := get("https://hacker-news.firebaseio.com/v0/item/" + strconv.Itoa(value) + ".json")
  27. 		if err != nil {
  28. 			fmt.Println(err)
  29. 		} else {
  30. 			contents = append(contents, body...)
  31. 		}
  32. 	}
  34. 	fmt.Println(contents)
  35. }
  37. func get(url string) ([]byte, error) {
  38. 	res, err := http.Get(url)
  39. 	if err != nil {
  40. 		return nil, err
  41. 	}
  43. 	body, err := ioutil.ReadAll(res.Body)
  44. 	res.Body.Close()
  46. 	return body, err
  47. }

运行时，它在迭代的get请求中抛出EOF json错误，但是当我单独访问这些URL时，它们似乎没有格式错误。

我漏掉了什么？

英文:

I have the following Go program: https://play.golang.org/p/-TUtJ7DIhi

  1. package main
  3. import (
  4. 	"encoding/json"
  5. 	"fmt"
  6. 	"io/ioutil"
  7. 	"net/http"
  8. 	"strconv"
  9. )
  11. func main() {
  12. 	body, err := get("https://hacker-news.firebaseio.com/v0/topstories.json")
  14. 	if err != nil {
  15. 		panic(err)
  16. 	}
  18. 	var ids [500]int
  19. 	if err = json.Unmarshal(body, &ids); err != nil {
  20. 		panic(err)
  21. 	}
  23. 	var contents []byte
  24. 	for _, value := range ids[0:10] {
  25. 		body, err := get("https://hacker-news.firebaseio.com/v0/item/" + strconv.Itoa(value) + ".json")
  27. 		if err != nil {
  28. 			fmt.Println(err)
  29. 		} else {
  30. 			contents = append(contents, body...)
  31. 		}
  32. 	}
  34. 	fmt.Println(contents)
  35. }
  37. func get(url string) ([]byte, error) {
  38. 	res, err := http.Get(url)
  39. 	if err != nil {
  40. 		return nil, err
  41. 	}
  43. 	body, err := ioutil.ReadAll(res.Body)
  44. 	res.Body.Close()
  46. 	return body, err
  47. }

When run it throws EOF json errors on the iterative get requests, but when I hit the URLs individually they do not appear to be malformed.

What am I missing?

答案1

得分: 1

看起来他们的服务器出了一些问题，没有发送Connection: close头部就关闭了连接。根据HTTP/1.1规范，客户端会尝试重用连接。

你可以通过创建自己的请求并设置Close = true，或者使用自定义的Transport并设置DisableKeepAlives = true来解决这个问题。

  1. req, err := http.NewRequest("GET", url, nil)
  2. if err != nil {
  3.     return nil, err
  4. }
  5. req.Close = true
  7. res, err := http.DefaultClient.Do(req)
  8. if err != nil {
  9.     return nil, err
  10. }
英文:

It looks like there's something wrong with their server, and it's closing connections without sending a Connection: close header. The client therefore tries to reuse the connection per the HTTP/1.1 specification.

You can work around this by creating your own request, and setting Close = true, or using a custom Transport with DisableKeepAlives = true

  1. req, err := http.NewRequest("GET", url, nil)
  2. if err != nil {
  3.     return nil, err
  4. }
  5. req.Close = true
  7. res, err := http.DefaultClient.Do(req)
  8. if err != nil {
  9. 	return nil, err
  10. }

huangapple
  • 本文由 发表于 2015年4月2日 23:59:28
  • 转载请务必保留本文链接：https://go.coder-hub.com/29417363.html
  • go
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定