Requesting multiple URLs in Go

huangapple go评论85阅读模式
英文:

Requesting multiple URLs in Go

问题

我有以下的Go程序:https://play.golang.org/p/-TUtJ7DIhi

package main

import (
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
	"strconv"
)

func main() {
	body, err := get("https://hacker-news.firebaseio.com/v0/topstories.json")

	if err != nil {
		panic(err)
	}

	var ids [500]int
	if err = json.Unmarshal(body, &ids); err != nil {
		panic(err)
	}

	var contents []byte
	for _, value := range ids[0:10] {
		body, err := get("https://hacker-news.firebaseio.com/v0/item/" + strconv.Itoa(value) + ".json")

		if err != nil {
			fmt.Println(err)
		} else {
			contents = append(contents, body...)
		}
	}

	fmt.Println(contents)
}

func get(url string) ([]byte, error) {
	res, err := http.Get(url)
	if err != nil {
		return nil, err
	}

	body, err := ioutil.ReadAll(res.Body)
	res.Body.Close()

	return body, err
}

运行时,它在迭代的get请求中抛出EOF json错误,但是当我单独访问这些URL时,它们似乎没有格式错误。

我漏掉了什么?

英文:

I have the following Go program: https://play.golang.org/p/-TUtJ7DIhi

package main

import (
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
	"strconv"
)

func main() {
	body, err := get("https://hacker-news.firebaseio.com/v0/topstories.json")

	if err != nil {
		panic(err)
	}

	var ids [500]int
	if err = json.Unmarshal(body, &ids); err != nil {
		panic(err)
	}

	var contents []byte
	for _, value := range ids[0:10] {
		body, err := get("https://hacker-news.firebaseio.com/v0/item/" + strconv.Itoa(value) + ".json")

		if err != nil {
			fmt.Println(err)
		} else {
			contents = append(contents, body...)
		}
	}

	fmt.Println(contents)
}

func get(url string) ([]byte, error) {
	res, err := http.Get(url)
	if err != nil {
		return nil, err
	}

	body, err := ioutil.ReadAll(res.Body)
	res.Body.Close()

	return body, err
}

When run it throws EOF json errors on the iterative get requests, but when I hit the URLs individually they do not appear to be malformed.

What am I missing?

答案1

得分: 1

看起来他们的服务器出了一些问题,没有发送Connection: close头部就关闭了连接。根据HTTP/1.1规范,客户端会尝试重用连接。

你可以通过创建自己的请求并设置Close = true,或者使用自定义的Transport并设置DisableKeepAlives = true来解决这个问题。

req, err := http.NewRequest("GET", url, nil)
if err != nil {
    return nil, err
}
req.Close = true

res, err := http.DefaultClient.Do(req)
if err != nil {
    return nil, err
}
英文:

It looks like there's something wrong with their server, and it's closing connections without sending a Connection: close header. The client therefore tries to reuse the connection per the HTTP/1.1 specification.

You can work around this by creating your own request, and setting Close = true, or using a custom Transport with DisableKeepAlives = true

req, err := http.NewRequest("GET", url, nil)
if err != nil {
    return nil, err
}
req.Close = true

res, err := http.DefaultClient.Do(req)
if err != nil {
	return nil, err
}

huangapple
  • 本文由 发表于 2015年4月2日 23:59:28
  • 转载请务必保留本文链接:https://go.coder-hub.com/29417363.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定