在两个函数之间传递一个`Reader`对象

huangapple go评论70阅读模式
英文:

Sending a `Reader` between two functions

问题

我正在尝试进行一个GET请求,然后将获取到的内容发送到另一个站点的POST请求中。以下代码可以正常工作:

func deliver(sourceURL *url.URL, destinationURL *url.URL) {
    client := &http.Client{}
    sourceResponse, err := http.Get(sourceURL.String())
    if err == nil {
        body, _ := ioutil.ReadAll(sourceResponse.Body)
        client.Post(destinationURL.String(), "text/html", strings.NewReader(string(body)))
    }
}

但是这种方式将整个响应读入内存中,我该如何将响应流式传输到POST请求中呢?以下是我尝试的方式:

func deliver(sourceURL *url.URL, destinationURL *url.URL) {
    client := &http.Client{}
    sourceResponse, err := http.Get(sourceURL.String())
    if err == nil {
        client.Post(destinationURL.String(), "text/html", sourceResponse.Body)
    }
}

我原以为这样可以工作,但实际上它只发送了一个空请求(没有正文)。为什么会这样?难道client.Post函数不会尝试读取响应吗?

这里有一个在Go Playground上的示例:http://play.golang.org/p/_EO80aj6wr
dataReader而不是sourceResponse.Body发送过去可以完美地工作,但直接发送sourceResponse.Body却不行。

示例链接:

英文:

I'm trying to make a GET request and then send the content I get to another site in a POST request. The following code works fine:

func deliver(sourceURL *url.URL, destinationURL *url.URL) {
	client := &http.Client{}
	sourceResponse, err := http.Get(sourceURL.String())
	if err == nil {
		body, _ := ioutil.ReadAll(sourceResponse.Body)
		client.Post(destinationURL.String(), "text/html", strings.NewReader(string(body)))
	}
}

But this reads the entire response into memory - how to I stream the response into the POST request instead? Here's what I'm trying:

func deliver(sourceURL *url.URL, destinationURL *url.URL) {
	client := &http.Client{}
	sourceResponse, err := http.Get(sourceURL.String())
	if err == nil {
		client.Post(destinationURL.String(), "text/html", sourceResponse.Body)
	}
}

I'd assume this would work, but it sends only an empty request (no body). Why is this? Doesn't the client.Post function actually attempt to read the response?

Here's an example on the Go playground: http://play.golang.org/p/_EO80aj6wr
Sending the dataReader instead of sourceResponse.Body works perfectly, but sending the sourceResponse.Body directly doesn't work.

Example Links:

http://localhost:3000/fetch?dest=http://requestb.in/177ji621&src=http://www.google.com
http://requestb.in/177ji621?inspect

答案1

得分: 2

我做了一个小测试程序来尝试一下:

package main

import (
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
	"net/url"
)

var client = &http.Client{}

func httpd() {
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		log.Printf("%s Request from %s", r.Method, r.RemoteAddr)
		if r.Method == "GET" {
			fmt.Fprintf(w, "Here is a GET body")
			return
		}
		body, err := ioutil.ReadAll(r.Body)
		if err != nil {
			log.Fatalf("body read failed: %v", err)
		}
		log.Printf("Body received from %s: %q", r.Method, body)
	})
	log.Fatal(http.ListenAndServe(":8080", nil))
}

func deliver(sourceURL *url.URL, destinationURL *url.URL) {
	sourceResponse, err := http.Get(sourceURL.String())
	if err != nil {
		log.Fatalf("post failed: %v", err)
	}
	postResponse, err := client.Post(destinationURL.String(), "text/html", sourceResponse.Body)
	if err != nil {
		log.Fatalf("post failed: %v", err)
	}
	postResponse.Body.Close()
	sourceResponse.Body.Close()
}

func main() {
	go httpd()
	src, err := url.Parse("http://127.0.0.1:8080/")
	if err != nil {
		log.Fatalf("src parse failed: %v", err)
	}
	dst, err := url.Parse("http://127.0.0.1:8080/")
	if err != nil {
		log.Fatalf("dst parse failed: %v", err)
	}
	deliver(src, dst)
}

它正常工作,并产生以下输出:

2014/10/04 08:40:26 GET Request from 127.0.0.1:48910
2014/10/04 08:40:26 POST Request from 127.0.0.1:48911
2014/10/04 08:40:26 Body received from POST: "Here is a GET body"

所以你的代码问题可能会通过你没有检查的错误来揭示出来,最有可能是在 Post 中的错误。

另外请注意,如果可能的话,你希望重用 http.Client,这样可以使用连接池。

更新:在查看问题的更新后,我使用 http://requestb.in/ 运行了我的测试程序,它没有显示请求体。但是我看到了 Transfer-Encoding: chunked,所以我怀疑 requestb.in 不支持分块传输编码。

如果你向请求提供了完整的请求体,那么 Go 会设置 Content-Length 头部,但是如果你提供了一个流,Go 就会使用分块传输编码。尽管分块传输编码是 HTTP/1.1 规范的一个非可选部分,但有很多服务器不正确地支持它。

英文:

I made a little test program to try this out

package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"net/url"
)
var client = &http.Client{}
func httpd() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
log.Printf("%s Request from %s", r.Method, r.RemoteAddr)
if r.Method == "GET" {
fmt.Fprintf(w, "Here is a GET body")
return
}
body, err := ioutil.ReadAll(r.Body)
if err != nil {
log.Fatalf("body read failed: %v", err)
}
log.Printf("Body received from %s: %q", r.Method, body)
})
log.Fatal(http.ListenAndServe(":8080", nil))
}
func deliver(sourceURL *url.URL, destinationURL *url.URL) {
sourceResponse, err := http.Get(sourceURL.String())
if err != nil {
log.Fatalf("post failed: %v", err)
}
postResponse, err := client.Post(destinationURL.String(), "text/html", sourceResponse.Body)
if err != nil {
log.Fatalf("post failed: %v", err)
}
postResponse.Body.Close()
sourceResponse.Body.Close()
}
func main() {
go httpd()
src, err := url.Parse("http://127.0.0.1:8080/")
if err != nil {
log.Fatalf("src parse failed: %v", err)
}
dst, err := url.Parse("http://127.0.0.1:8080/")
if err != nil {
log.Fatalf("dst parse failed: %v", err)
}
deliver(src, dst)
}

It worked fine, producing this output

2014/10/04 08:40:26 GET Request from 127.0.0.1:48910
2014/10/04 08:40:26 POST Request from 127.0.0.1:48911
2014/10/04 08:40:26 Body received from POST: "Here is a GET body"

So the problem with your code will probably be revealed by the errors you aren't checking - the one in the Post most likely.

Also note that you want to re-use http.Client if you can, so you can get connection pooling.


Update after reviewing update to question.

I tried my test program with http://requestb.in/ and it doesn't show the body. However I do see this Transfer-Encoding: chunked so my suspicion is that requestb.in doesn't support chunked transfer encoding.

If you provide the entire body to the request then go sets the Content-Length header, however if you provide a stream then go uses chunked transfer encoding. Even though chunked transfer encoding is a non optional part of the HTTP/1.1 spec there are quite a few servers which don't support it properly.

huangapple
  • 本文由 发表于 2014年10月4日 14:01:52
  • 转载请务必保留本文链接:https://go.coder-hub.com/26190173.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定