io.Copy 写入:大文件的管道中断

huangapple go评论75阅读模式
英文:

io.Copy write: broken pipe for big file

问题

我有一个文件服务器和一个代理服务器,通过代理服务器可以通过http访问文件服务器。

简单的文件服务器:

	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		http.ServeFile(w, r, "tmp/1.mp4")
	})

	if err := http.ListenAndServe("localhost:3000", nil); err != http.ErrServerClosed {
		log.Fatalf("HTTP server ListenAndServe: %v", err)
	}

和代理服务器:

type Server struct {
	conn net.Conn
}

func (s *Server) Handle(w http.ResponseWriter, r *http.Request) error {
	if err := r.Write(s.conn); err != nil {
		log.Fatal("req.Write ", err)
	}

	resp, err := http.ReadResponse(bufio.NewReader(s.conn), r)
	if err != nil {
		return fmt.Errorf("http.ReadResponse: %s", err)
	}

	defer func() {
		if resp.Body != nil {
			if err := resp.Body.Close(); err != nil && err != io.ErrUnexpectedEOF {
				fmt.Printf("resp.Body Close error: %s", err)
			}
		}
	}()

	copyHeader(w.Header(), resp.Header)
	w.WriteHeader(resp.StatusCode)

	if _, err := io.Copy(w, resp.Body); err != nil {
		if err == io.ErrUnexpectedEOF {
			fmt.Println("Client closed the connection, couldn't copy response")
		} else {
			fmt.Printf("copy err: %s\n", err)
		}
	}
	return nil
}

func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	if err := s.Handle(w, r); err != nil {
		http.Error(w, err.Error(), http.StatusBadGateway)
	}
}

func copyHeader(dst, src http.Header) {
	for k, v := range src {
		vv := make([]string, len(v))
		copy(vv, v)
		dst[k] = vv
	}
}

func main() {

	conn, err := net.Dial("tcp", "localhost:3000")

	if err != nil {
		fmt.Errorf("net.Dial(tcp, localhost:3000) %s", err)
	}
	server := &Server{conn}
	log.Fatal(http.ListenAndServe("localhost:8000", server))

}

如果文件服务器分发一个小图片,那么一切正常工作。
如果文件服务器提供一个大的视频文件(1.mp4),那么我会得到以下错误:

copy err: readfrom tcp 127.0.0.1:8000->127.0.0.1:58964: write tcp 127.0.0.1:8000->127.0.0.1:58964: write: broken pipe

可能的问题是什么?

英文:

I have a file server and a proxy server through which I can access the file server via http

Simple file server :


	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		http.ServeFile(w, r, "tmp/1.mp4")
	})

	if err := http.ListenAndServe("localhost:3000", nil); err != http.ErrServerClosed {
		log.Fatalf("HTTP server ListenAndServe: %v", err)
	}

and proxy server :


type Server struct {
conn net.Conn
}
func (s *Server) Handle(w http.ResponseWriter, r *http.Request) error {
if err := r.Write(s.conn); err != nil {
log.Fatal("req.Write ", err)
}
resp, err := http.ReadResponse(bufio.NewReader(s.conn), r)
if err != nil {
return fmt.Errorf("http.ReadResponse: %s", err)
}
defer func() {
if resp.Body != nil {
if err := resp.Body.Close(); err != nil && err != io.ErrUnexpectedEOF {
fmt.Printf("resp.Body Close error: %s", err)
}
}
}()
copyHeader(w.Header(), resp.Header)
w.WriteHeader(resp.StatusCode)
if _, err := io.Copy(w, resp.Body); err != nil {
if err == io.ErrUnexpectedEOF {
fmt.Println("Client closed the connection, couldn't copy response")
} else {
fmt.Printf("copy err: %s\n", err)
}
}
return nil
}
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if err := s.Handle(w, r); err != nil {
http.Error(w, err.Error(), http.StatusBadGateway)
}
}
func copyHeader(dst, src http.Header) {
for k, v := range src {
vv := make([]string, len(v))
copy(vv, v)
dst[k] = vv
}
}
func main() {
conn, err := net.Dial("tcp", "localhost:3000")
if err != nil {
fmt.Errorf("net.Dial(tcp, localhost:3000) %s", err)
}
server := &Server{conn}
log.Fatal(http.ListenAndServe("localhost:8000", server))
}

if the file server distributes a small picture, then everything works correctly
if the file server is serving a large video file (1.mp4) then I get the error:

copy err: readfrom tcp 127.0.0.1:8000->127.0.0.1:58964: write tcp 127.0.0.1:8000->127.0.0.1:58964: write: broken pipe

What could be the problem ?

答案1

得分: 2

broken pipe是与错误码32 EPIPE相关联的Golang消息。

在处理TCP连接时,当你的进程在另一端已经关闭连接时向其写入数据时,会触发该错误。


根据你对“客户端关闭连接”的理解,如果你的意图是寻找一个指示连接到你的代理的外部客户端提前关闭连接的指标,那么你应该查找EPIPE错误,而不是io.ErrUnexpectedEOF

可以使用以下代码:

func isEPIPE(err error) bool {
    var syscallErr syscall.Errno
    b := errors.As(err, &syscallErr) && syscallErr == syscall.EPIPE

    return b
}


...

if isEPIPE(err) {
    fmt.Println("客户端关闭了连接,无法复制响应")
} else {
    fmt.Printf("复制错误:%s\n", err)
}

io.ErrUnexpectedEOF可能在某些情况下发生,但它是由与文件服务器的连接引发的,而不是由客户端发起的连接。

英文:

broken pipe is the golang message linked to errno 32 EPIPE.

When dealing with a TCP connection, it is triggered when your process writes to a connection when the other end has already closed it.


Depening on what you mean by "Client closed the connection", if your intention is to look for an indicator that the outside client that connected to your proxy closed its connection early, then you should be looking for EPIPE errors, not for io.ErrUnexpectedEOF.

Something like :

func isEPIPE(err error) bool {
var syscallErr syscall.Errno
b := errors.As(err, &syscallErr) && syscallErr == syscall.EPIPE
return b
}
...
if isEPIPE(err) {
fmt.Println("Client closed the connection, couldn't copy response")
} else {
fmt.Printf("copy err: %s\n", err)
}

io.ErrUnexpectedEOF could probably occur in some situation, but it would be triggered from the connection with your file server, not from the connection initiated by the client.

答案2

得分: 0

当浏览器接收到一个mp4文件时,它会切换到流媒体模式。也就是说,它开始发送带有范围头的HTTP请求,并期望得到一个状态码为206的响应。

我的代理服务器总是以状态码200作为响应。

copyHeader(w.Header(), resp.Header)
w.WriteHeader(resp.StatusCode)

这会强制浏览器断开连接。

英文:

The browser, when receiving an mp4 file, switches to streaming mode
That is, it starts sending http requests with the range header and expects a response with a status of 206

My proxy server always responds with the status 200

 copyHeader(w.Header(), resp.Header)
w.WriteHeader(resp.StatusCode)

this forces the browser to disconnect the connection

huangapple
  • 本文由 发表于 2021年7月13日 18:28:44
  • 转载请务必保留本文链接:https://go.coder-hub.com/68360697.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定