高效地从Go服务器流式传输大文件(视频)

huangapple go评论111阅读模式
英文:

efficiently stream large file (video) from go server

问题

我正在使用Go语言实现视频流服务器。
我目前正在使用标准的fileserver,但我不确定它在处理大文件(4GB+)时是否高效。

有没有一种在Go语言中高效地提供大文件的方法?

英文:

I am implementing video stream server in Go.
I am currently using standard fileserver but I am unsure if it is efficient with large files (4GB+).

Is there a way to efficiently serve large file in Go?

答案1

得分: 2

我不确定你所说的“高效”是什么意思,所以我会假设你的意思是大文件是以流的方式传输而不是缓冲的。

标准的http.FileServer最终使用serveContent函数来写入数据,该函数需要一个io.ReadSeeker作为内容(幸运的是,文件就是这样的类型)。

然后使用io.Copy来复制内容,一般情况下(尽管可能不是你常见的情况,见下文),会使用copyBuffer来使用32KB的缓冲区。

因此,假设http.ResponseWriter的实现没有缓冲其输入(它没有,参见chunked writer),内存利用应该是恒定的。

然而,当写入器支持ReadFrom方法时,io.Copy将使用该方法。因为http.responseResponseWriter接口的标准实现)实现了ReadFrom方法,所以它将被用于替代copyBuffer。这个实现将尝试在可能的情况下使用sendfile系统调用(对于os.File就是这种情况),这是更高效的意思(数据根本不需要通过进程的内存空间,所以效率非常高)。

换句话说,我认为内置的net/http包已经支持对大文件进行高效的流式传输。

英文:

I'm not sure what you mean by "efficient" so I will assume you mean that large files are streamed and not buffered.

The standard http.FileServer eventually writes data using the serveContent function, which requires an io.ReadSeeker as the contents (fortunately files are just that).

The contents is then copied using io.Copy, which in the general case (though probably not your common case, see below) means copyBuffer. will use a 32KB buffer.

Therefore, assuming the implementation of http.ResponseWriter does not buffer its input (it doesn't, see also chunked writer), memory utilization should be constant.

However when the writer supports a ReadFrom method, io.Copy will use that instead. Because http.response (the standard implementation of ResponseWriter interface) implements the ReadFrom method, it'll be used instead of copyBuffer. This implementation in turn will try use the sendfile system call when possible (as is the case for os.File), which is a stronger meaning of efficient (the data doesn't have to go through the process memory space at all, so about as good as it gets).

In other words, I think it's reasonable to say that the built in net/http package already supports efficient streaming of large files.

huangapple
  • 本文由 发表于 2017年2月19日 02:06:09
  • 转载请务必保留本文链接:https://go.coder-hub.com/42318795.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定