英文:
efficiently stream large file (video) from go server
问题
我正在使用Go语言实现视频流服务器。
我目前正在使用标准的fileserver,但我不确定它在处理大文件(4GB+)时是否高效。
有没有一种在Go语言中高效地提供大文件的方法?
英文:
I am implementing video stream server in Go.
I am currently using standard fileserver but I am unsure if it is efficient with large files (4GB+).
Is there a way to efficiently serve large file in Go?
答案1
得分: 2
我不确定你所说的“高效”是什么意思,所以我会假设你的意思是大文件是以流的方式传输而不是缓冲的。
标准的http.FileServer
最终使用serveContent函数来写入数据,该函数需要一个io.ReadSeeker
作为内容(幸运的是,文件就是这样的类型)。
然后使用io.Copy
来复制内容,一般情况下(尽管可能不是你常见的情况,见下文),会使用copyBuffer
来使用32KB的缓冲区。
因此,假设http.ResponseWriter
的实现没有缓冲其输入(它没有,参见chunked writer),内存利用应该是恒定的。
然而,当写入器支持ReadFrom
方法时,io.Copy
将使用该方法。因为http.response
(ResponseWriter
接口的标准实现)实现了ReadFrom方法,所以它将被用于替代copyBuffer
。这个实现将尝试在可能的情况下使用sendfile
系统调用(对于os.File
就是这种情况),这是更高效的意思(数据根本不需要通过进程的内存空间,所以效率非常高)。
换句话说,我认为内置的net/http
包已经支持对大文件进行高效的流式传输。
英文:
I'm not sure what you mean by "efficient" so I will assume you mean that large files are streamed and not buffered.
The standard http.FileServer
eventually writes data using the serveContent
function, which requires an io.ReadSeeker
as the contents (fortunately files are just that).
The contents is then copied using io.Copy
, which in the general case (though probably not your common case, see below) means copyBuffer
. will use a 32KB buffer.
Therefore, assuming the implementation of http.ResponseWriter
does not buffer its input (it doesn't, see also chunked writer), memory utilization should be constant.
However when the writer supports a ReadFrom
method, io.Copy
will use that instead. Because http.response
(the standard implementation of ResponseWriter
interface) implements the ReadFrom method, it'll be used instead of copyBuffer
. This implementation in turn will try use the sendfile
system call when possible (as is the case for os.File
), which is a stronger meaning of efficient (the data doesn't have to go through the process memory space at all, so about as good as it gets).
In other words, I think it's reasonable to say that the built in net/http
package already supports efficient streaming of large files.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论