Golang net/http上传大文件时出现未定义错误。

huangapple go评论103阅读模式
英文:

Golang net/http uploading large files, undefined errors

问题

我正在上传大文件到使用golang的默认net/http包创建的服务器时遇到一个相当难以调试的错误。上传的代码如下:

uploadForm.onsubmit = () => {
    const formData = new FormData(uploadForm);
    const isPublic : boolean = (<HTMLInputElement>document.getElementById('public_switch')).checked;
    formData.append('file', (<HTMLInputElement>document.getElementById('file')).files[0]);
    formData.append('compression', (<HTMLInputElement>document.getElementById('compression')).value);
    formData.append('public', String(isPublic));
    const xhr = new XMLHttpRequest();
    xhr.open("POST", "/upload/");
    xhr.send(formData);
    xhr.onreadystatechange = function() {
        console.log(xhr.responseText);
    }
}

我有一个用golang编写的服务器,我按照以下方式启动:

var server = &http.Server{
    Addr:         ":" + Configuration.Port,
    ReadTimeout:  300 * time.Second,
    WriteTimeout: 300 * time.Second,
    ReadHeaderTimeout: 300 * time.Second,
    MaxHeaderBytes: 500000000}

http.HandleFunc("/upload/", uploadFile)

server.ListenAndServe()

最后,我使用以下代码接受并解析文件:

func uploadFile(w http.ResponseWriter, r *http.Request) {
    //将上传参数解析为我们将使用的值
    r.ParseMultipartForm(5000000000000000)

    file, _, err := r.FormFile("file")
    //等等
}

现在,代码在r.FormFile("file")处失败,并显示非常描述性的错误消息:"multipart: NextPart: EOF"。

是否有一些文件限制或超时设置,我可能没有在go代码或JavaScript中设置?我尝试上传的文件大小约为1.7GB,因此明显符合http支持的限制。

有没有办法在不必深入研究FormFile或捕获请求本身的情况下更好地调试此问题?该代码在较小的文件(几MB)上运行得很好。

英文:

I'm having an error that is rather hard to debug when uploading large files to a server made using golang's default net/http package. The upload code looks like this:

    uploadForm.onsubmit = () =&gt; {
        const formData = new FormData(uploadForm);
        const isPublic : boolean = (&lt;HTMLInputElement&gt;document.getElementById(&#39;public_switch&#39;)).checked;
        formData.append(&#39;file&#39;, (&lt;HTMLInputElement&gt;document.getElementById(&#39;file&#39;)).files[0]);
        formData.append(&#39;compression&#39;, (&lt;HTMLInputElement&gt;document.getElementById(&#39;compression&#39;)).value);
        formData.append(&#39;public&#39;, String(isPublic));
        const xhr = new XMLHttpRequest();
        xhr.open(&quot;POST&quot;, &quot;/upload/&quot;);
        xhr.send(formData);
        xhr.onreadystatechange = function() {
            console.log(xhr.responseText);
        }
    }

I have a server written in golang which I start as follows:

var server = &amp;http.Server{
	Addr:         &quot;:&quot; + Configuration.Port,
	ReadTimeout:  300 * time.Second,
	WriteTimeout: 300 * time.Second,
	ReadHeaderTimeout: 300 * time.Second,
	MaxHeaderBytes: 500000000}

http.HandleFunc(&quot;/upload/&quot;, uploadFile)

server.ListenAndServe()

Finally I accept the and parse the file using the following code

func uploadFile(w http.ResponseWriter, r *http.Request) {
	//Parsing the upload arguments into the values we shall be working with
	r.ParseMultipartForm(5000000000000000)

	file, _, err := r.FormFile(&quot;file&quot;)
	//etc

Now, the code itself fails at 'r.FormFile("file")' with the very descriptive error message: "multipart: NextPart: EOF"

Is there some sort of setting on file limit or timeouts which I might not be setting either in the go code or in javascript ? The file I'm trying to upload is ~1.7GB so clearly fits within the limits supported by http.

Any idea how I could debug this issue a bit better without having to delve into FormFile or capture the request itself ? The code works just fine with smaller files ( a few Mb's).

答案1

得分: 0

好的,以下是翻译好的内容:

好的,为了回答我自己的问题,以防这些问题再次出现。

问题出在JavaScript端,我担心这与表单和文件的Web API不够完善有关。

这是我在JavaScript端想出的解决方案:

const uploadFile = (form_id) => {
    const uploadForm = document.getElementById(form_id);
    document.getElementById("submit_form").addEventListener("click", function(e) {
        e.preventDefault()

        let reader = new FileReader();
        reader.readAsArrayBuffer(document.getElementById('file').files[0]);

        reader.onload = function(evt) {

            const formData = new FormData(uploadForm);
            const isPublic = document.getElementById('public_switch').checked;
            formData.append('file', evt.target.result);
            formData.append('compression', document.getElementById('compression').value);
            formData.append('public', String(isPublic));
            const xhr = new XMLHttpRequest();
            xhr.open("POST", "/upload/");
            xhr.send(formData);
            xhr.onreadystatechange = function() {
                console.log(xhr.responseText + '  \n status is: ' + xhr.statusText);
            }

        };
    });

}

非常简单,但是花了很长时间才找到,因为在典型的JavaScript社区精神中,所有在线示例都杂乱无章。

处理大文件应该使用FileReader对象,基本上可以这样调用:

let reader = new FileReader();
reader.readAsArrayBuffer(dom_element_with_file_input.files[0]);

然后它就像任何其他异步Web API一样使用:

reader.onload = function(e) {
    const your_file = e.target.result
}

变量'your_file'然后用于对文件进行任何你想做的操作。在我的情况下,是将其附加到表单数据中,可以这样做:

reader.onload = function(e) {
    const your_file = e.target.result
    const formData = new FormData(dom_element_of_type_form);
    formData.append('my_file', your_file);
    formData.append('other_stuff', 'a string with metadata');
}

我将这个作为示例留下,因为实际上我找不到任何一个清晰的文件读取器API示例,除了那些杂乱地包含了数百行代码的示例。我几乎完全按照这个示例的方式进行操作,这个示例在这里:

https://developer.mozilla.org/en-US/docs/Using_files_from_web_applications。但是出于速度考虑,我使用了'readAsArrayBuffer'。

最后需要注意的是,读取器API似乎非常慢并且消耗大量资源,这意味着对于任意大小的文件,浏览器可能会崩溃。我在chromium上遇到了这个问题,仍然无法解决,所以我希望能得到建议。如果我找到解决浏览器崩溃问题的方法(或者至少找到一种优雅处理它们的方法),我将更新问题并将其标记为答案。同时,如果有人找到了与我的答案重复的答案或者解决浏览器崩溃问题的方法,请告诉我。

英文:

OK, so, to answer my own question in case this stuff comes up again.

The problem is poorly javascript side and I'm afraid it has to do with the fact that the web apis for form&files aren't really up to scruff.

This is the solution I came up with javascript side:

const uploadFile = (form_id: string) =&gt; {
    const uploadForm: HTMLFormElement = &lt;HTMLFormElement&gt;document.getElementById(form_id);
    document.getElementById(&quot;submit_form&quot;).addEventListener(&quot;click&quot;, function(e) {
        e.preventDefault()

        let reader: FileReader = new FileReader();
        reader.readAsArrayBuffer((&lt;HTMLInputElement&gt;document.getElementById(&#39;file&#39;)).files[0]);

        reader.onload = function(evt) {

            const formData = new FormData(uploadForm);
            const isPublic: boolean = (&lt;HTMLInputElement&gt;document.getElementById(&#39;public_switch&#39;)).checked;
            formData.append(&#39;file&#39;, (&lt;any&gt;evt.target).result);
            formData.append(&#39;compression&#39;, (&lt;HTMLInputElement&gt;document.getElementById(&#39;compression&#39;)).value);
            formData.append(&#39;public&#39;, String(isPublic));
            const xhr = new XMLHttpRequest();
            xhr.open(&quot;POST&quot;, &quot;/upload/&quot;);
            xhr.send(formData);
            xhr.onreadystatechange = function() {
                console.log(xhr.responseText + &#39;  \n status is: &#39; + xhr.statusText);
            }

        };
    });

}

Quite simple really, but it took a long time to find since in the typical javascipt-community spirit all online examples are cluttered with shit.

The handling of large files should be done using a FileReader object which is basically called as such:

let reader = new FileReader();
reader.readAsArrayBuffer(dom_element_with_file_input.files[0]);

Then its used much like any other async web api:

reader.onload = function(e) {
    const your_file = e.target.result
}

The variable 'your_file' is then used for whatever you want to do with the file. In my case it was appending to form data which is done like this:

reader.onload = function(e) {
    const your_file = e.target.result
    const formData = new FormData(dom_element_of_type_form);
    formData.append(&#39;my_file&#39;, your_file);
    formData.append(&#39;other_stuff&#39;, &#39;a string with metadata&#39;);
}

I'm leaving this as an example since I could not actually find any so post with a clear example of the file Reader API, expect for ones cluttered in literally hundreds of lines of code. The example I followed almost to the T is the one here:

https://developer.mozilla.org/en-US/docs/Using_files_from_web_applications. But I used 'readAsArrayBuffer' instead for speed purposes.

One last note is that the reader API seems to be very slow and consumes a lot of ressource, which means that for arbitrary file sizes browser may simply crash, I had this problem with chromioum and I still can't fix it, so I'm open to suggestions, if I find a fix for browser crashes (or at least a way to handle them gracefully) I will update the question and mark it as the answer, in the meanwhile if anyone find a duplicate of my answer or a way to fix the browser crashes please tell me.

huangapple
  • 本文由 发表于 2017年6月11日 00:50:46
  • 转载请务必保留本文链接:https://go.coder-hub.com/44475783.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定