Golang在简单的Web服务器上出现高CPU使用率,无法理解原因。

huangapple go评论76阅读模式
英文:

Golang high cpu usage on simple webserver unable to understand why?

问题

所以我有一个简单的net/http web服务器。它的功能只是传送100MB的随机字节,我打算用它来进行网络速度测试。我为100MB端点编写的处理程序非常简单(如下所示)。代码运行正常,我可以得到随机字节文件,问题是当我运行这个程序并有人下载这100兆字节时,该程序的CPU占用率会飙升到150%,并保持在这个水平,直到处理程序运行完成。我在这里做错了什么吗?我该如何改进这个处理程序的性能?

func downloadHandler(w http.ResponseWriter, r *http.Request) {
    str := RandStringBytes(8192) //生成8192字节的随机数据
    sz := 1000*1000*100 //100兆字节
    iter := sz/len(str)+1
    w.Header().Set("Content-Type", "application/octet-stream")
    w.Header().Set("Content-Length", strconv.Itoa(sz))
    for i := 0; i < iter; i++ {
        fmt.Fprintf(w, str)
    }
}
英文:

So I have a simple net/http webserver. All it does is is deliver 100MB of random bytes, which I intend to use for network speed testing. My handler for the 100mb endpoint is really simple (pasted below). The code works fine and I get my random byte file, the problem is when I run this and someone downloads these 100megabytes, the CPU for this program shoots up to 150% and stays there until this handler finishes running. Am I doing something very wrong here? What could I do to improve this handler's performance?

func downloadHandler(w http.ResponseWriter, r *http.Request) {
    str := RandStringBytes(8192); //generates 8192 bytes of randomness
    sz := 1000*1000*100; //100Megabytes
    iter := sz/len(str)+1;
    w.Header().Set(&quot;Content-Type&quot;, &quot;application/octet-stream&quot;)
    w.Header().Set(&quot;Content-Length&quot;, strconv.Itoa( sz ))
    for i := 0; i &lt; iter ; i++ {
	    fmt.Fprintf(w, str )
    }
}

答案1

得分: 6

问题在于fmt.Fprintf()函数需要一个格式字符串:

func Fprintf(w io.Writer, format string, a ...interface{}) (n int, err error)

而你传递给它一个8 KB大小的大字符串。fmt包需要分析格式字符串,它不会直接输出。这很可能是导致CPU占用过高的原因。

如果随机字符串中包含特殊字符%,那么情况会更糟,因为fmt.Fprintf()可能会期望有进一步的参数,而你没有提供,所以fmt包会在输出中包含错误信息,例如:

fmt.Fprintf(os.Stdout, "aaa%bbb%d")

输出:

aaa%!b(MISSING)bb%!d(MISSING)

你可以使用fmt.Fprint()代替,它不需要格式字符串:

fmt.Fprint(w, str)

或者更好的方法是,将随机字符串转换为字节切片,并直接写入:

data := []byte(str)
for i := 0; i < iter; i++ {
    if _, err := w.Write(data); err != nil {
        // 处理错误,例如返回
    }
}

如果要传递大量数据,除非你在循环中写入一个预先准备好的字节切片(如果你改变切片的大小可能会稍微快一点),否则你不会得到更快的解决方案。如果你的解决方案仍然“慢”,那可能是因为我们对你的RandStringBytes()函数一无所知,或者你的输出可能被压缩(使用其他处理程序或某些框架时可能会消耗较高的CPU)。另外,如果接收响应的客户端也在你的计算机上(例如浏览器),它(或防火墙/杀毒软件)可能会检查/分析响应中的恶意代码(这也可能消耗资源)。

英文:

The problem is that fmt.Fprintf() expects a format string:

func Fprintf(w io.Writer, format string, a ...interface{}) (n int, err error)

And you pass it a big, 8 KB format string. The fmt package has to analyze the format string, it is not something that gets to the output as is. Most definately this is what is eating your CPU.

If the random string contains the special % sign, that even makes your case worse, as then fmt.Fprintf() might expect further arguments which you don't "deliver", so the fmt package also has to (will) include error messages in the output, such as:

fmt.Fprintf(os.Stdout, &quot;aaa%bbb%d&quot;)

Output:

aaa%!b(MISSING)bb%!d(MISSING)

Use fmt.Fprint() instead which does not expect a format string:

fmt.Fprint(w, str)

Or even better, convert your random string to a byte slice once, and just keep writing that:

data := []byte(str)
for i := 0; i &lt; iter; i++ {
    if _, err := w.Write(data); err != nil {
        // Handle error, e.g. return
    }
}

Delivering large amount of data – you won't get a faster solution than writing a prepared byte slice in a loop (maybe slightly if you vary the size of the slice). If your solution is still "slow", that might be due to your RandStringBytes() function which we don't know anything about, or your output might be compressed (gzipped) if you use other handlers or some framework (which does use relatively high CPU). Also if the client that receives the response is also on your computer (e.g. a browser), it –or a firewall / antivirus software– may check / analyze the response for malicious code (which may also be resource intensive).

huangapple
  • 本文由 发表于 2016年10月31日 20:37:23
  • 转载请务必保留本文链接:https://go.coder-hub.com/40342340.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定