英文:
Golang high cpu usage on simple webserver unable to understand why?
问题
所以我有一个简单的net/http web服务器。它的功能只是传送100MB的随机字节,我打算用它来进行网络速度测试。我为100MB端点编写的处理程序非常简单(如下所示)。代码运行正常,我可以得到随机字节文件,问题是当我运行这个程序并有人下载这100兆字节时,该程序的CPU占用率会飙升到150%,并保持在这个水平,直到处理程序运行完成。我在这里做错了什么吗?我该如何改进这个处理程序的性能?
func downloadHandler(w http.ResponseWriter, r *http.Request) {
str := RandStringBytes(8192) //生成8192字节的随机数据
sz := 1000*1000*100 //100兆字节
iter := sz/len(str)+1
w.Header().Set("Content-Type", "application/octet-stream")
w.Header().Set("Content-Length", strconv.Itoa(sz))
for i := 0; i < iter; i++ {
fmt.Fprintf(w, str)
}
}
英文:
So I have a simple net/http webserver. All it does is is deliver 100MB of random bytes, which I intend to use for network speed testing. My handler for the 100mb endpoint is really simple (pasted below). The code works fine and I get my random byte file, the problem is when I run this and someone downloads these 100megabytes, the CPU for this program shoots up to 150% and stays there until this handler finishes running. Am I doing something very wrong here? What could I do to improve this handler's performance?
func downloadHandler(w http.ResponseWriter, r *http.Request) {
str := RandStringBytes(8192); //generates 8192 bytes of randomness
sz := 1000*1000*100; //100Megabytes
iter := sz/len(str)+1;
w.Header().Set("Content-Type", "application/octet-stream")
w.Header().Set("Content-Length", strconv.Itoa( sz ))
for i := 0; i < iter ; i++ {
fmt.Fprintf(w, str )
}
}
答案1
得分: 6
问题在于fmt.Fprintf()
函数需要一个格式字符串:
func Fprintf(w io.Writer, format string, a ...interface{}) (n int, err error)
而你传递给它一个8 KB大小的大字符串。fmt
包需要分析格式字符串,它不会直接输出。这很可能是导致CPU占用过高的原因。
如果随机字符串中包含特殊字符%
,那么情况会更糟,因为fmt.Fprintf()
可能会期望有进一步的参数,而你没有提供,所以fmt
包会在输出中包含错误信息,例如:
fmt.Fprintf(os.Stdout, "aaa%bbb%d")
输出:
aaa%!b(MISSING)bb%!d(MISSING)
你可以使用fmt.Fprint()
代替,它不需要格式字符串:
fmt.Fprint(w, str)
或者更好的方法是,将随机字符串转换为字节切片,并直接写入:
data := []byte(str)
for i := 0; i < iter; i++ {
if _, err := w.Write(data); err != nil {
// 处理错误,例如返回
}
}
如果要传递大量数据,除非你在循环中写入一个预先准备好的字节切片(如果你改变切片的大小可能会稍微快一点),否则你不会得到更快的解决方案。如果你的解决方案仍然“慢”,那可能是因为我们对你的RandStringBytes()
函数一无所知,或者你的输出可能被压缩(使用其他处理程序或某些框架时可能会消耗较高的CPU)。另外,如果接收响应的客户端也在你的计算机上(例如浏览器),它(或防火墙/杀毒软件)可能会检查/分析响应中的恶意代码(这也可能消耗资源)。
英文:
The problem is that fmt.Fprintf()
expects a format string:
func Fprintf(w io.Writer, format string, a ...interface{}) (n int, err error)
And you pass it a big, 8 KB format string. The fmt
package has to analyze the format string, it is not something that gets to the output as is. Most definately this is what is eating your CPU.
If the random string contains the special %
sign, that even makes your case worse, as then fmt.Fprintf()
might expect further arguments which you don't "deliver", so the fmt
package also has to (will) include error messages in the output, such as:
fmt.Fprintf(os.Stdout, "aaa%bbb%d")
Output:
aaa%!b(MISSING)bb%!d(MISSING)
Use fmt.Fprint()
instead which does not expect a format string:
fmt.Fprint(w, str)
Or even better, convert your random string to a byte slice once, and just keep writing that:
data := []byte(str)
for i := 0; i < iter; i++ {
if _, err := w.Write(data); err != nil {
// Handle error, e.g. return
}
}
Delivering large amount of data – you won't get a faster solution than writing a prepared byte slice in a loop (maybe slightly if you vary the size of the slice). If your solution is still "slow", that might be due to your RandStringBytes()
function which we don't know anything about, or your output might be compressed (gzipped) if you use other handlers or some framework (which does use relatively high CPU). Also if the client that receives the response is also on your computer (e.g. a browser), it –or a firewall / antivirus software– may check / analyze the response for malicious code (which may also be resource intensive).
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论