使用 pprof 对 kubectl 进行性能分析。

huangapple
go评论118阅读模式
英文:

profile kubectl using pprof

问题

在kubernetes源代码中，有一段处理性能分析的代码块，但我无法访问这些端点：

  1. 在kubernetes/pkgs/kubelet/server/stats/server.go中
  3. func (s *Server) InstallProfilingHandler(enableProfilingLogHandler bool, enableContentionProfiling bool) {
  4. 	s.addMetricsBucketMatcher("debug")
  5. 	if !enableProfilingLogHandler {
  6. 		s.restfulCont.Handle(pprofBasePath, getHandlerForDisabledEndpoint("profiling endpoint is disabled."))
  7. 		return
  8. 	}
  10. 	handlePprofEndpoint := func(req *restful.Request, resp *restful.Response) {
  11. 		name := strings.TrimPrefix(req.Request.URL.Path, pprofBasePath)
  12. 		switch name {
  13. 		case "profile":
  14. 			pprof.Profile(resp, req.Request)
  15. 		case "symbol":
  16. 			pprof.Symbol(resp, req.Request)
  17. 		case "cmdline":
  18. 			pprof.Cmdline(resp, req.Request)
  19. 		case "trace":
  20. 			pprof.Trace(resp, req.Request)
  21. 		default:
  22. 			pprof.Index(resp, req.Request)
  23. 		}
  24. 	}
  26. 	// 设置pprof处理程序。
  27. 	ws := new(restful.WebService).Path(pprofBasePath)
  28. 	ws.Route(ws.GET("/{subpath:*}").To(handlePprofEndpoint)).Doc("pprof endpoint")
  29. 	s.restfulCont.Add(ws)
  31. 	if enableContentionProfiling {
  32. 		goruntime.SetBlockProfileRate(1)
  33. 	}
  34. }

我不知道pprof工具使用的端口，但我通过以下命令找到了它：

  1. controller-0:/home/sysadmin/go/bin# netstat -atlpn | grep kubelet
  2. tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      184856/kubelet      
  3. tcp        0      0 192.168.206.2:49720     192.168.206.1:6443      ESTABLISHED 184856/kubelet      
  4. tcp6       0      0 :::10250                :::*                    LISTEN      184856/kubelet

然后我尝试了以下命令：

  1. controller-0:/home/sysadmin/go/bin# ./go tool pprof  http://localhost:6443/debug/pprof/mutex
  2. Fetching profile over HTTP from http://localhost:6443/debug/pprof/mutex
  3. http://localhost:6443/debug/pprof/mutex: server response: 400 Bad Request
  4. failed to fetch any source profiles

有没有人知道我应该尝试什么来访问pprof端点？或者如何尝试其他方法来对kubelet进程进行性能分析？

英文:

In the kubernetes source code there is a block of code that handles the profiling part but I can not acces the endpoints:

  1. in kubernetes/pkgs/kubelet/server/stats/server.go
  3. func (s *Server) InstallProfilingHandler(enableProfilingLogHandler bool, enableContentionProfiling bool) {
  4. 	s.addMetricsBucketMatcher("debug")
  5. 	if !enableProfilingLogHandler {
  6. 		s.restfulCont.Handle(pprofBasePath, getHandlerForDisabledEndpoint("profiling endpoint is disabled."))
  7. 		return
  8. 	}
  10. 	handlePprofEndpoint := func(req *restful.Request, resp *restful.Response) {
  11. 		name := strings.TrimPrefix(req.Request.URL.Path, pprofBasePath)
  12. 		switch name {
  13. 		case "profile":
  14. 			pprof.Profile(resp, req.Request)
  15. 		case "symbol":
  16. 			pprof.Symbol(resp, req.Request)
  17. 		case "cmdline":
  18. 			pprof.Cmdline(resp, req.Request)
  19. 		case "trace":
  20. 			pprof.Trace(resp, req.Request)
  21. 		default:
  22. 			pprof.Index(resp, req.Request)
  23. 		}
  24. 	}
  26. 	// Setup pprof handlers.
  27. 	ws := new(restful.WebService).Path(pprofBasePath)
  28. 	ws.Route(ws.GET("/{subpath:*}").To(handlePprofEndpoint)).Doc("pprof endpoint")
  29. 	s.restfulCont.Add(ws)
  31. 	if enableContentionProfiling {
  32. 		goruntime.SetBlockProfileRate(1)
  33. 	}
  34. }

I do not know the port used by the pprof tool but I found it using:

  1. controller-0:/home/sysadmin/go/bin# netstat -atlpn | grep kubelet
  2. tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      184856/kubelet      
  3. tcp        0      0 192.168.206.2:49720     192.168.206.1:6443      ESTABLISHED 184856/kubelet      
  4. tcp6       0      0 :::10250                :::*                    LISTEN      184856/kubelet

Then I have tried

  1. controller-0:/home/sysadmin/go/bin# ./go tool pprof  http://localhost:6443/debug/pprof/mutex
  2. Fetching profile over HTTP from http://localhost:6443/debug/pprof/mutex
  3. http://localhost:6443/debug/pprof/mutex: server response: 400 Bad Request
  4. failed to fetch any source profiles

Is there anyone that knows what should I try to access the pprof endpoints? or how to try a different approach to profile the kubelet process?

答案1

得分: 3

尝试一下：

  1. $ kubectl proxy &
  2. 开始在 127.0.0.1:8001 上提供服务
  3. $ go tool pprof "http://localhost:8001/api/v1/nodes/${NODE}/proxy/debug/pprof/profile"

当你启动 kubectl proxy 后，所有对 http://localhost:8001/api/v1/nodes/${NODE}/proxy/ 的请求都会发送到运行在 ${NODE} 上的 kubelet。你可以添加任何你想要的路径。比如 /debug/pprof/profile 或者 debug/pprof/heap，或者其他任何你想要的路径。

英文:

Try:

  1. $ kubectl proxy &
  2. Starting to serve on 127.0.0.1:8001
  3. $ go tool pprof "http://localhost:8001/api/v1/nodes/${NODE}/proxy/debug/pprof/profile"

When you start kubectl proxy, all requests to http://localhost:8001/api/v1/nodes/${NODE}/proxy/ go to the kubelet running on the ${NODE}. You can add any path you want and it. Let it be /debug/pprof/profile or debug/pprof/heap or whatever you want.

huangapple
  • 本文由 发表于 2021年6月11日 19:25:21
  • 转载请务必保留本文链接：https://go.coder-hub.com/67936168.html
  • go
  • kubelet
  • pprof
  • profiling
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定