使用 pprof 对 kubectl 进行性能分析。

huangapple go评论79阅读模式
英文:

profile kubectl using pprof

问题

在kubernetes源代码中,有一段处理性能分析的代码块,但我无法访问这些端点:

在kubernetes/pkgs/kubelet/server/stats/server.go中

func (s *Server) InstallProfilingHandler(enableProfilingLogHandler bool, enableContentionProfiling bool) {
	s.addMetricsBucketMatcher("debug")
	if !enableProfilingLogHandler {
		s.restfulCont.Handle(pprofBasePath, getHandlerForDisabledEndpoint("profiling endpoint is disabled."))
		return
	}

	handlePprofEndpoint := func(req *restful.Request, resp *restful.Response) {
		name := strings.TrimPrefix(req.Request.URL.Path, pprofBasePath)
		switch name {
		case "profile":
			pprof.Profile(resp, req.Request)
		case "symbol":
			pprof.Symbol(resp, req.Request)
		case "cmdline":
			pprof.Cmdline(resp, req.Request)
		case "trace":
			pprof.Trace(resp, req.Request)
		default:
			pprof.Index(resp, req.Request)
		}
	}

	// 设置pprof处理程序。
	ws := new(restful.WebService).Path(pprofBasePath)
	ws.Route(ws.GET("/{subpath:*}").To(handlePprofEndpoint)).Doc("pprof endpoint")
	s.restfulCont.Add(ws)

	if enableContentionProfiling {
		goruntime.SetBlockProfileRate(1)
	}
}

我不知道pprof工具使用的端口,但我通过以下命令找到了它:

controller-0:/home/sysadmin/go/bin# netstat -atlpn | grep kubelet
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      184856/kubelet      
tcp        0      0 192.168.206.2:49720     192.168.206.1:6443      ESTABLISHED 184856/kubelet      
tcp6       0      0 :::10250                :::*                    LISTEN      184856/kubelet 

然后我尝试了以下命令:

controller-0:/home/sysadmin/go/bin# ./go tool pprof  http://localhost:6443/debug/pprof/mutex
Fetching profile over HTTP from http://localhost:6443/debug/pprof/mutex
http://localhost:6443/debug/pprof/mutex: server response: 400 Bad Request
failed to fetch any source profiles

有没有人知道我应该尝试什么来访问pprof端点?或者如何尝试其他方法来对kubelet进程进行性能分析?

英文:

In the kubernetes source code there is a block of code that handles the profiling part but I can not acces the endpoints:

in kubernetes/pkgs/kubelet/server/stats/server.go

func (s *Server) InstallProfilingHandler(enableProfilingLogHandler bool, enableContentionProfiling bool) {
	s.addMetricsBucketMatcher("debug")
	if !enableProfilingLogHandler {
		s.restfulCont.Handle(pprofBasePath, getHandlerForDisabledEndpoint("profiling endpoint is disabled."))
		return
	}

	handlePprofEndpoint := func(req *restful.Request, resp *restful.Response) {
		name := strings.TrimPrefix(req.Request.URL.Path, pprofBasePath)
		switch name {
		case "profile":
			pprof.Profile(resp, req.Request)
		case "symbol":
			pprof.Symbol(resp, req.Request)
		case "cmdline":
			pprof.Cmdline(resp, req.Request)
		case "trace":
			pprof.Trace(resp, req.Request)
		default:
			pprof.Index(resp, req.Request)
		}
	}

	// Setup pprof handlers.
	ws := new(restful.WebService).Path(pprofBasePath)
	ws.Route(ws.GET("/{subpath:*}").To(handlePprofEndpoint)).Doc("pprof endpoint")
	s.restfulCont.Add(ws)

	if enableContentionProfiling {
		goruntime.SetBlockProfileRate(1)
	}
}

I do not know the port used by the pprof tool but I found it using:

controller-0:/home/sysadmin/go/bin# netstat -atlpn | grep kubelet
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      184856/kubelet      
tcp        0      0 192.168.206.2:49720     192.168.206.1:6443      ESTABLISHED 184856/kubelet      
tcp6       0      0 :::10250                :::*                    LISTEN      184856/kubelet 

Then I have tried

controller-0:/home/sysadmin/go/bin# ./go tool pprof  http://localhost:6443/debug/pprof/mutex
Fetching profile over HTTP from http://localhost:6443/debug/pprof/mutex
http://localhost:6443/debug/pprof/mutex: server response: 400 Bad Request
failed to fetch any source profiles

Is there anyone that knows what should I try to access the pprof endpoints? or how to try a different approach to profile the kubelet process?

答案1

得分: 3

尝试一下:

$ kubectl proxy &
开始在 127.0.0.1:8001 上提供服务
$ go tool pprof "http://localhost:8001/api/v1/nodes/${NODE}/proxy/debug/pprof/profile"

当你启动 kubectl proxy 后,所有对 http://localhost:8001/api/v1/nodes/${NODE}/proxy/ 的请求都会发送到运行在 ${NODE} 上的 kubelet。你可以添加任何你想要的路径。比如 /debug/pprof/profile 或者 debug/pprof/heap,或者其他任何你想要的路径。

英文:

Try:

$ kubectl proxy &
Starting to serve on 127.0.0.1:8001
$ go tool pprof "http://localhost:8001/api/v1/nodes/${NODE}/proxy/debug/pprof/profile"

When you start kubectl proxy, all requests to http://localhost:8001/api/v1/nodes/${NODE}/proxy/ go to the kubelet running on the ${NODE}. You can add any path you want and it. Let it be /debug/pprof/profile or debug/pprof/heap or whatever you want.

huangapple
  • 本文由 发表于 2021年6月11日 19:25:21
  • 转载请务必保留本文链接:https://go.coder-hub.com/67936168.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定