How to show custom application metrics in Prometheus captured using the golang client library from all pods running in Kubernetes

I am trying to get some custom application metrics captured in golang using the prometheus client library to show up in Prometheus.

I have the following working:

• I have a go application which is exposing metrics on localhost:8080/metrics as described in this article:

  https://godoc.org/github.com/prometheus/client_golang/prometheus

• I have a kubernates minikube running which has Prometheus, Grafana and AlertManager running using the operator from this article:

  https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus

• I created a docker image for my go app, when I run it and go to localhost:8080/metrics I can see the prometheus metrics showing up in a browser.

• I use the following pod.yaml to deploy my docker image to a pod in k8s

> apiVersion: v1
> kind: Pod
> metadata:
> name: my-app-pod
> labels:
> zone: prod
> version: v1
> annotations:
> prometheus.io/scrape: 'true'
> prometheus.io/port: '8080'
>
> spec:
> containers:
> - name: my-container
> image: name/my-app:latest
> imagePullPolicy: IfNotPresent
> ports:
> - containerPort: 8080

• If I connect to my pod using:

> kubectl exec -it my-app-pod -- /bin/bash

then do wget on "localhost:8080/metrics", I can see my metrics

So far so good, here is where I am hitting a wall. I could have multiple pods running this same image. I want to expose all the images to prometheus as targets. How do I configure my pods so that they show up in prometheus so I can report on my custom metrics?

Thanks for any help offered!

The kubernetes_sd_config directive can be used to discover all pods with a given tag. Your Prometheus.yml config file should have something like so:

- job_name: 'some-app'
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app]
      regex: python-app
      action: keep

The source label [__meta_kubernetes_pod_label_app] is basically using the Kubernetes api to look at pods that have a label of 'app' and whose value is captured by the regex expression, given on the line below (in this case, matching 'python-app').

Once you've done this Prometheus will automatically discover the pods you want and start scraping the metrics from your app.

Hope that helps. You can follow blog post here for more detail.

Note: it is worth mentioning that at the time of writing, kubernetes_sd_config is still in beta. Thus breaking changes to configuration may occur in future releases.

Can you share the prometheus config that you are using to scrape the metrics. The config will control what all sources to scrape the metrics from. Here are a few links that you can refer to : https://groups.google.com/forum/#!searchin/prometheus-users/Application$20metrics$20monitoring$20of$20Kubernetes$20Pods%7Csort:relevance/prometheus-users/uNPl4nJX9yk/cSKEBqJlBwAJ

You need 2 things:

• a ServiceMonitor for the Prometheus Operator, which specifies which services will be scraped for metrics
• a Service which matches the ServiceMonitor and points to your pods

There is an example in the docs over here: https://coreos.com/operators/prometheus/docs/latest/user-guides/running-exporters.html