How to check the KAFKA health in my situation

Our application is built on top of K8S Cluster, that is to manage large amount of devices. One of the feature is, if any device becomes offline, a notification would be triggered and an email would be sent out to the user.

But, since 2 weeks ago, such feature no longer works, and emails are not sent.

And since the guy used to work on the kafka part has left the company, I need to figure out if kafka service is still working health myself.

I am quite new to kafka, when I login the master node on k8s, I have see such pods that kafka related.

k8s -n kafka get pods
NAME                              READY   STATUS    RESTARTS   AGE
kafka-0                           1/1     Running   779        466d
kafka-1                           1/1     Running   848        2y266d
kafka-2                           1/1     Running   797        466d
kafka-exporter-58c76747c6-j2cv2   1/1     Running   306        292d
kafka-manager-844657d4bf-q2j7n    1/1     Running   1647       2y215d
kafka-zookeeper-0                 1/1     Running   574        466d
kafka-zookeeper-1                 1/1     Running   573        2y266d
kafka-zookeeper-2                 1/1     Running   569        466d

Where should I start any type of sanity checking on Kafka service

You have both kafka-exporter and kafka-manager services.

Both of these have their own ability to check the health of Kafka via HTTP requests, either to the exporter's metrics endpoint (or Prometheus / Grafana, if these also exist in your cluster), or to the kafka-manager UI, rather than Kafka itself, and are not specific to kubectl commands.

More information

• https://github.com/danielqsj/kafka_exporter
• https://github.com/yahoo/CMAK (previously known as kafka-manager)

Alternatively, you could migrate to an installation of https://strimzi.io , which comes with its own monitoring stack based on Prometheus.

You can check pod logs with this command:

$ kubectl logs -n kafka [POD_NAME]

and you can check pod resource usage you can use this:

$ kubectl describe pod [POD_NAME] -n kafka

These two commands can help you.

If the check is done in an k8s idiomatic way then an liveness probe is used:

Execute:

kubectl get pod <pod name> -o yaml

and search for livenessProbe.