Prometheus metrics are overwritten

I have a deployment on openshift - host status monitor app.

I also have a deployment for prometheus instance and operator.

This is a golang application which performs some operations and publish the metrics on prometheus.

I have created a service and a route in openshift and using this route I can see all my prometheus metrics. I am using the link in the route as prmotheus data source for grafana.
<br><br>
Now the problem is, if I increase the pod count for my deployment - host status monitor app to say 2, the 2nd pod overwrites all the prometheus metrics published by the 1st pod.
<br><br>
I want to have multiple pods for my deployment and all of their prometheus metrics should come in the same route link without affecting the metrics for other pods. How can I achieve it??

I am using the below function to publish the metrics:

func PublishMetrics(hostName, originalState, swarch, server, provtype string) {
	states := []string{STATE_ACTIVE, STATE_UNRESPONSIVE, STATE_INACTIVE, STATE_DECOMISSIONED, STATE_REMOVED}

	for _, state := range states {
		HostStateGauge.With(prometheus.Labels{
			"host":     hostName,
			"state":    state,
			"swarch":   swarch,
			"server":   server,
			"provtype": provtype,
		}).Set(0)
	}
	HostStateGauge.With(prometheus.Labels{
		"host":     hostName,
		"state":    STATE_UNKNOWN,
		"swarch":   "",
		"server":   "",
		"provtype": "",
	}).Set(0)
	HostStateGauge.With(prometheus.Labels{
		"host":     hostName,
		"state":    originalState,
		"swarch":   swarch,
		"server":   server,
		"provtype": provtype,
	}).Set(1)
}

<br>
PS: I am new to openshift and prometheus.
<br><br>
Thanks.

The issue you are facing is because the PublishMetrics() function is using the same set of Prometheus labels for each pod, so the metrics from each pod are being aggregated together by Prometheus.

To solve this issue and ensure that metrics from each pod are separate and distinct, you can modify the labels used in PublishMetrics() to include a unique identifier for each pod.

First read the pod name and then add the pod label to all your metrics.

// Get the name of the current pod
podName := os.Getenv("HOSTNAME")


    "pod":      podName, // Add pod name as a label
    "host":     hostName,
    "state":    state,
    "swarch":   swarch,
    "server":   server,
    "provtype": provtype,

By including the pod name as a label, Prometheus will treat the metrics from each pod as separate time series, even if they have the same label values for the other labels.

Here are some logs:

# TYPE hoststatusmonitorapp_request_duration_seconds histogram
hoststatusmonitorapp_request_duration_seconds_bucket{name="MQTTMsgHandler",pod_name="hoststatusmonitorapp_host_status_monitor_app_5888b67d67_x74vm",result="",le="0.005"} 2776
hoststatusmonitorapp_request_duration_seconds_bucket{name="MQTTMsgHandler",pod_name="hoststatusmonitorapp_host_status_monitor_app_5888b67d67_x74vm",result="",le="0.01"} 3146
hoststatusmonitorapp_request_duration_seconds_bucket{name="MQTTMsgHandler",pod_name="hoststatusmonitorapp_host_status_monitor_app_5888b67d67_x74vm",result="",le="0.025"} 3220
hoststatusmonitorapp_request_duration_seconds_bucket{name="MQTTMsgHandler",pod_name="hoststatusmonitorapp_host_status_monitor_app_5888b67d67_x74vm",result="",le="0.05"} 3250
hoststatusmonitorapp_request_duration_seconds_bucket{name="MQTTMsgHandler",pod_name="hoststatusmonitorapp_host_status_monitor_app_5888b67d67_x74vm",result="",le="0.1"} 3327

It is a different function:

func InitializePrometheusMetrics(appName string) {
	RequestHistogram = prometheus.NewHistogramVec(
		prometheus.HistogramOpts{
			Name:    fmt.Sprintf("%s_request_duration_seconds", appName),
			Help:    fmt.Sprintf("Time (in seconds) spent serving %s request", appName),
			Buckets: prometheus.DefBuckets,
		},
		[]string{"pod_name", "name", "result"}) //request name and request result != "" => Error Msg
}

// Prometheus Middleware for MQTT Message Received Handler
func MqttMessageHandlerPrometheusMiddleware(
	msgHandler func(topic string, payload []byte, mongoClient *mongo.Client, ctx context.Context) (string, string, error),
	topic string,
	payload []byte,
	mongoClient *mongo.Client,
	ctx context.Context,
) {
	start := time.Now()
	reqHandlerName, reqHandlerResult, err := msgHandler(topic, payload, mongoClient, ctx)
	if err != nil {
		LogError(fmt.Errorf("In processing topic: %s, epochtime: %s, err: %v", topic, payload, err))
	}
	RequestHistogram.With(prometheus.Labels{
		"pod_name": GetNormalizedHostname(APP_NAME),
		"name":     reqHandlerName,
		"result":   reqHandlerResult,
	}).Observe(time.Since(start).Seconds())
}

For the above function I found a way around.