Golang prometheus: is there a way to measure metrics for outbound requests?

I have a service A where I am scrapping metrics using prometheus, A makes some outbound requests to another service B. Is there a way to get metrics for requets from A to B?

Like, I want a measure count and duration of requets from A -> B

The metrics can be record by using prometheus.NewCounterVec and prometheus.NewHistogramVec (or proemetheus.NewSummaryVec, depends on the use case).

We could use custom labels to identify who is the client and who is the server. By this way, can filter out a client's request count and duration metrics to a particular server.

Here is the sample snippet

client.go

package main
import (
"net/http"
"time"
"github.com/labstack/echo/v4"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
var (
requestCounter = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "client_request_count",
Help: "Total number of requests from client",
},
[]string{"client", "server", "method", "route", "status"},
)
durationHistorgram = prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name:    "client_request_duration_secs",
Help:    "Duration of requests from Client",
Buckets: []float64{0.1, 0.5, 1, 2, 5},
},
[]string{"client", "server", "method", "route", "status"},
)
)
func init() {
prometheus.MustRegister(requestCounter)
prometheus.MustRegister(durationHistorgram)
}
func main() {
e := echo.New()
e.GET("/", func(c echo.Context) error {
// the request start time
start := time.Now()
req, err := http.NewRequest("GET", "http://localhost:8002", nil)
if err != nil {
return c.JSON(http.StatusInternalServerError, err.Error())
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return c.JSON(http.StatusInternalServerError, err.Error())
}
defer resp.Body.Close()
labels := prometheus.Labels{
"client": "Server A", // defines the client server
"server": "Server B", // defines the outbound request server
"method": "GET",// HTTP method
"route":  "/", // Request route
"status": resp.Status,// Response status
}
duration := time.Since(start).Seconds()
// the duration
durationHistorgram.With(labels).Observe(duration)
// request api count
requestCounter.With(labels).Inc()
return c.String(http.StatusOK, "[Client -> Server] Request send and received")
})
e.GET("/metrics", echo.WrapHandler(promhttp.Handler()))
if err := e.Start(":8001"); err != nil {
panic(err)
}
}

server.go

package main
import (
"net/http"
"github.com/labstack/echo/v4"
)
func main() {
e := echo.New()
e.GET("/", func(c echo.Context) error {
return c.JSON(http.StatusOK, map[string]interface{}{
"message": "response sending from server",
})
})
if err := e.Start(":8002"); err != nil {
panic(err)
}
}

As you can see in the code, these are our custom labels.

 []string{"client", "server", "method", "route", "status"},

and these are the label values

labels := prometheus.Labels{
"client": "Server A", // defines the client server
"server": "Server B", // defines the outbound request server
"method": "GET", // HTTP method
"route":  "/", // Request route
"status": resp.Status,// Response status
}

and finally we can query

client_request_count {client="Server A", server="Server B"}

The request Server A send to the Server B

References

• Prometheus HistorGram
• Prometheus Summary
• Histogram And Summary
• Examples

You can implement interceptor for http requests - https://morioh.com/p/c67b06c88598

Capture start and end time for the requests to calculate the duration. Also increment the gauge counter to get the number of requests made from service A to Service B.