consul health check (All service checks failing)

I have read the similar question related to this problem. It ended up with frustration.

1. I installed consul with docker. I run the following command. docker run --name consul -d -p 8500:8500 consul, then I tested if the registration of service works using postman PUT method, but didn't enable health check, the service register into consul successfully in the consul web UI. For now everything works as expected.
2. I have a local GRPC service written in go. The local service address is 127.0.0.1:8880. The address of consul server is 127.0.0.1:8500. Then the service is registered into consul, but health check didn't work, gets All service failing. Then I tried following:

• 1. Use local IP address which is 192.168.0.152:8500 as address of consul server and 192.168.0.152:8880 as address of gprc service. Not working.
• 2. Use local IP address which is 192.168.0.152:8500 as address of consul server and 127.0.0.1:8880 as address of gprc service. Not working.
• 3. Check if HTTP service works. There is a local web service written in GIN. The port is 1010. AgentServiceCheck is following. Tried method above. Not working.

check := &api.AgentServiceCheck{
HTTP:                           "http://127.0.0.1:1010/health",
Timeout:                        "5s",
Interval:                       "5s",
DeregisterCriticalServiceAfter: "15s",
}

• 4. Run consul members gets

Node          Address         Status  Type    Build   Protocol  DC   Segment
aae2e6ac1ff8  127.0.0.1:8301  alive   server  1.10.3  2         dc1  <all>

Updates:

• After changing "127.0.0.1:8880/health" to "127.0.0.1:8880" in the following config, it surprisingly works. Don't know why..

check := &api.AgentServiceCheck{
GRPC:                           "127.0.0.1:8880",
Timeout:                        "5s",
Interval:                       "5s",
DeregisterCriticalServiceAfter: "10s",
}

The code register grpc service to consul server and enable health check is following:

func main() {
IP := flag.String("IP", "127.0.0.1", "IP address")
Port := flag.Int("Port", 8880, "Port")
flag.Parse()
// initialize logger, configuation file and database
initialize.InitLogger()
initialize.InitConfig()
initialize.InitDB()
zap.S().Info(global.ServerConfig)
zap.S().Info("IP: ", *IP)
zap.S().Info("Port: ", *Port)
server := grpc.NewServer()
proto.RegisterUserServer(server, &handler.UserServer{})
l, err := net.Listen("tcp", fmt.Sprintf("%s:%d", *IP, *Port))
if err != nil {
panic("failed to listen" + err.Error())
}
// register health check
grpc_health_v1.RegisterHealthServer(server, health.NewServer())
cfg := api.DefaultConfig()
// get cfg.address from configuration file, in which host is 127.0.0.1 port 8880
cfg.Address = fmt.Sprintf("%s:%d", global.ServerConfig.ConsulInfo.Host,
global.ServerConfig.ConsulInfo.Port)
client, err := api.NewClient(cfg)
if err != nil {
panic(err)
}
// generate health check instance
check := &api.AgentServiceCheck{
GRPC:                           "127.0.0.1:8880/health",
Timeout:                        "5s",
Interval:                       "5s",
DeregisterCriticalServiceAfter: "15s",
}
registration := new(api.AgentServiceRegistration)
registration.Name = global.ServerConfig.Name
registration.ID = global.ServerConfig.Name
registration.Address = "127.0.0.1"
registration.Port = 8880
registration.Tags = []string{"user-srv", "user"}
registration.Check = check
err = client.Agent().ServiceRegister(registration)
if err != nil {
panic(err)
}
err = server.Serve(l)
if err != nil {
panic("failed to start grpc" + err.Error())
}

Based on the documentation of the Consul API (https://www.consul.io/api-docs/agent/check#grpc), it seems that the GRPC field that you're using

> supports the standard gRPC health checking protocol.

Therefore, you only need to define the gRPC endpoint there, and the protocol will handle the health checks.

You can opt-out from this to use an HTTP endpoint for health checks by defining the HTTP field in your Agent Service Check config. You can find more information about that in the documentation (https://www.consul.io/api-docs/agent/check#http)

There are several problems about the "right" code:

1. the address of consul is equal to grpc service? Isn't cfg.Address consul's Addr?

// get cfg.address from configuration file, in which host is 127.0.0.1 port 8880
   cfg.Address = fmt.Sprintf("%s:%d", global.ServerConfig.ConsulInfo.Host,
      global.ServerConfig.ConsulInfo.Port)
   client, err := api.NewClient(cfg)

2. why the right code shows GRPC:"127.0.0.1:8880/health", the conclusion is "After changing "127.0.0.1:8880/health" to "127.0.0.1:8880" in the following config, it surprisingly works. Don't know why..". Which is inconsistent !!!!