Golang http server app have many Socket (CLOSE_WAIT)

App can works few days. But in some moment, app has many socket with CLOSE_WAIT state, and cannot receive new client.

Maybe it is some kind of flooding(Example: Sync-flood)?

netstat -ant | grep CLOSE_WAIT | wc

3258 19548 260640

3258 - socket in CLOSE_WAIT state

Update:

Code some handler:

 func GetScore(mongo *mgo.Session, redisConn redis.Conn, renderer handlers.Render) http.Handler {
mutex := sync.Mutex{}
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
id := bson.ObjectIdHex(r.FormValue("id"))
banner := models.Banner{}
err := mongo.DB("db").C("collection").FindId(id).One(&banner)
if err != nil {
log.Panicf("Banner selecting error: %s", err);
}
mutex.Lock();
sports, _ := util.GetSports(redisConn)
mutex.Unlock();
sport, _ := sports.FindSport(banner.SportId)
comp, err := sport.FindCompetition(banner.CompetitionId)
if err != nil {
comp, _ = sport.FindCompetition(0);
log.Println("Competition not found");
}
game, err := comp.FindGame(banner.GameId)
if err != nil {
game, _ = comp.FindGame(0)
}
mutex.Lock();
scores := util.GetScore(redisConn, game.ID)
mutex.Unlock();
game.Score1 = scores[0]
game.Score2 = scores[1]
w.Header().Set("Content-Type", "application/json;application/json;charset=utf-8")
renderer.RenderJson(w, 200, &game)
}
func GetScore(redisConn redis.Conn, gameId int) ([]float32) {
redisKey :=  fmt.Sprintf("game-%d", gameId);
bBody, err := redis.Bytes(redisConn.Do("GET", redisKey))
if err != nil || len(bBody) == 0 {
response, err := http.DefaultClient.Get(fmt.Sprintf("%s%d", GameApi, gameId))
if err != nil {
log.Panicf("GetScore error: %s", err)
}
bBody, _ = ioutil.ReadAll(response.Body);
redisConn.Send("SET", redisKey, bBody)
redisConn.Send("EXPIRE", redisKey, 60 * 5)
redisConn.Flush()
}
events := GameJson{};
err = json.Unmarshal(bBody, &events)
if err != nil {
log.Panicf("GetScore json error: %s", err)
}
var event []struct {
Name string `json:"name"`
Price float32 `json:"price"`
}
if len(events.AllEvents.P1XP2) != 0 {
event = events.AllEvents.P1XP2[0].Events
} else {
event = events.AllEvents.Other[0].Events
}
return []float32{event[0].Price, event[1].Price}
}

CLOSE-WAIT means that TCP is waiting for the local application to close the socket, having already received a close from the peer.

> Maybe it is some kind of flooding(Example: Sync-flood)?

No. It is a bug in your code. You aren't closing a socket somewhere.

> The problem seems to be not in the code.

The problem is in your code.

> Code tested several times.

But not for this condition, or under the conditions that produce this problem.

> But yesterday I did not have this problem.

So yesterday those condtiions did not occur.

> (Code was not changed.)

So the bug has always been there.

Emulation of situation:

When client timeout works. Socket on server will not closed 15 sec.
In my case, when app have many request, I have > 10k socket in CLOSE_WAIT STATE.

Emulation of mutex:

time.Sleep(30 * time.Second)

Server:

package main
import "net/http"
import "time";
func main() {
http.ListenAndServe(":81", http.HandlerFunc(func (w http.ResponseWriter, r *http.Request) {
println("Sleep")
time.Sleep(30 * time.Second)
println("After sleep");
}));
}

Client:

import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.URL;
public class Main {
public static void main(String[] args) throws IOException, InterruptedException {
HttpURLConnection connection = (HttpURLConnection) new URL("http://link").openConnection();
connection.setRequestMethod("GET");
connection.setDoOutput(false);
connection.setDoInput(true);
connection.setReadTimeout(15000);
connection.getInputStream().read();
System.out.println("Close");
connection.disconnect();
}
}