Closing a channel

I've created a simple channel to make asynchronous HTTP requests based on the following example:

http://matt.aimonetti.net/posts/2012/11/27/real-life-concurrency-in-go/

What would be the best pattern to close the channel, once all the requests have finished?

type HttpRequest struct {
	url        string
}

type HttpResponse struct {
	request  HttpRequest
	response *http.Response
	err      error
}

func asyncHttpGets(requests []HttpRequest) {
	ch := make(chan *HttpResponse)
	for _, request := range requests {
		go func(url string) {
			resp, err := http.Get(url)
			ch <- &HttpResponse{request, resp, err}
		}(request.url)
	}

	for {
		select {
		case r := <-ch:
			processResponse(r)
		}
	}
}

The code, written like this, will produce a deadlock. But, the channel does not have necessarily to be closed. There are multiple ways to solve this issue.

For instance, you could replace the for/select loop by:

n := len(requests)
for r := range ch {
    processResponse(r)
    n--
    if n == 0 {
        break
    }
}

Here we assume that the potential timeouts are managed in each goroutine.

Another solution, which really relies on closing the channel could be written as follows:

func asyncHttpGets(requests []HttpRequest) {

    ch := make(chan *HttpResponse)
    var wg sync.WaitGroup
    for _, request := range requests {
        wg.Add(1)
        go func(r HttpRequest) {
            defer wg.Done()
            resp, err := http.Get(r.url)
            ch <- &HttpResponse{r, resp, err}
        }(request)
    }

    go func() {
        wg.Wait()
        close(ch)
    }()
    for r := range ch {
        processResponse(r)
    }
}

Note that compared the initial code, the request variable is not accessed from the goroutine, but passed as a parameter. The output data structure posted via the channel is therefore consistent. This was an issue in the initial code. See more information about this specific topic at: https://github.com/golang/go/wiki/CommonMistakes

Yet another solution would be to count the responses in the goroutines using an atomic counter, and explicitly close the channel when the counter reaches the limit. But dealing with sync/atomic is often error-prone, so it is probably not a good idea here.

Finally, sometimes you need to get more control in order to properly manage timeouts, errors, etc ... The tomb package can help you to manage the lifecycle of the goroutines in a safe way.

See https://github.com/go-tomb/tomb/tree/v2