Process output of arbitrary number of goroutines as they finish

WaitGroups are used to wait for all goroutines to finish before continuing execution, but how do you process their outputs as they finish?

This method works okay

c := make(chan string)
rc := 0
for _, url := range urls {
	rc++
	go func(url string) {
		data := get(url)
		c <- data
	}(url)
}
for i := 0; i < rc; i++ {
    data <- c
}

It stops working okay however when you have to call goroutines from goroutines

for _, url := range urls {
	go func(url string) {
		data := get(url)
        urls := get_urls(data)
        for _, url := range urls {
            go func(url){
                data := get(url)
                c <- data
            }(url)
        }
	}(url)
}

This time we will have to put rc++ inside of goroutine which will result in undefined behavior. What do?

There are plenty of ways to solve this, but the most idiomatic (thus probably "best") is as colm.anseo suggested in a comment: have the "process output" routine use a ranged for loop, and have the generating routines use a sync.WaitGroup counter with a closing goroutine, like this:

// result channel:
c := make(chan string)

// wait-group:
var wg sync.WaitGroup

// run over the URLs
for _, url := range urls {
    wg.Add(1) // count up another start()
    go start(&wg, url, c)
}
go func() {
    // wait for all start()s to say they are done
    wg.Wait()
    // and now close c
    close(c)
}()
for data := range c {
    // deal with data
}

Now we can see how each start function works:

// start() loads up a set of URLs from a URL,
// then spins off worker goroutines that read from
// each of those URLs, sending data to channel c.
// When all of its subsidiary goroutines have finished,
// start() signals that it is done.
func start(wg *sync.Waitgroup, url string, c chan string) {
    defer wg.Done()
    data := get(url)
    urls := get_urls(data)
    var subWG sync.WaitGroup
    for _, url := range urls {
        subWG.Add(1)
        go func(url) {
            defer subWG.Done()
            data := get(url)
            c <- data
        }(url)
    }
    subWG.Wait()
}

Function start can of course be inlined, as you did originally; I wrote it this way for clarity.

(In general, rather than spinning off some mystery number of goroutines per URL, you really want a "limited number of workers" pattern here, i.e., a worker pool. There are lots of examples of this on the web.)

When you have an indeterminate number of results - its best to let the workers and the worker manager goroutine manage the results channel and close it when all results are complete. This avoid messy counting/mutex logic.

So to convert your first example:

c := make(chan string)

var wg sync.WaitGroup // used by manager goroutine to determine finish

for _, url := range urls {
	wg.Add(1) // about to start worker
	go func(url string) {
		defer wg.Done() // worker is complete
		data := get(url)
		c <- data
	}(url)
}

// manager goroutine
go func() {
	wg.Wait() // all workers are done ...
	close(c) // ... so signal this via channel close
}()

// results collection is then very simple
for data := range c {
	fmt.Println(data)
}

This design applies identically to your situation where the workers create more goroutines. But as @torek noted an extra sync.WaitGroup is needed:

c := make(chan string)

var wg sync.WaitGroup

for _, url := range urls {
	wg.Add(1) // new worker
	go func(url string) {
		defer wg.Done() // worker done
		data := get(url)
		urls := get_urls(data)
        var wg2 sync.WaitGroup
		for _, url := range urls {
            wg2.Add(1)
			go func(url) {
                defer wg2.Done()
				data := get(url)
				c <- data
			}(url)
		}
        wg2.Wait()
	}(url)
}

// manager goroutine
go func() {
	wg.Wait() // all workers are done ...
	close(c)  // ... so signal this via channel close
}()

for data := range c {
	fmt.Println(data)
}