How to find out nothing is being received in an unbuffered channel without closing it?

Is there a way to know if all the values in channel has been consumed? I'm making a crawler which recursively fetches sites from seed site. I'm not closing the channel because it consumes from the server and should crawl every time new site is sent. For a given seed site, I can't find a better way to know completion of a subtask other than timing out. If there was a way to know that there is no value in channel(left to be consumed), my program could get out of the sub task and continue listening to the server.

There is no such things as "queued in an unbuffered channel." If the channel is unbuffered, it is by definition always empty. If it is buffered, then it may have some number of elements in it up to its size. But trying to read how many elements are in it is always going to cause race conditions, so don't design that way (it's also impossible in Go).

Ideally, avoid designs that need to know when children are complete, but when you must, send them a channel to respond to you on. When they respond, then you know they're complete.

The kind of problem you're describing is well covered in the Go blogs and talks:

• Go Concurrency Patterns: Pipelines and cancellation
• Go Concurrency Patterns: Context
• Concurrency is not paralellism
• Go Concurrency Patterns
• Advanced Go Concurrency Patterns

You can determine whether or not a goroutine is blocked on the other end of a channel by using default in a select statement. For example:

package main

import (
	"fmt"
	"time"
)

var c = make(chan int)

func produce(i int) {
	c <- i
}

func consume() {
	for {
		select {
		case i := <-c:
			fmt.Println(i)
		default:
			return
		}
	}
}

func main() {
	for i := 0; i < 10; i++ {
		go produce(i)
	}
	time.Sleep(time.Millisecond)
	consume()
}

Keep in mind that this isn't a queue though. If you were to have 1 producing goroutine that looped and produced multiple values between the time it took to send one value and get back around the loop again the default case would happen and your consumer would move on.

You could use a timeout:

case <-time.After(time.Second):

Which would give your producer a second to produce another value, but you're probably better off using a terminal value. Wrap whatever you're sending in a struct:

type message struct {
    err error
    data theOriginalType
}

And send that thing instead. Then use io.EOF or a custom error var Done = errors.New("DONE") to signal completion.

Since you have a recursive problem why not use a WaitGroup? Each time you start a new task increment the wait group, and each time a task completes, decrement it. Then have an outer task waiting on completion. For example here's a really inefficient way of calculating a fibonacci number:

package main

import (
	"fmt"
	"sync"
)

var wg sync.WaitGroup

func fib(c chan int, n int) {
	defer wg.Done()
	if n < 2 {
		c <- n
	} else {
		wg.Add(2)
		go fib(c, n - 1)
		go fib(c, n - 2)
	}
}

func main() {
	wg.Add(1)
	c := make(chan int)
	go fib(c, 18)
	go func() {
		wg.Wait()
		close(c)
	}()
	sum := 0
	for i := range c {
		sum += i
	}
	fmt.Println(sum)
}