Processing a channel concurrently results in unexpected output

I have an unbuffered channel that i amount of workers take a value from (a filesystem path) and process it (send the file contents over HTTP). I'm running into problem when I increase i.

When I run this:

paths := make(chan string)

for i := 0; i < 5; i++ {
	go func() {
		for path := range paths {
			fmt.Println(path)
		}
	}()
}

walkFn := func(path string, info os.FileInfo, err error) error {
	if !info.IsDir() {
		paths <- path
	}
	return nil
}

filepath.Walk("/tmp/foo", walkFn)
close(paths)

It works expectedly and outputs all the contents of /tmp/foo:

/tmp/foo/2
/tmp/foo/file9
/tmp/foo/file91
/tmp/foo/file90
/tmp/foo/file900
/tmp/foo/file901
/tmp/foo/file902
/tmp/foo/file92
/tmp/foo/file97
/tmp/foo/file93
/tmp/foo/file94
/tmp/foo/file95
/tmp/foo/file96
/tmp/foo/file98
/tmp/foo/file99

But when I send the file contents over HTTP, the number of affected files suddenly goes down:

for i := 0; i < 5; i++ {
	go func() {
		for path := range paths {
			resp, err := http.Head("https://example.com/" + strings.TrimPrefix(path, rootDir+"/"))
			if err != nil {
				fmt.Printf("Error: %s\n", err)
				return
			}

			fmt.Printf("%s: %s\n", path, resp.Status)
		}
	}()
}

the number of affected files reduces from 15 (which is how many exist in the directory), down to 10:

/tmp/foo/2: 404 Not Found
/tmp/foo/file901: 404 Not Found
/tmp/foo/file900: 404 Not Found
/tmp/foo/file9: 404 Not Found
/tmp/foo/file90: 404 Not Found
/tmp/foo/file902: 404 Not Found
/tmp/foo/file91: 404 Not Found
/tmp/foo/file92: 404 Not Found
/tmp/foo/file93: 404 Not Found
/tmp/foo/file94: 404 Not Found

Here's a table that relates the value of i to the number of output lines:

+-----+-------+
| `i` | lines |
+-----+-------+
| 1   | 15    |
| 5   | 10    |
| 6   | 9     |
| 15  | 0     |
+-----+-------+

Why does this happen and how can I process all the channel entries concurrently? Is it a problem with http requests?

The problem is that after this line:

filepath.Walk("/tmp/foo", walkFn)

All paths have been sent through the paths channel, this implies that someone received them. However, it does not imply that those receiving goroutines have finished completely.

So when your program exits after close(paths), there are still goroutines working and they get killed because main is finished.

https://golang.org/ref/spec#Program_execution

> Program execution begins by initializing the main package and then invoking the function main. When that function invocation returns, the program exits. It does not wait for other (non-main) goroutines to complete.

One simple solution is to add

select{}

at the end of your program. This will make it block forever.