Golang: How do I determine the number of lines in a file efficiently?

In Golang, I am looking for an efficient way to determine the number of lines a file has.

Of course, I can always loop through the entire file, but does not seem very efficient.

file, _ := os.Open("/path/to/filename")
fileScanner := bufio.NewScanner(file)
lineCount := 0
for fileScanner.Scan() {
	lineCount++
}
fmt.Println("number of lines:", lineCount)

Is there a better (quicker, less expensive) way to find out how many lines a file has?

Here's a faster line counter using bytes.Count to find the newline characters.

It's faster because it takes away all the extra logic and buffering required to return whole lines, and takes advantage of some assembly optimized functions offered by the bytes package to search characters in a byte slice.

Larger buffers also help here, especially with larger files. On my system, with the file I used for testing, a 32k buffer was fastest.

func lineCounter(r io.Reader) (int, error) {
	buf := make([]byte, 32*1024)
	count := 0
	lineSep := []byte{'\n'}

	for {
		c, err := r.Read(buf)
		count += bytes.Count(buf[:c], lineSep)

		switch {
		case err == io.EOF:
			return count, nil

		case err != nil:
			return count, err
		}
	}
}

and the benchmark output:

BenchmarkBuffioScan	  500	   6408963 ns/op	 4208 B/op	  2 allocs/op
BenchmarkBytesCount	  500	   4323397 ns/op	 8200 B/op	  1 allocs/op
BenchmarkBytes32k     500	   3650818 ns/op	 65545 B/op	  1 allocs/op

The most efficient way I found is using IndexByte of the byte packet, it is at least four times faster than using bytes.Count and depending on the size of the buffer it uses much less memory.

func LineCounter(r io.Reader) (int, error) {

	var count int
	const lineBreak = '\n'

	buf := make([]byte, bufio.MaxScanTokenSize)

	for {
		bufferSize, err := r.Read(buf)
		if err != nil && err != io.EOF {
			return 0, err
		}

		var buffPosition int
		for {
			i := bytes.IndexByte(buf[buffPosition:], lineBreak)
			if i == -1 || bufferSize == buffPosition {
				break
			}
			buffPosition += i + 1
			count++
		}
		if err == io.EOF {
			break
		}
	}

	return count, nil
}

Benchmark

BenchmarkIndexByteWithBuffer  2000000	       653 ns/op	    1024 B/op	       1 allocs/op
BenchmarkBytes32k             500000	      3189 ns/op	   32768 B/op	       1 allocs/op

There is no approach that is significantly faster than yours as there is no meta-data on how many lines a file has. You could reach a little speed up by manually looking for newline-characters:

func lineCount(r io.Reader) (int n, error err) {
    buf := make([]byte, 8192)

    for {
        c, err := r.Read(buf)
        if err != nil {
            if err == io.EOF && c == 0 {
                break
            } else {
                return
            }
        }

        for _, b := range buf[:c] {
            if b == '\n' {
                n++
            }
        }
    }

    if err == io.EOF {
        err = nil
    }
}