Reading a file line by line in Go

I'm unable to find file.ReadLine function in Go.

How does one read a file line by line?

In Go 1.1 and newer the most simple way to do this is with a bufio.Scanner. Here is a simple example that reads lines from a file:

package main

import (
	"bufio"
	"fmt"
	"log"
	"os"
)

func main() {
	file, err := os.Open("/path/to/file.txt")
	if err != nil {
		log.Fatal(err)
	}
	defer file.Close()

	scanner := bufio.NewScanner(file)
    // optionally, resize scanner's capacity for lines over 64K, see next example
	for scanner.Scan() {
		fmt.Println(scanner.Text())
	}

	if err := scanner.Err(); err != nil {
		log.Fatal(err)
	}
}

This is the cleanest way to read from a Reader line by line.

There is one caveat: Scanner will error with lines longer than 65536 characters. If you know your line length is greater than 64K, use the Buffer() method to increase the scanner's capacity:

...
scanner := bufio.NewScanner(file)

const maxCapacity int = longLineLen  // your required line length
buf := make([]byte, maxCapacity)
scanner.Buffer(buf, maxCapacity)

for scanner.Scan() {
...

NOTE: The accepted answer was correct in early versions of Go. See the highest voted answer contains the more recent idiomatic way to achieve this.

There is function ReadLine in package <code>bufio</code>.

Please note that if the line does not fit into the read buffer, the function will return an incomplete line. If you want to always read a whole line in your program by a single call to a function, you will need to encapsulate the <code>ReadLine</code> function into your own function which calls <code>ReadLine</code> in a for-loop.

bufio.ReadString('\n') isn't fully equivalent to ReadLine because ReadString is unable to handle the case when the last line of a file does not end with the newline character.

EDIT: As of go1.1, the idiomatic solution is to use bufio.Scanner

I wrote up a way to easily read each line from a file. The Readln(*bufio.Reader) function returns a line (sans \n) from the underlying bufio.Reader struct.

// Readln returns a single line (without the ending \n)
// from the input buffered reader.
// An error is returned iff there is an error with the
// buffered reader.
func Readln(r *bufio.Reader) (string, error) {
  var (isPrefix bool = true
       err error = nil
       line, ln []byte
      )
  for isPrefix && err == nil {
      line, isPrefix, err = r.ReadLine()
      ln = append(ln, line...)
  }
  return string(ln),err
}

You can use Readln to read every line from a file. The following code reads every line in a file and outputs each line to stdout.

f, err := os.Open(fi)
if err != nil {
    fmt.Printf("error opening file: %v\n",err)
    os.Exit(1)
}
r := bufio.NewReader(f)
s, e := Readln(r)
for e == nil {
    fmt.Println(s)
    s,e = Readln(r)
}

Cheers!

There two common way to read file line by line.

1. Use bufio.Scanner
2. Use ReadString/ReadBytes/... in bufio.Reader

In my testcase, ~250MB, ~2,500,000 lines, bufio.Scanner(time used: 0.395491384s) is faster than bufio.Reader.ReadString(time_used: 0.446867622s).

Source code: https://github.com/xpzouying/go-practice/tree/master/read_file_line_by_line

Read file use bufio.Scanner,

func scanFile() {
    f, err := os.OpenFile(logfile, os.O_RDONLY, os.ModePerm)
    if err != nil {
        log.Fatalf("open file error: %v", err)
        return
    }
    defer f.Close()

    sc := bufio.NewScanner(f)
    for sc.Scan() {
        _ = sc.Text()  // GET the line string
    }
    if err := sc.Err(); err != nil {
        log.Fatalf("scan file error: %v", err)
        return
    }
}

Read file use bufio.Reader,

func readFileLines() {
    f, err := os.OpenFile(logfile, os.O_RDONLY, os.ModePerm)
    if err != nil {
        log.Fatalf("open file error: %v", err)
        return
    }
    defer f.Close()

    rd := bufio.NewReader(f)
    for {
        line, err := rd.ReadString('\n')
        if err != nil {
            if err == io.EOF {
                break
            }

            log.Fatalf("read file line error: %v", err)
            return
        }
        _ = line  // GET the line string
    }
}

Example from this gist

func readLine(path string) {
  inFile, err := os.Open(path)
  if err != nil {
     fmt.Println(err.Error() + `: ` + path)
     return
  }
  defer inFile.Close()

  scanner := bufio.NewScanner(inFile)
  for scanner.Scan() {
    fmt.Println(scanner.Text()) // the line
  }
}

but this gives an error when there is a line that larger than Scanner's buffer.

When that happened, what I do is use reader := bufio.NewReader(inFile) create and concat my own buffer either using ch, err := reader.ReadByte() or len, err := reader.Read(myBuffer)

Another way that I use (replace os.Stdin with file like above), this one concats when lines are long (isPrefix) and ignores empty lines:

func readLines() []string {
  r := bufio.NewReader(os.Stdin)
  bytes := []byte{}
  lines := []string{}
  for {
    line, isPrefix, err := r.ReadLine()
    if err != nil {
      break
    }
    bytes = append(bytes, line...)
    if !isPrefix {
      str := strings.TrimSpace(string(bytes))
      if len(str) > 0 {
        lines = append(lines, str)
        bytes = []byte{}
      }
    }
  }
  if len(bytes) > 0 {
    lines = append(lines, string(bytes))
  }
  return lines
}

You can also use ReadString with \n as a separator:

  f, err := os.Open(filename)
  if err != nil {
    fmt.Println("error opening file ", err)
    os.Exit(1)
  }
  defer f.Close()
  r := bufio.NewReader(f)
  for {
    path, err := r.ReadString(10) // 0x0A separator = newline
    if err == io.EOF {
      // do something here
      break
    } else if err != nil {
      return err // if you return error
    }
  }

Another method is to use the io/ioutil and strings libraries to read the entire file's bytes, convert them into a string and split them using a "\n" (newline) character as the delimiter, for example:

import (
    "io/ioutil"
    "strings"
)

func main() {
	bytesRead, _ := ioutil.ReadFile("something.txt")
	fileContent := string(bytesRead)
	lines := strings.Split(fileContent, "\n")
}

Technically you're not reading the file line-by-line, however you are able to parse each line using this technique. This method is applicable to smaller files. If you're attempting to parse a massive file use one of the techniques that reads line-by-line.

bufio.Reader.ReadLine() works well. But if you want to read each line by a string, try to use ReadString('\n'). It doesn't need to reinvent the wheel.

// strip '\n' or read until EOF, return error if read error  
func readline(reader io.Reader) (line []byte, err error) {   
	line = make([]byte, 0, 100)                              
	for {                                                    
		b := make([]byte, 1)                                 
		n, er := reader.Read(b)                              
		if n > 0 {                                           
			c := b[0]                                        
			if c == '\n' { // end of line                    
				break                                        
			}                                                
			line = append(line, c)                           
		}                                                    
		if er != nil {                                       
			err = er                                         
			return                                           
		}                                                    
	}                                                        
	return                                                   
}                                    

In the code bellow, I read the interests from the CLI until the user hits enter and I'm using Readline:

interests := make([]string, 1)
r := bufio.NewReader(os.Stdin)
for true {
	fmt.Print("Give me an interest:")
	t, _, _ := r.ReadLine()
	interests = append(interests, string(t))
	if len(t) == 0 {
		break;
	}
}
fmt.Println(interests)

import (
     "bufio"
     "os"
)

var (
    reader = bufio.NewReader(os.Stdin)
)

func ReadFromStdin() string{
	result, _ := reader.ReadString('\n')
	witl := result[:len(result)-1]
	return witl
}

Here is an example with function ReadFromStdin() it's like fmt.Scan(&name) but its takes all strings with blank spaces like: "Hello My Name Is ..."

var name string = ReadFromStdin()

println(name)

The Scan* functions are of great user here. Here is a slightly modified version of word scanner example from go-lang docs to scan lines from a file.

package main

import (
	"bufio"
	"fmt"
	"os"
	"strings"
)

func main() {
	// An artificial input source.
	const input = "Now is the winter of our discontent,\nMade glorious summer by this sun of York.\n"
	scanner := bufio.NewScanner(strings.NewReader(input))
	// Set the split function for the scanning operation.
	scanner.Split(bufio.ScanLines)
	// Count the lines.
	count := 0
	for scanner.Scan() {
		fmt.Println(scanner.Text())
		count++
	}
	if err := scanner.Err(); err != nil {
		fmt.Fprintln(os.Stderr, "reading input:", err)
	}
	fmt.Printf("%d\n", count)
}

In the new version of Go 1.16 we can use package embed to read the file contents as shown below.

package main

import _"embed"


func main() {
	//go:embed "hello.txt"
	var s string
	print(s)

    //go:embed "hello.txt"
    var b []byte
    print(string(b))

    //go:embed hello.txt
    var f embed.FS
    data, _ := f.ReadFile("hello.txt")
    print(string(data))
}

For more details go through https://tip.golang.org/pkg/embed/
And
https://golangtutorial.dev/tips/embed-files-in-go/