Why reading and writing files in Go is much slower than in Perl?

I am using Go to improve code efficiency, but when I use Go to read and write files, I find that its reading and writing efficiency is not as high as that of Perl. Is it the problem of my code or other reasons?

Build input file:

# Input File:
for i in $(seq 1 600000) do     echo SERVER$((RANDOM%800+100)),$RANDOM,$RANDOM,$RANDOM >> sample.csv done

Read and write files with Perl：

time cat sample.csv | perl -ne 'chomp;print"$_"' > out.txt

real	0m0.249s
user	0m0.083s
sys	0m0.049s

Read and write files with Go:

package main

import (
	"bufio"
	"fmt"
	"io"
	"os"
	"strings"
)

func main() {

	filepath := "./sample.csv"
	file, err := os.OpenFile(filepath, os.O_RDWR, 0666)
	if err != nil {
		fmt.Println("Open file error!", err)
		return
	}
	defer file.Close()
	buf := bufio.NewReader(file)
	for {
		line, err := buf.ReadString('\n')
		line = strings.TrimSpace(line)
		fmt.Println(line)
		if err != nil {
			if err == io.EOF {
				fmt.Println("File read ok!")
				break
			} else {
				fmt.Println("Read file error!", err)
				return
			}
		}
	}
}

Then I run:

time go run read.go > out.txt

real	0m2.332s
user	0m0.326s
sys	0m2.038s

Why are read and write in Go almost 10 times slower than Perl?

You're comparing apples to oranges.

At least two methodological errors:

1. Your Perl incantation measures how cat reads the file and sends its contents over pipe(2), and perl reads data from there, processes it and writes the results to its stdout.

2. Your Go incantation

   • measures full build pass of the Go toolchain (which includes compilation, linking and writing out an executable image file) and then a run
     of the compiled program¹, and
   • measures unbuffered writes to stdout (fmt.Print* calls), while in the Perl code writes to the standard output - to cite the docs - 'typically can be line buffered if output is to the terminal and block buffered otherwise.'

Let's try to compare apples to apples.

First, here's a comparable Go implementation:

package main

import (
	"bufio"
	"bytes"
	"fmt"
	"os"
)

func main() {
	in := bufio.NewScanner(os.Stdin)
	out := bufio.NewWriter(os.Stdout)

	for in.Scan() {
		s := bytes.TrimSpace(in.Bytes())

		if _, err := out.Write(s); err != nil {
			fmt.Fprint(os.Stderr, "failed to write file:", err)
			os.Exit(1)
		}
	}

	if err := out.Flush(); err != nil {
		fmt.Fprint(os.Stderr, "failed to write file:", err)
		os.Exit(1)
	}

	if err := in.Err(); err != nil {
		fmt.Fprint(os.Stderr, "reading failed:", err)
		os.Exit(1)
	}
}

Let's save it as chomp.go and measure:

1. Build the code:

   $ go build chomp.go

2. Generate the input file:

   $ for i in $(seq 1 600000); do echo
SERVER$((RANDOM%800+100)),$RANDOM,$RANDOM,$RANDOM; done >sample.csv

3. Run the Perl code:

   $ time { perl -ne 'chomp; print "$_";' <sample.csv >out1.txt; }
   
   real	0m0.226s
   user	0m0.102s
   sys	0m0.048s
   

4. Run it again to make sure it had read the input file from the filesystem cache:

   $ time { perl -ne 'chomp; print "$_";' <sample.csv >out1.txt; }
   
   real	0m0.123s
   user	0m0.090s
   sys	0m0.033s
   

   Note how the execution time has gone down.

5. Run the Go code on the cached input:

   $ time { ./chomp <sample.csv >out2.txt; }
   
   real	0m0.063s
   user	0m0.032s
   sys	0m0.032s
   

6. Make sure the results are the same:

   $ cmp out1.txt out2.txt

As you can see, on my linux/amd64 system with an SSD the results are in the same ballpark.

Well, I should also state that to get sensible results, you'd need to run each command, like, 1000 times and average the results in each batch, and compare those numbers instead, but I think it's enough to demonstrate what the problems with your approach were.

One more thing to consider: the run time of these two programs is overwhelmingly dominated by the filesystem I/O, so if you thought Go would be faster at that, your expectation was unfounded: both programs most of the time sleep in the kernel's system calls read(2) and write(2). A Go program might be faster than a Perl program in certain cases involving CPU crunching (especially if it's written to take advantage of multi-core systems), but your example is simply not that case.

Oh, and just to make the unstated fact explicit: while the Go language specification does not say anything on how the runtime system of a Go implementation must be done, both two existing state-of-the art Go implementations (one of which you're ostensibly using) rely on AOT, and go run is a hack for one-off throw-away gigs not intended neither for serious work nor for executing code of any serious level of complexity. In simpler words, Go-that-you-are-using is not an interpreted language even though the availability of go run might make it appear so. In fact it does what a normal go build would do then runs the resulting executable file then throws it away.

¹ You might be tempted to state that Perl also deals with "the source code" but the Perl interpreter is highly optimized to deal with scripts, and the Go's build toolchain–while being crazy fast in comparison to most other compiled languages–is not optimized for that.
What is possibly a more glaring distinction, is that the Perl interpreter actually interprets your (very simple) script, and chomp and print are the so-called "built-ins"–functions, readily provided to the executing script by the interpreter. Compare that to building of a Go program which involves the compiler parsing the source code file and transforming it to the machine code, linker actually reading the files of the Go standard library's compiled packages—all those which are imported,–taking bits of code from them, combining all those bits of machine code and writing out an executable image file (which is much like what the perl binary itself is!); sure thing, this is a very resource-consuming process which has nothing to do with actual program execution.