What is the fastest way to rewrite file with go

I have a large file(can't fit entirely in memory) containing strings of various sizes. I want to rewrite these strings to another file, but with each string capitalized. What is the fastest way to achieve this in Go?

Here is the most efficient way that I could come up with. Any ideas on how to make it faster?

package main

import (
	"bufio"
	"log"
	"os"
	"strings"
)

func main() {
	inputFile, err := os.Open("input.txt")
	if err != nil {
		log.Fatal(err)
	}
	defer inputFile.Close()

	outputFile, err := os.Create("output.txt")
	if err != nil {
		log.Fatal(err)
	}
	defer outputFile.Close()

	scanner := bufio.NewScanner(inputFile)
	writer := bufio.NewWriter(outputFile)

	for scanner.Scan() {
		line := scanner.Text()
		capitalized := strings.ToUpper(line)
		_, err := writer.WriteString(capitalized + "\\n")
		if err != nil {
			log.Fatal(err)
		}
	}

	err = writer.Flush()
	if err != nil {
		log.Fatal(err)
	}
}

One place to start is to run Go testing package benchmarks.

For benchmark data I use a 275,502 word, largely lowercase, 3,077,701 byte, Linux dictionary file: /usr/share/dict/brazilian. It's the best I could do given your vague description of your file. To avoid benchmark disk I/O, I use bytes.Reader for io.Reader and ioutil.Discard for io.Writer.

The results for your code:

$ go test upper_so_test.go -run=! -benchmem -bench=.
BenchmarkSO-12   48  22765120 ns/op  8143216 B/op  550993 allocs/op

The results for Blunderific's code:

BenchmarkB-12    94  13061407 ns/op  3782866 B/op  275505 allocs/op

As a Proof of Concept (PoC), using the dictionary file, I wrote code which uses minimal CPU and memory. The results, so far, for my PoC code:

BenchmarkTU-12  182   6457334 ns/op     8240 B/op       3 allocs/op

Running my PoC code as a program, using SSD file storage for reading and writing the dictionary file, takes a few milliseconds:

$ time ./upper
real	0m0.031s
user	0m0.014s
sys	    0m0.009s

Without even a small sample of your file, it is not possible to make concrete recommendations for performance improvement. However, using the dictionary file, my PoC benchmark results versus your benchmark results (6,457,334 ns/op vs. 22,765,120, 8,240 B/op vs. 8,143,216, 3 allocs/op vs. 550,993) do make it likely that your profligate use of CPU and memory is hurting performance.

upper_so_test.go:

package main

import (
    "bufio"
    "bytes"
    "io"
    "io/ioutil"
    "os"
    "strings"
    "testing"
)

func SOToUpper(r io.Reader, w io.Writer) error {
    scanner := bufio.NewScanner(r)
    writer := bufio.NewWriter(w)
    for scanner.Scan() {
	    line := scanner.Text()
	    capitalized := strings.ToUpper(line)
	    _, err := writer.WriteString(capitalized + "\n")
	    if err != nil {
		    return err
	    }
    }
    err := writer.Flush()
    if err != nil {
	    return err
    }
    return nil
}

var benchData = func () []byte {
    data, err := os.ReadFile(`/usr/share/dict/brazilian`)
    if err != nil {
	    panic(err)
    }
    return data
}()

func BenchmarkSO(b *testing.B) {
    for i := 0; i < b.N; i++ {
	    r := bytes.NewReader(benchData)
        w := ioutil.Discard
	    err := SOToUpper(r, w)
	    if err != nil {
		    b.Error(err)
	    }
    }
}

Use []byte instead of string in the inner loop to avoid conversions from []byte to string.

The Scanner.String() method copies the data to a new string.
The Scanner.Bytes() returns a slice on the scanner's buffer.

for scanner.Scan() {
	line := scanner.Bytes() 
	capitalized := bytes.ToUpper(line)
	_, err := writer.Write(capitalized)
	if err != nil {
		log.Fatal(err)
	}
	err = writer.WriteByte('\n') 
	if err != nil {
		log.Fatal(err)
	}
}