Standard xml parser has very low performance in Golang

I have a 100GB XML file and parse it with SAX method in go with this code

    file, err := os.Open(filename)
	handle(err)
	defer file.Close()
	buffer := bufio.NewReaderSize(file, 1024*1024*256) // 33554432
	decoder := xml.NewDecoder(buffer)
    for {
    		t, _ := decoder.Token()
    		if t == nil {
    			break
    		}
    		switch se := t.(type) {
    		case xml.StartElement:
    			if se.Name.Local == "House" {
    				house := House{}
    				err := decoder.DecodeElement(&house, &se)
    				handle(err)
    			}
    		}
    	}

But golang working very slow, its seems by execution time and disk usage. My HDD capable to read data with speed around 100-120 MB/s, but golang uses only 10-13 MB/s.

For experiment I rewrite this code in C#:

    using (XmlReader reader = XmlReader.Create(filename)
                {
                    while (reader.Read())
                    {
                        switch (reader.NodeType)
                        {
                            case XmlNodeType.Element:
                                if (reader.Name == "House")
                                {
                                    //Code
                                }
                                break;
                        }
                    }
                }

And I got full HDD loaded, c# read data with 100-110 MB/s speed. And execution time around 10 times lower.

How can I improve XML parse performance using golang?

These 5 things can help increase speed using the encoding/xml library:
(Tested against XMB with 75k entries, 20MB, %s are applied to previous bullet)

1. Use well defined structures
2. Implement xml.Unmarshaller on all your structures
   • Lots of code
   • Saves 20% time and 15% allocs
3. Replace d.DecodeElement(&foo, &token) with foo.UnmarshallXML(d, &token)
   • Almost 100% safe
   • Saves 10% time & allocs
4. Use d.RawToken() instead of d.Token()
   • Needs manual handling of nested objects and namespaces
   • Saves 10% time & 20% allocs
5. If use use d.Skip(), reimplement it using d.RawToken()

I reduced time and allocs by 40% on my specific usecase at the cost of more code, boileplate, and potentially worse handling of corner cases, but my inputs are fairly consistent, however it's not enough.

benchstat first.bench.txt parseraw.bench.txt 
name          old time/op    new time/op    delta
Unmarshal-16     1.06s ± 6%     0.66s ± 4%  -37.55%  (p=0.008 n=5+5)

name          old alloc/op   new alloc/op   delta
Unmarshal-16     461MB ± 0%     280MB ± 0%  -39.20%  (p=0.029 n=4+4)

name          old allocs/op  new allocs/op  delta
Unmarshal-16     8.42M ± 0%     5.03M ± 0%  -40.26%  (p=0.016 n=4+5)

On my experiments, the lack of memoizing issue is the reason for large time/allocs on the XML parser which slows down significantly, mostly because of Go copying by value.

To answer your question "How can i improve xml parse performance using golang?"

Using the common xml.NewDecoder / decoder.Token, I was seeing 50 MB/s locally. By using https://github.com/tamerh/xml-stream-parser I was able to double the parse speed.

To test I used Posts.xml (68 GB) from the https://archive.org/details/stackexchange archive torrent.

package main

import (
	"bufio"
	"fmt"
	"github.com/tamerh/xml-stream-parser"
	"os"
	"time"
)

func main() {
	// Using `Posts.xml` (68 GB) from https://archive.org/details/stackexchange (in the torrent)
	f, err := os.Open("Posts.xml")
	if err != nil {
		panic(err)
	}
	defer f.Close()

	br := bufio.NewReaderSize(f, 1024*1024)
	parser := xmlparser.NewXmlParser(br, "row")

	started := time.Now()
	var previous int64 = 0

	for x := range *parser.Stream() {
		elapsed := int64(time.Since(started).Seconds())
		if elapsed > previous {
			kBytesPerSecond := int64(parser.TotalReadSize) / elapsed / 1024
			fmt.Printf("\r%ds elapsed, read %d kB/s (last post.Id %s)", elapsed, kBytesPerSecond, x.Attrs["Id"])
			previous = elapsed
		}
	}
}

This will output something along the line of:

...s elapsed, read ... kB/s (last post.Id ...)

Only caveat is that this does not give you the convenient unmarshal into struct.

As discussed in https://github.com/golang/go/issues/21823, speed seems to be general problem with the XML implementation in Golang and would require a rewrite / rethink of that part of the standard library.