Paradoxical performance serving samba files in go

I've written a program in go that acts as a simple HTTP interface to a samba share: the user makes a get request to http://proxy.example.com/?path=\\repository\foo\bar.txt and \\repository\foo\bar.txt is served up by way of http.ServeFile (more or less).

However, the performance is abysmal. I ran some benchmarks and the results have me stumped. For context, there are three machines in the picture: the samba server (where the file is located), the proxy server (where the go program runs) and the end user's machine (where I ultimately want the file to get). The samba server and the proxy server are co-located, and the end users are fairly far away.

A straight copy from the samba machine to the user machine using windows' copy runs at ~1.5MB/s. That's good enough for me, and what I'm aiming for in the proxy service.

Unfortunately curl 'http://proxy.example.com/?path=\\repository\foo\bar.txt' > bar.txt from a user's machine clocks in at about 150KB/s.

So, let's see if there's connectivity issues between the samba server and the proxy server. A copy from the samba server to the proxy server looks like it's getting... about 15MB/s.

Hm, maybe it's a go thing? I'll write a go program that benchmarks the transfer speed.

src, _ := os.Open("\\\\repository\\foo\\bar.txt")
start := time.Now()
written, _ := io.Copy(ioutil.Discard, src)
elapsed := time.Since(start)
bytesPerSecond := written/int64(elapsed/time.Second)

Dang, 15MB/s.

Ok, ok, maybe there's something else in the go code causing the issue. Remote onto the proxy server, start up IE, go to http://proxy.example.com/?path=\\repository\foo\bar.txt, 15MB/s.

Alright, so my code is obviously working great, it must be the connection between the proxy server and the end user. I'll copy bar.txt to the proxy server and use its local path in the url, \mycoolfiles\bar.txt. Huh, 1.5MB/s.

To make things even weirder, I just happen to have C:\mycoolfiles set up as a network share named \\alexscoolfiles, and http://proxy.example.com/?path=\\alexscoolfiles\bar.txt clocks in at, dun dun dun, 150KB/s.

Just to confirm this madness, I changed the go program to run in two steps:

1. Copy the file from the share to the local hard drive
2. http.SendFile from there

Lo and behold, after a short pause while the file transfers over at 15MB/s, the download begins at a solid 1.5MB/s.

So, share->proxy is 15MB/s, and proxy->user is 1.5MB/s, but share->proxy->user is... 150KB/s? Ten times slower than it should be? Except not when you're on the same machine as the proxy, because then it's exactly as fast as it should be? And further this problem exists even if it's the exact same file being accessed as long as one is a UNC path and the other is just a local path?

WHAT?

Please help, I just have no idea.

EDIT: So my hunch is (as was commented) that it has something to do with TCP. The offending code has been isolated to pretty much just io.Copy.

• I know that when the reader is a samba file, and the writer is ioutil.Discard, I get about maximum throughput.
• I know that when the reader is a local file and the writer is an http.Response, I get about maximum throughput regardless of the bandwidth and RRT of the client consuming the response.
• I know that when the reader is a samba file, the writer is an http.Response, and the connection is local, I get about maximum throughput.
• I know that when the reader is a samba file, the writer is an http.Response, and the connection is not local, I get terrible (~1/10) throughput.

Looking through io.Copy, it seems the only thing that could cause the issue is in the interplay between the timings of reading the samba file and writing the response; a sufficiently fast writer makes reading a samba file reach max throughput, a sufficiently fast reader makes http.Response.Write reach max throughput, but combining them makes everything suck.

What would be very helpful is... What is actually happening and, more importantly, how can I make this problem go away.

After trying to track down exactly where and under what circumstances I could reproduce the issue, I finally got it down to one line: if I commented out where io.Copy uses ReadFrom (currently io/io.go, line 358), I got maximum throughput.

Checking out where ReadFrom is implemented (net/http/server.go, line 381):

// ReadFrom is here to optimize copying from an *os.File regular file
// to a *net.TCPConn with sendfile.

I don't really have the will to dig any deeper, but I'm guessing this calls net/sendfile_windows.go, which calls the TransmitFile syscall, and somewhere between that and our various server configs something not good happens.

The solution was to copy and paste the things I needed from http.ServeFile, and then turn the ResponseWriter into an io.Writer (or whatever) before passing it to io.Copy:

type writerOnly struct {
    io.Writer
}

//...
io.Copy(writerOnly{w}, f)
//...