How do I retrieve file data over a socket in Go?

I've got two small programs communicating nicely over a socket where the receiving side is in Go. Everything works peachy when my messages are tiny enough to fit in the 1024 byte buffer and can be received in a single Read from the connection but now I want to transfer data from an image that is 100k+ or more. I'm assuming the correct solution is not to increase the buffer until any image can fit inside.

Pseudo-go:

var buf = make([]byte,1024)
conn, err := net.Dial("tcp", ":1234")

for {
    r, err := conn.Read(buf[0:])
    go readHandler(string(buf[0:r]),conn)
}

How can I improve my socket read routine to accept both simple messages of a few bytes and also larger data? Bonus points if you can turn the total image data into an io.Reader for use in image.Decode.

I have no direct experience with TCP in Go but to me it seems that you fell victim of a quite typical misunderstanding of what guarntees TCP offers.

The thing is, in contrast with, say, UDP and SCTP, TCP does not have the concept of message boundaries because it's stream-oriented. It means, TCP transports opaque streams of bytes and you have very little control of "chunking" that stream with regard to the receiving side.

I suspect what you observe as "sending a 100k+ message" is the runtime/network library on the sender side typically "deceiving" you by consuming your "message" into its internal buffers and then streaming it in whatever chunks OS's TCP stack allows it to (on ubiquitous hardware/software it's usually about 8k). The size of pieces the receiver gets that stream is completely undefined; the only thing defined is ordering of the bytes in the stream, which is preserved.

Hence it might turn out you have to resonsider your approach to receiving data. The exact approach varies depending on the nature of the data being streamed:

• The easiest way (if you have the control over the application-level protocol) is to pass the length of the following "message payload" in a special length field of fixed format. Then destreaming the whole message is a two-step process: 1) receive that many bytes to get the length field, read it, check the value for sanity, then 2) read that many following bytes and be done with it.
• If you have no control over the app-level protocol, parsing messages becomes more involved and usually requires some sort of complicated state machine.

For more info, look at this and this.

You can use <code>io.ReadFull</code> to read a <code>[]byte</code> of a specific length. This assumes that you know beforehand how many bytes you need to read.

As for <code>image.Decode</code>, it should be possible to pass the <code>conn</code> directly to the <code>image.Decode</code> function. This assumes that you do not perform any reads from the connection until the image is decoded.

Your code

for {
    r, err := conn.Read(buf[0:])
    go readHandler(string(buf[0:r]),conn)
}

seems to be suggesting that the goroutine you are starting is reading from <code>conn</code> This doesn't seem like a good idea, because you will end up having multiple concurrent reads from the connection (without having control over the order in which the reads will happen): one in the for-loop, another one in <code>readHandler</code>.