Filtering non-json content in a json stream in Go

I'm working with an input stream of json structures in Go. I receive the input stream from another application on my stdin and I can't alter the communications protocol.

The problem I have is that every json structure is terminated by a non-json string line: "end" (without the quotes).

I'm using the Golang encoder/json package to decode the json I'm receiving from stdin. The problem is that the decoder produces an error the second time I call it with the msg: "invalid character 'e' looking for beginning of value".

The issue, of course is, that the "end" string is not json encoded. I would like to know how I can have Go's json decoder skip over this string?

Some sample input:

{"command": "ack", "id": "1231231"}
end
{"command": "fail", "id": "1231231"}
end
{
    "command": "log",
    // the message to log
    "msg": "hello world!"
}
end

Things I've tried:

• I declared: endStr := make([]byte, 10)
• I've tried to use fmt.Fscanf(os.Stdin, "%s", endStr), to read past the string, but no data are read.
• I've tried to use os.Stdin.Read(endStr), but it also returns no data.
• After I read the first json structure, dec.Buffered() returns an io.Reader containing the "end" string, but I don't know how to tell the decoder to skip over this.

Any help would be appreciated.

So the best solution I've been able to come up with is:

1. Ditch the json Decoders,
2. read a byte slice from stdin,
3. trim the slice to exclude the ("\nend\n") character string
4. pass the trimmed slice to a json Unmarshaller

The code I had to write:

// Create a buffer to hold the stream data
data := make([]byte, 5000)

// Read data from stdin in a loop
for {
    _, err = os.Stdin.Read(data)
    if err != nil {
        panic(err)
    }

    index := bytes.Index(data, []byte("\n"))
    data = data[:index]
	
    var myStruct MyStruct
    err = json.Unmarshal(data, &myStruct)
    if err != nil {
        panic(err)
    }

    //(Do something with myStruct)
}

This is messy, but it will do the trick:

package main

import "fmt"
import "encoding/json"
import "bytes"
import "io"
import "bufio"

var input = `{"command": "ack", "id": "1231231"}
end
{"command": "fail", "id": "1231231"}
end
`

func main() {
	// make an io.Reader out of our input constant
	var input = bytes.NewBuffer([]byte(input))
	// we're going to need a buffered reader so we can Peek
	var buf = bufio.NewReader(input)

	// This is the result of the decode.  Use whatever makes sense for you
	var res map[string]interface{}
	var err error
	var dec *json.Decoder
	// We're going to loop until we get an error (hopefully it will be io.EOF
	for err == nil {
		if dec != nil {
			// This is the tricky bit:  json.Decoder has its own buffer.
			// it will read more than the data it needs.  In my simple test, 
			// it buffers all of the data.  What we're doing here is constructing
			// a new bufio.Reader using the remaining bytes in the json decoder's buffer
			// and whatever hasn't been read from our original buffer.
			buf = bufio.NewReader(io.MultiReader(dec.Buffered(), buf))
		}
		// Now let's try to drop an 'end' statement from the buffer
		dropEnd(buf)
		// We need a new json.Decoder each time since the old one contains unusable
		// data in its internal buffer.
		dec = json.NewDecoder(buf)
		// do the decode
		if err = dec.Decode(&res); err == nil {
			fmt.Println("Read:", res)
		}
	}
	if err != io.EOF {
		fmt.Println("Unexpected error:", err)
	}
}

func dropEnd(buf *bufio.Reader) {
	var check = make([]byte, 4)
	// If the next 4 bytes (either "\nend" or "end\n") contain "end", drop read them off the buffer
	if check, _ = buf.Peek(4); bytes.Contains(check, []byte("end")) {
		buf.Read(check)
	}
}

You can play with this code here: http://play.golang.org/p/7NER_fTzXI

You can wrap your os.Stdin in bufio.Reader from the bufio package. Then use buf.Peek(num) to look ahead before you Decode.

You can also use a custom Scanner to delimit the JSON chunks.

What's nice about using bufio vs a static buffer is it'll work on a stream.

If you can restrict your JSON objects to a single line, you just have to break the input by line and ignore what doesn't Unmarshal. Here's a small piece of code.

package main

import (
	"encoding/json"
	"fmt"
	"strings"
)

var input = `{"command": "ack", "id": "1231231"}
end
{"command": "fail", "id": "1231231"}
end
`

type Obj struct {
	Command string
	Msg     string
	Id      string
}

func DoSomethingCool(o Obj) {
	// Do something cool here
	fmt.Println(o)
}

func main() {
	inputs := strings.Split(input, "\n")
	for _, v := range inputs {
		var obj Obj
		if err := json.Unmarshal([]byte(v), &obj); err == nil {
			DoSomethingCool(obj) // Got a valid JSON
		}
	}
}