Unmarshal a json stream (not newline-separated)

I want to turn a stream of JSON into a stream of objects. This is easy to do with newline-separated JSON. From the Go docs: https://golang.org/pkg/encoding/json/#Decoder.Buffered

However, I need to generate a stream from JSON arrays like this one:

        [{"Name": "Ed", "Text": "Knock knock."},
        {"Name": "Sam", "Text": "Who's there?"},
        {"Name": "Ed", "Text": "Go fmt."},
        {"Name": "Sam", "Text": "Go fmt who?"},
        {"Name": "Ed", "Text": "Go fmt yourself!"}]

What is a performant way to do this?

I've considered this method:

1. Drop the outside angle brackets
2. When there are matching top-level curly braces, unmarshal the string between the braces (inclusive) to get one top-level object at a time.

I don't want to do it because of the performance implications of scanning each portion of the string twice.

The best alternative I can thing of is to copy the source code for the decoder in the Golang encoding/json package and modify it so it returns a Reader that spits out one object at a time. But that seems like too much work for such a simple requirement.

Is there a better way to decode a stream that is a JSON array?

EDIT

I'm looking to parse JSON with nested objects and arbitrary structure.

You can use a streaming parser. For example megajson's scanner:

package main

import (
	"fmt"
	"strings"

	"github.com/benbjohnson/megajson/scanner"
)

func main() {
	// our incoming data
	rdr := strings.NewReader(`[
		{"Name": "Ed", "Text": "Knock knock."},
        {"Name": "Sam", "Text": "Who's there?"},
        {"Name": "Ed", "Text": "Go fmt."},
        {"Name": "Sam", "Text": "Go fmt who?"},
        {"Name": "Ed", "Text": "Go fmt yourself!"}
	]`)

	// we want to create a list of these
	type Object struct {
		Name string
		Text string
	}
	objects := make([]Object, 0)

	// scan the JSON as we read
	s := scanner.NewScanner(rdr)

	// this is how we keep track of where we are parsing the JSON
	// if you needed to support nested objects you would need to
	// use a stack here ([]state{}) and push / pop each time you
	// see a brace
	var state struct {
		inKey   bool
		lastKey string
		object  Object
	}
	for {
		tok, data, err := s.Scan()
		if err != nil {
			break
		}

		switch tok {
		case scanner.TLBRACE:
			// just saw '{' so start a new object
			state.inKey = true
			state.lastKey = ""
			state.object = Object{}
		case scanner.TRBRACE:
			// just saw '}' so store the object
			objects = append(objects, state.object)
		case scanner.TSTRING:
			// for `key: value`, we just parsed 'key'
			if state.inKey {
				state.lastKey = string(data)
			} else {
				// now we are on `value`
				if state.lastKey == "Name" {
					state.object.Name = string(data)
				} else {
					state.object.Text = string(data)
				}
			}
			state.inKey = !state.inKey
		}
	}
	fmt.Println(objects)
}

This is probably as efficient as you can get, but it does require a lot of manual processing.

Assume the json stream like:

{"Name": "Ed", "Text": "Knock knock."}{"Name": "Sam", "Text": "Who's there?"}{"Name": "Ed", "Text": "Go fmt."}

I have idea, pseudo code like below:

1: skip prefix whitespace
2: if first char not {, throw error
3: load some chars, and find the first "}"
    4: if found, try json.Unmarshal()
        5: if unmarshal fail, load more chars, and find second "}"
             6: redo STEP 4

Below is an implementation, already working in my project:

package json

import (
	"bytes"
	j "encoding/json"
	"errors"
	"io"
	"strings"
)

// Stream represent a json stream
type Stream struct {
	stream *bytes.Buffer
	object *bytes.Buffer
	scrap  *bytes.Buffer
}

// NewStream return a Stream that based on src
func NewStream(src []byte) *Stream {
	return &Stream{
		stream: bytes.NewBuffer(src),
		object: new(bytes.Buffer),
		scrap:  new(bytes.Buffer),
	}
}

// Read read a json object
func (s *Stream) Read() ([]byte, error) {
	var obj []byte

	for {
		// read a rune from stream
		r, _, err := s.stream.ReadRune()
		switch err {
		case nil:
		case io.EOF:
			if strings.TrimSpace(s.object.String()) != "" {
				return nil, errors.New("Invalid JSON")
			}

			fallthrough
		default:
			return nil, err
		}

		// write the rune to object buffer
		if _, err := s.object.WriteRune(r); err != nil {
			return nil, err
		}

		if r == '}' {
			obj = s.object.Bytes()

			// check whether json string valid
			err := j.Compact(s.scrap, obj)
			s.scrap.Reset()
			if err != nil {
				continue
			}

			s.object.Reset()

			break
		}
	}

	return obj, nil
}

Usage like below:

func process(src []byte) error {
	s := json.NewStream(src)

	for {
		obj, err := s.Read()
		switch err {
		case nil:
		case io.EOF:
			return nil 
		default:
			return err 
		}   

		// now you can try to decode the obj to a struct/map/...
        // it is also support mix stream, ex.:
        a = new(TypeOne)
        b = new(TypeTwo)
        if err := j.Unmarshal(obj, a); err == nil && a.Error != "" {
             // it is a TypeOne object
        } else if err := j.Unmarshal(obj, b); err == nil && a.ID != "" {
             // it is a TypeTwo object
        } else {
             // unkown type
        }
	}

	return nil
}