handle unstructured JSON data using the standard Go unmarshaller

Some context, I'm designing a backend that will receive JSON post data, but the nature of the data is that it has fields that are unstructured. My general research tells me this is a static language vs unstructured data problem.

Normally if you can create a struct for it if the data is well known and just unmarshal into the struct. I have create custom unmarshaling functions for nested objects.

The issue now is that one of the fields could contain an object with an arbitrary number of keys. To provide some code context:

properties: {
    "k1": "v1",
    "k2": "v2",
    "k3": "v3",
    ...
}



type Device struct {
    id: string,
    name: string,
    status: int,
    properties: <what would i put here?>
}

So its hard to code an explicit unmarshaling function for it. Should put a type of map[string]string{}? How would it work if the values were not all strings then? And what if that object itself had nested values/objects as well?

You can make the Properties field as map[string]interface{} so that it can accommodate different types of values.I created a small code for your scenario as follows:

package main

import (
	"encoding/json"
	"fmt"
)

type Device struct {
	Id         string
	Name       string
	Status     int
	Properties map[string]interface{}
}

func main() {
	devObj := Device{}
	data := []byte(`{"Id":"101","Name":"Harold","Status":1,"properties":{"key1":"val1"}}`)
	if err := json.Unmarshal(data, &devObj); err != nil {
		panic(err)
	}

	fmt.Println(devObj)

	devObj2 := Device{}
	data2 := []byte(`{"Id":"102","Name":"Thanor","Status":1,"properties":{"k1":25,"k2":"someData"}}`)
	if err := json.Unmarshal(data2, &devObj2); err != nil {
		panic(err)
	}

	fmt.Println(devObj2)

	devObj3 := Device{}
	data3 := []byte(`{"Id":"101","Name":"GreyBeard","Status":1,"properties":{"k1":25,"k2":["data1","data2"]}}`)
	if err := json.Unmarshal(data3, &devObj3); err != nil {
		panic(err)
	}
	fmt.Println(devObj3)

}

Output:

{101 Harold 1 map[key1:val1]}
{102 Thanor 1 map[k1:25 k2:someData]}
{101 GreyBeard 1 map[k1:25 k2:[data1 data2]]}

I would use one of the popular Go JSON parsers that don't require parsing to a pre-defined struct. An added benefit, and the primary reason they were created, is that they are much faster than encoding/json because they don't use reflection, interface{} or certain other approaches.

Here are two:

• https://github.com/buger/jsonparser - 4.1k GitHub stars
• https://github.com/valyala/fastjson - 1.3k GitHub stars

Using github.com/buger/jsonparser, a property could be retrieved using the GetString function:

func GetString(data []byte, keys ...string) (val string, err error)

Here's a full example:

package main

import (
	"fmt"
	"strconv"

	"github.com/buger/jsonparser"
)

func main() {
	jsondata := []byte(`
	{
		"properties": {
			"k1": "v1",
			"k2": "v2",
			"k3": "v3"
		}
	}`)

	for i := 1; i > 0; i++ {
		key := "k" + strconv.Itoa(i)
		val, err := jsonparser.GetString(jsondata, "properties", key)
		
        if err == jsonparser.KeyPathNotFoundError {
			break
		} else if err != nil {
			panic(err)
		}

		fmt.Printf("found: key [%s] val [%s]\n", key, val)
	}
}

See it run on Go Playground: https://play.golang.org/p/ykAM4gac8zT