How to flatten JSON for a generic type in Go

I'm trying to implement HAL in Go, just to see if I can. This means that I've got a HAL type that is generic over the payload, and also contains the _links:

type HAL[T any] struct {
	Payload T
	Links   Linkset `json:"_links,omitempty"`
}

In the HAL spec, the payload is actually at the top level and not nested inside it - like, e.g. Siren would be. So that means given the following:

type TestPayload struct {
	Name   string `json:"name"`
	Answer int    `json:"answer"`
}

	hal := HAL[TestPayload]{
		Payload: TestPayload{
			Name:   "Graham",
			Answer: 42,
		},
		Links: Linkset{
			"self": {
				{Href: "/"},
			},
		},
	}

The resulting JSON should be:

{
    "name": "Graham",
    "answer": 42,
    "_links": {
      "self": {"href": "/"}
    }
}

But I can't work out a good way to get this JSON marshalling to work.

I've seen suggestions of embedding the payload as an anonymous member, which works great if it's not generic. Unfortunately, you can't embed generic types in that way so that's a non-starter.

I probably could write a MarshalJSON method that will do the job, but I'm wondering if there's any standard way to achieve this instead?

I've got a Playground link with this working code to see if it helps: https://go.dev/play/p/lorK5Wv-Tri

Cheers

Yes, unfortunately you can't embed the type parameter T. I'll also argue that in the general case you shouldn't attempt to flatten the output JSON. By constraining T with any, you are admitting literally any type, however not all types have fields to promote into your HAL struct.

This is semantically inconsistent.

If you attempt to embed a type with no fields, the output JSON will be different. Using the solution with reflect.StructOf as an example, nothing stops me from instantiating HAL[[]int]{ Payload: []int{1,2,3}, Links: ... }, in which case the output would be:

{"X":[1,2,3],"Links":{"self":{"href":"/"}}}

This makes your JSON serialization change with the types used to instantiate T, which is not easy to spot for someone who reads your code. The code is less predictable, and you are effectively working against the generalization that type parameters provide.

Using the named field Payload T is just better, as:

• the output JSON is always (for most intents and purposes) consistent with the actual struct
• unmarshalling also keeps a predictable behavior
• scalability of the code is not an issue, as you don't have to repeat all of the fields of HAL to build an anonymous struct

OTOH, if your requirements are precisely to marshal structs as flattened and everything else with a key (as it might be the case with HAL types), at the very least make it obvious by checking reflect.ValueOf(hal.Payload).Kind() == reflect.Struct in the MarshalJSON implementation, and provide a default case for whatever else T could be. Will have to be repeated in JSONUnmarshal.

Here is a solution with reflection that works when T is not a struct and scales when you add more fields to the main struct:

// necessary to marshal HAL without causing infinite loop
// can't declare inside the method due to a current limitation with Go generics
type tmp[T any] HAL[T]

func (h HAL[T]) MarshalJSON() ([]byte, error) {
	// examine Payload, if it isn't a struct, i.e. no embeddable fields, marshal normally
	v := reflect.ValueOf(h.Payload)
	if v.Kind() == reflect.Pointer || v.Kind() == reflect.Interface {
		v = v.Elem()
	}
	if v.Kind() != reflect.Struct {
		return json.Marshal(tmp[T](h))
	}

	// flatten all fields into a map
	m := make(map[string]any)
	// flatten Payload first
	for i := 0; i < v.NumField(); i++ {
		key := jsonkey(v.Type().Field(i))
		m[key] = v.Field(i).Interface()
	}
	// flatten the other fields
	w := reflect.ValueOf(h)
	// start at 1 to skip the Payload field
	for i := 1; i < w.NumField(); i++ {
		key := jsonkey(w.Type().Field(i))
		m[key] = w.Field(i).Interface()
	}
	return json.Marshal(m)
}

func jsonkey(field reflect.StructField) string {
	// trickery to get the json tag without omitempty and whatnot
	tag := field.Tag.Get("json")
	tag, _, _ = strings.Cut(tag, ",")
	if tag == "" {
		tag = field.Name
	}
	return tag
}

With HAL[TestPayload] or HAL[*TestPayload] it outputs:

{"answer":42,"name":"Graham","_links":{"self":{"href":"/"}}}

With HAL[[]int] it outputs:

{"Payload":[1,2,3],"_links":{"self":{"href":"/"}}}

Playground: https://go.dev/play/p/bWGXWj_rC5F

I'd make a custom JSON codec that inserts _links field at the end of the JSON jenerated for the payload.

Marshaller.

type Link struct {
Href string `json:"href"`
}
type Linkset map[string]Link
type HAL[T any] struct {
Payload T
Links   Linkset `json:"_links,omitempty"`
}
func (h HAL[T]) MarshalJSON() ([]byte, error) {
payloadJson, err := json.Marshal(h.Payload)
if err != nil {
return nil, err
}
if len(payloadJson) == 0 {
return nil, fmt.Errorf("Empty payload")
}
if h.Links != nil {
return appendField(payloadJson, "_links", h.Links)
}
return payloadJson, nil
}
func appendField[T any](raw []byte, fieldName string, v T) ([]byte, error) {
// The JSON data must be braced in {}
if raw[0] != '{' || raw[len(raw)-1] != '}' {
return nil, fmt.Errorf("Not an object: %s", string(raw))
}
valJson, err := json.Marshal(v)
if err != nil {
return nil, err
}
// Add the field at the end of the json text
result := bytes.NewBuffer(raw[:len(raw)-1])
// Append `"<fieldName>":value`
// Insert comma if the `raw` object is not empty
if len(raw) > 2 {
result.WriteByte(',')
}
// tag
result.WriteByte('"')
result.WriteString(fieldName)
result.WriteByte('"')
// colon
result.WriteByte(':')
// value
result.Write(valJson)
// closing brace
result.WriteByte('}')
return result.Bytes(), nil
}

The marshaller returns an error if Payload serializes to something other than JSON object. The reason is that the codec can add _links field to objects only.

Unmarshaller:

func (h *HAL[T]) UnmarshalJSON(raw []byte) error {
// Unmarshal fields of the payload first.
// Unmarshal the whole JSON into the payload, it is safe:
// decorer ignores unknow fields and skips "_links".
if err := json.Unmarshal(raw, &h.Payload); err != nil {
return err
}
// Get "_links": scan trough JSON until "_links" field
links := make(Linkset)
exists, err := extractField(raw, "_links", &links)
if err != nil {
return err
}
if exists {
h.Links = links
}
return nil
}
func extractField[T any](raw []byte, fieldName string, v *T) (bool, error) {
// Scan through JSON until field is found
decoder := json.NewDecoder(bytes.NewReader(raw))
t := must(decoder.Token())
// should be `{`
if t != json.Delim('{') {
return false, fmt.Errorf("Not an object: %s", string(raw))
}
t = must(decoder.Token())
if t == json.Delim('}') {
// Empty object
return false, nil
}
for decoder.More() {
name, ok := t.(string)
if !ok {
return false, fmt.Errorf("must never happen: expected string, got `%v`", t)
}
if name != fieldName {
skipValue(decoder)
} else {
if err := decoder.Decode(v); err != nil {
return false, err
}
return true, nil
}
if decoder.More() {
t = must(decoder.Token())
}
}
return false, nil
}
func skipValue(d *json.Decoder) {
braceCnt := 0
for d.More() {
t := must(d.Token())
if t == json.Delim('{') || t == json.Delim('[') {
braceCnt++
}
if t == json.Delim('}') || t == json.Delim(']') {
braceCnt--
}
if braceCnt == 0 {
return
}
}
}

The unmarshaller fails on non-object as well. It is required to read _links field. For that the input must be an object.

The full example: https://go.dev/play/p/E3NN2T7Fbnm

func main() {
hal := HAL[TestPayload]{
Payload: TestPayload{
Name:   "Graham",
Answer: 42,
},
Links: Linkset{
"self": Link{Href: "/"},
},
}
bz := must(json.Marshal(hal))
println(string(bz))
var halOut HAL[TestPayload]
err := json.Unmarshal(bz, &halOut)
if err != nil {
println("Decode failed: ", err.Error())
}
fmt.Printf("%#v\n", halOut)
}

Output:

{"name":"Graham","answer":42,"_links":{"self":{"href":"/"}}}
main.HAL[main.TestPayload]{Payload:main.TestPayload{Name:"Graham", Answer:42}, Links:main.Linkset{"self":main.Link{Href:"/"}}}

Yes, embedding is the easiest way, and as you wrote, you can't currently embed a type parameter.

You may however construct a type that embeds the type param using reflection. We may instantiate this type and marshal it instead.

For example:

func (hal HAL[T]) MarshalJSON() ([]byte, error) {
t := reflect.StructOf([]reflect.StructField{
{
Name:      "X",
Anonymous: true,
Type:      reflect.TypeOf(hal.Payload),
},
{
Name: "Links",
Type: reflect.TypeOf(hal.Links),
},
})
v := reflect.New(t).Elem()
v.Field(0).Set(reflect.ValueOf(hal.Payload))
v.Field(1).Set(reflect.ValueOf(hal.Links))
return json.Marshal(v.Interface())
}

This will output (try it on the Go Playground):

{"name":"Graham","answer":42,"Links":{"self":{"href":"/"}}}

See related: https://stackoverflow.com/questions/42709680/adding-arbitrary-fields-to-json-output-of-an-unknown-struct/42715610#42715610

Keep it simple.

Yes it would be nice to embed the type - but since it's not currently possible (as of go1.19) to embed a generic type - just write it out inline:

body, _ = json.Marshal(
struct {
TestPayload
Links       Linkset `json:"_links,omitempty"`
}{
TestPayload: hal.Payload,
Links:       hal.Links,
},
)

https://go.dev/play/p/8yrB-MzUVK-

{
"name": "Graham",
"answer": 42,
"_links": {
"self": {
"href": "/"
}
}
}

Yes, the constraint type needs to be referenced twice - but all the customization is code-localized, so no need for a custom marshaler.