How to flatten JSON for a generic type in Go

huangapple go评论88阅读模式
英文:

How to flatten JSON for a generic type in Go

问题

我正在尝试在Go中实现HAL,只是为了看看我能不能做到。这意味着我有一个HAL类型,它对有效负载是通用的,并且还包含_links

type HAL[T any] struct {
	Payload T
	Links   Linkset `json:"_links,omitempty"`
}

在HAL规范中,有效负载实际上是在顶层而不是嵌套在其中的,就像Siren一样。所以,给定以下代码:

type TestPayload struct {
	Name   string `json:"name"`
	Answer int    `json:"answer"`
}

hal := HAL[TestPayload]{
	Payload: TestPayload{
		Name:   "Graham",
		Answer: 42,
	},
	Links: Linkset{
		"self": {
			{Href: "/"},
		},
	},
}

生成的JSON应该是:

{
    "name": "Graham",
    "answer": 42,
    "_links": {
      "self": {"href": "/"}
    }
}

但是我无法找到一个好的方法来使这个JSON编组工作。

我看到有人建议将有效负载作为匿名成员嵌入,这在非通用情况下效果很好。不幸的是,你不能以这种方式嵌入通用类型,所以这是行不通的。

我可能可以编写一个MarshalJSON方法来完成这个任务,但我想知道是否有任何标准的方法来实现这个?

我在这里提供了一个Playground链接,其中包含这个可工作的代码,希望能对你有所帮助:https://go.dev/play/p/lorK5Wv-Tri

谢谢!

英文:

I'm trying to implement HAL in Go, just to see if I can. This means that I've got a HAL type that is generic over the payload, and also contains the _links:

type HAL[T any] struct {
	Payload T
	Links   Linkset `json:"_links,omitempty"`
}

In the HAL spec, the payload is actually at the top level and not nested inside it - like, e.g. Siren would be. So that means given the following:

type TestPayload struct {
	Name   string `json:"name"`
	Answer int    `json:"answer"`
}

	hal := HAL[TestPayload]{
		Payload: TestPayload{
			Name:   "Graham",
			Answer: 42,
		},
		Links: Linkset{
			"self": {
				{Href: "/"},
			},
		},
	}

The resulting JSON should be:

{
    "name": "Graham",
    "answer": 42,
    "_links": {
      "self": {"href": "/"}
    }
}

But I can't work out a good way to get this JSON marshalling to work.

I've seen suggestions of embedding the payload as an anonymous member, which works great if it's not generic. Unfortunately, you can't embed generic types in that way so that's a non-starter.

I probably could write a MarshalJSON method that will do the job, but I'm wondering if there's any standard way to achieve this instead?

I've got a Playground link with this working code to see if it helps: https://go.dev/play/p/lorK5Wv-Tri

Cheers

答案1

得分: 0

是的,不幸的是,你不能嵌入类型参数T。我还会认为在一般情况下,你不应该尝试展平输出的 JSON。通过使用any约束T,你允许任何类型,然而,并不是所有类型都有字段可以提升到你的HAL结构中。

这在语义上是不一致的。

如果你尝试嵌入一个没有字段的类型,输出的 JSON 将会不同。以使用reflect.StructOf的解决方案为例,没有任何限制阻止我实例化HAL[[]int]{ Payload: []int{1,2,3}, Links: ... },在这种情况下,输出将是:

{"X":[1,2,3],"Links":{"self":{"href":"/"}}}

这使得你的 JSON 序列化随着用于实例化T的类型而改变,这对于阅读你的代码的人来说并不容易发现。代码的可预测性降低了,你实际上是在与类型参数提供的泛化相抵触。

使用命名字段Payload T更好,因为:

  • 输出的 JSON 总是(在大多数情况下)与实际结构一致
  • 反序列化也保持了可预测的行为
  • 代码的可扩展性不是问题,因为你不需要重复构建匿名结构的所有字段

另一方面,如果你的要求确切地是将结构体序列化为展平形式,而其他类型使用键(就像 HAL 类型可能的情况),至少在MarshalJSON实现中通过检查reflect.ValueOf(hal.Payload).Kind() == reflect.Struct来明确表示,并为T可能是其他类型的情况提供一个默认情况。这将不得不在JSONUnmarshal中重复。

下面是一个使用反射的解决方案,当T不是结构体时可以工作,并且在主结构体中添加更多字段时可以扩展:

// 为了在不引起无限循环的情况下序列化 HAL,必须声明在方法外部,因为 Go 泛型目前有一个限制
type tmp[T any] HAL[T]

func (h HAL[T]) MarshalJSON() ([]byte, error) {
	// 检查 Payload,如果它不是结构体,即没有可嵌入的字段,就正常序列化
	v := reflect.ValueOf(h.Payload)
	if v.Kind() == reflect.Pointer || v.Kind() == reflect.Interface {
		v = v.Elem()
	}
	if v.Kind() != reflect.Struct {
		return json.Marshal(tmp[T](h))
	}

	// 将所有字段展平为一个映射
	m := make(map[string]interface{})
	// 先展平 Payload
	for i := 0; i < v.NumField(); i++ {
		key := jsonkey(v.Type().Field(i))
		m[key] = v.Field(i).Interface()
	}
	// 展平其他字段
	w := reflect.ValueOf(h)
	// 从 1 开始跳过 Payload 字段
	for i := 1; i < w.NumField(); i++ {
		key := jsonkey(w.Type().Field(i))
		m[key] = w.Field(i).Interface()
	}
	return json.Marshal(m)
}

func jsonkey(field reflect.StructField) string {
	// 通过一些技巧获取 json 标签,不包括 omitempty 等内容
	tag := field.Tag.Get("json")
	tag, _, _ = strings.Cut(tag, ",")
	if tag == "" {
		tag = field.Name
	}
	return tag
}

使用HAL[TestPayload]HAL[*TestPayload]输出:

{"answer":42,"name":"Graham","_links":{"self":{"href":"/"}}}

使用HAL[[]int]输出:

{"Payload":[1,2,3],"_links":{"self":{"href":"/"}}}

Playground: https://go.dev/play/p/bWGXWj_rC5F

英文:

Yes, unfortunately you can't embed the type parameter T. I'll also argue that in the general case you shouldn't attempt to flatten the output JSON. By constraining T with any, you are admitting literally any type, however not all types have fields to promote into your HAL struct.

This is semantically inconsistent.

If you attempt to embed a type with no fields, the output JSON will be different. Using the solution with reflect.StructOf as an example, nothing stops me from instantiating HAL[[]int]{ Payload: []int{1,2,3}, Links: ... }, in which case the output would be:

{&quot;X&quot;:[1,2,3],&quot;Links&quot;:{&quot;self&quot;:{&quot;href&quot;:&quot;/&quot;}}}

This makes your JSON serialization change with the types used to instantiate T, which is not easy to spot for someone who reads your code. The code is less predictable, and you are effectively working against the generalization that type parameters provide.

Using the named field Payload T is just better, as:

  • the output JSON is always (for most intents and purposes) consistent with the actual struct
  • unmarshalling also keeps a predictable behavior
  • scalability of the code is not an issue, as you don't have to repeat all of the fields of HAL to build an anonymous struct

OTOH, if your requirements are precisely to marshal structs as flattened and everything else with a key (as it might be the case with HAL types), at the very least make it obvious by checking reflect.ValueOf(hal.Payload).Kind() == reflect.Struct in the MarshalJSON implementation, and provide a default case for whatever else T could be. Will have to be repeated in JSONUnmarshal.

Here is a solution with reflection that works when T is not a struct and scales when you add more fields to the main struct:

// necessary to marshal HAL without causing infinite loop
// can&#39;t declare inside the method due to a current limitation with Go generics
type tmp[T any] HAL[T]

func (h HAL[T]) MarshalJSON() ([]byte, error) {
	// examine Payload, if it isn&#39;t a struct, i.e. no embeddable fields, marshal normally
	v := reflect.ValueOf(h.Payload)
	if v.Kind() == reflect.Pointer || v.Kind() == reflect.Interface {
		v = v.Elem()
	}
	if v.Kind() != reflect.Struct {
		return json.Marshal(tmp[T](h))
	}

	// flatten all fields into a map
	m := make(map[string]any)
	// flatten Payload first
	for i := 0; i &lt; v.NumField(); i++ {
		key := jsonkey(v.Type().Field(i))
		m[key] = v.Field(i).Interface()
	}
	// flatten the other fields
	w := reflect.ValueOf(h)
	// start at 1 to skip the Payload field
	for i := 1; i &lt; w.NumField(); i++ {
		key := jsonkey(w.Type().Field(i))
		m[key] = w.Field(i).Interface()
	}
	return json.Marshal(m)
}

func jsonkey(field reflect.StructField) string {
	// trickery to get the json tag without omitempty and whatnot
	tag := field.Tag.Get(&quot;json&quot;)
	tag, _, _ = strings.Cut(tag, &quot;,&quot;)
	if tag == &quot;&quot; {
		tag = field.Name
	}
	return tag
}

With HAL[TestPayload] or HAL[*TestPayload] it outputs:

{&quot;answer&quot;:42,&quot;name&quot;:&quot;Graham&quot;,&quot;_links&quot;:{&quot;self&quot;:{&quot;href&quot;:&quot;/&quot;}}}

With HAL[[]int] it outputs:

{&quot;Payload&quot;:[1,2,3],&quot;_links&quot;:{&quot;self&quot;:{&quot;href&quot;:&quot;/&quot;}}}

Playground: https://go.dev/play/p/bWGXWj_rC5F

答案2

得分: 0

我会为你翻译这段代码。以下是翻译的结果:

我会创建一个自定义的JSON编解码器,将_links字段插入到生成的JSON负载的末尾。

编组器:

type Link struct {
	Href string `json:"href"`
}

type Linkset map[string]Link

type HAL[T any] struct {
	Payload T
	Links   Linkset `json:"_links,omitempty"`
}

func (h HAL[T]) MarshalJSON() ([]byte, error) {
	payloadJson, err := json.Marshal(h.Payload)
	if err != nil {
		return nil, err
	}
	if len(payloadJson) == 0 {
		return nil, fmt.Errorf("Empty payload")
	}
	if h.Links != nil {
		return appendField(payloadJson, "_links", h.Links)
	}
	return payloadJson, nil
}

func appendField[T any](raw []byte, fieldName string, v T) ([]byte, error) {
	// JSON数据必须用{}括起来
	if raw[0] != '{' || raw[len(raw)-1] != '}' {
		return nil, fmt.Errorf("Not an object: %s", string(raw))
	}
	valJson, err := json.Marshal(v)
	if err != nil {
		return nil, err
	}
	// 在json文本末尾添加字段
	result := bytes.NewBuffer(raw[:len(raw)-1])
	// 添加 "<fieldName>":value
	// 如果"raw"对象不为空,则插入逗号
	if len(raw) > 2 {
		result.WriteByte(',')
	}
	result.WriteByte('"')
	result.WriteString(fieldName)
	result.WriteByte('"')
	result.WriteByte(':')
	result.Write(valJson)
	result.WriteByte('}')
	return result.Bytes(), nil
}

如果Payload序列化为非JSON对象,编组器将返回错误。原因是编解码器只能向对象添加_links字段。

解码器:

func (h *HAL[T]) UnmarshalJSON(raw []byte) error {
	// 首先解组负载的字段。
	// 将整个JSON解组为负载,是安全的:
	// 解码器会忽略未知字段并跳过"_links"。
	if err := json.Unmarshal(raw, &h.Payload); err != nil {
		return err
	}
	// 获取"_links":扫描JSON直到找到"_links"字段
	links := make(Linkset)
	exists, err := extractField(raw, "_links", &links)
	if err != nil {
		return err
	}
	if exists {
		h.Links = links
	}
	return nil
}

func extractField[T any](raw []byte, fieldName string, v *T) (bool, error) {
	// 扫描JSON直到找到字段
	decoder := json.NewDecoder(bytes.NewReader(raw))
	t := must(decoder.Token())
	// 应该是`{`
	if t != json.Delim('{') {
		return false, fmt.Errorf("Not an object: %s", string(raw))
	}
	t = must(decoder.Token())
	if t == json.Delim('}') {
		// 空对象
		return false, nil
	}
	for decoder.More() {
		name, ok := t.(string)
		if !ok {
			return false, fmt.Errorf("must never happen: expected string, got `%v`", t)
		}
		if name != fieldName {
			skipValue(decoder)
		} else {
			if err := decoder.Decode(v); err != nil {
				return false, err
			}
			return true, nil
		}
		if decoder.More() {
			t = must(decoder.Token())
		}
	}
	return false, nil
}

func skipValue(d *json.Decoder) {
	braceCnt := 0
	for d.More() {
		t := must(d.Token())
		if t == json.Delim('{') || t == json.Delim('[') {
			braceCnt++
		}
		if t == json.Delim('}') || t == json.Delim(']') {
			braceCnt--
		}
		if braceCnt == 0 {
			return
		}
	}
}

解码器也会在非对象上失败。它需要读取_links字段。为此,输入必须是一个对象。

完整示例:https://go.dev/play/p/E3NN2T7Fbnm

func main() {
	hal := HAL[TestPayload]{
		Payload: TestPayload{
			Name:   "Graham",
			Answer: 42,
		},
		Links: Linkset{
			"self": Link{Href: "/"},
		},
	}
	bz := must(json.Marshal(hal))
	println(string(bz))

	var halOut HAL[TestPayload]
	err := json.Unmarshal(bz, &halOut)
	if err != nil {
		println("Decode failed: ", err.Error())
	}
	fmt.Printf("%#v\n", halOut)
}

输出:

{"name":"Graham","answer":42,"_links":{"self":{"href":"/"}}}
main.HAL[main.TestPayload]{Payload:main.TestPayload{Name:"Graham", Answer:42}, Links:main.Linkset{"self":main.Link{Href:"/"}}}
英文:

I'd make a custom JSON codec that inserts _links field at the end of the JSON jenerated for the payload.

Marshaller.


type Link struct {
Href string `json:&quot;href&quot;`
}
type Linkset map[string]Link
type HAL[T any] struct {
Payload T
Links   Linkset `json:&quot;_links,omitempty&quot;`
}
func (h HAL[T]) MarshalJSON() ([]byte, error) {
payloadJson, err := json.Marshal(h.Payload)
if err != nil {
return nil, err
}
if len(payloadJson) == 0 {
return nil, fmt.Errorf(&quot;Empty payload&quot;)
}
if h.Links != nil {
return appendField(payloadJson, &quot;_links&quot;, h.Links)
}
return payloadJson, nil
}
func appendField[T any](raw []byte, fieldName string, v T) ([]byte, error) {
// The JSON data must be braced in {}
if raw[0] != &#39;{&#39; || raw[len(raw)-1] != &#39;}&#39; {
return nil, fmt.Errorf(&quot;Not an object: %s&quot;, string(raw))
}
valJson, err := json.Marshal(v)
if err != nil {
return nil, err
}
// Add the field at the end of the json text
result := bytes.NewBuffer(raw[:len(raw)-1])
// Append `&quot;&lt;fieldName&gt;&quot;:value`
// Insert comma if the `raw` object is not empty
if len(raw) &gt; 2 {
result.WriteByte(&#39;,&#39;)
}
// tag
result.WriteByte(&#39;&quot;&#39;)
result.WriteString(fieldName)
result.WriteByte(&#39;&quot;&#39;)
// colon
result.WriteByte(&#39;:&#39;)
// value
result.Write(valJson)
// closing brace
result.WriteByte(&#39;}&#39;)
return result.Bytes(), nil
}

The marshaller returns an error if Payload serializes to something other than JSON object. The reason is that the codec can add _links field to objects only.

Unmarshaller:

func (h *HAL[T]) UnmarshalJSON(raw []byte) error {
// Unmarshal fields of the payload first.
// Unmarshal the whole JSON into the payload, it is safe:
// decorer ignores unknow fields and skips &quot;_links&quot;.
if err := json.Unmarshal(raw, &amp;h.Payload); err != nil {
return err
}
// Get &quot;_links&quot;: scan trough JSON until &quot;_links&quot; field
links := make(Linkset)
exists, err := extractField(raw, &quot;_links&quot;, &amp;links)
if err != nil {
return err
}
if exists {
h.Links = links
}
return nil
}
func extractField[T any](raw []byte, fieldName string, v *T) (bool, error) {
// Scan through JSON until field is found
decoder := json.NewDecoder(bytes.NewReader(raw))
t := must(decoder.Token())
// should be `{`
if t != json.Delim(&#39;{&#39;) {
return false, fmt.Errorf(&quot;Not an object: %s&quot;, string(raw))
}
t = must(decoder.Token())
if t == json.Delim(&#39;}&#39;) {
// Empty object
return false, nil
}
for decoder.More() {
name, ok := t.(string)
if !ok {
return false, fmt.Errorf(&quot;must never happen: expected string, got `%v`&quot;, t)
}
if name != fieldName {
skipValue(decoder)
} else {
if err := decoder.Decode(v); err != nil {
return false, err
}
return true, nil
}
if decoder.More() {
t = must(decoder.Token())
}
}
return false, nil
}
func skipValue(d *json.Decoder) {
braceCnt := 0
for d.More() {
t := must(d.Token())
if t == json.Delim(&#39;{&#39;) || t == json.Delim(&#39;[&#39;) {
braceCnt++
}
if t == json.Delim(&#39;}&#39;) || t == json.Delim(&#39;]&#39;) {
braceCnt--
}
if braceCnt == 0 {
return
}
}
}

The unmarshaller fails on non-object as well. It is required to read _links field. For that the input must be an object.

The full example: https://go.dev/play/p/E3NN2T7Fbnm

func main() {
hal := HAL[TestPayload]{
Payload: TestPayload{
Name:   &quot;Graham&quot;,
Answer: 42,
},
Links: Linkset{
&quot;self&quot;: Link{Href: &quot;/&quot;},
},
}
bz := must(json.Marshal(hal))
println(string(bz))
var halOut HAL[TestPayload]
err := json.Unmarshal(bz, &amp;halOut)
if err != nil {
println(&quot;Decode failed: &quot;, err.Error())
}
fmt.Printf(&quot;%#v\n&quot;, halOut)
}

Output:

{&quot;name&quot;:&quot;Graham&quot;,&quot;answer&quot;:42,&quot;_links&quot;:{&quot;self&quot;:{&quot;href&quot;:&quot;/&quot;}}}
main.HAL[main.TestPayload]{Payload:main.TestPayload{Name:&quot;Graham&quot;, Answer:42}, Links:main.Linkset{&quot;self&quot;:main.Link{Href:&quot;/&quot;}}}

答案3

得分: -1

是的,嵌入是最简单的方法,正如你所写的,目前无法嵌入类型参数。

但是,你可以使用反射构造一个嵌入类型参数的类型。我们可以实例化这个类型并进行编组。

例如:

func (hal HAL[T]) MarshalJSON() ([]byte, error) {
    t := reflect.StructOf([]reflect.StructField{
        {
            Name:      "X",
            Anonymous: true,
            Type:      reflect.TypeOf(hal.Payload),
        },
        {
            Name: "Links",
            Type: reflect.TypeOf(hal.Links),
        },
    })

    v := reflect.New(t).Elem()
    v.Field(0).Set(reflect.ValueOf(hal.Payload))
    v.Field(1).Set(reflect.ValueOf(hal.Links))

    return json.Marshal(v.Interface())
}

这将输出(在Go Playground上尝试):

{"name":"Graham","answer":42,"Links":{"self":{"href":"/"}}}

参考链接:https://stackoverflow.com/questions/42709680/adding-arbitrary-fields-to-json-output-of-an-unknown-struct/42715610#42715610

英文:

Yes, embedding is the easiest way, and as you wrote, you can't currently embed a type parameter.

You may however construct a type that embeds the type param using reflection. We may instantiate this type and marshal it instead.

For example:

func (hal HAL[T]) MarshalJSON() ([]byte, error) {
t := reflect.StructOf([]reflect.StructField{
{
Name:      &quot;X&quot;,
Anonymous: true,
Type:      reflect.TypeOf(hal.Payload),
},
{
Name: &quot;Links&quot;,
Type: reflect.TypeOf(hal.Links),
},
})
v := reflect.New(t).Elem()
v.Field(0).Set(reflect.ValueOf(hal.Payload))
v.Field(1).Set(reflect.ValueOf(hal.Links))
return json.Marshal(v.Interface())
}

This will output (try it on the Go Playground):

{&quot;name&quot;:&quot;Graham&quot;,&quot;answer&quot;:42,&quot;Links&quot;:{&quot;self&quot;:{&quot;href&quot;:&quot;/&quot;}}}

See related: https://stackoverflow.com/questions/42709680/adding-arbitrary-fields-to-json-output-of-an-unknown-struct/42715610#42715610

答案4

得分: -1

保持简单。

是的,嵌入类型会很好,但由于当前(截至go1.19)无法嵌入通用类型,所以只需内联写出它:

body, _ = json.Marshal(
	struct {
		TestPayload
		Links       Linkset `json:"_links,omitempty"`
	}{
		TestPayload: hal.Payload,
		Links:       hal.Links,
	},
)

这里是一个示例。

{
    "name": "Graham",
    "answer": 42,
    "_links": {
        "self": {
            "href": "/"
        }
    }
}

是的,约束类型需要引用两次,但所有的自定义都是局部于代码的,所以不需要自定义编组器。

英文:

Keep it simple.

Yes it would be nice to embed the type - but since it's not currently possible (as of go1.19) to embed a generic type - just write it out inline:

body, _ = json.Marshal(
struct {
TestPayload
Links       Linkset `json:&quot;_links,omitempty&quot;`
}{
TestPayload: hal.Payload,
Links:       hal.Links,
},
)

https://go.dev/play/p/8yrB-MzUVK-

{
&quot;name&quot;: &quot;Graham&quot;,
&quot;answer&quot;: 42,
&quot;_links&quot;: {
&quot;self&quot;: {
&quot;href&quot;: &quot;/&quot;
}
}
}

Yes, the constraint type needs to be referenced twice - but all the customization is code-localized, so no need for a custom marshaler.

huangapple
  • 本文由 发表于 2022年10月15日 19:02:04
  • 转载请务必保留本文链接:https://go.coder-hub.com/74078845.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定