解析JSON结构时出现问题,导致REST请求失败。

huangapple go评论87阅读模式
英文:

Marshal Unmarshal JSON struct problems causing REST request to fail

问题

我正在更新问题描述,因为nipuna提供了解决方案。问题可以更好地描述为:“为什么在对JSON进行解组和组合后,会导致JSON的更改与REST API请求不兼容?”发现的原因是,文本字段有时需要包含在JSON数组中,有时需要省略,而组合数据结构不支持省略它们。该结构对这些字段使用了“omitempty”一词,因此可以认为该指令会省略它们,但这是不够的。我还需要将文本字段设置为指针。原始问题描述如下。最终的数据结构同时使用“omitempty”来处理文本字段,并将匿名结构设置为指针。(请参阅已接受的答案)

我有一个REST请求,在使用Go的Unmarshal / Marshal时失败,但是从文件中使用相同的JSON数据时却可以正常工作。我已经验证,如果我将读取文件得到的确切[]byte进行往返的UnMarshal / Marshal,得到的[]byte是不同的。结果,我的REST请求失败了。下面是一个示意图:文件JSON -> []byte = 字节集A。文件JSON -> []byte -> UnMarshal -> Marshal -> []byte = 字节集B。字节集A != 字节集B。我不关心这一点,除非我需要修改Go结构中的数据,然后进行Marshal,但是我不理解的字节更改导致请求失败。

这是测试代码:

type SlackRequestData struct {
	Blocks []struct {
		BlockID  string `json:"block_id"`
		Elements []struct {
			ActionID string `json:"action_id"`
			Style    string `json:"style"`
			Text     struct {
				Text string `json:"text"`
				Type string `json:"type"`
			} `json:"text"`
			Type  string `json:"type"`
			Value string `json:"value"`
		} `json:"elements"`
		Text struct {
			Text string `json:"text"`
			Type string `json:"type"`
		} `json:"text"`
		Type string `json:"type"`
	} `json:"blocks"`
}

func MarshalFailTest(){
	bytes_io, err := ioutil.ReadFile("../../slack/slack-message.json")
	if err != nil {
		fmt.Printf("Error: %+v\n", err)	// handle err
	}
	new_req := SlackRequestData{}
	err = json.Unmarshal(bytes_io, &new_req)
	if err != nil {
		fmt.Printf("Error: %+v\n", err)	// handle err
	}
	new_json, err := json.Marshal(new_req)
	if err != nil {
		fmt.Printf("Error: %+v\n", err)	// handle err
	}
	fmt.Printf("New req:\n%+v\n", new_req)
	fmt.Printf("New Json:\n%+v\n", new_json)
	fmt.Printf("Bytes:\n%+v\n", bytes_io)
}

这是JSON文件的数据:

{
  "blocks": [{
    "type": "section",
    "text": {
      "type": "mrkdwn",
      "text": "Service Is Ready. Now?"
    }
  },
	{
	  "type": "actions",
	  "block_id": "deploy_id",
	  "elements": [{
	    "type": "button",
      "action_id": "yes_button",
	    "text": {
		    "type": "plain_text",
		    "text": "Yes"
	      },
	    "style": "danger",
	    "value": "yes"
	  },
    {
      "type": "button",
      "action_id": "yes_toll_button",
      "text": {
        "type": "plain_text",
        "text": "Yes To All"
      },
      "style": "danger",
      "value": "yes"
    },
	  {
	    "type": "button",
      "action_id": "no_button",
	    "text": {
	 	    "type": "plain_text",
		    "text": "No"
	    },
	    "style": "primary",
      "value": "no"
	  },
    {
      "type": "button",
      "action_id": "no_toall_button",
      "text": {
        "type": "plain_text",
        "text": "No To All"
      },
      "style": "primary",
      "value": "no"
    }]
	}]
}

请注意,我使用了几个转换器将文件JSON转换为SlackRequestData结构,并且在调试器中,UnMarshal似乎完美地工作,因为结构中的所有数据都在正确的位置上。Marshal / UnMarshal也没有错误。那么为什么[]byte会不同呢?

数据集很大,所以我只会在这里分享每个数据集的第一行。
集合A:
[123 10 32 32 34 98 108 111 99 107 115 34 58 32 91 123 10 32 32 32 32 34 116 121 112 101 34 58 32 34 115 101 99 116 105 111 110 34 44 10 32 32 32 32 34
集合B:
[123 34 98 108 111 99 107 115 34 58 91 123 34 98 108 111 99 107 95 105 100 34 58 34 34 44 34 101 108 101 109 101 110 116 115 34 58 110 117 108 108 44

下面是将两个集合从[]byte转换为字符串的结果,这也很有趣。集合A保留了空格,而集合B没有空格,而且元素的顺序也发生了改变。我想知道结构是否不正确?然而,到目前为止,我一直在使用这种将转换器用于从JSON创建结构的技术,它们一直工作得很好。

集合A:

{
  "blocks": [{
    "type": "section",
    "text": {
      "type": "mrkdwn",
      "text": "Service Is Ready. Now?"
    }
  },
	{
	  "type": "actions",
	  "block_id": "deploy_id",
	  "elements": [{
	    "type": "button",
      "action_id": "yes_button",
	    "text": {
		    "type": "plain_text",
		    "text": "Yes"
	      },
	    "style": "danger",
	    "value": "yes"
	  },
    {
      "type": "button",
      "action_id": "yes_toll_button",
      "text": {
        "type": "plain_text",
        "text": "Yes To All"
      },
      "style": "danger",
      "value": "yes"
    },
	  {
	    "type": "button",
      "action_id": "no_button",
	    "text": {
	 	    "type": "plain_text",
		    "text": "No"
	    },
	    "style": "primary",
      "value": "no"
	  },
    {
      "type": "button",
      "action_id": "no_toall_button",
      "text": {
        "type": "plain_text",
        "text": "No To All"
      },
      "style": "primary",
      "value": "no"
    }]
	}]
}

集合B:

{"blocks":[{"block_id":"","elements":null,"text":{"text":"Service Is Ready. Now?","type":"mrkdwn"},"type":"section"},{"block_id":"deploy_id","elements":[{"action_id":"yes_button","style":"danger","text":{"text":"Yes","type":"plain_text"},"type":"button","value":"yes"},{"action_id":"yes_toll_button","style":"danger","text":{"text":"Yes To All","type":"plain_text"},"type":"button","value":"yes"},{"action_id":"no_button","style":"primary","text":{"text":"No","type":"plain_text"},"type":"button","value":"no"},{"action_id":"no_toall_button","style":"primary","text":{"text":"No To All","type":"plain_text"},"type":"button","value":"no"}],"text":{"text":"","type":""},"type":"actions"}]}

nipuna建议我尝试使用JSON的美化,所以我这样做了,下面是每个集合的字符串版本。
集合A:

{
  "blocks": [{
    "type": "section",
    "text": {
      "type": "mrkdwn",
      "text": "Service Is Ready. Now?"
    }
  },
	{
	  "type": "actions",
	  "block_id": "deploy_id",
	  "elements": [{
	    "type": "button",
      "action_id": "yes_button",
	    "text": {
		    "type": "plain_text",
		    "text": "Yes"
	      },
	    "style": "danger",
	    "value": "yes"
	  },
    {
      "type": "button",
      "action_id": "yes_toll_button",
      "text": {
        "type": "plain_text",
        "text": "Yes To All"
      },
      "style": "danger",
      "value": "yes"
    },
	  {
	    "type": "button",
      "action_id": "no_button",
	    "text": {
	 	    "type": "plain_text",
		    "text": "No"
	    },
	    "style": "primary",
      "value": "no"
	  },
    {
      "type": "button",
      "action_id": "no_toall_button",
      "text": {
        "type": "plain_text",
        "text": "No To All"
      },
      "style": "primary",
      "value": "no"
    }]
	}]
}

集合B:

{
  "blocks": [
    {
      "block_id": "",
      "elements": null,
      "text": {
        "text": "Service Is Ready. Now?",
        "type": "mrkdwn"
      },
      "type": "section"
    },
    {
      "block_id": "deploy_id",
      "elements": [
        {
          "action_id": "yes_button",
          "style": "danger",
          "text": {
            "text": "Yes",
            "type": "plain_text"
          },
          "type": "button",
          "value": "yes"
        },
        {
          "action_id": "yes_toll_button",
          "style": "danger",
          "text": {
            "text": "Yes To All",
            "type": "plain_text"
          },
          "type": "button",
          "value": "yes"
        },
        {
          "action_id": "no_button",
          "style": "primary",
          "text": {
            "text": "No",
            "type": "plain_text"
          },
          "type": "button",
          "value": "no"
        },
        {
          "action_id": "no_toall_button",
          "style": "primary",
          "text": {
            "text": "No To All",
            "type": "plain_text"
          },
          "type": "button",
          "value": "no"
        }
      ],
      "text": {
        "text": "",
        "type": ""
      },
      "type": "actions"
    }
  ]
}
英文:

I am updating the problem statement since input from nipuna, who solved the problem. The question is better described as, "why does unmarshal, then marshal of JSON result in changes to the JSON that are not compatible with a REST API request?" The reason discovered, and that I could not have discovered alone, were that the text fields sometimes needed to be included and sometimes omitted in a JSON array, and the marshal data structure did not support omitting them. The structure did use the term "omitempty" for these fields, so it could be believed this directive would omit them, however that is not sufficient. I also needed for the text fields to be pointers. The original problem description follows below. The final data structure uses both "omitempty" for the text fields and also makes the anonymous struct a pointer. (Please see the accepted answer)

I have a REST request that fails when I use Go Unmarshal / Marshal but it works on the same JSON data from a file. I have verified that the resulting []byte from reading the file is different if I take that exact []byte and roundtrip UnMarshal / Marshal it. As a result, my REST request fails. Here it is diagramatically: file json -> []byte = set A of bytes. file json -> []byte -> UnMarshal -> Marshal -> []byte = set B of bytes. Set A of bytes != set B of bytes. I would not care except that I need to modify the data in the structure of Go, then Marshal it, however the byte changes that I do not understand causes the request to fail.

Here is test code:

type SlackRequestData struct {
	Blocks []struct {
		BlockID  string `json:"block_id"`
		Elements []struct {
			ActionID string `json:"action_id"`
			Style    string `json:"style"`
			Text     struct {
				Text string `json:"text"`
				Type string `json:"type"`
			} `json:"text"`
			Type  string `json:"type"`
			Value string `json:"value"`
		} `json:"elements"`
		Text struct {
			Text string `json:"text"`
			Type string `json:"type"`
		} `json:"text"`
		Type string `json:"type"`
	} `json:"blocks"`
}

func MarshalFailTest(){
	bytes_io, err := ioutil.ReadFile("../../slack/slack-message.json")
	if err != nil {
		fmt.Printf("Error: %+v\n", err)	// handle err
	}
	new_req := SlackRequestData{}
	err = json.Unmarshal(bytes_io, &new_req)
	if err != nil {
		fmt.Printf("Error: %+v\n", err)	// handle err
	}
	new_json, err := json.Marshal(new_req)
	if err != nil {
		fmt.Printf("Error: %+v\n", err)	// handle err
	}
	fmt.Printf("New req:\n%+v\n", new_req)
	fmt.Printf("New Json:\n%+v\n", new_json)
	fmt.Printf("Bytes:\n%+v\n", bytes_io)
}

Here is the json file data:

{
  "blocks": [{
    "type": "section",
    "text": {
      "type": "mrkdwn",
      "text": "Service Is Ready. Now?"
    }
  },
	{
	  "type": "actions",
	  "block_id": "deploy_id",
	  "elements": [{
	    "type": "button",
      "action_id": "yes_button",
	    "text": {
		    "type": "plain_text",
		    "text": "Yes"
	      },
	    "style": "danger",
	    "value": "yes"
	  },
    {
      "type": "button",
      "action_id": "yes_toll_button",
      "text": {
        "type": "plain_text",
        "text": "Yes To All"
      },
      "style": "danger",
      "value": "yes"
    },
	  {
	    "type": "button",
      "action_id": "no_button",
	    "text": {
	 	    "type": "plain_text",
		    "text": "No"
	    },
	    "style": "primary",
      "value": "no"
	  },
    {
      "type": "button",
      "action_id": "no_toall_button",
      "text": {
        "type": "plain_text",
        "text": "No To All"
      },
      "style": "primary",
      "value": "no"
    }]
	}]
}

Note that I used several converters to take the file json and create the SlackRequestData struct, and the UnMarshal appears to work perfectly as the struct in the debugger has all the data in the right place. There are no errors with Marshal / UnMarshal either. So why does the []byte differ?

The data sets are large, so I'll just share the first line of each set here.
Set A:
[123 10 32 32 34 98 108 111 99 107 115 34 58 32 91 123 10 32 32 32 32 34 116 121 112 101 34 58 32 34 115 101 99 116 105 111 110 34 44 10 32 32 32 32 34
Set B:
[123 34 98 108 111 99 107 115 34 58 91 123 34 98 108 111 99 107 95 105 100 34 58 34 34 44 34 101 108 101 109 101 110 116 115 34 58 110 117 108 108 44

Here are the two sets converted from []byte to strings, as this is quite interesting too. The set A keeps whitespace, the set B has no whitespace but also the elements have been rearranged. I wonder if the struct is not correct? However I've been using this technique of using converters to create structs out of json and they have worked well to this point.

Set A:

{
  "blocks": [{
    "type": "section",
    "text": {
      "type": "mrkdwn",
      "text": "Service Is Ready. Now?"
    }
  },
	{
	  "type": "actions",
	  "block_id": "deploy_id",
	  "elements": [{
	    "type": "button",
      "action_id": "yes_button",
	    "text": {
		    "type": "plain_text",
		    "text": "Yes"
	      },
	    "style": "danger",
	    "value": "yes"
	  },
    {
      "type": "button",
      "action_id": "yes_toll_button",
      "text": {
        "type": "plain_text",
        "text": "Yes To All"
      },
      "style": "danger",
      "value": "yes"
    },
	  {
	    "type": "button",
      "action_id": "no_button",
	    "text": {
	 	    "type": "plain_text",
		    "text": "No"
	    },
	    "style": "primary",
      "value": "no"
	  },
    {
      "type": "button",
      "action_id": "no_toall_button",
      "text": {
        "type": "plain_text",
        "text": "No To All"
      },
      "style": "primary",
      "value": "no"
    }]
	}]
}

Set B:

{"blocks":[{"block_id":"","elements":null,"text":{"text":"Service Is Ready. Now?","type":"mrkdwn"},"type":"section"},{"block_id":"deploy_id","elements":[{"action_id":"yes_button","style":"danger","text":{"text":"Yes","type":"plain_text"},"type":"button","value":"yes"},{"action_id":"yes_toll_button","style":"danger","text":{"text":"Yes To All","type":"plain_text"},"type":"button","value":"yes"},{"action_id":"no_button","style":"primary","text":{"text":"No","type":"plain_text"},"type":"button","value":"no"},{"action_id":"no_toall_button","style":"primary","text":{"text":"No To All","type":"plain_text"},"type":"button","value":"no"}],"text":{"text":"","type":""},"type":"actions"}]}

nipuna suggested I try using beautify of the JSON so I did this and there is the string version of the []byte for each set.
Set A:

{
  "blocks": [{
    "type": "section",
    "text": {
      "type": "mrkdwn",
      "text": "Service Is Ready. Now?"
    }
  },
	{
	  "type": "actions",
	  "block_id": "deploy_id",
	  "elements": [{
	    "type": "button",
      "action_id": "yes_button",
	    "text": {
		    "type": "plain_text",
		    "text": "Yes"
	      },
	    "style": "danger",
	    "value": "yes"
	  },
    {
      "type": "button",
      "action_id": "yes_toll_button",
      "text": {
        "type": "plain_text",
        "text": "Yes To All"
      },
      "style": "danger",
      "value": "yes"
    },
	  {
	    "type": "button",
      "action_id": "no_button",
	    "text": {
	 	    "type": "plain_text",
		    "text": "No"
	    },
	    "style": "primary",
      "value": "no"
	  },
    {
      "type": "button",
      "action_id": "no_toall_button",
      "text": {
        "type": "plain_text",
        "text": "No To All"
      },
      "style": "primary",
      "value": "no"
    }]
	}]
}

Set B:

{
  "blocks": [
    {
      "block_id": "",
      "elements": null,
      "text": {
        "text": "Service Is Ready. Now?",
        "type": "mrkdwn"
      },
      "type": "section"
    },
    {
      "block_id": "deploy_id",
      "elements": [
        {
          "action_id": "yes_button",
          "style": "danger",
          "text": {
            "text": "Yes",
            "type": "plain_text"
          },
          "type": "button",
          "value": "yes"
        },
        {
          "action_id": "yes_toll_button",
          "style": "danger",
          "text": {
            "text": "Yes To All",
            "type": "plain_text"
          },
          "type": "button",
          "value": "yes"
        },
        {
          "action_id": "no_button",
          "style": "primary",
          "text": {
            "text": "No",
            "type": "plain_text"
          },
          "type": "button",
          "value": "no"
        },
        {
          "action_id": "no_toall_button",
          "style": "primary",
          "text": {
            "text": "No To All",
            "type": "plain_text"
          },
          "type": "button",
          "value": "no"
        }
      ],
      "text": {
        "text": "",
        "type": ""
      },
      "type": "actions"
    }
  ]
}

答案1

得分: 1

我可以看到以下的差异,并提出一些建议。

可选字段

你的数据包含一个blocks数组,在第一个块中,只有两个字段texttype,而在第二个块中,有所有其他字段但没有text。因此,text是可选的,当它是一个空结构时需要省略它。所以你需要用omitempty和指针类型来定义你的SlackRequestData中的Text字段。

建议的SlackRequestData结构如下所示。

type Text struct {
	Type string `json:"type,omitempty"`
	Text string `json:"text,omitempty"`
}

type SlackRequestData struct {
	Blocks []struct {
		BlockID  string `json:"block_id,omitempty"`
		Elements []struct {
			ActionID string `json:"action_id,omitempty"`
			Style    string `json:"style,omitempty"`
			Text     *Text `json:"text,omitempty"`
			Type  string `json:"type,omitempty"`
			Value string `json:"value,omitempty"`
		} `json:"elements,omitempty"`
		Text *Text `json:"text,omitempty"`
		Type string `json:"type,omitempty"`
	} `json:"blocks,omitempty"`
}

编组数据顺序

你的文件JSON数据的字段顺序和SlackRequestData的字段顺序是不同的。因此,显然编组后的字符串和文件字符串是不同的。JSON编组在编组时不保证顺序,如这里所述。所以如果你需要比较这两个,可以对这些字节进行排序,然后进行比较。

JSON缩进

在你的文件数据中,数据有一定的JSON美化格式,而JSON编组返回的是紧凑的输出。如果你需要获得相同的编组输出,你需要使用MarshalIndent并使用文件中使用的缩进。

因此,如果你需要比较这两个内容,请使用omitempty进行编组,对两个结果(文件字节数组和JSON编组字节数组)进行排序。然后移除空格,进行比较。这样你可以得到正确的相同结果。

英文:

I can see following differences and I have mention some suggestions.

Optional fields

Your data consist of array of blocks, in first block, there is only two fields text and type and in second block there is all other fields but no text. So that text is optional and it need to be omit when it is an empty struct. So you need to define your SlackRequestData with omitempty and pointer type to Text fields.

Suggested SlackRequestData struct is something like below.

type Text struct {
	Type string `json:"type,omitempty"`
	Text string `json:"text,omitempty"`
}

type SlackRequestData struct {
	Blocks []struct {
		BlockID  string `json:"block_id,omitempty"`
		Elements []struct {
			ActionID string `json:"action_id,omitempty"`
			Style    string `json:"style,omitempty"`
			Text     *Text `json:"text,omitempty"`
			Type  string `json:"type,omitempty"`
			Value string `json:"value,omitempty"`
		} `json:"elements,omitempty"`
		Text *Text `json:"text,omitempty"`
		Type string `json:"type,omitempty"`
	} `json:"blocks,omitempty"`
}

Marshalled Data Order

your file json data's field order and SlackRequestData's field order is different. So obviously that marshalled string and file string is different. json marshalling not guarantees that order when marshalling as mentiond in here. So if you need to compare those two, please sort those bytes in someway and compare.

Json Indent

In your file data is somewhat json beautified data and json marshal return compacted output. If you need to get same marshal output, you need to use MarshalIndent and do indent you used in file.

So if you need to compare both things, marshal with omitempty, sort both results (file byte array and json marshalled byte array). and remove whitespaces and then compare. You can get correct same result.

huangapple
  • 本文由 发表于 2021年8月21日 11:45:45
  • 转载请务必保留本文链接:https://go.coder-hub.com/68869965.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定