英文:
Deserializing external JSON payload to protobuf Any
问题
我有一个处理API分页结果的protobuf定义:
message ArrayRespone {
int32 count = 1;
string next_url = 2;
string request_id = 3;
repeated google.protobuf.Any results = 4;
string status = 5;
}
目标是将API的分页响应反序列化,并将每个页面的结果提取到适当类型的切片中。我用Go编写了以下代码来实现:
func getData[T ~proto.Message](data []byte) ([]T, error) {
var resp *ArrayRespone
if err := json.Unmarshal(data, &resp); err != nil {
return nil, err
}
var items []T
for _, result := range resp.Results {
var item T
if err := result.UnmarshalTo(&item); err != nil {
return nil, err
}
items = append(items, item)
}
return items, nil
}
我遇到的问题是,在测试这段代码时,遇到了以下错误:
proto: mismatched message type: got "X", want ""
从这个错误信息可以看出,Protobuf没有足够的信息来确定它正在处理的是哪种类型。查看Any
的定义,我发现它有一个TypeUrl
字段和一个Value
字段。看起来类型URL是空的,但实际上不应该是空的。所以,我的想法是,如果我将其设置为"X",错误就会消失,但这也不起作用,因为Value
字段仍然是空的,我的JSON数据被忽略了。
你如何让这段代码工作起来?
英文:
I have a protobuf definition to handle paged results from an API:
message ArrayRespone {
int32 count = 1;
string next_url = 2;
string request_id = 3;
repeated google.protobuf.Any results = 4;
string status = 5;
}
The goal here is to deserialize the paged responses from this API and then extract the results from each page into slices of the appropriate type. I wrote code in Go that does this:
func getData[T ~proto.Message](data []byte) ([]T, error) {
var resp *ArrayRespone
if err := json.Unmarshal(data, &resp); err != nil {
return nil, err
}
var items []T
for _, result := range resp.Results {
var item T
if err := result.UnmarshalTo(item); err != nil {
return nil, err
}
items = append(items, item)
}
return items, nil
}
The problem I'm running into is that, when testing this code, I run into the following error:
> proto: mismatched message type: got "X", want ""
From this, I can understand that Protobuf doesn't have the information necessary to determine which type it's working with. Looking at the definition for Any
, I can see that it has a TypeUrl
field and a Value
field. It appears that the type URL is empty but shouldn't be. So, my thought was that if I were to set it to X
, the error would go away, but that wouldn't work either because the Value
field was still empty; my JSON data had been ignored.
How can I get this code working?
答案1
得分: 1
我找到了两种解决这个问题的潜在方法,但它们都涉及到自定义实现UnmarshalJSON
。首先,我尝试修改我的proto定义,使results
的类型为bytes
,但是JSON反序列化失败,因为源数据既不是字符串,也不是可以直接反序列化为[]byte
的类型。所以,我不得不自己动手:
使用Struct
使用google.protobuf.Struct
类型,我修改了我的ArrayResponse
如下:
message ArrayRespone {
int32 count = 1;
string next_url = 2;
string request_id = 3;
repeated google.protobuf.Struct results = 4;
string status = 5;
}
然后,我编写了一个自定义的UnmarshalJSON
实现,如下所示:
// UnmarshalJSON将JSON数据转换为Providers.Polygon.ArrayResponse
func (resp *ArrayRespone) UnmarshalJSON(data []byte) error {
// 首先,将JSON反序列化为键字段和值之间的映射
// 如果失败,则返回错误
var mapped map[string]interface{}
if err := json.Unmarshal(data, &mapped); err != nil {
return fmt.Errorf("无法执行第一次解组,错误:%v", err)
}
// 接下来,从映射中提取计数;如果失败,则返回错误
if err := extractValue(mapped, "count", &resp.Count); err != nil {
return err
}
// 从映射中提取下一个URL;如果失败,则返回错误
if err := extractValue(mapped, "next_url", &resp.NextUrl); err != nil {
return err
}
// 从映射中提取请求ID;如果失败,则返回错误
if err := extractValue(mapped, "request_id", &resp.RequestId); err != nil {
return err
}
// 从映射中提取状态;如果失败,则返回错误
if err := extractValue(mapped, "status", &resp.Status); err != nil {
return err
}
// 现在,将结果数组提取到临时变量中;如果失败,则返回错误
var results []interface{}
if err := extractValue(mapped, "results", &results); err != nil {
return err
}
// 最后,遍历每个结果并将其添加到结果切片中,尝试将其转换为Struct;如果有任何转换失败,则返回错误
resp.Results = make([]*structpb.Struct, len(results))
for i, result := range results {
if value, err := structpb.NewStruct(result.(map[string]interface{})); err == nil {
resp.Results[i] = value
} else {
return fmt.Errorf("无法从结果%d创建结构体,错误:%v", i, err)
}
}
return nil
}
// 辅助函数,尝试从标准接口映射中提取值,并将其设置为字段,如果类型兼容
func extractValue[T any](mapping map[string]interface{}, field string, value *T) error {
if raw, ok := mapping[field]; ok {
if inner, ok := raw.(T); ok {
*value = inner
} else {
return fmt.Errorf("无法将值%v设置为字段%s(%T)", raw, field, *value)
}
}
return nil
}
然后,在我的服务代码中,我修改了代码的反序列化部分,以使用Struct
对象。这段代码依赖于mapstructure包:
func getData[T ~proto.Message](data []byte) ([]T, error) {
var resp *ArrayRespone
if err := json.Unmarshal(data, &resp); err != nil {
return nil, err
}
items := make([]T, len(resp.Results))
for i, result := range resp.Results {
var item T
if err := mapstructure.Decode(result.AsMap(), &item); err != nil {
return nil, err
}
items[i] = item
}
return items, nil
}
只要所有字段都可以轻松反序列化为google.protobuf.Value
类型的字段,这个方法就可以工作。然而,对于我来说,并非如此,因为我使用getData
调用的类型中的几个字段都有自定义的UnmarshalJSON
实现。因此,我实际选择的解决方案是使用bytes
:
使用Bytes
对于这个实现,我不需要依赖任何导入的类型,因此消息本身更容易处理:
message ArrayRespone {
int32 count = 1;
string next_url = 2;
string request_id = 3;
bytes results = 4;
string status = 5;
}
这仍然需要开发一个自定义的UnmarshalJSON
实现,但这个实现也更简单:
func (resp *ArrayRespone) UnmarshalJSON(data []byte) error {
// 首先,将JSON反序列化为键字段和值之间的映射
// 如果失败,则返回错误
var mapped map[string]*json.RawMessage
if err := json.Unmarshal(data, &mapped); err != nil {
return fmt.Errorf("无法执行第一次解组,错误:%v", err)
}
// 接下来,从映射中提取计数;如果失败,则返回错误
if err := extractValue(mapped, "count", &resp.Count); err != nil {
return err
}
// 从映射中提取下一个URL;如果失败,则返回错误
if err := extractValue(mapped, "next_url", &resp.NextUrl); err != nil {
return err
}
// 从映射中提取请求ID;如果失败,则返回错误
if err := extractValue(mapped, "request_id", &resp.RequestId); err != nil {
return err
}
// 从映射中提取状态;如果失败,则返回错误
if err := extractValue(mapped, "status", &resp.Status); err != nil {
return err
}
// 最后,将每个结果迭代并将其添加到结果切片中,尝试将其转换为Struct;如果有任何转换失败,则返回错误
if raw, ok := mapped["results"]; ok {
resp.Results = *raw
}
return nil
}
// 辅助函数,尝试从标准接口映射中提取值,并将其设置为字段,如果类型兼容
func extractValue[T any](mapping map[string]*json.RawMessage, field string, value *T) error {
if raw, ok := mapping[field]; ok {
if err := json.Unmarshal(*raw, &value); err != nil {
return fmt.Errorf("无法将值%s设置为字段%s(%T)", *raw, field, *value)
}
}
return nil
}
然后,我修改了我的getData
函数如下:
func getData[T ~proto.Message](data []byte) ([]T, error) {
var resp *ArrayRespone
if err := json.Unmarshal(data, &resp); err != nil {
return nil, err
}
var items []T
if err := json.Unmarshal(resp.Results, &items); err != nil {
return nil, err
}
return items, nil
}
显然,这个实现更简单,需要一个较少的反序列化步骤,这意味着比Struct
实现更少的反射。
英文:
I found two potential solutions to this problem but they both involve a custom implementation of UnmarshalJSON
. First, I tried modifying my proto definition so that results
was of type bytes
, but the JSON deserialization failed because the source data wasn't a string or anything that could be deserialized to []byte
directly. So, I had to roll my own:
Using Struct
Using the google.protobuf.Struct
type, I modified my ArrayResponse
to look like this:
message ArrayRespone {
int32 count = 1;
string next_url = 2;
string request_id = 3;
repeated google.protobuf.Struct results = 4;
string status = 5;
}
and then wrote a custom implementation of UnmarshalJSON
that worked like this:
// UnmarshalJSON converts JSON data into a Providers.Polygon.ArrayResponse
func (resp *ArrayRespone) UnmarshalJSON(data []byte) error {
// First, deserialize the JSON into a mapping between key fields and values
// If this fails then return an error
var mapped map[string]interface{}
if err := json.Unmarshal(data, &mapped); err != nil {
return fmt.Errorf("failed to perform first-pass unmarshal, error: %v", err)
}
// Next, extract the count from the mapping; if this fails return an error
if err := extractValue(mapped, "count", &resp.Count); err != nil {
return err
}
// Extract the next URL from the mapping; if this fails return an error
if err := extractValue(mapped, "next_url", &resp.NextUrl); err != nil {
return err
}
// Extract the request ID from the mapping; if this fails return an error
if err := extractValue(mapped, "request_id", &resp.RequestId); err != nil {
return err
}
// Extract the status from the mapping; if this fails return an error
if err := extractValue(mapped, "status", &resp.Status); err != nil {
return err
}
// Now, extract the results array into a temporary variable; if this fails return an error
var results []interface{}
if err := extractValue(mapped, "results", &results); err != nil {
return err
}
// Finally, iterate over each result and add it to the slice of results by attempting
// to convert it to a Struct; if any of these fail to convert then return an error
resp.Results = make([]*structpb.Struct, len(results))
for i, result := range results {
if value, err := structpb.NewStruct(result.(map[string]interface{})); err == nil {
resp.Results[i] = value
} else {
return fmt.Errorf("failed to create struct from result %d, error: %v", i, err)
}
}
return nil
}
// Helper function that attempts to extract a value from a standard mapping of interfaces
// and set a field with it if the types are compatible
func extractValue[T any](mapping map[string]interface{}, field string, value *T) error {
if raw, ok := mapping[field]; ok {
if inner, ok := raw.(T); ok {
*value = inner
} else {
return fmt.Errorf("failed to set value %v to field %s (%T)", raw, field, *value)
}
}
return nil
}
Then, in my service code, I modified the unmarshalling portion of my code to consume the Struct
objects. This code relies on the mapstructure package:
func getData[T ~proto.Message](data []byte) ([]T, error) {
var resp *ArrayRespone
if err := json.Unmarshal(data, &resp); err != nil {
return nil, err
}
items := make([]T, len(resp.Results))
for i, result := range resp.Results {
var item T
if err := mapstructure.Decode(result.AsMap(), &item); err != nil {
return nil, err
}
items[i] = item
}
return items, nil
}
This works so long as all your fields can be easily deserialized to a field on the google.protobuf.Value
type. However, this wasn't the case for me as several of the fields in types that I would call getData
with have custom implementations of UnmarshalJSON
. So, the solution I actually chose was to use bytes
instead:
Using Bytes
For this implementation, I didn't need to rely on any imported types so the message itself was much easier to work with:
message ArrayRespone {
int32 count = 1;
string next_url = 2;
string request_id = 3;
bytes results = 4;
string status = 5;
}
This still necessitated the development of a custom implementation for UnmarshalJSON
, but that implementation was also simpler:
func (resp *ArrayRespone) UnmarshalJSON(data []byte) error {
// First, deserialize the JSON into a mapping between key fields and values
// If this fails then return an error
var mapped map[string]*json.RawMessage
if err := json.Unmarshal(data, &mapped); err != nil {
return fmt.Errorf("failed to perform first-pass unmarshal, error: %v", err)
}
// Next, extract the count from the mapping; if this fails return an error
if err := extractValue(mapped, "count", &resp.Count); err != nil {
return err
}
// Extract the next URL from the mapping; if this fails return an error
if err := extractValue(mapped, "next_url", &resp.NextUrl); err != nil {
return err
}
// Extract the request ID from the mapping; if this fails return an error
if err := extractValue(mapped, "request_id", &resp.RequestId); err != nil {
return err
}
// Extract the status from the mapping; if this fails return an error
if err := extractValue(mapped, "status", &resp.Status); err != nil {
return err
}
// Finally, iterate over each result and add it to the slice of results by attempting
// to convert it to a Struct; if any of these fail to convert then return an error
if raw, ok := mapped["results"]; ok {
resp.Results = *raw
}
return nil
}
// Helper function that attempts to extract a value from a standard mapping of interfaces
// and set a field with it if the types are compatible
func extractValue[T any](mapping map[string]*json.RawMessage, field string, value *T) error {
if raw, ok := mapping[field]; ok {
if err := json.Unmarshal(*raw, &value); err != nil {
return fmt.Errorf("failed to set value %s to field %s (%T)", *raw, field, *value)
}
}
return nil
}
Then, I modified my getData
function to be:
func getData[T ~proto.Message](data []byte) ([]T, error) {
var resp *ArrayRespone
if err := json.Unmarshal(data, &resp); err != nil {
return nil, err
}
var items []T
if err := json.Unmarshal(resp.Results, &items); err != nil {
return nil, err
}
return items, nil
}
Clearly, this implementation is simpler and requires one less deserialization step, which means less reflection than the Struct
implementation.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论