英文:
Hash protobuf object to a string as the key for redis database
问题
我有一个复杂的protobuf对象。它是发送到我的GRPC端点的请求。如果之前没有处理过,我想要对它进行处理。所以我想要将该对象哈希为字符串,并将其存储在我的Redis数据库中。我使用了ObjectHash-Proto,但是在使用新版本的protobuf编译器时,我遇到了以下错误:
got an unexpected struct of type 'impl.MessageState' for field {Name:state PkgPath:... Type:impl.MessageState Tag: Offset:0 Index:[0] Anonymous:false}
看起来它不支持结构体,而新版本的protobuf编译器生成的代码包含结构体。
我无法为每个请求生成某种ID。实际上,该ID就是整个对象的哈希值。
英文:
I have some sort of complex protobuf object. It's a request sent to my GRPC endpoint. I want to just process it if I didn't before. So I want to hash the object to some string and store it in my Redis database. I used ObjectHash-Proto but with new versions of protobuf-compiler I got this error:
got an unexpected struct of type 'impl.MessageState' for field {Name:state PkgPath:... Type:impl.MessageState Tag: Offset:0 Index:[0] Anonymous:false}
Seems it does not support structs and new version of protobuf-compiler generates a code containing struct.
I can't generate some sort of ID for each request. The ID is actually the hash of the whole of the object.
答案1
得分: 2
如果你有一个proto.Message
1,那么你可以免费获得一个Marshal函数[2]。
因此,在你对消息进行Marshal之后,你可以将字节直接传递给base64
或者md5
,或者任何你想要的方式:
package main
import (
"encoding/base64"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/known/structpb"
)
func hash(m proto.Message) (string, error) {
b, err := proto.Marshal(m)
if err != nil {
return "", err
}
return base64.StdEncoding.EncodeToString(b), nil
}
func main() {
m, err := structpb.NewStruct(map[string]interface{}{
"month": 12, "day": 31,
})
if err != nil {
panic(err)
}
s, err := hash(m)
if err != nil {
panic(err)
}
println(s) // ChIKBW1vbnRoEgkRAAAAAAAAKEAKEAoDZGF5EgkRAAAAAAAAP0A=
}
- https://godocs.io/google.golang.org/protobuf/proto#Message
- https://godocs.io/google.golang.org/protobuf/proto#Marshal
英文:
If you have a proto.Message
1, then you get a Marshal function [2] for free.
So after you Marshal the message, you can just pass the bytes to base64
, or
md5
or whatever you want:
package main
import (
"encoding/base64"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/known/structpb"
)
func hash(m proto.Message) (string, error) {
b, err := proto.Marshal(m)
if err != nil {
return "", err
}
return base64.StdEncoding.EncodeToString(b), nil
}
func main() {
m, err := structpb.NewStruct(map[string]interface{}{
"month": 12, "day": 31,
})
if err != nil {
panic(err)
}
s, err := hash(m)
if err != nil {
panic(err)
}
println(s) // ChIKBW1vbnRoEgkRAAAAAAAAKEAKEAoDZGF5EgkRAAAAAAAAP0A=
}
答案2
得分: 2
Proto序列化不稳定,因此您不能依赖于编组和哈希输出以产生相同消息的相同哈希。
来自https://developers.google.com/protocol-buffers/docs/reference/go/faq#hash
如何将协议缓冲区消息用作哈希键?
您需要规范化序列化,其中协议缓冲区消息的编组输出保证随时间稳定。不幸的是,目前没有规范化序列化的规范。您需要编写自己的规范化序列化方法,或者找到一种避免需要规范化序列化的方法。
我找到的最接近的解决方案是deepmind objecthash-proto,但在过去的4年中没有任何贡献,所以我认为它可能已经过时了。
英文:
Proto serialization is not stable, thus you can't rely on marshalling and hashing the output to produce the same hash for the same message.
From https://developers.google.com/protocol-buffers/docs/reference/go/faq#hash
> ## How do I use a protocol buffer message as a hash key? ##
> You need canonical serialization, where the marshaled output of a protocol buffer message is guaranteed to be stable over time. Unfortunately, no specification for canonical serialization exists at this time. You'll need to write your own or find a way to avoid needing one.
The closest solution I could find is deepmind objecthash-proto but there have been no contributions in last 4 years so I assume it might be outdated
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论