将 protobuf 对象哈希为字符串,作为 Redis 数据库的键。

huangapple go评论70阅读模式
英文:

Hash protobuf object to a string as the key for redis database

问题

我有一个复杂的protobuf对象。它是发送到我的GRPC端点的请求。如果之前没有处理过,我想要对它进行处理。所以我想要将该对象哈希为字符串,并将其存储在我的Redis数据库中。我使用了ObjectHash-Proto,但是在使用新版本的protobuf编译器时,我遇到了以下错误:

got an unexpected struct of type 'impl.MessageState' for field {Name:state PkgPath:... Type:impl.MessageState Tag: Offset:0 Index:[0] Anonymous:false}

看起来它不支持结构体,而新版本的protobuf编译器生成的代码包含结构体。

我无法为每个请求生成某种ID。实际上,该ID就是整个对象的哈希值。

英文:

I have some sort of complex protobuf object. It's a request sent to my GRPC endpoint. I want to just process it if I didn't before. So I want to hash the object to some string and store it in my Redis database. I used ObjectHash-Proto but with new versions of protobuf-compiler I got this error:

got an unexpected struct of type 'impl.MessageState' for field {Name:state PkgPath:... Type:impl.MessageState Tag: Offset:0 Index:[0] Anonymous:false}

Seems it does not support structs and new version of protobuf-compiler generates a code containing struct.

I can't generate some sort of ID for each request. The ID is actually the hash of the whole of the object.

答案1

得分: 2

如果你有一个proto.Message 1,那么你可以免费获得一个Marshal函数[2]。
因此,在你对消息进行Marshal之后,你可以将字节直接传递给base64或者md5,或者任何你想要的方式:

package main

import (
   "encoding/base64"
   "google.golang.org/protobuf/proto"
   "google.golang.org/protobuf/types/known/structpb"
)

func hash(m proto.Message) (string, error) {
   b, err := proto.Marshal(m)
   if err != nil {
      return "", err
   }
   return base64.StdEncoding.EncodeToString(b), nil
}

func main() {
   m, err := structpb.NewStruct(map[string]interface{}{
      "month": 12, "day": 31,
   })
   if err != nil {
      panic(err)
   }
   s, err := hash(m)
   if err != nil {
      panic(err)
   }
   println(s) // ChIKBW1vbnRoEgkRAAAAAAAAKEAKEAoDZGF5EgkRAAAAAAAAP0A=
}
  1. https://godocs.io/google.golang.org/protobuf/proto#Message
  2. https://godocs.io/google.golang.org/protobuf/proto#Marshal
英文:

If you have a proto.Message 1, then you get a Marshal function [2] for free.
So after you Marshal the message, you can just pass the bytes to base64, or
md5 or whatever you want:

package main

import (
   "encoding/base64"
   "google.golang.org/protobuf/proto"
   "google.golang.org/protobuf/types/known/structpb"
)

func hash(m proto.Message) (string, error) {
   b, err := proto.Marshal(m)
   if err != nil {
      return "", err
   }
   return base64.StdEncoding.EncodeToString(b), nil
}

func main() {
   m, err := structpb.NewStruct(map[string]interface{}{
      "month": 12, "day": 31,
   })
   if err != nil {
      panic(err)
   }
   s, err := hash(m)
   if err != nil {
      panic(err)
   }
   println(s) // ChIKBW1vbnRoEgkRAAAAAAAAKEAKEAoDZGF5EgkRAAAAAAAAP0A=
}
  1. https://godocs.io/google.golang.org/protobuf/proto#Message
  2. https://godocs.io/google.golang.org/protobuf/proto#Marshal

答案2

得分: 2

Proto序列化不稳定,因此您不能依赖于编组和哈希输出以产生相同消息的相同哈希。

来自https://developers.google.com/protocol-buffers/docs/reference/go/faq#hash

如何将协议缓冲区消息用作哈希键?

您需要规范化序列化,其中协议缓冲区消息的编组输出保证随时间稳定。不幸的是,目前没有规范化序列化的规范。您需要编写自己的规范化序列化方法,或者找到一种避免需要规范化序列化的方法。

我找到的最接近的解决方案是deepmind objecthash-proto,但在过去的4年中没有任何贡献,所以我认为它可能已经过时了。

英文:

Proto serialization is not stable, thus you can't rely on marshalling and hashing the output to produce the same hash for the same message.

From https://developers.google.com/protocol-buffers/docs/reference/go/faq#hash

> ## How do I use a protocol buffer message as a hash key? ##
> You need canonical serialization, where the marshaled output of a protocol buffer message is guaranteed to be stable over time. Unfortunately, no specification for canonical serialization exists at this time. You'll need to write your own or find a way to avoid needing one.

The closest solution I could find is deepmind objecthash-proto but there have been no contributions in last 4 years so I assume it might be outdated

huangapple
  • 本文由 发表于 2021年8月28日 22:11:06
  • 转载请务必保留本文链接:https://go.coder-hub.com/68965130.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定