将protoc生成的结构体转换为bson结构体的最佳方法是什么?

huangapple go评论82阅读模式
英文:

What would be the best approach to converting protoc generated structs from bson structs?

问题

我正在使用Golang编写一个RESTful API,该API还具有gRPC API。该API连接到一个MongoDB数据库,并使用结构体来映射实体。我还有一个.proto定义,它与我用于MongoDB的结构体完全匹配。

我想知道是否有一种方法可以共享或重用用于MongoDB调用的.proto定义的代码。我注意到protoc生成的结构体每个字段都有json标签,但显然没有bson标签等。

我有类似这样的代码...

// 菜单 -
type Menu struct {
    ID          bson.ObjectId      `json:"id" bson:"_id"`
    Name        string             `json:"name" bson:"name"`
    Description string             `json:"description" bson:"description"`
    Mixers      []mixers.Mixer     `json:"mixers" bson:"mixers"`
    Sections    []sections.Section `json:"sections" bson:"sections"`
}

但是我还有protoc生成的代码...

type Menu struct {
    Id          string     `protobuf:"bytes,1,opt,name=id" json:"id,omitempty"`
    Name        string     `protobuf:"bytes,2,opt,name=name" json:"name,omitempty"`
    Description string     `protobuf:"bytes,3,opt,name=description" json:"description,omitempty"`
    Mixers      []*Mixer   `protobuf:"bytes,4,rep,name=mixers" json:"mixers,omitempty"`
    Sections    []*Section `protobuf:"bytes,5,rep,name=sections" json:"sections,omitempty"`
}

目前,我不得不根据我的操作在这两个结构体之间进行转换。这很繁琐,而且可能会带来相当大的性能损失。所以是否有更好的方法在这两者之间进行转换,或者重用其中一个来完成两个任务?

英文:

I'm writing a RESTful API in Golang, which also has a gRPC api. The API connects to a MongoDB database, and uses structs to map out entities. I also have a .proto definition which matches like for like the struct I'm using for MongoDB.

I just wondered if there was a way to share, or re-use the .proto defined code for the MongoDB calls also. I've noticed the strucs protoc generates has json tags for each field, but obviously there aren't bson tags etc.

I have something like...

// Menu -
type Menu struct {
	ID          bson.ObjectId      `json:"id" bson"_id"`
	Name        string             `json:"name" bson:"name"`
	Description string             `json:"description" bson:"description"`
	Mixers      []mixers.Mixer     `json:"mixers" bson:"mixers"`
	Sections    []sections.Section `json:"sections" bson:"sections"`
}

But then I also have protoc generated code...

type Menu struct {
	Id          string     `protobuf:"bytes,1,opt,name=id" json:"id,omitempty"`
	Name        string     `protobuf:"bytes,2,opt,name=name" json:"name,omitempty"`
	Description string     `protobuf:"bytes,3,opt,name=description" json:"description,omitempty"`
	Mixers      []*Mixer   `protobuf:"bytes,4,rep,name=mixers" json:"mixers,omitempty"`
	Sections    []*Section `protobuf:"bytes,5,rep,name=sections" json:"sections,omitempty"`
}

Currently I'm having to convert between the two structs depending what I'm doing. Which is tedious and I'm probably quite a considerable performance hit. So is there a better way of converting between the two, or re-using one of them for both tasks?

答案1

得分: 2

解决这个问题有几种方法,根据两种一般方法进行分类:

  1. 使用相同的数据类型
  2. 使用两种不同的结构类型并在它们之间进行映射

如果你想使用相同的数据类型,你需要修改代码生成部分

你可以使用像gogoprotobuf这样的工具,它有一个扩展可以添加标签。这样你的结构体中就会有bson标签。

你也可以使用后处理生成的文件,可以使用正则表达式或者涉及Go抽象语法树的更复杂的方法。

如果你选择进行映射:

  1. 使用反射。你可以编写一个包,接受两个结构体并尝试将一个结构体的值应用到另一个结构体上。你需要处理一些边缘情况(轻微的命名差异,哪些类型是等价的等等),但是如果出现这些情况,你将对边缘情况有更好的控制。

  2. 使用JSON作为中介。只要生成的JSON标签匹配,这将是一个快速的编码练习,如果这不是你代码中的紧密循环,序列化和反序列化的性能损耗可能是可以接受的。

  3. 手动编写或使用代码生成映射函数。根据你有多少个结构体,你可以编写一系列函数来在两者之间进行转换。

在我们的工作场所,我们最终采用了上述所有方法:分叉protoc生成器以进行自定义标签,基于反射的结构体叠加包用于映射任意结构体,以及在性能敏感或不太自动化的映射中手动编写的方法。

英文:

Having lived with this same issue, there's a couple methods of solving it. They fall into two general methods:

  1. Use the same data type
  2. Use two different struct types and map between them

If you want to use the same data type, you'll have to modify the code generation

You can use something like gogoprotobuf which has an extension to add tags. This should give you bson tags in your structs.

You could also post-process your generated files, either with regular expressions or something more complicated involving the go abstract syntax tree.

If you choose to map between them:

  1. Use reflection. You can write a package that will take two structs and try to take the values from one and apply it to another. You'll have to deal with edge cases (slight naming differences, which types are equivalent, etc), but you'll have better control over edge cases if they ever come up.

  2. Use JSON as an intermediary. As long as the generated json tags match, this will be a quick coding exercise and the performance hit of serializing and deserializing might be acceptable if this isn't in a tight loop in your code.

  3. Hand-write or codegen mapping functions. Depending on how many structs you have, you could write out a bunch of functions that translate between the two.

At my workplace, we ended up doing a bit of all of them: forking the protoc generator to do some custom tags, a reflection based structs overlay package for mapping between arbitrary structs, and some hand-written ones in more performance sensitive or less automatable mappings.

答案2

得分: 2

我已经使用以下内容进行了翻译:

我已经使用它并且有一个可工作的示例,其中包含以下内容:

github.com/gogo/protobuf v1.3.1
go.mongodb.org/mongo-driver v1.4.0
google.golang.org/grpc v1.31.0

首先,我想分享我的 proto/contract/example.proto 文件:

syntax = "proto2";

package protobson;

import "gogoproto/gogo.proto";

option (gogoproto.sizer_all) = true;
option (gogoproto.marshaler_all) = true;
option (gogoproto.unmarshaler_all) =  true;
option go_package = "gitlab.com/8bitlife/proto/go/protobson";

service Service {
    rpc SayHi(Hi) returns (Hi) {}
}

message Hi {
    required bytes id = 1 [(gogoproto.customtype) = "gitlab.com/8bitlife/protobson/custom.BSONObjectID", (gogoproto.nullable) = false, (gogoproto.moretags) = "bson:\"_id\""] ;
    required int64 limit = 2  [(gogoproto.nullable) = false, (gogoproto.moretags) = "bson:\"limit\""] ;
}

它包含一个简单的 gRPC 服务 Service,其中有一个带有请求类型 HiSayHi 方法。它包含一组选项:gogoproto.sizer_allgogoproto.marshaler_allgogoproto.unmarshaler_all。你可以在扩展页面上找到它们的含义。Hi 本身包含两个字段:

  1. id,它有指定的附加选项:gogoproto.customtypegogoproto.moretags
  2. limit,只有 gogoproto.moretags 选项

BSONObjectIDid 字段的 gogoproto.customtype 中使用,它是我在 custom/objectid.go 中定义的自定义类型:

package custom

import (
    "go.mongodb.org/mongo-driver/bson/bsontype"
	"go.mongodb.org/mongo-driver/bson/primitive"
)

type BSONObjectID primitive.ObjectID

func (u BSONObjectID) Marshal() ([]byte, error) {
    return u[:], nil
}

func (u BSONObjectID) MarshalTo(data []byte) (int, error) {
    return copy(data, (u)[:]), nil
}

func (u *BSONObjectID) Unmarshal(d []byte) error {
    copy((*u)[:], d)
	return nil
}

func (u *BSONObjectID) Size() int {
    return len(*u)
}

func (u *BSONObjectID) UnmarshalBSONValue(t bsontype.Type, d []byte) error {
    copy(u[:], d)
	return nil
}

func (u BSONObjectID) MarshalBSONValue() (bsontype.Type, []byte, error) {
	return bsontype.ObjectID, u[:], nil
}

这是必需的,因为我们需要为协议缓冲区和 MongoDB 驱动程序定义自定义的编组和解组方法。这允许我们在 MongoDB 中将此类型用作对象标识符。为了向 MongoDB 驱动程序“解释”它,我在 proto 文件中使用了 (gogoproto.moretags) = "bson:\"_id\"" 选项,给它加上了一个 bson 标签。

为了从 proto 文件生成源代码,我使用了以下命令:

protoc \
	--plugin=/Users/pstrokov/go/bin/protoc-gen-gogo \
	--plugin=/Users/pstrokov/go/bin/protoc-gen-go \
	-I=/Users/pstrokov/Workspace/protobson/proto/contract \
	-I=/Users/pstrokov/go/pkg/mod/github.com/gogo/protobuf@v1.3.1 \
	--gogo_out=plugins=grpc:. \
	example.proto

我在我的 MacOS 上进行了测试,使用运行中的 MongoDB 实例:docker run --name mongo -d -p 27017:27017 mongo

package main

import (
	"context"
	"log"
	"net"
	"time"

	"gitlab.com/8bitlife/protobson/gitlab.com/8bitlife/proto/go/protobson"
	"go.mongodb.org/mongo-driver/bson"
	"go.mongodb.org/mongo-driver/mongo"
	"go.mongodb.org/mongo-driver/mongo/options"
	"google.golang.org/grpc"
)

type hiServer struct {
	mgoClient *mongo.Client
}

func (s *hiServer) SayHi(ctx context.Context, hi *protobson.Hi) (*protobson.Hi, error) {
	collection := s.mgoClient.Database("local").Collection("bonjourno")
    res, err := collection.InsertOne(ctx, bson.M{"limit": hi.Limit})
	if err != nil { panic(err) }
	log.Println("generated _id", res.InsertedID)

	out := &protobson.Hi{}
	if err := collection.FindOne(ctx, bson.M{"_id": res.InsertedID}).Decode(out); err != nil { return nil, err }
	log.Println("found", out.String())
	return out, nil
}

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()
	lis, err := net.Listen("tcp", "localhost:0")
	if err != nil { log.Fatalf("failed to listen: %v", err) }
	clientOptions := options.Client().ApplyURI("mongodb://localhost:27017")
	clientOptions.SetServerSelectionTimeout(time.Second)
	client, err := mongo.Connect(ctx, clientOptions)
	if err != nil { log.Fatal(err) }
	if err := client.Ping(ctx, nil); err != nil { log.Fatal(err) }
	grpcServer := grpc.NewServer()
	protobson.RegisterServiceServer(grpcServer, &hiServer{mgoClient: client})
	go grpcServer.Serve(lis); defer grpcServer.Stop()
	conn, err := grpc.Dial(lis.Addr().String(), grpc.WithInsecure())
	if err != nil { log.Fatal(err) }; defer conn.Close()
	hiClient := protobson.NewServiceClient(conn)
	response, err := hiClient.SayHi(ctx, &protobson.Hi{Limit: 99})
	if err != nil { log.Fatal(err) }
	if response.Limit != 99 { log.Fatal("unexpected limit", response.Limit) }
	if response.Id.Size() == 0 { log.Fatal("expected a valid ID of the new entity") }
	log.Println(response.String())
}

对于最后一个代码片段的格式问题,我很抱歉 将protoc生成的结构体转换为bson结构体的最佳方法是什么?
希望这可以帮到你。

英文:

I have played with it and have a working example with:

github.com/gogo/protobuf v1.3.1
go.mongodb.org/mongo-driver v1.4.0
google.golang.org/grpc v1.31.0

First of all I would like to share my proto/contract/example.proto file:

syntax = "proto2";

package protobson;

import "gogoproto/gogo.proto";

option (gogoproto.sizer_all) = true;
option (gogoproto.marshaler_all) = true;
option (gogoproto.unmarshaler_all) =  true;
option go_package = "gitlab.com/8bitlife/proto/go/protobson";

service Service {
    rpc SayHi(Hi) returns (Hi) {}
}

message Hi {
    required bytes id = 1 [(gogoproto.customtype) = "gitlab.com/8bitlife/protobson/custom.BSONObjectID", (gogoproto.nullable) = false, (gogoproto.moretags) = "bson:\"_id\""] ;
    required int64 limit = 2  [(gogoproto.nullable) = false, (gogoproto.moretags) = "bson:\"limit\""] ;
}

It contains a simple gRPC service Service that has SayHi method with request type Hi. It includes a set of options: gogoproto.sizer_all, gogoproto.marshaler_all, gogoproto.unmarshaler_all. Their meaning you can find at extensions page. The Hi itself contains two fields:

  1. id that has additional options specified: gogoproto.customtype and gogoproto.moretags
  2. limit with only gogoproto.moretags option

BSONObjectID used in gogoproto.customtype for id field is a custom type that I defined as custom/objectid.go:

package custom

import (
    "go.mongodb.org/mongo-driver/bson/bsontype"
	"go.mongodb.org/mongo-driver/bson/primitive"
)

type BSONObjectID primitive.ObjectID

func (u BSONObjectID) Marshal() ([]byte, error) {
    return u[:], nil
}

func (u BSONObjectID) MarshalTo(data []byte) (int, error) {
    return copy(data, (u)[:]), nil
}

func (u *BSONObjectID) Unmarshal(d []byte) error {
    copy((*u)[:], d)
	return nil
}

func (u *BSONObjectID) Size() int {
    return len(*u)
}

func (u *BSONObjectID) UnmarshalBSONValue(t bsontype.Type, d []byte) error {
    copy(u[:], d)
	return nil
}

func (u BSONObjectID) MarshalBSONValue() (bsontype.Type, []byte, error) {
	return bsontype.ObjectID, u[:], nil
}

It is needed because we need to define a custom marshaling and un-marshaling methods for both: protocol buffers and mongodb driver. This allows us to use this type as an object identifier in mongodb. And to "explaine" it to mongodb driver I marked it with a bson tag by using (gogoproto.moretags) = "bson:\"_id\"" option in proto file.

To generate source code from the proto file I used:

protoc \
	--plugin=/Users/pstrokov/go/bin/protoc-gen-gogo \
	--plugin=/Users/pstrokov/go/bin/protoc-gen-go \
	-I=/Users/pstrokov/Workspace/protobson/proto/contract \
	-I=/Users/pstrokov/go/pkg/mod/github.com/gogo/protobuf@v1.3.1 \
	--gogo_out=plugins=grpc:. \
	example.proto

I have tested it on my MacOS with running MongoDB instance: docker run --name mongo -d -p 27017:27017 mongo:

package main

import (
	"context"
	"log"
	"net"
	"time"

	"gitlab.com/8bitlife/protobson/gitlab.com/8bitlife/proto/go/protobson"
	"go.mongodb.org/mongo-driver/bson"
	"go.mongodb.org/mongo-driver/mongo"
	"go.mongodb.org/mongo-driver/mongo/options"
	"google.golang.org/grpc"
)

type hiServer struct {
	mgoClient *mongo.Client
}

func (s *hiServer) SayHi(ctx context.Context, hi *protobson.Hi) (*protobson.Hi, error) {
	collection := s.mgoClient.Database("local").Collection("bonjourno")
    res, err := collection.InsertOne(ctx, bson.M{"limit": hi.Limit})
	if err != nil { panic(err) }
	log.Println("generated _id", res.InsertedID)

	out := &protobson.Hi{}
	if err := collection.FindOne(ctx, bson.M{"_id": res.InsertedID}).Decode(out); err != nil { return nil, err }
	log.Println("found", out.String())
	return out, nil
}

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()
	lis, err := net.Listen("tcp", "localhost:0")
	if err != nil { log.Fatalf("failed to listen: %v", err) }
	clientOptions := options.Client().ApplyURI("mongodb://localhost:27017")
	clientOptions.SetServerSelectionTimeout(time.Second)
	client, err := mongo.Connect(ctx, clientOptions)
	if err != nil { log.Fatal(err) }
	if err := client.Ping(ctx, nil); err != nil { log.Fatal(err) }
	grpcServer := grpc.NewServer()
	protobson.RegisterServiceServer(grpcServer, &hiServer{mgoClient: client})
	go grpcServer.Serve(lis); defer grpcServer.Stop()
	conn, err := grpc.Dial(lis.Addr().String(), grpc.WithInsecure())
	if err != nil { log.Fatal(err) }; defer conn.Close()
	hiClient := protobson.NewServiceClient(conn)
	response, err := hiClient.SayHi(ctx, &protobson.Hi{Limit: 99})
	if err != nil { log.Fatal(err) }
	if response.Limit != 99 { log.Fatal("unexpected limit", response.Limit) }
	if response.Id.Size() == 0 { log.Fatal("expected a valid ID of the new entity") }
	log.Println(response.String())
}

Sorry for the formatting of the last code snippet 将protoc生成的结构体转换为bson结构体的最佳方法是什么?
I hope this can help.

答案3

得分: 1

我正在进行测试,并可能很快提供代码(如果你没有看到并且需要的话,请提醒我),但是https://godoc.org/go.mongodb.org/mongo-driver/bson/bsoncodec 看起来是一个不错的选择。protoc会生成你的结构体,你不需要自定义它们。然后你可以自定义mongo-driver来为你执行某些类型的映射,他们的库看起来非常好用。

这很棒,因为如果我使用protogen结构体,我希望它们属于我的应用程序核心/领域层。我不想在那里担心与mongoDB的兼容性。

所以现在,对我来说,@Liyan Chang 的回答似乎不一定是必须的。因为你可以选择使用一个数据类型。

你可以使用一个生成的类型,并使用这个编解码系统来处理与数据库的数据获取和设置。

参考https://stackoverflow.com/a/59561699/8546258 - bson结构体标签并不是唯一的选择。看起来编解码器完全可以帮助解决这个问题。

参考https://stackoverflow.com/a/58985629/8546258 了解有关编解码器的一般信息。

请记住,这些编解码器是在mongodb go驱动程序的1.3版本中发布的。我找到了这个链接,它引导我到那里:https://developer.mongodb.com/community/forums/t/mgo-setbson-to-mongo-golang-driver/2340/2?u=yehuda_makarov

英文:

I'm in the process of testing and may provide code shortly, (ping me if you don't see it and you want it) but https://godoc.org/go.mongodb.org/mongo-driver/bson/bsoncodec looks like the ticket. protoc will make your structs and you don't have to mess with customizing them. Then you can customize the mongo-driver to do the mapping of certain types for you and it looks like their library for this is pretty good.

This is great because if I use the protogen structs then I'd like that to my application core / domain layer. I don't want to be concerned about mongoDB compatibility over there.

So right now, it seems to me that @Liyan Chang 's answer saying
> If you want to use the same data type, you'll have to modify the code generation
doesn't necessarily have to be the case. Because you can opt to use 1 datatype.

You can use one generated type and account for seemingly whatever you need to in terms of getting and setting data to the DB with this codec system.

See https://stackoverflow.com/a/59561699/8546258 - the bson struct tags are not an end all be all. looks like codec can totally help with this.

See https://stackoverflow.com/a/58985629/8546258 fo a nice write up about codecs in general.

Please keep in mind these codecs were released in 1.3 of the mongodb go driver. I found this which directed me there: https://developer.mongodb.com/community/forums/t/mgo-setbson-to-mongo-golang-driver/2340/2?u=yehuda_makarov

答案4

得分: 0

注意:截至2023年5月,有多个相互矛盾/过时的答案,protobuf 3是最新版本。经过大量的研究,我得出了以下结论:

  • brew install protobuf - 首先我们需要安装protoc,即proto编译器。
  • go install google.golang.org/protobuf/cmd/protoc-gen-go - 全局安装protoc-gen-go。这是用于生成Go代码的Google协议缓冲区编译器的插件。
  • go install github.com/favadi/protoc-go-inject-tag@latest - 这是一个扩展,可以在生成的Go结构体上添加任何自定义标签。我们需要使用bson。
  • 更新你的.bashrc/.zshrc文件(Mac/Linux)。你需要设置路径指向goroot。否则,你将无法运行protoc
export PATH=~/flutter/bin:$PATH
export PATH=~/.local/bin/:$PATH
export LANG=en_US.UTF-8
export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export GOBIN=$GOPATH/bin
export PATH=$PATH:$GOROOT:$GOPATH:$GOBIN
  • cd your-project - 可以一次为整个项目生成Protobuf文件。
  • protoc --go_out=. **/*.proto - 这个命令会生成*.pb.go文件。这些文件在编组和解组Kafka消息时是必需的。
  • string id = 1; // @gotags: bson:"id,omitempty" - 在需要使用bson标签的proto字段上方或右侧添加此注释。
  • protoc-go-inject-tag -remove_tag_comment -input="**/**/*.pb.go - 在使用protoc生成go文件后运行此命令。

详细说明

  • Protobuf默认不生成bson标签,Go语言扩展也不生成。可以在这个Github问题中找到原因:MutateHook for protoc-gen-go。StackOverflow上的一些旧答案建议我们编写自己的脚本来添加这些缺失的标签。只要有一些可以完成这个任务的GitHub仓库,我就不想自己去做这个任务。What would be the best approach to converting protoc generated structs from bson structs? | StackOverflow

    我可以自定义protoc-gen-go生成的代码吗?一般来说,不能。协议缓冲区旨在成为一种与语言无关的数据交换格式,而实现特定的自定义与此目的相悖。

    这个问题以前已经有过各种讨论,但决定通常是,“我们认为这不是一个我们可以或应该添加的功能。”不幸的是,这个包需要严格遵守protobuf标准,该标准旨在支持多种语言,并需要确保最大的兼容性。我们之前已经因为添加了json标签而遇到过问题,因为在Go语言中这样做非常容易。但现在protobuf有了一个标准的JSON映射,这些JSON标签现在不符合规范,标准库encoding/json不能被改造以使其符合规范。然而,由于人们一直依赖于json标签,我们不能只是删除它们,尽管它们是一个错误。由于这个历史原因,我们对于单方面添加任何东西都非常不情愿,整个protobuf项目都不鼓励添加特定于语言的功能,因为正如前面提到的,它需要是与语言无关的。已经有人提供了执行此任务的工具,但官方的golang protobuf模块不太可能采用未经广泛protobuf标准同意的东西。

  • srikrsna/protoc-gen-gotag - 不起作用 - 最初我在这个Github问题中找到了这个库protoc-gen-gotag (PGGT),该库似乎已经过时和废弃。我不明白它应该如何使用。说明没有提供明确的前进路径。网络上也没有任何资源。即使这个明显不错的教程也没有给出一个好的指示,让tagger.tags起作用:New official MongoDB Go Driver and Google Protobuf — making them work together

  • favadi/protoc-go-inject-tag - 起作用 - 经过几个小时的搜索,我再次偶然发现了这个Github问题:protoc-gen-go: support go_tag option to specify custom struct tags。再次阅读后,我找到了一个使用魔术注释语法来添加缺失的bson标签的库:protoc-go-inject-tag。幸运的是,它适用于最新的protobuf 3。它似乎也有更好的推广。更好的是,语法不会干扰go结构体,因此保持了生成的结构体的可读性。

    • go install github.com/favadi/protoc-go-inject-tag@latest - 安装这个扩展
    • protoc --go_out=. **/*.proto - 像往常一样生成go protobuf
    • string id = 1; // @gotags: bson:"id,omitempty" - 在需要使用bson标签的proto字段上方或右侧添加此注释。
    • protoc-go-inject-tag -remove_tag_comment -input="**/**/*.pb.go - 在使用protoc生成go文件后运行此命令。不幸的是,这个第二步是无法绕过的。注意:由于某种原因,通配符语法不能进入深层嵌套的文件夹。因此,我们必须重复多次。这意味着如果我们有3层深的proto文件,这个命令将无法匹配它们。-remove_tag_comment将从生成的结构体中删除@gotag注释。(如果你找到了解决通配符模式的方法,请告诉我)
英文:

Note: As of May 2023 there are multiple contradicting/outdated answers and protobuf 3 is the latest version. After a lot of digging I came up with this:

  • brew install protobuf - First we need to install protoc, proto compiler.
  • go install google.golang.org/protobuf/cmd/protoc-gen-go - Installs protoc-gen-go globally. This is a plugin for the Google protocol buffer compiler to generate Go code.
  • go install github.com/favadi/protoc-go-inject-tag@latest - Extension that can add any custom tags on the generated go structs. We need bson.
  • Update your .bashrc/.zshrc file. (Mac/linux). You will need to setup the path to point to goroot. Otherwise you wont be able to run protoc
export PATH=~/flutter/bin:$PATH
export PATH=~/.local/bin/:$PATH
export LANG=en_US.UTF-8
export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export GOBIN=$GOPATH/bin
export PATH=$PATH:$GOROOT:$GOPATH:$GOBIN
  • cd your-project - Protobuf files can be generated for the entire project in one command.
  • protoc --go_out=. **/*.proto - This command generates *.pb.go files. These are necessary to marshal and unmarshal our kafka messages.
  • string id = 1; // @gotags: bson:"id,omitempty" - Add this comment above or right side of the proto fields that need to be tagged with bson tags.
  • protoc-go-inject-tag -remove_tag_comment -input="**/**/*.pb.go - Run this command after you generated go files using protoc.

Detailed Explanation

  • Protobuf does not generate by default bson tags. Neither the go golang extension. The reasoning can be found here in this Github ticket: MutateHook for protoc-gen-go. Some older answers on StackOverflow suggest that we should write our own script to add these missing tags. That is by no means a task that I'd like to carry out as long as there's some github repo that ca do it. What would be the best approach to converting protoc generated structs from bson structs? | StackOverflow

    > Can I customize the code generated by protoc-gen-go? In general, no. Protocol buffers are intended to be a language-agnostic data interchange format, and implementation-specific customizations run counter to that intent.

    > This has been variously discussed before, but the decision usually settles on, “we do not think this is a feature we can or should add.” Unfortunately, this package has a need to stick strictly to the protobuf standards, which by design targets multiple languages, and needs to ensure maximum compatibility. We have already been bit before by adding json tags, because in Go doing so was so easy to do. But now that protobuf has a standard JSON mapping, those JSON tags are now non-compliant, and the standard library encoding/json cannot be retrofitted to make it compliant. However, because people have been relying on the json tags, we cannot just remove them, even though they were a mistake. Because of this history, we’re quite reluctant to add anything unilaterally, and the protobuf project as a whole frowns upon adding language-specific features, because as mentioned, it needs to be language-agnostic. There have been people presenting tools to perform this ask, but the official golang protobuf module is unlikely to ever take up things that have not been agreed upon by the wider protobuf standard.

  • srikrsna/protoc-gen-gotag - Not working - Initially I found this library protoc-gen-gotag (PGGT) in this github ticket MutateHook for protoc-gen-go. The library seems outdated and abandoned. I simply don't get it how it's supposed to be used. The instructions don't offer a clear path forward. Neither any resources on the web. Not even this apparently decent tutorial provided a good indication of what to do to make tagger.tags work: New official MongoDB Go Driver and Google Protobuf — making them work together.

  • favadi/protoc-go-inject-tag - Working - After hours of digging on the web I stumbled once again on this github ticket: protoc-gen-go: support go_tag option to specify custom struct tags. Reading again I found a library that uses magic comment syntax to add the missing bson tags: protoc-go-inject-tag. Fortunately it works with latest protobuf 3. It also seems to have better traction. And even better is that the syntax is not distracting from the go struct, thus maintaining decent readability of the generated structs.

    • go install github.com/favadi/protoc-go-inject-tag@latest - Installs the extension
    • protoc --go_out=. **/*.proto - generate go protobuf as usual
    • string id = 1; // @gotags: bson:"id,omitempty" - Add this comment above or right side of the proto fields that need to be tagged with bson tags.
    • protoc-go-inject-tag -remove_tag_comment -input="**/**/*.pb.go - Run this command after you generated go files using protoc. Sadly there's no way around this second step. Note: for some reason the glob syntax does not go into deep nested folders. So we have to repeat for all levels. This means that if we have proto files 3 folders deep, this command wont match them. -remove_tag_comment will remove the @gotag comments form the generated struct. (Let me know if you find a fix for the glob pattern)

huangapple
  • 本文由 发表于 2017年7月18日 06:49:01
  • 转载请务必保留本文链接:https://go.coder-hub.com/45154825.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定