How to use a package of generated protobuf inside a go module?

huangapple go评论75阅读模式
英文:

How to use a package of generated protobuf inside a go module?

问题

我在Go模块管理和生成protobuf(使用go1.16和protoc-gen-go@latest)方面遇到了问题。

我有以下项目结构:

subproj
├── go.mod         (模块为company.tld/proj/subproj)
├── subproj.go     (入口点:package main)
├── proto          (包含.proto文件的文件夹)
├── packageFolder
|   └── file1.go   (包packageFolder)
└── Makefile       (用于生成*.pb.go和构建subproj二进制文件)

proto文件夹被其他项目使用(显然...)(通过git子模块)。
protobuf文件如下所示:

syntax = "proto3"
option csharp_namespace = "Proj.Proto";
option go_package = "company.tld/proj/projpb";
package entity.proj
...

由于消息的不同版本,一些protobuf文件需要位于另一个"namespace"中:

option go_package = "company.tld/proj/projpb/other";
package entity.proj.other

在我的Makefile中,我尝试在正确的位置生成正确的*.pb.go文件:

# Proto sources
PROTO= $(wildcard ${PROTODIR}/*.proto)
PBGO=  $(PROTO:.proto=.pb.go)

MODULE_NAME=company.tld/proj
GO_OPT_FLAG=   --go_opt=module=${MODULE_NAME}     
GRPC_OPT_FLAG= --go-grpc_opt=module=${MODULE_NAME}
#GO_OPT_FLAG=   --go_opt=paths=import
#GRPC_OPT_FLAG= --go-grpc_opt=paths=import

.PHONY: clean install proto

## Builds the project
build: proto
    go build ${LDFLAGS} -o ${BINARY}

$(PROTOBUF_GO_PLUGIN):
    go install google.golang.org/protobuf/cmd/protoc-gen-go@latest

$(GRPC_GO_PLUGIN):
    go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

%.pb.go: %.proto | $(PROTOBUF_GO_PLUGIN) $(GRPC_GO_PLUGIN)
    protoc --proto_path=${PROTODIR} --go_out=. ${GO_OPT_FLAG} --go-grpc_out=. ${GRPC_OPT_FLAG} $<

proto: $(PBGO)

因此,根据protoc编译器使用的选项:
→ 使用--go_opt=paths=import

protoc会在项目根目录下创建一个名为company.tld/proj/projpb的文件夹。每个对象都位于名为projpbother的包中,位于subpackage other中。

生成的Proto对象包括other命名空间的对象,其导入路径为import other "company.tld/proj/projpb/other"(由go_package选项提供,但是这是错误的,因为它不是一个现有的模块 - go mod tidy/vendor会报错找不到它)。

普通项目文件需要以下导入路径才能访问生成的Proto对象:
import pb "company.tld/proj/subproj/company.tld/proj/projpb"
这看起来很奇怪,不是正确的做法。

→ 使用--go_opt=module=company.tld/proj

protoc会在项目根目录下创建一个名为projpb的文件夹,每个生成的.pb.go文件都有名为projpbother的包,位于subpackage other中。

生成的Proto对象,包括other命名空间的对象,仍然具有导入路径import other "company.tld/proj/projpb/other"(仍然由go_package选项提供,仍然是错误的,因为这仍然是一个不存在的模块 - 这些是生成的文件...为什么我要为它们创建一个模块呢?)。

好处是,使用这个go_opt,访问生成的类型看起来更正常,如下所示:
import pb "company.tld/proj/subproj/projpb"

最后,我尝试了以下方法:

  • 在.proto文件的go_package选项中使用本地导入路径(在构建时被拒绝,因为生成的protobuf对象中会有import other "./projpb/other"
  • 在go.mod文件中使用replace指令,如下所示:
    company.tld/proj/projpb => ./projpb
    company.tld/proj/projpb/other => ./projpb/other
)

(但是go mod tidy/vendor会报错找不到生成的文件夹./projpb中的go.mod文件)

有人遇到过类似的问题吗?或者我是否缺少一个命令选项来告诉Go,“我在一个包中生成了protobuf对象,或者在一个包中生成了包,我只想使用它们。它们不是一个模块,所以请提供正确的导入路径给生成的对象,并让我在我的代码中使用它们”。

英文:

I encounter an issue with Go Module management and a generation of protobuffers (using go1.16, protoc-gen-go@latest).

I have this project structure:

subproj
├── go.mod         (module company.tld/proj/subproj)
├── subproj.go     (entry point : package main)
├── proto          (folder containing .proto files)
├── packageFolder
|   └── file1.go   (package packageFolder)
└── Makefile       (used to generate *.pb.go and build subproj binary)

The proto folder is used by other projects (obviously...) (via git submodule).
Protos are like the following:

syntax = &quot;proto3&quot;
option csharp_namespace = &quot;Proj.Proto&quot;;
option go_package = &quot;company.tld/proj/projpb&quot;;
package entity.proj
...

because of different version of messages, few protobuffer files need to be in another "namespace":

option go_package = &quot;company.tld/proj/projpb/other&quot;;
package entity.proj.other

In my Makefile, I tried to generate the right *.pb.go at the right place:

# Proto sources
PROTO= $(wildcard ${PROTODIR}/*.proto)
PBGO=  $(PROTO:.proto=.pb.go)

MODULE_NAME=company.tld/proj
GO_OPT_FLAG=   --go_opt=module=${MODULE_NAME}     
GRPC_OPT_FLAG= --go-grpc_opt=module=${MODULE_NAME}
#GO_OPT_FLAG=   --go_opt=paths=import
#GRPC_OPT_FLAG= --go-grpc_opt=paths=import

.PHONY: clean install proto

## Builds the project
build: proto
    go build ${LDFLAGS} -o ${BINARY}

$(PROTOBUF_GO_PLUGIN):
    go install google.golang.org/protobuf/cmd/protoc-gen-go@latest

$(GRPC_GO_PLUGIN):
    go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

%.pb.go: %.proto | $(PROTOBUF_GO_PLUGIN) $(GRPC_GO_PLUGIN)
    protoc --proto_path=${PROTODIR} --go_out=. ${GO_OPT_FLAG} --go-grpc_out=. ${GRPC_OPT_FLAG} $&lt;

proto: $(PBGO)

So, depending on the option used with the protoc compiler:
→ With --go_opt=paths=import

A folder tree company.tld/proj/projpb is created by protoc at the project's root. Each object is in a package called projpb or other, in the subpackage other.

Generated Proto objects, that include the other namespace-d objects, have the import path import other &quot;company.tld/proj/projpb/other&quot; (which is brought by the go_package option, but which is wrong because it is not an existing module - go mod tidy/vendor is complaining that it cannot find it).

Normal project files need the following import path to reach the Generated Proto objects :
import pb &quot;company.tld/proj/subproj/company.tld/proj/projpb&quot;
which seems odd and not the proper way to do.

→ With --go_opt=module=company.tld/proj

A folder projpb is created by protoc at the project's root and each generated .pb.go has the package projpb or other, in the subpackage other.

Generated Proto objects, that include the other namespace-d objects, still have the import path import other &quot;company.tld/proj/projpb/other&quot; (which is still brought by the go_package option and is still wrong because this is still a non-existing module - these are generated files... why would I want to create a module of these ?).

The cool thing is that with this go_opt, accessing generated types looks much more normal with
import pb &quot;company.tld/proj/subproj/projpb&quot;.

Finally, I tried

  • using local import path on the go_package option in the .proto files (that is refused on build time, because there would be an import other &quot;./projpb/other&quot; in generated protobuffer object)
  • to use the replace instruction in the go.mod file like this :
replace (
    company.tld/proj/projpb =&gt; ./projpb
    company.tld/proj/projpb/other =&gt; ./projpb/other
)

(but go mod tidy/vendor is complaining that it cannot find the go.mod file inside the generated folder ./projpb)

Has someone encountered a similar problem? Or am I missing a command option to tell to Go, «I generate protobuffer objects in a package, or package in a package, and I simply want to use them. They are not a module, so please, provide the right import paths to the generated object and let me use them in my code».


[Update 01]
I gave a try to the go_opt=paths=source_relative (inspired by this ticket).
I created the folder in the Makefile, protoc generates files inside.
Notes:

  • generated protos use the full path, specified with the go_package option, to relate to one another.
  • As long as go_package option needs a full path, Go (go mod tidy/vendor) will want to search for a go.mod file inside the created folder, containing generated protos.

What is the correct way to tell Go that I am not looking for a Module, yet still satisfy the go_package option's full path constraint in the protobuffer file ?

答案1

得分: 2

在多次更改proto文件中的go_package选项,更改protoc编译器命令中的go_opt选项后,我发现唯一能够编译生成的protobuffer并符合Go约束的方法是动态创建go.mod文件...

最终proto文件的“header”(符合go_package选项中的完整路径)

syntax = "proto3";
option csharp_namespace = "Proj.Proto";
option go_package = "company.tld/proj/projpb";
// 或者对于子包...
option csharp_namespace = "Proj.Proto.Other";
option go_package = "company.tld/proj/projpb/other";

我的Makefile(为生成的proto文件创建go.mod文件)


# Proto sources
PROTO= $(shell find ${PROTODIR} -type f -name '*.proto')
PBGO=  $(PROTO:.proto=.pb.go)

DEST_DIR=.
MODULE_NAME=company.tld/proj
GO_OPT_FLAG=   --go_opt=module=${MODULE_NAME}
GRPC_OPT_FLAG= --go-grpc_opt=module=${MODULE_NAME}

PROTO_PKG_DIR=projpb
PROTO_MODULE_NAME=${MODULE_NAME}/${PROTO_PKG_DIR}
PROTO_GOMOD_FILE=${PROTO_PKG_DIR}/go.mod

.PHONY: clean install proto gomod

build: proto gomod
	go build ${LDFLAGS} -o ${BINARY}

%.pb.go: %.proto | $(PROTOBUF_GO_PLUGIN) $(GRPC_GO_PLUGIN) $(DEST_DIR)
	${PROTOC} --proto_path=${PROTODIR} --go_out=${DEST_DIR} ${GO_OPT_FLAG} --go-grpc_out=${DEST_DIR} ${GRPC_OPT_FLAG} $<;

proto: $(PBGO)

gomod: ${PROTO_GOMOD_FILE}

${PROTO_GOMOD_FILE}:
	cd ${PROTO_PKG_DIR} && go mod init ${PROTO_MODULE_NAME} && cd ..

我的主要go.mod文件(将动态创建的模块重定向到项目范围内的本地文件夹)

module company.tld/proj/subproj

go 1.16

require (
	// ...
	company.tld/proj/projpb v0.0.0
)

replace company.tld/proj/projpb v0.0.0 => ./projpb

通过replace指令,go mod tidy/vendor很高兴,并且不会尝试在远程存储库中搜索模块。
生成的*.pb.go文件具有正确的导入路径:company.tld/proj/projpb(对于子包为company.tld/proj/projpb/other)。
在主项目中,使用生成的proto的导入语句工作正常。

我希望有一个更简单和更漂亮的解决方案,但不幸的是...

对于任何问题,抱歉,并感谢那些思考过的人!

英文:

After changing a numerous amount of time the go_package option in the proto files, changing the go_opt on the protoc compiler command, the only way I found to compile my project with my generated protobuffers, respecting every Go constraints, is by creating a go.mod file on-the-fly...

final proto «header» (respects the full puth in the go_package option)

syntax = &quot;proto3&quot;;
option csharp_namespace = &quot;Proj.Proto&quot;;
option go_package = &quot;company.tld/proj/projpb&quot;;
// or for subpackages...
option csharp_namespace = &quot;Proj.Proto.Other&quot;;
option go_package = &quot;company.tld/proj/projpb/other&quot;;

my Makefile (creates a go.mod file for the generated proto files)


# Proto sources
PROTO= $(shell find ${PROTODIR} -type f -name &#39;*.proto&#39;)
PBGO=  $(PROTO:.proto=.pb.go)

DEST_DIR=.
MODULE_NAME=company.tld/proj
GO_OPT_FLAG=   --go_opt=module=${MODULE_NAME}
GRPC_OPT_FLAG= --go-grpc_opt=module=${MODULE_NAME}

PROTO_PKG_DIR=projpb
PROTO_MODULE_NAME=${MODULE_NAME}/${PROTO_PKG_DIR}
PROTO_GOMOD_FILE=${PROTO_PKG_DIR}/go.mod

.PHONY: clean install proto gomod

build: proto gomod
	go build ${LDFLAGS} -o ${BINARY}

%.pb.go: %.proto | $(PROTOBUF_GO_PLUGIN) $(GRPC_GO_PLUGIN) $(DEST_DIR)
	${PROTOC} --proto_path=${PROTODIR} --go_out=${DEST_DIR} ${GO_OPT_FLAG} --go-grpc_out=${DEST_DIR} ${GRPC_OPT_FLAG} $&lt;

proto: $(PBGO)

gomod: ${PROTO_GOMOD_FILE}

${PROTO_GOMOD_FILE}:
	cd ${PROTO_PKG_DIR} &amp;&amp; go mod init ${PROTO_MODULE_NAME} &amp;&amp; cd ..

my main go.mod file (redirects the on-the-fly-created module to a local folder inside the project's scope)

module company.tld/proj/subproj

go 1.16

require (
	// ...
	company.tld/proj/projpb v0.0.0
)

replace company.tld/proj/projpb v0.0.0 =&gt; ./projpb

Thanks to the replace instruction, go mod tidy/vendor is happy and do not try to search the module in a remote repository.
Generated *.pb.go files have the right import path : company.tld/proj/projpb (and company.tld/proj/projpb/other for the subpackages).
And the import statement to use generated protos are working fine in the main project.

I hoped there was a simpler and more prettier solution, but alas...

Sorry for the any and thanks to those who gave it a thought !

huangapple
  • 本文由 发表于 2021年6月24日 23:55:50
  • 转载请务必保留本文链接:https://go.coder-hub.com/68119052.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定