Trying to write shared protobuf definitions

I'm trying to create shared protobuf definitions in a shared go module, but I'm not having much luck. Specifically:<br/>
- I want a proto file in module protos that contains shared definitions.<br/>
- Other proto files in module protos would reference the shared definitions.<br/>
- Module protos-use would reference module protos (via go.mod)<br/>
- Module protos-use go code would import and use definitions from module protos<br/>
- Both modules are stored in github.com

I'm unable to get the various package names to align. No matter what I use, something fails. Currently, module protos doesn't have any errors, but go mod tidy on module protos-use fails.

Module protos is at https://github.com/rob-woerner/protos&lt;br/>
Module protos-use is at https://github.com/rob-woerner/protos-use

I haven't found an example that addresses this scenario. If I simply duplicate the shared proto definitions everywhere, I can get it to work. The most recent error is:<br/>
$ go mod tidy<br/>
go: finding module for package github.com/rob-woerner/protos/messages<br/>
go: finding module for package github.com/rob-woerner/protos/common<br/>
github.com/rob-woerner/protos-use/code imports<br/>
github.com/rob-woerner/protos/common: module github.com/rob-woerner/protos@latest found (XXX), but does not contain package github.com/rob-woerner/protos/common<br/>
github.com/rob-woerner/protos-use/code imports<br/>
github.com/rob-woerner/protos/messages: module github.com/rob-woerner/protos@latest found (XXX), but does not contain package github.com/rob-woerner/protos/messages<br/>

But I emphasize that I get lots of different errors, depending on what package names I use in the dozen or so places that need package names.

If anyone knows of a working example, or if this is not supported, I'd appreciate it.

Code snippets are elided for compactness

Module protos:

go.mod<br/>
Makefile<br/>
protobuf:<br/>
&nbsp;&nbsp;&nbsp;common.proto<br/>
&nbsp;&nbsp;&nbsp;messages.proto<br/>
pb:<br/>
&nbsp;&nbsp;&nbsp;common.pb.go<br/>
&nbsp;&nbsp;&nbsp;messages.pb.go<br/>

Module protos-use:<br/>
go.mod<br/>
code:<br/>
&nbsp;&nbsp;&nbsp;project.go<br/>

Module protos:
go.mod:<br/><br/>
module protos<br/>
go 1.16<br/>
require google.golang.org/protobuf v1.27.1<br/>

protobuf/common.proto:<br/>
package common;<br/>
option go_package = "github.com/rob-woerner/protos";<br/>
enum MyState { LOGGED_OUT = 0; }<br/>

protobuf/messages.proto:<br/>
import "common.proto";<br/>
package messages;<br/>
option go_package = "github.com/rob-woerner/protos";<br/>
message MyRequest { common.MyState state = 1; }<br/>

Makefile:<br/>
protos:<br/>
&nbsp;&nbsp;&nbsp;protoc --proto_path=protobuf --go_out=plugins=grpc:pb --go_opt=module=github.com/rob-woerner/protos protobuf/*.proto<br/>

Module protos-use:

go.mod:<br/>
module github.com/rob-woerner/protos-use<br/>
go 1.16<br/>
require (<br/>
&nbsp;&nbsp;&nbsp;github.com/rob-woerner/protos latest<br/>
)<br/>

code/project.go:<br/>
package code<br/>
import (<br/>
&nbsp;&nbsp;&nbsp;"github.com/rob-woerner/protos/common"<br/>
&nbsp;&nbsp;&nbsp;"github.com/rob-woerner/protos/messages"<br/>
)<br/>
func MyFunc() messages.MyMessage {<br/>
&nbsp;&nbsp;&nbsp;return messages.MyMessage { state: common.MyState_LOGGED_OUT}<br/>
}<br/>

This is challenging with protobufs and is poorly documented.

Part of the complexity -- I sense -- is that the protobuf maintainers are trying to solve for the myriad runtimes and packages that may be used.

If I understand your problem correctly, with what you have, protos-use is struggling because code/product.go should:

import (
    "github.com/rob-woerner/protos/pb/common" // /pb/
    ...
)

I've got a working mechanism that I'll try to apply to your mechanism. I think the references to common aren't your issue but the overall structure and modules vs. packages imports:

• github.com/foo/protos
• github.com/bar/app

In foo/protos, I have *.proto files in the root (but this is just a convention and need not be the case):

• example.proto

syntax = "proto3";

package baz;

option go_package = "github.com/foo/protos;baz";

service Some {...}

> NOTE The ;baz this describe the eventual Go package name (baz) of the generated code. It need not match the the repo name (protos) and I don't want this because (see below), I'm putting the generated code in a non-root directory (/some/baz)

Then:

MODULE="github.com/foo/protos"
protoc \
--include_imports \
--include_source_info \
--proto_path=. \
--descriptor_set_out=foo.pb \
--go_out=./some/baz \
--go_opt=module=${MODULE} \
--go-grpc_out=./some/baz \
--go-grpc_opt=module=${MODULE} \
./*.proto

> NOTE This will give us a package name of github.com/foo/protos/some/baz

This will yield:

.
├── example.proto
├── some
│   └── baz
│       ├── example_grpc.pb.go
│       └── example.pb.go
├── go.mod
├── go.sum
└── protoc-3.17.3-linux-x86_64

Then, from bar/app, I have:

go.mod:

module github.com/bar/app

go 1.16

require (
    github.com/foo/protos v0.0.1
    ...
)

And, within that module, I alias the imports:

package main

import (
    ...
	pb "github.com/foo/protos/some/baz"
)

func main() {
    ...
    pb.RegisterSomeServer(grpcServer, ...)

Update

> NOTE Responding to the comment thread

common and messages are either in one package or in two.

Let's do two:

common.proto:

syntax = "proto3";

package common;

option go_package = "github.com/rob-woerner/protos/common;common";

enum MyState { LOGGED_OUT = 0; }

messages.proto:

syntax = "proto3";

package messages;

import "common.proto";

option go_package = "github.com/rob-woerner/protos/messages;messages";

message MyRequest { common.MyState state = 1; }

> NOTE Because we want 2 packages, we need to give the protos (!) unique package, go_package names ./protos/common;common. Although we could have ./protos/common;freddie, the issue is that the path must be unique to give a unique package.

Then:

MODULE="github.com/rob-woerner/protos"

protoc \
--proto_path=. \
--go_out=./some \
--go_opt=module=${MODULE} \
--go-grpc_out=./some \
--go-grpc_opt=module=${MODULE}

> NOTE The use of some here is solely to ensure that the generated code ends up aggregated by some package (some) below the repo(==module) root. You can remove /some if you want the packages directly beneath the module root.

yields:

some
├── common
│   └── common.pb.go
└── messages
    └── messages.pb.go

> NOTE some becomes the root package github.com/rob-woerner/protos/some and common.pb.go is in the sub-package github.com/rob-woerner/protos/some/common

So, we can now:

main.go:

package main

import (
	"github.com/rob-woerner/protos/some/common"
	"github.com/rob-woerner/protos/some/messages"
)

func main() {
	rqst := messages.MyRequest{
		state: common.MyState_LOGGED_OUT,
	}
    // do something with `rqst`
}

If you want common and messages to be in the same package, then you can just:

package protos;

option go_package = "github.com/rob-woerner/protos;protos`;

And:

message MyRequest { MyState state = 1; } // Now in same package

And then:

MODULE="github.com/rob-woerner/protos"

protoc \
--proto_path=. \
--go_out=./protos \
--go_opt=module=${MODULE} \
--go-grpc_out=./protos \
--go-grpc_opt=module=${MODULE}

> NOTE In this case, we need _out=./protos so that the generated files get put under the correct package (protos

main.go:

package main

import (
    pb "github.com/rob-woerner/protos"
)

func main() {
    rqst := pb.MyRequest{
        state: pb.MyState_LOGGED_OUT,
    }
    // do something with `rqst`
}

Phew! I said it is challenging