protoc --go_opt=paths=source_relative vs --go-grpc_opt=paths=source_relative

I am having a hard time figuring out protoc command and go plugin.

What is the different between:

protoc \
   # Directory where you want the compiler to write your Go output.
   --go_out=.
   # vs ?
   --go_opt=paths=source_relative
   # vs ?
   --go-grpc_out=.
   # vs ?
   --go-grpc_opt=paths=source_relative

If --go_opt generate

• <name>.pb.go file

and --go-grpc_opt generate

• <name>_grpc.pb.go file

why even have --go_out?

Can you shed some light on protoc - the doc do not say anything about --go-grpc_opt?

And and protoc -h do not even list go as an OUT_DIR?

Note: I install using this doc

> why even have --go_out?

So, the thing to understand here is that gRPC is not the same as Protocol Buffers, gRPC uses Protocol Buffers but there are other frameworks that are using them as well. So we need to generate both.

Now, in order to generate the Protocol buffer related code, you need to use --go_out as you mentioned. but for the gRPC code you need to use --go-grpc_out.

> and --go-grpc_opt generate <name>_grpc.pb.go file

No, --go-grpc_out does.

> Can you shade some light on protoc - the doc do not stay anything about --go-grpc_opt?

Then, before the generation of code you can pass some options and that's what --go_opt and --go-grpc_opt are for. The first one passes options for Protobuf generation and the second one for gRPC generation. Options are pretty obscure, and there is not official list of all of them, but you used source_relative (which tell protoc to use relative paths) for path and there is also the module option (which help protoc know the go module name to generate in proper folder)

> And and protoc -h do not even list go as an OUT_DIR?

And finally, protoc doesn't officially support Go as output, you need to install an external plugin and that's why protoc -h doesn't show --go_out. A related discussion can be found here.

The protoc compiler supports different flags or options and the flags you use on the command line decides where the generated go code will be placed.

The official docs for these flags(atleast for paths=source_relative and module=$PREFIX) are not very clear and can be hard to understand.

paths=source_relative

Here is what official docs says

> If the paths=source_relative flag is specified, the output file is
> placed in the same relative directory as the input file. For example,
> an input file protos/buzz.proto results in an output file at
> protos/buzz.pb.go.

The above statement might be confusing because it is not giving complete context and folder layout of how the files are placed on the disk.

To me what this flag means is that when used generates the go code in the directory specified by --go_out and ensures that directory tree structure of the generated go code files matches the directory tree structure of proto files.

Let's say we have the following directory structure

❯ tree
.
├── go.mod
└── src
    └── protos
        ├── bar
        │   └── baz.proto
        └── foo.proto

5 directories, 3 files

Consider the following examples

protoc --proto_path=src/protos --go_out=. --go_opt=paths=source_relative foo.proto bar/baz.proto

❯ ls -l
drwxrwxr-x  3 rbhanot  staff    96 Jul  5 20:52 bar
-rw-rw-r--  1 rbhanot  staff  3912 Jul  5 20:52 foo.pb.go
-rw-rw-r--  1 rbhanot  staff    45 Jul  5 16:12 go.mod
drwxr-xr-x  3 rbhanot  staff    96 Jul  5 18:12 src

❯ ls -l bar
-rw-rw-r--  1 rbhanot  staff  4053 Jul  5 20:52 baz.pb.go

In the above example we set the --proto_path=src/protos which means the directory path for the actual proto files would be foo.proto and bar/baz.proto, and the pb files were created in the current directory (--go_out=.) as foo.pb.go and bar/baz.pb.go.

Now let's change the --proto_path in the above command to src and see what happnes.

❯ protoc --proto_path=src --go_out=. --go_opt=paths=source_relative protos/foo.proto protos/bar/baz.proto

❯ ls -l
-rw-rw-r--  1 rbhanot  staff   45 Jul  5 16:12 go.mod
drwxrwxr-x  4 rbhanot  staff  128 Jul  5 21:00 protos
drwxr-xr-x  3 rbhanot  staff   96 Jul  5 18:12 src

~/dummy_go  
❯ ls -l protos
drwxrwxr-x  3 rbhanot  staff    96 Jul  5 21:00 bar
-rw-rw-r--  1 rbhanot  staff  4158 Jul  5 21:00 foo.pb.go

This time a new protos directory was created under which we have the generated go files, why ? Because when we changed the --proto-path=src the directory structure of proto files changed to protos/foo.proto and protos/bar/baz.proto.

Now let's finally throw the --go_out as well in here and see what happens

❯ mkdir out
❯ protoc --proto_path=src --go_out=out --go_opt=paths=source_relative protos/foo.proto protos/bar/baz.proto
❯ ls -l        
total 8
-rw-rw-r--  1 rbhanot  staff  45 Jul  5 16:12 go.mod
drwxrwxr-x  3 rbhanot  staff  96 Jul  5 21:05 out
drwxr-xr-x  3 rbhanot  staff  96 Jul  5 18:12 src

~/dummy_go  
❯ ls -lR out
drwxrwxr-x  4 rbhanot  staff  128 Jul  5 21:05 protos

out/protos:
drwxrwxr-x  3 rbhanot  staff    96 Jul  5 21:05 bar
-rw-rw-r--  1 rbhanot  staff  4158 Jul  5 21:05 foo.pb.go

out/protos/bar:
-rw-rw-r--  1 rbhanot  staff  4298 Jul  5 21:05 baz.pb.go

This is exactly similar to last example except that we provided a custom directory to hold the generated code.

module=$PREFIX

> If the module=$PREFIX flag is specified, the output file is placed in
> a directory named after the Go package’s import path, but with the
> specified directory prefix removed from the output filename. For
> example, an input file protos/buzz.proto with a Go import path of
> example.com/project/protos/fizz and example.com/project specified as
> the module prefix results in an output file at protos/fizz/buzz.pb.go.
> Generating any Go packages outside the module path results in an
> error. This mode is useful for outputting generated files directly
> into a Go module.

Lets see this as well as in action, consider the following proto file

syntax = "proto3";

package foo;

option go_package = "github.com/rbhanot/dummy-app/greet";

message Foo {
    string name = 1;
}

Note carefully I have intentionally remove the src from the go_package to show the behaviour of this flag

❯ protoc --proto_path=src --go_out=. --go_opt=module=github.com/rbhanot/dummy-app protos/foo.proto protos/bar/baz.proto

❯ ls -l
-rw-rw-r--  1 rbhanot  staff   45 Jul  5 16:12 go.mod
drwxrwxr-x  3 rbhanot  staff   96 Jul  5 21:14 greet
drwxr-xr-x  4 rbhanot  staff  128 Jul  5 21:14 src

❯ ls -l greet
-rw-rw-r--  1 rbhanot  staff  4274 Jul  5 21:17 baz.pb.go
-rw-rw-r--  1 rbhanot  staff  4133 Jul  5 21:17 foo.pb.go

We see the greet directory with the pb files in it. So it basically removed the prefix(github.com/rbhanot/dummy-app) from the go_package and then actually created the go package in the current directory(--go_out=.).

Now lets change go_package = "github.com/rbhanot/dummy-app/src/greet"; and running the above command yields following

❯ protoc --proto_path=src --go_out=. --go_opt=module=github.com/rbhanot/dummy-app protos/foo.proto protos/bar/baz.proto

❯ ls -l src
drwxrwxr-x  4 rbhanot  staff  128 Jul  5 21:22 greet
drwxr-xr-x  4 rbhanot  staff  128 Jul  5 17:37 protos

This time we see the greet package generated inside our src directory.

Finally lets throw in the same --go_out=out here as well

❯ mkdir out

❯ protoc --proto_path=src --go_out=out --go_opt=module=github.com/rbhanot/dummy-app protos/foo.proto protos/bar/baz.proto

❯ ls -l out     
total 0
drwxrwxr-x  3 rbhanot  staff  96 Jul  5 21:24 src

❯ ls -lR out
total 0
drwxrwxr-x  3 rbhanot  staff  96 Jul  5 21:24 src

out/src:
total 0
drwxrwxr-x  4 rbhanot  staff  128 Jul  5 21:24 greet

out/src/greet:
total 32
-rw-rw-r--  1 rbhanot  staff  4298 Jul  5 21:24 baz.pb.go
-rw-rw-r--  1 rbhanot  staff  4158 Jul  5 21:24 foo.pb.go

This time the generated code was not put in src but inside the out and note the directory structure as well src/greet/foo.pb.go.

I hope this makes things more clear (it atleast did for me) wrt to how these flags behave.
1: https://protobuf.dev/reference/go/go-generated/#invocation