go reflect for metaprogramming/templates

I have some code that receives protobuf messages that is basically duplicated in a couple places so I want to put it into a library. The problem is that the exact protobuf message that's used is different for each set of code.

EDIT: And I don't have the flexiblity of restructuring them.

I'm not entirely sure this is possible to solve without duplicate code in go, but I did make an attempt (below). Am I doing something wrong, or is this not something that's not possible? (Note: This is stripped down code, in the real code the objects have lots of additional fields)

Example.proto:

package testmsg;

enum RepStatus {
    DONE_OK = 0;
    DONE_ERROR = 1;
}

message ReqHeader {
    optional int64 user_id = 1;
}

message RespHeader {
    optional RepStatus status = 1;
    optional string error_msg = 2;
}

message PostReq {
    optional ReqHeader header = 1;
    optional bytes post_data = 2;
}

message PostResp {
    optional RespHeader header = 1;
}

message StatusReq {
    optional ReqHeader header = 1;
    optional string id = 2;
}

message StatusRep {
    optional RespHeader header = 1;
    optional string status = 2;
}

mini-service/service.go:

package miniservice

import "reflect"
import "github.com/golang/protobuf/proto"
import "testmsg"

type MiniService struct {
    name string
    reqType reflect.Type
    repType reflect.Type
}

func NewService(name string, reqPort int, reqType proto.Message, repType proto.Message) *MiniService {
    ms := new(MiniService)
    ms.name = name
    ms.reqType = reflect.TypeOf(reqType)
    ms.repType = reflect.TypeOf(repType)
    return ms
}

func (ms *MiniService) Handler(msgs []string) (string) {
    resp := reflect.New(ms.repType.Elem())

    msg := msgs[0]
    req := reflect.New(ms.reqType.Elem())
    err := proto.Unmarshal([]byte(msg), req)
    //add some error handling, or just get set _

    resp.Header = &testmsg.RespHeader{}
    //Call handler function that is unique per service
    //the signature will be something like:
    //handleRequest(reqType, respType) & called like:
    //handleRequest(req, resp)
    resp.Header.Status = testmsg.RepStatus_DONE_OK.Enum()

    respMsg, _ := proto.Marshal(resp)
    return string(respMsg)
}

testservice.go:

package main
import "github.com/golang/protobuf/proto"
import "testmsg"
import "mylibs/mini-service"

func main() {
    //fake out a zmq message
    req := &testmsg.PostReq{}
    req.Header = &testmsg.ReqHeader{}
    req.Header.MessageId = proto.Int64(10)
    reqMsg, _ := proto.Marshal(req)
    reqMsgs := []string{string(reqMsg)}

    ms := miniservice.NewService("tester", 5555, testmsg.PostReq, testmsg.PostResp)
    //What will be called when receiving a zmq request
    resp := ms.Handler(reqMsgs)
    log.Info(resp)
}

When I try to compile I get errors like:

resp.Header undefined (type reflect.Value has no field or method Header)

cannot use resp (type reflect.Value) as type proto.Message in argument to proto.Marshal:
reflect.Value does not implement proto.Message (missing ProtoMessage method)

Which make complete sense since the resp isn't connected to ms.respType.

From my point of view, your Protobuf definition is too specific. I cleaned it down a great deal. For example: There is no need to have a different request and response header per type when all they differ in is the content. The most obvious is that I eliminated the specific request and reponse types, because again, all they differed in was their semantic meaning, which on the other hand is rather obvious from the surrounding code. This way, we have eliminated a lot of redundancy. In sum, different types of requests con be identified by the headers, be it either the presence or absence of a user_id field or the evaluation of the content field. Of course, you can expand the headers value selection by what you need.

// exchange.proto
syntax = "proto2";
package main;
enum Status {
DONE_OK = 0;
DONE_ERROR = 1;
}
message Header {
required string name = 1;
oneof value {
int32 user_id = 2;
Status status = 3;
string content= 4;
}
}
message Exchange {
repeated Header header = 1;
optional bytes content = 2;
}

Then, I see your miniservice as rather odd. You'd usually set up a service with things like DAOs, maybe other services and have them handle individual requests taking in a request object and returning a response object. For gRPC services are defined with a .proto file like this (staying within your example)

service Miniservice {
rpc UserInfo(Exchange) returns (Exchange)
}

Which after compiling your .proto basically defines the following interface

type Miniservice interface {
UserInfo(ctx context.Context, in *Exchange) (*Exchange, error)
}

You don't have to use grpc, but it shows how to deal with services, because everything else, like DAOs, loggers and such needs to be a field in the struct implementing said interface. A small example without grpc

//go:generate protoc --go_out=. exchange.proto
package main
import (
"fmt"
"log"
"os"
)
var (
statusName = "Status"
userIdName = "uid"
)
func main() {
logger := log.New(os.Stderr, "SRVC ", log.Ltime|log.Lshortfile)
logger.Println("Main: Setting up dao…")
dao := &daoMock{
Users:  []string{"Alice", "Bob", "Mallory"},
Logger: logger,
}
logger.Println("Main: Setting up service…")
service := &Miniservice{
DAO:    dao,
Logger: logger,
}
// First, we do a valid request
req1 := &Exchange{
Header: []*Header{
&Header{
Value: &Header_UserId{UserId: 0},
},
},
}
if resp1, err := service.UserInfo(req1); err != nil {
logger.Printf("Main: error was returned on request: %s\n", err.Error())
} else {
fmt.Println(">", string(resp1.GetContent()))
}
// A missing UserIdHeader causes an error to be returned
// Header creation compacted for brevity
noUserIdHeader := &Exchange{Header: []*Header{&Header{Value: &Header_Content{Content: "foo"}}}}
if resp2, err := service.UserInfo(noUserIdHeader); err != nil {
logger.Printf("Main: error was returned by service: %s\n", err.Error())
} else {
fmt.Println(">", string(resp2.GetContent()))
}
// Self explanatory
outOfBounds := &Exchange{Header: []*Header{&Header{Value: &Header_UserId{UserId: 42}}}}
if resp3, err := service.UserInfo(outOfBounds); err != nil {
logger.Printf("Main: error was returned by service: %s\n", err.Error())
} else {
fmt.Println(">", string(resp3.GetContent()))
}
}
type daoMock struct {
Users  []string
Logger *log.Logger
}
func (d *daoMock) Get(id int) (*string, error) {
d.Logger.Println("DAO: Retrieving data…")
if id > len(d.Users) {
d.Logger.Println("DAO: User not in 'database'...")
return nil, fmt.Errorf("id %d not in users", id)
}
d.Logger.Println("DAO: Returning data…")
return &d.Users[id], nil
}
type Miniservice struct {
Logger *log.Logger
DAO    *daoMock
}
func (s *Miniservice) UserInfo(in *Exchange) (out *Exchange, err error) {
var idHdr *Header_UserId
s.Logger.Println("UserInfo: retrieving ID header")
// Here is where the magic happens:
// You Identify different types of requests by the presence or absence
// of certain headers
for _, hdr := range in.GetHeader() {
v := hdr.GetValue()
if i, ok := v.(*Header_UserId); ok {
idHdr = i
}
}
if idHdr == nil {
s.Logger.Println("UserInfo: invalid request")
return nil, fmt.Errorf("invalid request")
}
u, err := s.DAO.Get(int(idHdr.UserId))
if err != nil {
s.Logger.Printf("UserInfo: accessing user data: %s", err.Error())
return nil, fmt.Errorf("error accessing user data: %s", err.Error())
}
/* ----------------- create the response ----------------- */
statusHeader := &Header{
Name:  &statusName,
Value: &Header_Status{Status: Status_DONE_OK},
}
userHeader := &Header{
Name:  &userIdName,
Value: &Header_UserId{UserId: idHdr.UserId},
}
s.Logger.Println("UserInfo: sending response")
return &Exchange{
Header:  []*Header{statusHeader, userHeader},
Content: []byte(*u),
}, nil
}

Now, your Requests and Responses are more generic and are suitable for being used in various types of requests, without changing the format and without the need for reflection. I am not saying that this is the golden bullet, however. Others might come up with solutions which better fit your needs. But I hth.

I ended up completely abandoning reflect. I could work on generic objects, but I could not pass them on to handlers. Not being able to do this made it not worth using a library, so it seemed like a bad approach.

I created a simple "template" that I could copy and drop in the protobuf message names into. I then used go generate to build the messages that I needed. This let me put special go generate comments in my code that specified the types - so even though there is templating, filling it in and using it is done in a single go file.

So I put the base template in src/mylibs/req-handlers/base.tmp.go. I wanted to keep .go as the extension for syntax highlighting. In that file, I had generic things like {{RequestProto}} that would get replaced.

This script defined a ReqHandler type using some template variables:

type ReqHandlerFunc func(req *testmsg.{{RequestProto}}, resp *testmsg.{{ResponseProto}}) error

And I created an object that reference the handler function:

func NewReqHandler(name string, handler ReqHandlerFunc) *ReqHandler {
...
rh.handler = handler
return rh
}

and later in the code I called the handler function where it was needed:

err = rh.handler(req, resp)

In the bin directory, I added this script, which copies the template, and uses sed to replace some keywords with words I can specify in go code:

#!/bin/bash
if [ "$#" -ne 3 ] && [ "$#" -ne 4 ]; then
echo "Usage: build_handler (Package Name) (Request Proto Name) (Response Proto Name) [Logger Name]"
exit 1
fi
LIB=$1
REQ=$2
REP=$3
PKG="${LIB//-/}"
if [ "$#" -ne 4 ]; then
LOG=${PKG}
else
LOG=$4
fi
HANDLERS_DIR=$(dirname "$0")/../src/mylibs/req-handlers
#Generate go code
mkdir -p ${HANDLERS_DIR}/${LIB}/
GEN_FILE=${HANDLERS_DIR}/${LIB}/${LIB}_handler.go
cp ${HANDLERS_DIR}/base.tmpl.go ${GEN_FILE}
sed -i"" -e "s/{{PackageName}}/${PKG}/g" ${GEN_FILE}
sed -i"" -e "s/{{LoggerName}}/${LOG}/g" ${GEN_FILE}
sed -i"" -e "s/{{RequestProto}}/${REQ}/g" ${GEN_FILE}
sed -i"" -e "s/{{ResponseProto}}/${REP}/g" ${GEN_FILE}

Finally to make use of it, testservice.go would then have something like:

//go:generate build_handler testservicelib PostReq PostResp
import "mylibs/req-handlers/testservicelib"

When I run go generate, build_handler is called, which creates the mylibs/req-handlers/testservicelib library, which has a request handler with the PostReq and PostResp types. So I create a handler function that will have those as inputs:

func handleRequest(req *testmsg.PostReq, resp *testmsg.PostResp) error {
...
}

and pass that to my generated library:

reqHandler := testservicelib.NewReqHandler("test", handleRequest)

And life is good.

To build, the Makefile needed an extra step. Both go generate and go build/install steps are needed:

go generate testservice
go install testservice

Note that the generate call will run all the //go:generate comments in testservice.go, so in some cases I have more than 1 handler created.