strategy for REST API in go

In my database, each row corresponds to a struct

type datum struct{
    Id *string `json:"task_id"`
    Status *string `json:"status"`
    AccountId *string `json:"account_id"`
    .... // many more fields, all of pointer types
}

On the webpage, the user can query on several fields of datum (say account_id and status). The server will return all data that satisfy the query with a projection of the fields (say Id, account_id and status).

Right now, I wrote a HTTP handler to

Extract the query as a datum object from the request:

body, err := ioutil.ReadAll(r.Body)
condition := datum{} 
err = json.Unmarshal(body, &condition)

Use the partially filled datum object to query the database, only the non-nil fields translate to SELECT ... WHERE ..=... The query result is saved in query_result []datum

Write the query_result into json object for reply:

reply := map[string]interface{}{
			"reply": query_result,
		}
data, err := json.Marshal(reply)

The problem is that in the reply many of the fields are nil, but I still send them, which is wasteful. On the other hand, I don't want to change the datum struct to include omitempty tag because in the database a value entry has all fields non-nil.

• In this case, shall I define a new struct just for the reply? Is there a way to define this new struct using datum struct, instead of hard code one?
• Is there a better design for this query feature?

You have several options, with choice depending what is more wasteful/expensive in your particular case:

1. Just use pointers+omitempty in the original struct.
2. Prepare a custom response object. But you'll need to copy/convert the values from the original struct into its export version.
3. Write a custom marshaller, that will be exploring your struct and creating an export-ready variant, this way being more dynamic/automatic that #1.

While #1 needs no comments, and #2 to some extend covered by Gepser above, here's how you can address this with a custom marshaller (the idea is to re-assemble your output skipping nil fields):

package main
import (
"fmt"
"encoding/json"
"reflect"
)
type datum struct {
Id        *string `json:"task_id"`
Status    *string `json:"status"`
AccountId *string `json:"account_id"`
}
type Response struct {
Reply []datum `json:"reply"`
}
func main() {
var query_result []datum
// mocking a query result with records with nil fields
val_id_a := "id-a"
val_status := "status-b"
d1 := datum{
Id:     &val_id_a,
Status: &val_status,
}
query_result = append(query_result, d1)
val_id_b := "id-b"
val_account_id := "account-id-b"
d2 := datum{
Id:        &val_id_b,
AccountId: &val_account_id,
}
query_result = append(query_result, d2)
reply := &Response{
Reply: query_result,
}
data, err := json.Marshal(reply)
if err != nil {
panic(err)
}
fmt.Printf("%+v\n", string(data))
}
// MarshalJSON is a custom JSON marshaller implementation for Response object.
func (r *Response) MarshalJSON() ([]byte, error) {
a := struct {
Reply []map[string]interface{} `json:"reply"`
}{}
for _, v := range r.Reply {
a.Reply = append(a.Reply, converter(v))
}
return json.Marshal(a)
}
// converter converts a struct into a map, skipping fields with nil values.
func converter(in interface{}) map[string]interface{} {
out := make(map[string]interface{})
v := reflect.ValueOf(in)
for i := 0; i < v.NumField(); i++ {
f := v.Type().Field(i)
tag := f.Tag.Get("json")
if tag != "" && !v.Field(i).IsNil() {
out[tag] = v.Field(i).Interface()
}
}
return out
}

The approach I suggest (is the one I use) is the new struct with omitempty tag, for example:

type datumResponse struct{
Id *string `json:"task_id,omitempty"`
Status *string `json:"status,omitempty"`
AccountId *string `json:"account_id,omitempty"`
.... // many more fields
}

and there is no option to write your new struct using the fields of the old one if there is not substructs or you don't write an array of structs.