英文:
Correctly log protobuf messages as unescaped JSON with zap logger
问题
我有一个Go项目,在其中使用Zap结构化日志记录结构的内容。这是我初始化日志记录器的方式:
zapLog, err := zap.NewProductionConfig().Build()
if err != nil {
panic(err)
}
最初,我使用自己的带有json标签的结构体,一切都运行得很完美:
zapLog.Info("Event persisted", zap.Any("event", &event))
结果:
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
"msg":"Event persisted","event":{"sourceType":4, "sourceId":"some-source-id",
"type":"updated", "value":"{...}", "context":{"foo":"bar"}}}
现在我切换到了protobuf,我正在努力实现相同的结果。最初,当使用zap.Any()时,我只得到了"reflected map"版本:
zapLog.Info("Event persisted", zap.Any("event", &event))
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
"msg":"Event persisted","event":"sourceType:TYPE_X sourceId:\"some-source-id\",
type:\"updated\" value:{...}, context:<key: foo, value:bar>"}
我尝试使用jsonpb编组器对对象进行编组,它本身生成了正确的输出,但是当我在zap.String()
中使用它时,字符串被转义了,所以在每个引号前面多了一组'\'。由于日志在稍后的处理中进行处理,这在那里引起了问题,因此我想避免它:
m := jsonpb.Marshaler{}
var buf bytes.Buffer
if err := m.Marshal(&buf, msg); err != nil {
// 处理错误
}
zapLog.Info("Event persisted", zap.ByteString("event", buf.Bytes()))
结果:
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
"msg":"Event persisted","event":"{\"sourceType\":\"TYPE_X\", \"sourceId\":\"some-source-id\",
\"type\":\"updated\", \"value\":\"{...}\", \"context\":{\"foo\":\"bar\"}}"}
然后,我尝试使用zap.Reflect()
而不是zap.Any()
,这是我能够接近所需结果的最接近的方法,只是枚举值被呈现为它们的数值(初始解决方案中没有枚举值,因此在protobuf之前的解决方案中也不起作用):
zapLog.Info("Event persisted", zap.Reflect("event", &event))
结果:
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
"msg":"Event persisted","event":{"sourceType":4, "sourceId":"some-source-id",
"type":"updated", "value":"{...}", "context":{"foo":"bar"}}}
到目前为止,我唯一看到的选择是编写自己的MarshalLogObject()
函数:
type ZapEvent struct {
event *Event
}
func (z *ZapEvent) MarshalLogObject(encoder zapcore.ObjectEncoder) error {
encoder.AddString("sourceType", z.event.SourceType.String()
// 为每个属性实现编码器
}
func processEvent(e Event) {
...
zapLog.Info("Event persisted", zap.Object("event", &ZapEvent{event: &e}))
}
但由于它是一个复杂的结构体,我宁愿使用一个不那么容易出错和维护繁重的解决方案。理想情况下,我希望告诉zap以某种方式使用jsonpb编组器,但我不知道是否可能。
英文:
I have a Go project where I'm using Zap structured logging to log the contents of structs. That's how I initialise the logger:
zapLog, err := zap.NewProductionConfig().Build()
if err != nil {
panic(err)
}
Initially I started with my own structs with json tags and it all worked perfectly:
zapLog.Info("Event persisted", zap.Any("event", &event))
Result:
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
"msg":"Event persisted","event":{"sourceType":4, "sourceId":"some-source-id",
"type":"updated", "value":"{...}", "context":{"foo":"bar"}}}
I now switched to protobuf and I'm struggling to achieve the same result. Initially I just got the "reflected map" version, when using zap.Any():
zapLog.Info("Event persisted", zap.Any("event", &event))
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
"msg":"Event persisted","event":"sourceType:TYPE_X sourceId:\"some-source-id\",
type:\"updated\" value:{...}, context:<key: foo, value:bar>}
I tried marshalling the object with the jsonpb marshaller, which generated the correct output on itself, however, when I use it in zap.String()
, the string is escaped, so I get an extra set of '\' in front of each quotation mark. Since there's processing of the logs at a later point, this causes problems there and hence I want to avoid it:
m := jsonpb.Marshaler{}
var buf bytes.Buffer
if err := m.Marshal(&buf, msg); err != nil {
// handle error
}
zapLog.Info("Event persisted", zap.ByteString("event", buf.Bytes()))
Result:
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
"msg":"Event persisted","event":"{\"sourceType\":\"TYPE_X\", \"sourceId\":\"some-source-id\",
\"type\":\"updated\", \"value\":\"{...}\", \"context\":{\"foo\":"bar\"}}"}
I then tried using zap.Reflect()
instead of zap.Any()
which was the closest thing I could get to what I need, except that enums are rendered as their numerical values (the initial solution did not have enums, so that didn't work in the pre-protobuf solution either):
zapLog.Info("Event persisted", zap.Reflect("event", &event))
Result:
{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
"msg":"Event persisted","event":{"sourceType":4, "sourceId":"some-source-id",
"type":"updated", "value":"{...}", "context":{"foo":"bar"}}}
The only option I see so far is to write my own MarshalLogObject()
function:
type ZapEvent struct {
event *Event
}
func (z *ZapEvent) MarshalLogObject(encoder zapcore.ObjectEncoder) error {
encoder.AddString("sourceType", z.event.SourceType.String()
// implement encoder for each attribute
}
func processEvent(e Event) {
...
zapLog.Info("Event persisted", zap.Object("event", &ZapEvent{event: &e}))
}
But since it's a complex struct, I would rather use a less error prone and maintenance heavy solution. Ideally, I would tell zap to use the jsonpb marshaller somehow, but I don't know if that's possible.
答案1
得分: 3
使用zap.Any
和json.RawMessage
。你可以直接将jsonpb.Marshaler
的字节输出进行转换:
foo := &pb.FooMsg{
Foo: "blah",
Bar: 1,
}
m := jsonpb.Marshaler{}
var buf bytes.Buffer
if err := m.Marshal(&buf, foo); err != nil {
// 处理错误
}
logger, _ := zap.NewDevelopment()
logger.Info("事件已持久化", zap.Any("event", json.RawMessage(buf.Bytes())))
字节将被打印为:
事件已持久化 {"event": {"foo":"blah","bar":"1"}}
我认为这是最简单的方法,但我也知道一个包kazegusuri/go-proto-zap-marshaler
(我与之无关),它作为protoc插件生成MarshalLogObject()
实现。你也可以看看那个。
英文:
Use zap.Any
with a json.RawMessage
. You can convert directly the byte output of jsonpb.Marshaler
:
foo := &pb.FooMsg{
Foo: "blah",
Bar: 1,
}
m := jsonpb.Marshaler{}
var buf bytes.Buffer
if err := m.Marshal(&buf, foo); err != nil {
// handle error
}
logger, _ := zap.NewDevelopment()
logger.Info("Event persisted", zap.Any("event", json.RawMessage(buf.Bytes())))
The bytes will be printed as:
> Event persisted {"event": {"foo":"blah","bar":"1"}}`
I believe that's the easiest way, however I'm also aware of a package kazegusuri/go-proto-zap-marshaler
(I'm not affiliated to it) that generates MarshalLogObject()
implementations as a protoc plugin. You may want to take a look at that too.
答案2
得分: 1
我使用了另一种方法来将 protos 转换为 JSON。
由于 protos 可以自然地进行编组,我只是将它们包装在严格的 JSON 编组器中。
你可以修改内部使用 protojson(较新的 jsonpb)。
与之前的解决方案中的编组器不同,这个编组器不需要预先处理。
type jsonObjectMarshaler struct {
obj any
}
func (j *jsonObjectMarshaler) MarshalJSON() ([]byte, error) {
bytes, err := json.Marshal(j.obj)
// bytes, err := protojson.Marshal(j.obj)
if err != nil {
return nil, fmt.Errorf("json marshaling failed: %w", err)
}
return bytes, nil
}
func ZapJsonable(key string, obj any) zap.Field {
return zap.Reflect(key, &jsonObjectMarshaler{obj: obj})
}
然后使用它,只需:
logger, _ := zap.NewDevelopment()
logger.Info("Event persisted", ZapJsonable("event", buf))
英文:
I used another way to jsonify protos.
Since protos can be naturally marshaled, I just wrapped them in the strict-to-json marshaler.
And you can modify the internals to use protojson (newer jsonpb).
Unlike the marshaler in the previous solution, this one doesn't require ahead-of-logging processing.
type jsonObjectMarshaler struct {
obj any
}
func (j *jsonObjectMarshaler) MarshalJSON() ([]byte, error) {
bytes, err := json.Marshal(j.obj)
// bytes, err := protojson.Marshal(j.obj)
if err != nil {
return nil, fmt.Errorf("json marshaling failed: %w", err)
}
return bytes, nil
}
func ZapJsonable(key string, obj any) zap.Field {
return zap.Reflect(key, &jsonObjectMarshaler{obj: obj})
}
Then to use it, just
logger, _ := zap.NewDevelopment()
logger.Info("Event persisted", ZapJsonable("event", buf))
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论