Error while using proto.Unmarshal for Protocol Buffers data: "string field contains invalid UTF-8

I'm working on a Go application that uses Protocol Buffers, and I'm encountering an issue while deserializing the data using proto.Unmarshal. The specific error message I'm getting is "string field contains invalid UTF-8".

Here is my deserialization code:

import (
    "github.com/example/pb" // Import the package containing the Protocol Buffers definition.
    "google.golang.org/protobuf/proto"
)

func deserializeData(res_data *pb.ResponseData) (*pb.NotfnMessg, error) {
    notfnMessg := &pb.NotfnMessg{}

    err := proto.Unmarshal(res_data.Payload.GetSerializedMessage(), notfnMessg)
    if err != nil {
        appLogger.Logging().Error("Error Unmarshalling the message: %v", err)
    }
    appLogger.Logging().Info("%v, %v", notfnMessg, res_data.Payload.GetSerializedMessage())
}

The res_data.Payload.GetSerializedMessage() function returns the serialized message data in a string format. The content of res_data is provided below as a demo.

Here's an example of the serialized data stored in res_data:

msg_hdr:{s_val:"CONFIG_GET_REP"}  payload:{payload_header:{s_val:"SUCCESS"}  serialized_message:"\n\xc3\x0b...<omitted for brevity>.../communication\n*\n\x05title\x12!\n\x17Pi Module ConfigurationZ\x06/title"}

As you can see, the serialized message contains binary data, including strings. It appears that there might be invalid UTF-8 characters in one of the string fields, causing the deserialization to fail.

I would like to know how to properly handle this issue and successfully deserialize the data without encountering the "string field contains invalid UTF-8" error.

Any guidance or suggestions on how to handle invalid UTF-8 characters during the deserialization process would be greatly appreciated. Thank you!

When deserializing a string field the Go protobuf code (and others) validate that strings are UTF-8. When they are not you get the error you are seeing.

As far as I am aware the simplest way around this is to edit the .proto and change the string fields to bytes, then rerun protoc (or buf etc) to regenerate your github.com/example/pb. string and bytes fields are wire compatible the difference is in the output types.

This will give you the data as a []byte in Go, you can then process it yourself (e.g. strings.ToValidUTF8("a\xc5z", "") as per this answer)