接收部分写入的Protobuf编码消息?

huangapple go评论89阅读模式
英文:

Receive protobuf encoded messages that can be partially written?

问题

我正在尝试在GoLang中通过TCP发送和接收使用Protobuf编码的消息,其中发送方可以在操作进行到一半时取消write(),而接收方可以正确接收部分消息。

请注意,我使用单个TCP连接来发送不同用户定义类型的消息,无限次数(这不是每个连接的消息情况)。

为了具体解释我的问题,首先我将介绍如何在没有部分写入的情况下实现发送/接收。

在我的程序中,有多种类型的消息,在一个.proto文件中定义。我将解释一种这样的消息类型的机制。

message MessageType {
  int64 sender = 1;
  int64 receiver = 2;
  int64 operation = 3;
  string message = 4;
}

然后我使用Golang Protobuf插件生成存根。

然后在发送方,以下是我发送消息的方式。

func send(w *bufio.Writer, code uint8, oriMsg MessageType) {
    err := w.WriteByte(code)
	data, err := proto.Marshal(oriMsg)
    lengthWritten := len(data)
    var b [8]byte
    bs := b[:8]
    binary.LittleEndian.PutUint64(bs, uint64(lengthWritten))
    _, err = w.Write(bs)
    _, err = w.Write(data)
    w.flush()
}

然后在接收方,以下是我接收消息的方式。

reader *bufio.Reader
for true {
		if msgType, err = reader.ReadByte(); err != nil {
			panic()
		}
		if msgType == 1 || msgType == 2{
			var b [8]byte
            bs := b[:8]

        	_, err := io.ReadFull(reader, bs) 
        	numBytes := binary.LittleEndian.Uint64(bs)
            data := make([]byte, numBytes)
            length, err := io.ReadFull(reader, data) 
            msg *MessageType = new(GenericConsensus) // an empty message 
            err = proto.Unmarshal(data[:length], msg) 
            // do something with the message 
			
		} else {
			// unknown message type handler
		}
	}

现在我的问题是,如果发送方在中途中止写入会怎么样,更具体地说,

  1. 情况1:如果发送方写入消息类型字节,然后中止怎么办?在这种情况下,接收方将读取消息类型字节,并等待接收8字节的消息长度,但发送方没有发送它。

  2. 情况2:这是情况1的扩展版本,其中发送方首先只发送消息类型字节,然后中止发送消息长度和编组消息,然后发送下一条消息:类型字节、长度和编码消息。现在在接收方,一切都出错了,因为消息的顺序(类型、长度和编码消息)被违反了。

所以我的问题是,我如何修改接收方,使其能够在发送方违反预先约定的类型:长度:编码消息顺序的情况下继续操作?

谢谢。

英文:

I am trying to send and receive protobuff encoded messages in GoLang over TCP, where the sender can cancel the write() halfway through the operation, and the receiver can correctly receive partial messages.

Note that I use a single TCP connection to send messages of different user defined types, infinitely (this is not a per connection message case)

To explain my question concretely, first I will present how I implement the send/receive without partial writes.

In my program, there are multiple types of messages, defined in a .proto file. I will explain the mechanism for one such message type.

message MessageType {
  int64 sender = 1;
  int64 receiver = 2;
  int64 operation = 3;
  string message = 4;
}

Then I use Golang Protobuf plugin to generate the stubs.

Then in the sender side, the following is how I send.

func send(w *bufio.Writer, code uint8, oriMsg MessageType) {
    err := w.WriteByte(code)
	data, err := proto.Marshal(oriMsg)
    lengthWritten := len(data)
    var b [8]byte
    bs := b[:8]
    binary.LittleEndian.PutUint64(bs, uint64(lengthWritten))
    _, err = w.Write(bs)
    _, err = w.Write(data)
    w.flush()
}

Then in the receiver side, the following is how I receive.

reader *bufio.Reader
for true {
		if msgType, err = reader.ReadByte(); err != nil {
			panic()
		}
		if msgType == 1 || msgType == 2{
			var b [8]byte
            bs := b[:8]

        	_, err := io.ReadFull(reader, bs) 
        	numBytes := binary.LittleEndian.Uint64(bs)
            data := make([]byte, numBytes)
            length, err := io.ReadFull(reader, data) 
            msg *MessageType = new(GenericConsensus) // an empty message 
            err = proto.Unmarshal(data[:length], msg) 
            // do something with the message 
			
		} else {
			// unknown message type handler
		}
	}

Now my question is, what if the sender aborts his writes in the middle: more concretely,

  1. Case 1: what if the sender writes the message type byte, and then abort? In this case the receiver will read the message type byte, and waits to receive an 8 byte message length, but the sender doesn't send it.

  2. Case 2: This is an extended version of case 1 where the sender first sends only the message type byte, and the aborts sending the message length and marshaled message, and then send the next message: the type byte, the length and encoded message. Now in the receiver side, everything goes wrong because the order of messages (type, length and encoded message) is violated.

So my question is, how can I modify the receiver such that it can continue to operate despite the sender violating the pre-agreed order of type:length:encoded-message?

Thanks

答案1

得分: 2

为什么发送方会中止一条消息,然后发送另一条消息?你是指发送方完全是拜占庭式的吗?还是在准备模糊测试?

如果你的API协议规定发送方必须始终发送正确的消息,那么接收方可以简单地忽略错误的消息,甚至在发现API协议违规时关闭连接。

如果你真的需要,以下是一些使其工作的想法:

  • 使用唯一的前导标记,但是你必须确保这个前导标记在数据中不会出现
  • 在发送消息到解码器之前,在消息中添加一个校验和。因此完整的数据包将是:[消息类型:消息长度:消息:校验和]。这样接收方就可以检查它是正确的消息还是格式错误的消息。

另外,目前的代码很容易因为发送一个64位的大小而导致崩溃。所以你还应该检查大小是否在一个有用的范围内。我会将其限制在32位内...

英文:

Why would the sender abort a message, but then send another message? You mean it's a fully byzantine sender? Or are you preparing for fuzzy-testing?

If your API contract says that the sender always needs to send a correct message, then the receiver can simply ignore wrong messages, or even close the connection if it sees a violation of the API contract.

If you really need it, here some ideas of how you could make it work:

  • start with a unique preamble - but then you will have to make sure this preamble never comes up in the data
  • add a checksum to the message before sending it to the decoder. So the full packet would be: [msg_type : msg_len : msg : chksum ]. This allows the receiver to check whether it's a correct message or a misformed one.

Also, as the code is currently, it is quite easy to crash by sending a size with the maximum of 64 bits. So you should also check for the size to be in a useful range. I would limit it to 32 bits...

huangapple
  • 本文由 发表于 2022年3月23日 20:55:50
  • 转载请务必保留本文链接:https://go.coder-hub.com/71587634.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定