如何反序列化非标准大小的字段?

huangapple go评论114阅读模式
英文:

How to deserialize a non-standard size field?

问题

我必须对来自另一个应用程序的一些二进制消息进行反序列化。我很想使用restruct.io,但是消息结构中的一些字段使用了“非标准”的位数(5位、3位、10位等)。

有没有办法处理这种类型的结构?我已经搜索了一段时间,没有任何成功的结果,所以任何帮助都将非常受欢迎。

提前感谢。

我将尝试给出一个示例来澄清我的问题。给定以下代码:

  1. package main
  2. import (
  3. "encoding/binary"
  4. "fmt"
  5. restruct "gopkg.in/restruct.v1"
  6. )
  7. type MessageType uint8
  8. const (
  9. MessageTypeOne MessageType = iota + 1
  10. MessageTypeTwo
  11. MessageTypeThree
  12. )
  13. // Message is the data to deserialize from the binary stream
  14. type Message struct {
  15. Length uint32 `struct:"uint32"` // message size in bytes (including length)
  16. Type MessageType `struct:"uint8"`
  17. Version uint8 `struct:"uint8:4"` // Just need 4 bits
  18. Subversion uint8 `struct:"uint8:2"` // just need 2 bits
  19. Optional uint8 `struct:"uint8:1"` // just one bit --> '1' means next field is NOT present
  20. NodeName string ``
  21. ANumber uint16 `struct:"uint16:10"` // just need 10 bits
  22. }
  23. // (length(4)+type(1)+(version(4bits)+Subversion(2bits)+Optional(1bit))) = 6 bytes
  24. // need 32bit alignment
  25. func main() {
  26. var inStream = []byte{0x08, // just 8 bytes needed
  27. 0x01, // messge type = MessageTypeOne
  28. 0x4a, // Version=0100 Subversion=10 Optional=1 ANumber = 0 (MSB bit)
  29. 0x00, 0x60, // ANumber(000 0000 011) Padding = 0 0000 for 32 bits alignment
  30. }
  31. var msg Message
  32. err := restruct.Unpack(inStream, binary.BigEndian, &msg)
  33. if err != nil {
  34. panic(err)
  35. }
  36. fmt.Println(msg)
  37. // Expected:
  38. // msg.Length = 8
  39. // msg.Type = 1
  40. // msg.Version = 4
  41. // msg.Subversion = 2
  42. // msg.Optional = 1
  43. // msg.NodeName = ""
  44. // msg.ANumber = 3
  45. }

我将从TCP连接接收inStream,并希望对二进制数据进行反序列化,并获得一个具有预期值的Message结构体。

希望这能澄清我的问题。

再次感谢;)

英文:

I have to deserialize some binary messages coming from another application. I would love to use restruct.io but some fields in the message structure use a "non-standard" number of bits ( 5 bits, 3 bits, ... 10 bits ... ).

Is there any way to handle this type of structs? I have been searching for some time without any success so any help will be very welcomed.

thanks in advance

I wil try to give an example to clarify my question. Given the code:

  1. package main
  2. import (
  3. "encoding/binary"
  4. "fmt"
  5. restruct "gopkg.in/restruct.v1"
  6. )
  7. type MessageType uint8
  8. const (
  9. MessageTypeOne MessageType = iota + 1
  10. MessageTypeTwo
  11. MessageTypeThree
  12. )
  13. // Message is the data to deserialize from the binary stream
  14. type Message struct {
  15. Length uint32 `struct:"uint32"` // message size in bytes (including length)
  16. Type MessageType `struct:"uint8"`
  17. Version uint8 `struct:"uint8:4"` // Just need 4 bits
  18. Subversion uint8 `struct:"uint8:2"` // just need 2 bits
  19. Optional uint8 `struct:"uint8:1"` // just one bit --> '1' means next field is NOT present
  20. NodeName string ``
  21. ANumber uint16 `struct:"uint16:10"` // just need 10 bits
  22. }
  23. // (length(4)+type(1)+(version(4bits)+Subversion(2bits)+Optional(1bit))) = 6 bytes
  24. // need 32bit alignment
  25. func main() {
  26. var inStream = []byte{0x08, // just 8 bytes needed
  27. 0x01, // messge type = MessageTypeOne
  28. 0x4a, // Version=0100 Subversion=10 Optional=1 ANumber = 0 (MSB bit)
  29. 0x00, 0x60, // ANumber(000 0000 011) Padding = 0 0000 for 32 bits alignment
  30. }
  31. var msg Message
  32. err := restruct.Unpack(inStream, binary.BigEndian, &msg)
  33. if err != nil {
  34. panic(err)
  35. }
  36. fmt.Println(msg)
  37. // Expected:
  38. // msg.Length = 8
  39. // msg.Type = 1
  40. // msg.Version = 4
  41. // msg.Subversion = 2
  42. // msg.Optional = 1
  43. // msg.NodeName = ""
  44. // msg.ANumber = 3
  45. }

I will receive inStream from a TCP connection and will want to deserialize the binary data and get a Message struct with the expected values ...

Hope this will clarify my question.

Thanks again 如何反序列化非标准大小的字段?

答案1

得分: 1

虽然可能没有通用的包来实现这种自定义结构的打包,但你可以很容易地创建自己的方法,仅提取每个字段所需的位。

  1. func (m *Message) UnmarshalBinary(data []byte) error {
  2. m.Length = binary.BigEndian.Uint32(data[:4])
  3. if int(m.Length) > len(data) {
  4. return fmt.Errorf("not enough bytes")
  5. }
  6. m.Type = MessageType(data[4])
  7. m.Version = data[5] >> 4
  8. m.Subversion = data[5] >> 2 & 0x03
  9. m.Optional = data[5] >> 1 & 0x01
  10. // 如果有可选字符串,将 ANumber 的索引移回
  11. idx := 6
  12. if m.Optional == 0 {
  13. // 移除 ANumber 的最后两个字节
  14. end := int(m.Length) - 2
  15. m.NodeName = string(data[6:end])
  16. idx = end
  17. }
  18. m.ANumber = uint16(data[idx]&0xc0)<<2 | uint16(data[idx]&0x3f<<2|data[idx+1]>>6)
  19. return nil
  20. }

当然,你可以添加更多的边界检查,以返回错误,而不是在索引越界时引发 panic。

我稍微修改了你的 inStream 切片以匹配你的定义,你可以在这里看到示例输出:https://play.golang.org/p/FoNoazluOF

英文:

While there's probably no generic package to implement this custom struct packing, you can easily create your own method extracting just the bits required for each field.

  1. func (m *Message) UnmarshalBinary(data []byte) error {
  2. m.Length = binary.BigEndian.Uint32(data[:4])
  3. if int(m.Length) &gt; len(data) {
  4. return fmt.Errorf(&quot;not enough bytes&quot;)
  5. }
  6. m.Type = MessageType(data[4])
  7. m.Version = data[5] &gt;&gt; 4
  8. m.Subversion = data[5] &gt;&gt; 2 &amp; 0x03
  9. m.Optional = data[5] &gt;&gt; 1 &amp; 0x01
  10. // move the index for ANumber back if there&#39;s an optional string
  11. idx := 6
  12. if m.Optional == 0 {
  13. // remove the last two bytes for ANumber
  14. end := int(m.Length) - 2
  15. m.NodeName = string(data[6:end])
  16. idx = end
  17. }
  18. m.ANumber = uint16(data[idx]&amp;0xc0)&lt;&lt;2 | uint16(data[idx]&amp;0x3f&lt;&lt;2|data[idx+1]&gt;&gt;6)
  19. return nil

}

You can of course add more bound checks to return errors rather than letting this panic when indexing out of bounds.

I modified your inStream slice slightly to match your definition, and you can see the example output here: https://play.golang.org/p/FoNoazluOF

答案2

得分: 0

我一直在为restruct.io编写一些补丁,以便能够处理位字段...尽管还没有完全测试,但似乎可以工作...

一旦测试完成,我将尝试发送一个拉取请求...

  1. func (e *encoder) writeBits(f field, inBuf []byte) {
  2. var inputLength uint8 = uint8(len(inBuf))
  3. if f.BitSize == 0 {
  4. // Having problems with complex64 type ... so we asume we want to read all
  5. //f.BitSize = uint8(f.Type.Bits())
  6. f.BitSize = 8 * inputLength
  7. }
  8. // destPos: Destination position ( in the result ) of the first bit in the first byte
  9. var destPos uint8 = 8 - e.bitCounter
  10. // originPos: Original position of the first bit in the first byte
  11. var originPos uint8 = f.BitSize % 8
  12. if originPos == 0 {
  13. originPos = 8
  14. }
  15. // numBytes: number of complete bytes to hold the result
  16. var numBytes uint8 = f.BitSize / 8
  17. // numBits: number of remaining bits in the first non-complete byte of the result
  18. var numBits uint8 = f.BitSize % 8
  19. // number of positions we have to shift the bytes to get the result
  20. var shift uint8
  21. if originPos > destPos {
  22. shift = originPos - destPos
  23. } else {
  24. shift = destPos - originPos
  25. }
  26. shift = shift % 8
  27. var inputInitialIdx uint8 = inputLength - numBytes
  28. if numBits > 0 {
  29. inputInitialIdx = inputInitialIdx - 1
  30. }
  31. if originPos < destPos {
  32. // shift left
  33. carry := func(idx uint8) uint8 {
  34. if (idx + 1) < inputLength {
  35. return (inBuf[idx+1] >> (8 - shift))
  36. }
  37. return 0x00
  38. }
  39. mask := func(idx uint8) uint8 {
  40. if idx == 0 {
  41. return (0x01 << destPos) - 1
  42. }
  43. return 0xFF
  44. }
  45. var idx uint8 = 0
  46. for inIdx := inputInitialIdx; inIdx < inputLength; inIdx++ {
  47. e.buf[idx] |= ((inBuf[inIdx] << shift) | carry(inIdx)) & mask(idx)
  48. idx++
  49. }
  50. } else {
  51. // originPos >= destPos => shift right
  52. var idx uint8 = 0
  53. // carry : is a little bit tricky in this case because of the first case
  54. // when idx == 0 and there is no carry at all
  55. carry := func(idx uint8) uint8 {
  56. if idx == 0 {
  57. return 0x00
  58. }
  59. return (inBuf[idx-1] << (8 - shift))
  60. }
  61. mask := func(idx uint8) uint8 {
  62. if idx == 0 {
  63. return (0x01 << destPos) - 1
  64. }
  65. return 0xFF
  66. }
  67. inIdx := inputInitialIdx
  68. for ; inIdx < inputLength; inIdx++ {
  69. //note: Should the mask be done BEFORE the OR with carry?
  70. e.buf[idx] |= ((inBuf[inIdx] >> shift) | carry(inIdx)) & mask(idx)
  71. idx++
  72. }
  73. if ((e.bitCounter + f.BitSize) % 8) > 0 {
  74. e.buf[idx] |= carry(inIdx)
  75. }
  76. }
  77. //now we should update buffer and bitCounter
  78. e.bitCounter = (e.bitCounter + f.BitSize) % 8
  79. // move the head to the next non-complete byte used
  80. headerUpdate := func() uint8 {
  81. if (e.bitCounter == 0) && ((f.BitSize % 8) != 0) {
  82. return (numBytes + 1)
  83. }
  84. return numBytes
  85. }
  86. e.buf = e.buf[headerUpdate():]
  87. return
  88. }
英文:

I have been working on some patch for restruct.io to be able to work with bitfields .... Still not fully tested but seems to work ...

Will try to send a pull request once tested ...

  1. func (e *encoder) writeBits(f field, inBuf []byte) {
  2. var inputLength uint8 = uint8(len(inBuf))
  3. if f.BitSize == 0 {
  4. // Having problems with complex64 type ... so we asume we want to read all
  5. //f.BitSize = uint8(f.Type.Bits())
  6. f.BitSize = 8 * inputLength
  7. }
  8. // destPos: Destination position ( in the result ) of the first bit in the first byte
  9. var destPos uint8 = 8 - e.bitCounter
  10. // originPos: Original position of the first bit in the first byte
  11. var originPos uint8 = f.BitSize % 8
  12. if originPos == 0 {
  13. originPos = 8
  14. }
  15. // numBytes: number of complete bytes to hold the result
  16. var numBytes uint8 = f.BitSize / 8
  17. // numBits: number of remaining bits in the first non-complete byte of the result
  18. var numBits uint8 = f.BitSize % 8
  19. // number of positions we have to shift the bytes to get the result
  20. var shift uint8
  21. if originPos &gt; destPos {
  22. shift = originPos - destPos
  23. } else {
  24. shift = destPos - originPos
  25. }
  26. shift = shift % 8
  27. var inputInitialIdx uint8 = inputLength - numBytes
  28. if numBits &gt; 0 {
  29. inputInitialIdx = inputInitialIdx - 1
  30. }
  31. if originPos &lt; destPos {
  32. // shift left
  33. carry := func(idx uint8) uint8 {
  34. if (idx + 1) &lt; inputLength {
  35. return (inBuf[idx+1] &gt;&gt; (8 - shift))
  36. }
  37. return 0x00
  38. }
  39. mask := func(idx uint8) uint8 {
  40. if idx == 0 {
  41. return (0x01 &lt;&lt; destPos) - 1
  42. }
  43. return 0xFF
  44. }
  45. var idx uint8 = 0
  46. for inIdx := inputInitialIdx; inIdx &lt; inputLength; inIdx++ {
  47. e.buf[idx] |= ((inBuf[inIdx] &lt;&lt; shift) | carry(inIdx)) &amp; mask(idx)
  48. idx++
  49. }
  50. } else {
  51. // originPos &gt;= destPos =&gt; shift right
  52. var idx uint8 = 0
  53. // carry : is a little bit tricky in this case because of the first case
  54. // when idx == 0 and there is no carry at all
  55. carry := func(idx uint8) uint8 {
  56. if idx == 0 {
  57. return 0x00
  58. }
  59. return (inBuf[idx-1] &lt;&lt; (8 - shift))
  60. }
  61. mask := func(idx uint8) uint8 {
  62. if idx == 0 {
  63. return (0x01 &lt;&lt; destPos) - 1
  64. }
  65. return 0xFF
  66. }
  67. inIdx := inputInitialIdx
  68. for ; inIdx &lt; inputLength; inIdx++ {
  69. //note: Should the mask be done BEFORE the OR with carry?
  70. e.buf[idx] |= ((inBuf[inIdx] &gt;&gt; shift) | carry(inIdx)) &amp; mask(idx)
  71. idx++
  72. }
  73. if ((e.bitCounter + f.BitSize) % 8) &gt; 0 {
  74. e.buf[idx] |= carry(inIdx)
  75. }
  76. }
  77. //now we should update buffer and bitCounter
  78. e.bitCounter = (e.bitCounter + f.BitSize) % 8
  79. // move the head to the next non-complete byte used
  80. headerUpdate := func() uint8 {
  81. if (e.bitCounter == 0) &amp;&amp; ((f.BitSize % 8) != 0) {
  82. return (numBytes + 1)
  83. }
  84. return numBytes
  85. }
  86. e.buf = e.buf[headerUpdate():]
  87. return
  88. }

huangapple
  • 本文由 发表于 2016年12月12日 23:22:09
  • 转载请务必保留本文链接:https://go.coder-hub.com/41104049.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定