goavro和原始的Go数据结构

huangapple
go评论118阅读模式
英文:

goavro and original Go data structure

问题

使用goavro库在Golang中如何使用avro重新创建原始数据结构？

使用这个库https://github.com/hamba/avro非常简单。

  1. out := SimpleRecord{}
  2. err = avro.Unmarshal(schema, data, &out)

变量out的类型是SimpleRecord。

假设我有以下结构体和avro模式：

  1. type SimpleRecord struct {
  2. 	F1           int      `avro:"f1"`
  3. 	F2           string   `avro:"f2"`
  4. 	F3           string   `avro:"f3"`
  5. 	Dependencies []string `avro:"dependencies"`
  6. }
  8. func main() {
  9. 	avro_schema_txt := `{
  10. 		"type": "record",
  11. 		"name": "AvroData",
  12. 		"namespace": "data.avro",
  13. 		"doc": "docstring",
  14. 		"fields": [
  15. 			{
  16. 				"name": "f1",
  17. 				"type": "int"
  18. 			},
  19. 			{
  20. 				"name": "f2",
  21. 				"type": "string"
  22. 			},
  23. 			{
  24. 				"name": "f3",
  25. 				"type": "string"
  26. 			},
  27. 			{
  28. 				"name": "dependencies",
  29. 				"type": {
  30. 					"type": "array",
  31. 					"items": "string"
  32. 				}
  33. 			}
  34. 		]
  35. 	}`
  36. }

然后使用以下代码：

  1. codec, err := goavro.NewCodec(avro_schema_txt)
  2. if err != nil {
  3. 	log.Fatal(err.Error())
  4. }
  5. out, _, err := codec.NativeFromBinary(data)
  6. if err != nil {
  7. 	log.Fatal(err.Error())
  8. }
  9. fmt.Println(out)

其中data是使用avro编组的，out的类型是interface{}，那么如何将其转换为SimpleRecord类型呢？

英文:

How can I recreate original data structure in Golang serialized with avro using goavro?

With this library https://github.com/hamba/avro it's quite easy.

  1. out := SimpleRecord{}
  2. err = avro.Unmarshal(schema, data, &out)

type of variable out is SimpleRecord.

Let's say I have this struct and avro schema:

  1. type SimpleRecord struct {
  2. 	F1           int      `avro:"f1"`
  3. 	F2           string   `avro:"f2"`
  4. 	F3           string   `avro:"f3"`
  5. 	Dependencies []string `avro:"dependencies"`
  6. }
  8. func main() {
  9. 	avro_schema_txt := `{
  10. 		"type": "record",
  11. 		"name": "AvroData",
  12. 		"namespace": "data.avro",
  13. 		"doc": "docstring",
  14. 		"fields": [
  15. 			{
  16. 				"name": "f1",
  17. 				"type": "int"
  18. 			},
  19. 			{
  20. 				"name": "f2",
  21. 				"type": "string"
  22. 			},
  23. 			{
  24. 				"name": "f3",
  25. 				"type": "string"
  26. 			},
  27. 			{
  28. 				"name": "dependencies",
  29. 				"type": {
  30. 					"type": "array",
  31. 					"items": "string"
  32. 				}
  33. 			}
  34. 		]
  35. 	}`
  36. }

and then

  1. codec, err := goavro.NewCodec(avro_schema_txt)
  2. 	if err != nil {
  3. 		log.Fatal(err.Error())
  4. 	}
  5. 	out, _, err := codec.NativeFromBinary(data)
  6. 	if err != nil {
  7. 		log.Fatal(err.Error())
  8. 	}
  9. 	fmt.Println(out)

where data is marshaled with avro, out is of type interface{}, so how can I "make" it SimpleRecord?

答案1

得分: 2

有两种方法可以实现，一种是将out强制转换为map[string]interface{}并获取值，另一种是使用codec.TextualFromNative。下面分别展示了这两种方法的代码：

方法一

将out从interface{}转换为map[string]interface{}并获取值：

  1. ...
  2. simpleMap := out.(map[string]interface{})
  3. f1 := simpleMap["f1"]
  4. f2 := simpleMap["f2"]
  5. ...
  7. SimpleRecord {
  8.     F1: f1,
  9.     F2: f2,
  10.     ...
  11. }

方法二

使用TextualFromNative，下面的代码展示了编码和解码的过程：

  1. var avro_schema_txt = `{
  2.     "type": "record",
  3.     "name": "AvroData",
  4.     "namespace": "data.avro",
  5.     "doc": "docstring",
  6.     "fields": [
  7.         {
  8.             "name": "f1",
  9.             "type": "int"
  10.         },
  11.         {
  12.             "name": "f2",
  13.             "type": "string"
  14.         },
  15.         {
  16.             "name": "f3",
  17.             "type": "string"
  18.         },
  19.         {
  20.             "name": "dependencies",
  21.             "type": {
  22.                 "type": "array",
  23.                 "items": "string"
  24.             }
  25.         }
  26.     ]
  27. }`
  29. // 添加json以匹配avro中的字段名
  30. type SimpleRecord struct {
  31.     F1           int      `avro:"f1" json:"f1"`
  32.     F2           string   `avro:"f2" json:"f2"`
  33.     F3           string   `avro:"f3" json:"f3"`
  34.     Dependencies []string `avro:"dependencies" json:"dependencies"`
  35. }
  37. func encodeDecode() {
  39.     data := SimpleRecord{
  40.         F1: 1,
  41.         F2: "tester2",
  42.         F3: "tester3",
  43.         Dependencies: []string { "tester4", "tester5" },
  44.     }
  46.     codec, err := goavro.NewCodec(avro_schema_txt)
  47.     if err != nil {
  48.         log.Fatal(err.Error())
  49.     }
  51.     // 编码
  53.     textualIn, err := json2.Marshal(data)
  54.     if err != nil {
  55.         log.Fatal(err.Error())
  56.     }
  58.     nativeIn, _, err := codec.NativeFromTextual(textualIn)
  59.     if err != nil {
  60.         log.Fatal(err.Error())
  61.     }
  63.     binaryIn, err := codec.BinaryFromNative(nil, nativeIn)
  64.     if err != nil {
  65.         log.Fatal(err.Error())
  66.     }
  68.     // 解码
  70.     nativeOut, _, err := codec.NativeFromBinary(binaryIn)
  71.     if err != nil {
  72.         log.Fatal(err.Error())
  73.     }
  75.     textualOut, err := codec.TextualFromNative(nil, nativeOut)
  76.     if err != nil {
  77.         log.Fatal(err.Error())
  78.     }
  80.     var out = SimpleRecord{}
  81.     err = json2.Unmarshal(textualOut, &out)
  82.     if err != nil {
  83.         log.Fatal(err.Error())
  84.     }
  86.     if !reflect.DeepEqual(data, out) {
  87.         log.Fatal("should be equal")
  88.     }
  89. }
英文:

There are two ways you can do it, either by casting out and doing a 'manual mapping' or by using codec.TextualFromNative. Both approaches are shown below for completeness,

Approach 1

Cast out from interface{} to map[string]interface{} and retrieve the values

  1. ...
  2. simpleMap := out.(map[string]interface{})
  3. f1 := simpleMap["f1"]
  4. f2 := simpleMap["f2"] 
  5. ...
  6. SimpleRecord {
  7. F1: f1,
  8. F2: f2,
  9. ...
  10. }

Approach 2

Use TextualFromNative, the below code shows both encode decode process

  1. var avro_schema_txt = `{
  2. "type": "record",
  3. "name": "AvroData",
  4. "namespace": "data.avro",
  5. "doc": "docstring",
  6. "fields": [
  7. {
  8. "name": "f1",
  9. "type": "int"
  10. },
  11. {
  12. "name": "f2",
  13. "type": "string"
  14. },
  15. {
  16. "name": "f3",
  17. "type": "string"
  18. },
  19. {
  20. "name": "dependencies",
  21. "type": {
  22. "type": "array",
  23. "items": "string"
  24. }
  25. }
  26. ]
  27. }`
  28. // added json to match field names in avro
  29. type SimpleRecord struct {
  30. F1           int      `avro:"f1" json:"f1"`
  31. F2           string   `avro:"f2" json:"f2"`
  32. F3           string   `avro:"f3" json:"f3"`
  33. Dependencies []string `avro:"dependencies" json:"dependencies"`
  34. }
  35. func encodeDecode() {
  36. data := SimpleRecord{
  37. F1: 1,
  38. F2: "tester2",
  39. F3: "tester3",
  40. Dependencies: []string { "tester4", "tester5" },
  41. }
  42. codec, err := goavro.NewCodec(avro_schema_txt)
  43. if err != nil {
  44. log.Fatal(err.Error())
  45. }
  46. // encode
  47. textualIn, err := json2.Marshal(data)
  48. if err != nil {
  49. log.Fatal(err.Error())
  50. }
  51. nativeIn, _, err := codec.NativeFromTextual(textualIn)
  52. if err != nil {
  53. log.Fatal(err.Error())
  54. }
  55. binaryIn, err := codec.BinaryFromNative(nil, nativeIn)
  56. if err != nil {
  57. log.Fatal(err.Error())
  58. }
  59. // decode
  60. nativeOut, _, err := codec.NativeFromBinary(binaryIn)
  61. if err != nil {
  62. log.Fatal(err.Error())
  63. }
  64. textualOut, err := codec.TextualFromNative(nil, nativeOut)
  65. if err != nil {
  66. log.Fatal(err.Error())
  67. }
  68. var out = SimpleRecord{}
  69. err = json2.Unmarshal(textualOut, &out)
  70. if err != nil {
  71. log.Fatal(err.Error())
  72. }
  73. if !reflect.DeepEqual(data, out) {
  74. log.Fatal("should be equal")
  75. }
  76. }

huangapple
  • 本文由 发表于 2021年9月8日 18:49:26
  • 转载请务必保留本文链接：https://go.coder-hub.com/69101669.html
  • avro
  • go
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定