Json转换为Avro在Python中

huangapple
go评论115阅读模式
英文:

Json to avro in python

问题

以下是您提供的代码的翻译部分：

  1. 我正在尝试使用以下代码将JSON转换为Avro：
  3. from fastavro import writer, reader, schema
  4. from rec_avro import to_rec_avro_destructive, from_rec_avro_destructive, rec_avro_schema
  6. avro_objects = (to_rec_avro_destructive(rec) for rec in all_obj[:100])
  8. with open('json_in_avro.avro', 'wb') as f_out:
  9.     writer(f_out, schema.parse_schema(rec_avro_schema()), avro_objects)

它能够正常工作，但是all_obj 中的JSON架构从这样：

  1. {
  2. 'Date': '2023-07-16',
  3. 'Url': 'google.pt',
  4. 'Item': {'Title': 'abababab',
  5. 'Id': '28e3c5n',
  6. 'DedupId': None,
  7. 'ImageHash': None,
  8. 'Attributes': {'Cost': None,
  9.   'Area': None},
  10. 'PropertyDetail':  []}
  11.  }

变成了这样：

  1. {
  2. 'Date': '2023-07-16',
  3. 'Url': 'google.pt',
  4. 'Item': {'_': {'Title': 'abababab',
  5. 'Id': '28e3c5n',
  6. 'DedupId': None,
  7. 'ImageHash': None,
  8. 'Attributes': {'_': {'Cost': None,
  9.   'Area': None}},
  10. 'PropertyDetail': {'_': []}}}
  11.  }

看起来为什么会创建这些'_'项呢？

感谢您的帮助！

  2. <details>
  3. <summary>英文:</summary>
  5. I am trying to convert json into avro using the following code:
  7.     from fastavro import writer, reader, schema
  8.     from rec_avro import to_rec_avro_destructive, from_rec_avro_destructive, rec_avro_schema
  9.     
  10.     avro_objects = (to_rec_avro_destructive(rec) for rec in all_obj[:100])
  12.     with open(&#39;json_in_avro.avro&#39;, &#39;wb&#39;) as f_out:
  13.         writer(f_out, schema.parse_schema(rec_avro_schema()), avro_objects)
  16. It works fine however the json schema within all_obj goes from this
  17.  
  19.     {
  20.     &#39;Date&#39;: &#39;2023-07-16&#39;,
  21.     &#39;Url&#39;: &#39;google.pt&#39;,
  22.     &#39;Item&#39;: {&#39;Title&#39;: &#39;abababab&#39;,
  23.     &#39;Id&#39;: &#39;28e3c5n&#39;,
  24.     &#39;DedupId&#39;: None,
  25.     &#39;ImageHash&#39;: None,
  26.     &#39;Attributes&#39;: {&#39;Cost&#39;: None,
  27.       &#39;Area&#39;: None},
  28.     &#39;PropertyDetail&#39;:  []}
  29.      }
  31. To this:
  33.     {
  34.     &#39;Date&#39;: &#39;2023-07-16&#39;,
  35.     &#39;Url&#39;: &#39;google.pt&#39;,
  36.     &#39;Item&#39;: {&#39;_&#39;: {&#39;Title&#39;: &#39;abababab&#39;,
  37.     &#39;Id&#39;: &#39;28e3c5n&#39;,
  38.     &#39;DedupId&#39;: None,
  39.     &#39;ImageHash&#39;: None,
  40.     &#39;Attributes&#39;: {&#39;_&#39;: {&#39;Cost&#39;: None,
  41.       &#39;Area&#39;: None}},
  42.     &#39;PropertyDetail&#39;: {&#39;_&#39;: []}}
  43.      }
  45. Any reason why it seems to create these &#39;_&#39; items?
  47. Thank you for the help!
  49. </details>
  52. # 答案1
  53. **得分**: 1
  55. ```python
  56. # 下划线（`_`）的内容来自于 `rec_avro` 库。然而，只要您定义了数据的模式（您应该这样做），就不需要使用该库。
  58. 我不知道您模式的所有细节，但我尽力猜测，并提供了一个非常接近您所需的脚本：
  60. ```python
  61. import fastavro
  63. schema = {
  64.   "name": "YourName",
  65.   "type": "record",
  66.   "fields": [
  67.     {"name": "Date", "type": "string"},
  68.     {"name": "Url", "type": "string"},
  69.     {
  70.       "name": "Item",
  71.       "type": {
  72.         "type": "record",
  73.         "name": "YourItem",
  74.         "fields": [
  75.           {"name": "Title", "type": "string"},
  76.           {"name": "Id", "type": ["null", "string"]},
  77.           {"name": "DedupId", "type": ["null", "string"]},
  78.           {"name": "ImageHash", "type": ["null", "string"]},
  79.           {
  80.            "name": "Attributes",
  81.             "type": {
  82.               "type": "record",
  83.               "name": "YourAttributes",
  84.               "fields": [
  85.                 {"name": "Cost", "type": ["null", "string"]},
  86.                 {"name": "Area", "type": ["null", "string"]},
  87.               ],
  88.             },
  89.           },
  90.           {"name": "PropertyDetail", "type": {"type": "array", "items": "string"}},
  91.         ],
  92.       },
  93.     },
  94.   ]
  95. }
  97. records = [
  98.     {
  99.         'Date': '2023-07-16',
  100.         'Url': 'google.pt',
  101.         'Item': {
  102.             'Title': 'abababab',
  103.             'Id': '28e3c5n',
  104.             'DedupId': None,
  105.             'ImageHash': None,
  106.             'Attributes': {
  107.                 'Cost': None,
  108.                 'Area': None,
  109.             },
  110.             'PropertyDetail':  [],
  111.         }
  112.     }
  113. ]
  115. with open('json_in_avro.avro', 'wb') as fp:
  116.     fastavro.writer(fp, schema, records)
  2. <details>
  3. <summary>英文:</summary>
  5. The `_` things come from the `rec_avro` library. However, you don&#39;t need to use that library as long as you define the schema for your data (which you should do).
  7. I don&#39;t know all the details of your schema, but I took my best guess and here&#39;s a script that should be pretty close to what you are looking for:
  9. ```python
  10. import fastavro
  12. schema = {
  13.   &quot;name&quot;: &quot;YourName&quot;,
  14.   &quot;type&quot;: &quot;record&quot;,
  15.   &quot;fields&quot;: [
  16.     {&quot;name&quot;: &quot;Date&quot;, &quot;type&quot;: &quot;string&quot;},
  17.     {&quot;name&quot;: &quot;Url&quot;, &quot;type&quot;: &quot;string&quot;},
  18.     {
  19.       &quot;name&quot;: &quot;Item&quot;,
  20.       &quot;type&quot;: {
  21.         &quot;type&quot;: &quot;record&quot;,
  22.         &quot;name&quot;: &quot;YourItem&quot;,
  23.         &quot;fields&quot;: [
  24.           {&quot;name&quot;: &quot;Title&quot;, &quot;type&quot;: &quot;string&quot;},
  25.           {&quot;name&quot;: &quot;Id&quot;, &quot;type&quot;: [&quot;null&quot;, &quot;string&quot;]},
  26.           {&quot;name&quot;: &quot;DedupId&quot;, &quot;type&quot;: [&quot;null&quot;, &quot;string&quot;]},
  27.           {&quot;name&quot;: &quot;ImageHash&quot;, &quot;type&quot;: [&quot;null&quot;, &quot;string&quot;]},
  28.           {
  29.            &quot;name&quot;: &quot;Attributes&quot;,
  30.             &quot;type&quot;: {
  31.               &quot;type&quot;: &quot;record&quot;,
  32.               &quot;name&quot;: &quot;YourAttributes&quot;,
  33.               &quot;fields&quot;: [
  34.                 {&quot;name&quot;: &quot;Cost&quot;, &quot;type&quot;: [&quot;null&quot;, &quot;string&quot;]},
  35.                 {&quot;name&quot;: &quot;Area&quot;, &quot;type&quot;: [&quot;null&quot;, &quot;string&quot;]},
  36.               ],
  37.             },
  38.           },
  39.           {&quot;name&quot;: &quot;PropertyDetail&quot;, &quot;type&quot;: {&quot;type&quot;: &quot;array&quot;, &quot;items&quot;: &quot;string&quot;}},
  40.         ],
  41.       },
  42.     },
  43.   ]
  44. }
  46. records = [
  47.     {
  48.         &#39;Date&#39;: &#39;2023-07-16&#39;,
  49.         &#39;Url&#39;: &#39;google.pt&#39;,
  50.         &#39;Item&#39;: {
  51.             &#39;Title&#39;: &#39;abababab&#39;,
  52.             &#39;Id&#39;: &#39;28e3c5n&#39;,
  53.             &#39;DedupId&#39;: None,
  54.             &#39;ImageHash&#39;: None,
  55.             &#39;Attributes&#39;: {
  56.                 &#39;Cost&#39;: None,
  57.                 &#39;Area&#39;: None,
  58.             },
  59.             &#39;PropertyDetail&#39;:  [],
  60.         }
  61.     }
  62. ]
  64. with open(&#39;json_in_avro.avro&#39;, &#39;wb&#39;) as fp:
  65.     fastavro.writer(fp, schema, records)

huangapple
  • 本文由 发表于 2023年7月18日 00:51:44
  • 转载请务必保留本文链接：https://go.coder-hub.com/76706589.html
  • avro
  • json
  • python
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定