如何在NIFI中使用Jolttransform规范进行映射？

huangapple
go评论97阅读模式
英文:

How to map using Jolttransform specification in NIFI?

问题

我正在准备jolt规范，但无法完成，请帮忙。

**注意：**有一个credits数组，该数组的值应与类型为credit record的模式字段进行映射。

输入：

  1. {
  2.   "schema": {
  3.     "fields": [
  4.       {
  5.         "name": "user",
  6.         "type": "STRING",
  7.         "mode": "NULLABLE"
  8.       },
  9.       {
  10.         "name": "billed",
  11.         "type": "STRING",
  12.         "mode": "NULLABLE"
  13.       },
  14.       {
  15.         "name": "currency",
  16.         "type": "STRING",
  17.         "mode": "NULLABLE"
  18.       },
  19.       {
  20.         "name": "currency_conversion_rate",
  21.         "type": "FLOAT",
  22.         "mode": "NULLABLE"
  23.       },
  24.       {
  25.         "name": "usage_unit",
  26.         "type": "STRING",
  27.         "mode": "NULLABLE"
  28.       },
  29.       {
  30.         "name": "adjustment_info_mode",
  31.         "type": "STRING",
  32.         "mode": "NULLABLE"
  33.       },
  34.       {
  35.         "name": "adjustment_info_type",
  36.         "type": "STRING",
  37.         "mode": "NULLABLE"
  38.       },
  39.       {
  40.         "name": "InvoiceMonth",
  41.         "type": "STRING",
  42.         "mode": "NULLABLE"
  43.       },
  44.       {
  45.         "name": "credits",
  46.         "type": "RECORD",
  47.         "mode": "REPEATED",
  48.         "fields": [
  49.           {
  50.             "name": "name",
  51.             "type": "STRING",
  52.             "mode": "NULLABLE"
  53.           },
  54.           {
  55.             "name": "amount",
  56.             "type": "FLOAT",
  57.             "mode": "NULLABLE"
  58.           },
  59.           {
  60.             "name": "full_name",
  61.             "type": "STRING",
  62.             "mode": "NULLABLE"
  63.           },
  64.           {
  65.             "name": "id",
  66.             "type": "STRING",
  67.             "mode": "NULLABLE"
  68.           },
  69.           {
  70.             "name": "type",
  71.             "type": "STRING",
  72.             "mode": "NULLABLE"
  73.           }
  74.         ]
  75.       },
  76.       {
  77.         "name": "resource_name",
  78.         "type": "STRING",
  79.         "mode": "NULLABLE"
  80.       },
  81.       {
  82.         "name": "resource_global_name",
  83.         "type": "STRING",
  84.         "mode": "NULLABLE"
  85.       },
  86.       {
  87.         "name": "label",
  88.         "type": "STRING",
  89.         "mode": "NULLABLE"
  90.       }
  91.     ]
  92.   },
  93.   "InvoiceMonth": "202006",
  94.   "credits": [
  95.     {
  96.       "v": {
  97.         "f": [
  98.           {
  99.             "v": "ae391_credit:0"
  100.           },
  101.           {
  102.             "v": "-0.010237"
  103.           },
  104.           {
  105.             "v": "Committed dollar"
  106.           },
  107.           {
  108.             "v": "ae31_credit:0"
  109.           },
  110.           {
  111.             "v": "COMMITTED_BASE"
  112.           }
  113.         ]
  114.       }
  115.     },
  116.     {
  117.       "v": {
  118.         "f": [
  119.           {
  120.             "v": "ae391_credit:0"
  121.           },
  122.           {
  123.             "v": "-0.010237"
  124.           },
  125.           {
  126.             "v": "Committed dollar"
  127.           },
  128.           {
  129.             "v": "ae31_credit:0"
  130.           },
  131.           {
  132.             "v": "COMMITTED_BASE"
  133.           }
  134.         ]
  135.       }
  136.     }
  137.   ],
  138.   "resource_name": "projects-0",
  139.   "resource_global_name": "//compute.googleapis.com/",
  140.   "label": ""
  141. }

我尝试了这个jolt规范：

  1. [
  2.   {
  3.     "operation": "shift",
  4.     "spec": {
  5.       "credits": {
  6.         "*": {
  7.           "*": {
  8.             "*": {
  9.               "*": {
  10.                 "@v": "[&3].@(5,schema.fields.fields[&].name)"
  11.               }
  12.             }
  13.           }
  14.         }
  15.       }
  16.     }
  17.   }
  18. ]

预期输出：

  1. [
  2.   {
  3.     "name": "ae391_credit:0",
  4.     "amount": "-0.010237",
  5.     "full_name": "Committed dollar",
  6.     "id": "ae31_credit:0",
  7.     "type": "COMMITTED_BASE"
  8.   },
  9.   {
  10.     "name": "ae391_credit:0",
  11.     "amount": "-0.010237",
  12.     "full_name": "Committed dollar",
  13.     "id": "ae31_credit:0",
  14.     "type": "COMMITTED_BASE"
  15.   }
  16. ]
英文:

I am preparing the jolt specification but I am unable to complete it, pls help out.

Note: There is a credits array, the values of that array should be mapped with schema fields which are of type credit record.

Input:

  1. {
  2.   "schema": {
  3.     "fields": [
  4.       {
  5.         "name": "user",
  6.         "type": "STRING",
  7.         "mode": "NULLABLE"
  8.       },
  9.       {
  10.         "name": "billed",
  11.         "type": "STRING",
  12.         "mode": "NULLABLE"
  13.       },
  14.       {
  15.         "name": "currency",
  16.         "type": "STRING",
  17.         "mode": "NULLABLE"
  18.       },
  19.       {
  20.         "name": "currency_conversion_rate",
  21.         "type": "FLOAT",
  22.         "mode": "NULLABLE"
  23.       },
  24.       {
  25.         "name": "usage_unit",
  26.         "type": "STRING",
  27.         "mode": "NULLABLE"
  28.       },
  29.       {
  30.         "name": "adjustment_info_mode",
  31.         "type": "STRING",
  32.         "mode": "NULLABLE"
  33.       },
  34.       {
  35.         "name": "adjustment_info_type",
  36.         "type": "STRING",
  37.         "mode": "NULLABLE"
  38.       },
  39.       {
  40.         "name": "InvoiceMonth",
  41.         "type": "STRING",
  42.         "mode": "NULLABLE"
  43.       },
  44.       {
  45.         "name": "credits",
  46.         "type": "RECORD",
  47.         "mode": "REPEATED",
  48.         "fields": [
  49.           {
  50.             "name": "name",
  51.             "type": "STRING",
  52.             "mode": "NULLABLE"
  53.           },
  54.           {
  55.             "name": "amount",
  56.             "type": "FLOAT",
  57.             "mode": "NULLABLE"
  58.           },
  59.           {
  60.             "name": "full_name",
  61.             "type": "STRING",
  62.             "mode": "NULLABLE"
  63.           },
  64.           {
  65.             "name": "id",
  66.             "type": "STRING",
  67.             "mode": "NULLABLE"
  68.           },
  69.           {
  70.             "name": "type",
  71.             "type": "STRING",
  72.             "mode": "NULLABLE"
  73.           }
  74.         ]
  75.       },
  76.       {
  77.         "name": "resource_name",
  78.         "type": "STRING",
  79.         "mode": "NULLABLE"
  80.       },
  81.       {
  82.         "name": "resource_global_name",
  83.         "type": "STRING",
  84.         "mode": "NULLABLE"
  85.       },
  86.       {
  87.         "name": "label",
  88.         "type": "STRING",
  89.         "mode": "NULLABLE"
  90.       }
  91.     ]
  92.   },
  93.   "InvoiceMonth": "202006",
  94.   "credits": [
  95.     {
  96.       "v": {
  97.         "f": [
  98.           {
  99.             "v": "ae391_credit:0"
  100.           },
  101.           {
  102.             "v": "-0.010237"
  103.           },
  104.           {
  105.             "v": "Committed dollar "
  106.           },
  107.           {
  108.             "v": "ae31_credit:0"
  109.           },
  110.           {
  111.             "v": "COMMITTED_BASE"
  112.           }
  113.         ]
  114.       }
  115.     },
  116.     {
  117.       "v": {
  118.         "f": [
  119.           {
  120.             "v": "ae391_credit:0"
  121.           },
  122.           {
  123.             "v": "-0.010237"
  124.           },
  125.           {
  126.             "v": "Committed dollar "
  127.           },
  128.           {
  129.             "v": "ae31_credit:0"
  130.           },
  131.           {
  132.             "v": "COMMITTED_BASE"
  133.           }
  134.         ]
  135.       }
  136.     }
  137.   ],
  138.   "resource_name": "projects-0",
  139.   "resource_global_name": "//compute.googleapis.com/",
  140.   "label": ""
  141. }

I have tried this jolt:

  1. [
  2.   {
  3.     "operation": "shift",
  4.     "spec": {
  5.       "credits": {
  6.         "*": {
  7.           "*": {
  8.             "*": {
  9.               "*": {
  10.                 "@v": "[&3].@(5,schema.fields.fields[&].name)"
  11.               }
  12.             }
  13.           }
  14.         }
  15.       }
  16.     }
  17.   }
  18. ]

Expected Output:

  1. [
  2.   {
  3.     "name": "ae391_credit:0",
  4.     "amount": "-0.010237",
  5.     "full_name": "Committed dollar",
  6.     "id": "ae31_credit:0",
  7.     "type": "COMMITTED_BASE"
  8.   },
  9.   {
  10.     "name": "ae391_credit:0",
  11.     "amount": "-0.010237",
  12.     "full_name": "Committed dollar",
  13.     "id": "ae31_credit:0",
  14.     "type": "COMMITTED_BASE"
  15.   }
  16. ]

答案1

得分: 1

你可以使用 shift 操作将内部的 fields 提取并在 credits 数组中使用另一个 shift 操作来获取名称作为键的值。

  1. [
  2.   {
  3.     "operation": "shift",
  4.     "spec": {
  5.       "credits": "&",
  6.       "schema": {
  7.         "fields": {
  8.           "*": {
  9.             "fields": "&"
  10.           }
  11.         }
  12.       }
  13.     }
  14.   },
  15.   {
  16.     "operation": "shift",
  17.     "spec": {
  18.       "credits": {
  19.         "*": {
  20.           "*": {
  21.             "*": {
  22.               "*": {
  23.                 "*": "[&4].@(6,fields[&1].name)"
  24.               }
  25.             }
  26.           }
  27.         }
  28.       }
  29.     }
  30.   }
  31. ]
英文:

You can use a shift operation and bring inner fields up and use another shift for getting the value of the name as your key in the credits array.

  1. [
  2.   {
  3.     "operation": "shift",
  4.     "spec": {
  5.       "credits": "&",
  6.       "schema": {
  7.         "fields": {
  8.           "*": {
  9.             "fields": "&"
  10.           }
  11.         }
  12.       }
  13.     }
  14.   },
  15.   {
  16.     "operation": "shift",
  17.     "spec": {
  18.       "credits": {
  19.         "*": {
  20.           "*": {
  21.             "*": {
  22.               "*": {
  23.                 "*": "[&4].@(6,fields[&1].name)"
  24.               }
  25.             }
  26.           }
  27.         }
  28.       }
  29.     }
  30.   }
  31. ]

huangapple
  • 本文由 发表于 2023年6月12日 16:52:41
  • 转载请务必保留本文链接：https://go.coder-hub.com/76454983.html
  • apache-nifi
  • jolt
  • json
  • transformation
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定