合并两个YAML文件深度合并在Bash中使用yq。

huangapple
go评论129阅读模式
英文:

Merge Two Yaml Files Deeply in Bash yq

问题

我有两个YAML文件：

file1.yaml

  1. name: "name-a"
  2. namespace: "name-a-space"
  4. livenessProbe:
  5.   path: is-alive
  7. readinessProbe:
  8.   path: is-ready
  10. env:
  11.   - name: key1
  12.     value: test1
  13.   - name: key2
  14.     value: test2
  15.   - name: KEY3
  16.     value: test2

和file2.yaml

  1. name: "name-b"
  2. namespace: "name-b-space"
  4. livenessProbe:
  5.   path: is-alive1
  7. readinessProbe:
  8.   path: is-ready1
  10. env:
  11.   - name: key1
  12.     value: test7

为了合并file1.yaml和file2.yaml，我正在使用yq v4.30.8

  1. yq '. *= load("file2.yml")' file1.yml

问题是，关于env键，它会覆盖file1中file2的所有env元素，但我只需要覆盖key1的不同值，所以输出应该是

  1. name: "name-b"
  2. namespace: "name-b-space"
  4. livenessProbe:
  5.   path: is-alive1
  7. readinessProbe:
  8.   path: is-ready1
  10. env:
  11.   - name: key1
  12.     value: test7
  13.   - name: key2
  14.     value: test2
  15.   - name: KEY3
  16.     value: test2

但我的代码生成

  1. name: "name-b"
  2. namespace: "name-b-space"
  4. livenessProbe:
  5.   path: is-alive1
  7. readinessProbe:
  8.   path: is-ready1
  10. env:
  11.   - name: key1
  12.     value: test7

参考链接

英文:

I have two yaml files:

file1.yaml

  1. name: "name-a"
  2. namespace: "name-a-space"
  4. livenessProbe:
  5.   path: is-alive
  7. readinessProbe:
  8.   path: is-ready
  10. env:
  11.   - name: key1
  12.     value: test1
  13.   - name: key2
  14.     value: test2
  15.   - name: KEY3
  16.     value: test2

and file2.yaml

  1. name: "name-b"
  2. namespace: "name-b-space"
  4. livenessProbe:
  5.   path: is-alive1
  7. readinessProbe:
  8.   path: is-ready1
  10. env:
  11.   - name: key1
  12.     value: test7

And to merge file1.yaml and file2.yaml I'm using yq v4.30.8

yq '. *= load("file2.yml")' file1.yml based on this reference

The problem is that regarding env key it overwriting all elements of env in file2 on file1, but I need to to just overwrite the different value of key1
so the output should be

  1. name: "name-b"
  2. namespace: "name-b-space"
  4. livenessProbe:
  5.   path: is-alive1
  7. readinessProbe:
  8.   path: is-ready1
  10. env:
  11.   - name: key1
  12.     value: test7
  13.   - name: key2
  14.     value: test2
  15.   - name: KEY3
  16.     value: test2

But my code generate

  1. name: "name-b"
  2. namespace: "name-b-space"
  4. livenessProbe:
  5.   path: is-alive1
  7. readinessProbe:
  8.   path: is-ready1
  10. env:
  11.   - name: key1
  12.     value: test7

答案1

得分: 3

使用 *d 来合并、深度合并数组，它会合并数组 "类似对象，以其索引作为键"，即 "合并数组中的第一项，然后不对[其他项]做任何操作"：

  1. yq '. *d load("file2.yaml")' file1.yaml
  1. name: "name-b"
  2. namespace: "name-b-space"
  3. livenessProbe:
  4.   path: is-alive1
  5. readinessProbe:
  6.   path: is-ready1
  7. env:
  8.   - name: key1
  9.     value: test7
  10.   - name: key2
  11.     value: test2
  12.   - name: KEY3
  13.     value: test2

正如转到的链接所示，您希望通过匹配项目对象中的特定字段来合并.env中的数组。为此，您可以使用.name作为字段名称（在合并时变为 "键"）将.env数组简化为一个对象，使用.[] as $item ireduce ({}; .[$item.name] = $item.value)。对于两侧的.env都执行此操作，然后使用常规合并 *。最后，使用map({ "name": key, "value": . })将索引对象映射回原始数组结构，使用多余的空格来突出显示重复的代码：

  1. yq '
  2.   ( .env |= (.[] as $item ireduce ({}; .[$item.name] = $item.value))) *
  3.   (load("file2.yaml") | .env |= (.[] as $item ireduce ({}; .[$item.name] = $item.value)))
  4.   | .env |= map({ "name": key, "value": . })
  5. ' file1.yaml

此外，还请查看手册中的合并对象数组，根据键匹配部分，该部分基于问题如何在"yq"中有条件地添加/替换数组元素。

最后，还请考虑使用kislyuk/yq（这是yq的另一个实现），或者itchyny/gojq。这两个工具都是YAML处理器，前者在内部将YAML转换为JSON，然后在底层使用stedolan/jq，后者是Go中jq的重新实现，实现了本机的YAML功能。因此，两者都可以使用mikefarah/yq尚未实现的jq功能。例如：

  1. # 使用 kislyuk/yq
  2. yq -sy '
  3.   map(.env |= INDEX(.name)) | .[0] * .[1] | .env |= map(.)
  4. ' file1.yaml file2.yaml
  6. # 使用 itchyny/gojq
  7. gojq -s --yaml-input --yaml-output '
  8.   map(.env |= INDEX(.name)) | .[0] * .[1] | .env |= map(.)
  9. ' file1.yaml file2.yaml
英文:

Use *d to Merge, deeply-merging arrays which merges arrays "like objects, with indices as their key", i.e. "merge the first item in the array and do nothing with the [other ones].":

  1. yq '. *d load("file2.yaml")' file1.yaml
  1. name: "name-b"
  2. namespace: "name-b-space"
  3. livenessProbe:
  4.   path: is-alive1
  5. readinessProbe:
  6.   path: is-ready1
  7. env:
  8.   - name: key1
  9.     value: test7
  10.   - name: key2
  11.     value: test2
  12.   - name: KEY3
  13.     value: test2

As it turned out, you want the arrays in .env to be merged by matching a specific field within the item objects. To this end, you could ireduce the .env array into one object using .name as field names (which becomes the "key" when merging) using .[] as $item ireduce ({}; .[$item.name] = $item.value). Do this for .env on both sides, then use regular merging with *. Eventually, map the indexed object back into your original array structure using map({"name": key, "value": .}) (using unnecessary spaces to highlight duplicate code):

  1. yq '
  2.   (                     .env |= (.[] as $item ireduce ({}; .[$item.name] = $item.value))) *
  3.   (load("file2.yaml") | .env |= (.[] as $item ireduce ({}; .[$item.name] = $item.value)))
  4.   |                     .env |= map({"name": key, "value": .})
  5. ' file1.yaml

Also, take a look at how to Merge arrays of objects together, matching on a key in the manual, which is based on the question How to Add / Replace array elements with "yq" conditionally.

Finally, also consider using kislyuk/yq (which is "the other" implementation of yq), or itchyny/gojq. Both are YAML processors with the former internally translating to JSON, then using stedolan/jq under the hood, and the latter being a reimplementation of jq in Go, implementing native YAML functionality. Thus, both can make use of jq features not (yet) implemted by mikefarah/yq. For example:

  1. # using kislyuk/yq
  2. yq -sy '
  3.   map(.env |= INDEX(.name)) | .[0] * .[1] | .env |= map(.)
  4. ' file1.yaml file2.yaml
  6. # using itchyny/gojq
  7. gojq -s --yaml-input --yaml-output '
  8.   map(.env |= INDEX(.name)) | .[0] * .[1] | .env |= map(.)
  9. ' file1.yaml file2.yaml

huangapple
  • 本文由 发表于 2023年5月29日 01:02:07
  • 转载请务必保留本文链接：https://go.coder-hub.com/76352612.html
  • bash
  • merge
  • yaml
  • yq
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定