Combining Dumper class with string representer to get exact required YAML output

huangapple
go评论111阅读模式
英文:

Combining Dumper class with string representer to get exact required YAML output

问题

我使用PyYAML 6.0和Python 3.9。

我正在尝试以下操作：

  1. 创建一个YAML列表
  2. 将此列表嵌入另一个YAML对象中作为多行字符串
  3. 将这个YAML对象替换到现有文档中
  4. 以能够通过YAML 1.2 linting的格式写回文档

我已经让这个过程运行起来了，除了YAML 1.2的要求，以下是相关的Python代码：

  1. import yaml
  3. # 这里有一些代码，它定义了str_presenter函数和DoYamlStuff类。

这会生成类似这样的内容：

  1. - patch: |
  2.     - op: replace
  3.       path: /spec/postRenderers
  4.       value:
  5.       - kustomize:
  6.           images:
  7.           - name: nginx:latest
  8.             newName: 12345678910.dkr.ecr.eu-west-1.amazonaws.com/nginx
  9.             newTag: latest    
  10.   target:
  11.     kind: HelmRelease
  12.     name: nginx
  13.     namespace: nginx

如您所见，这大部分都可以正常工作。有效的YAML，做了所需的操作等。

不幸的是，它没有缩进列表项2个空格，因此我们仓库的预提交中的YAML linter会对一切进行调整。这使得仓库变得混乱，并且导致PR经常包含不相关的更改。

随后，我尝试实现来自StackOverflow的PrettyDumper类，这反转了效果 - 我的缩进现在正确了，但我的标量根本不起作用。

我已经尝试将str_presenter函数与PrettyDumper类合并，但标量仍然无法正常工作。

如果我能将这两种方法合并到PrettyDumper类中，我认为它将满足我所有的需求。有人可以指点我正确的方向吗？

英文:

I'm using PyYAML 6.0 with Python 3.9.

In order, I am trying to...

  1. Create a YAML list
  2. Embed this list as a multi-line string in another YAML object
  3. Replace this YAML object in an existing document
  4. Write the document back, in a format that will pass YAML 1.2 linting

I have the process working, apart from the YAML 1.2 requirement, with the following code:

  1. import yaml
  3. def str_presenter(dumper, data):
  4.     """configures yaml for dumping multiline strings
  5.     Ref: https://stackoverflow.com/questions/8640959/how-can-i-control-what-scalar-form-pyyaml-uses-for-my-data"""
  6.     if data.count('\n') > 0:  # check for multiline string
  7.         return dumper.represent_scalar('tag:yaml.org,2002:str', data, style='|')
  8.     return dumper.represent_scalar('tag:yaml.org,2002:str', data)
  10. yaml.add_representer(str, str_presenter)
  11. yaml.representer.SafeRepresenter.add_representer(
  12.     str, str_presenter) 
  14. class DoYamlStuff:
  15.     def post_renderers(images):
  16.         return yaml.dump([
  17.             {
  18.                 "op": "replace",
  19.                 "path": "/spec/postRenderers",
  20.                 "value": [
  21.                     {
  22.                         "kustomize": {
  23.                             "images": images
  24.                         }
  25.                     }
  26.                 ]
  27.             }])
  29.     @classmethod
  30.     def images_patch(cls, chart, images, ecr_url):
  31.         return {
  32.             "target": {
  33.                 "kind": "HelmRelease",
  34.                 "name": chart,
  35.                 "namespace": chart
  36.             },
  37.             "patch": cls.post_renderers([x.patch(ecr_url) for x in images])

This produces something like this:

  1. - patch: |
  2.     - op: replace
  3.       path: /spec/postRenderers
  4.       value:
  5.       - kustomize:
  6.           images:
  7.           - name: nginx:latest
  8.             newName: 12345678910.dkr.ecr.eu-west-1.amazonaws.com/nginx
  9.             newTag: latest    
  10.   target:
  11.     kind: HelmRelease
  12.     name: nginx
  13.     namespace: nginx

As you can see, that's mostly working. Valid YAML, does what it needs to, etc.

Unfortunately... it doesn't indent the list item by 2 spaces, so the YAML linter in our repository's pre-commit then adjusts everything. Makes the repo messy, and causes PRs to regularly include changes that aren't relevant.

I then set out to implement this PrettyDumper class from StackOverflow. This reversed the effects - my indentation is now right, but my scalars aren't working at all:

  1.   - patch: "- op: replace\n  path: /spec/postRenderers\n  value:\n    - kustomize:\n\
  2.       \        images:\n          - name: nginx:latest\n           \
  3.       \ newName: 793961818876.dkr.ecr.eu-west-1.amazonaws.com/nginx\n        \
  4.       \    newTag: latest\n"
  5.     target:
  6.       kind: HelmRelease
  7.       name: nginx
  8.       namespace: nginx

I have tried to merge the str_presenter function with the PrettyDumper class, but the scalars still don't work:

  1. import yaml.emitter
  2. import yaml.serializer
  3. import yaml.representer
  4. import yaml.resolver
  7. class IndentingEmitter(yaml.emitter.Emitter):
  8.     def increase_indent(self, flow=False, indentless=False):
  9.         """Ensure that lists items are always indented."""
  10.         return super().increase_indent(
  11.             flow=False,
  12.             indentless=False,
  13.         )
  16. class PrettyDumper(
  17.     IndentingEmitter,
  18.     yaml.serializer.Serializer,
  19.     yaml.representer.Representer,
  20.     yaml.resolver.Resolver,
  21. ):
  22.     def __init__(
  23.         self,
  24.         stream,
  25.         default_style=None,
  26.         default_flow_style=False,
  27.         canonical=None,
  28.         indent=None,
  29.         width=None,
  30.         allow_unicode=None,
  31.         line_break=None,
  32.         encoding=None,
  33.         explicit_start=None,
  34.         explicit_end=None,
  35.         version=None,
  36.         tags=None,
  37.         sort_keys=True,
  38.     ):
  39.         IndentingEmitter.__init__(
  40.             self,
  41.             stream,
  42.             canonical=canonical,
  43.             indent=indent,
  44.             width=width,
  45.             allow_unicode=allow_unicode,
  46.             line_break=line_break,
  47.         )
  48.         yaml.serializer.Serializer.__init__(
  49.             self,
  50.             encoding=encoding,
  51.             explicit_start=explicit_start,
  52.             explicit_end=explicit_end,
  53.             version=version,
  54.             tags=tags,
  55.         )
  56.         yaml.representer.Representer.__init__(
  57.             self,
  58.             default_style=default_style,
  59.             default_flow_style=default_flow_style,
  60.             sort_keys=sort_keys,
  61.         )
  62.         yaml.resolver.Resolver.__init__(self)
  63.         
  64.         yaml.add_representer(str, self.str_presenter)
  65.         yaml.representer.SafeRepresenter.add_representer(
  66.             str, self.str_presenter) 
  68.     def str_presenter(self, data):
  69.         print(data)
  70.         """configures yaml for dumping multiline strings
  71.         Ref: https://stackoverflow.com/questions/8640959/how-can-i-control-what-scalar-form-pyyaml-uses-for-my-data"""
  72.         if data.count('\n') > 0:  # check for multiline string
  73.             return self.represent_scalar('tag:yaml.org,2002:str', data, style='|')
  74.         return self.represent_scalar('tag:yaml.org,2002:str', data)

If I could merge these two approaches into the PrettyDumper class, I think it would do everything I require. Can anyone point me in the right direction?

答案1

得分: 1

如果您需要通过YAML 1.2的验证，不应该使用PyYAML，因为它只支持（YAML 1.1的一个子集）。

ruamel.yaml可以处理更多内容，例如将序列用作映射键，这是PyYAML无法处理的，尽管它是有效的YAML 1.1。除此之外，它支持并默认使用YAML 1.2的加载/转储（免责声明：我是该软件包的作者）。

多年来，ruamel.yaml的往返模式，最初是为了保留注释而构建的，现在已经扩展，可以处理多余的引号、锚点/别名名称保留、不同格式的字符串标量、整数和浮点数等。您可以使用它的底层技术轻松获取您想要的内容，而无需处理表示器。

  1. import sys
  2. import io
  3. import ruamel.yaml
  5. images = [
  6.    dict(name='nginx:latest', newName='12345678910.dkr.ecr.eu-west-1.amazonaws.com/nginx', newTag='latest'),
  7. ]
  8. chart = 'nginx'
  10. def data_as_literal_scalar(d):
  11.     """将数据结构d转储并将其变成文字标量字符串以供进一步转储"""
  12.     yaml = ruamel.yaml.YAML()
  13.     yaml.indent(sequence=4, offset=2)  # 这会使根序列缩进2个额外位置
  14.     buf = io.StringIO()
  15.     yaml.dump(d, buf)
  16.     v = ''.join([x[2:] for x in buf.getvalue().splitlines(True)])  # 去除额外的位置
  17.     return ruamel.yaml.scalarstring.LiteralScalarString(v)
  19. data = [dict(patch=data_as_literal_scalar([{
  20.                                    "op": "replace",
  21.                                    "path": "/spec/postRenderers",
  22.                                    "value": [
  23.                                        {
  24.                                            "kustomize": {
  25.                                                "images": images
  26.                                            }
  27.                                        }
  28.                                    ]
  29.                                  }]),
  30.     target={
  31.                 "kind": "HelmRelease",
  32.                 "name": chart,
  33.                 "namespace": chart
  34.             },
  35. )]
  37. yaml = ruamel.yaml.YAML()
  38. yaml.dump(data, sys.stdout)

这会生成：

  1. - patch: |
  2. - op: replace
  3. path: /spec/postRenderers
  4. value:
  5. - kustomize:
  6. images:
  7. - name: nginx:latest
  8. newName: 12345678910.dkr.ecr.eu-west-1.amazonaws.com/nginx
  9. newTag: latest
  10. target:
  11. kind: HelmRelease
  12. name: nginx
  13. namespace: nginx
英文:

If you need to pass your output through YAML 1.2 linting, you should not use PyYAML as it only supports (a subset of) YAML 1.1.

ruamel.yaml can handle more, e.g using a sequence as a mapping key, something that PyYAML cannot handle at all, although it is
valid YAML 1.1. Apart from that it supports, and defaults to,
YAML 1.2 loading/dumping (disclaimer: I am the author of that package).

Over the years ruamel.yaml's round-trip mode, which was originally built to preserve comments,
has been extended and now
handles superfluous quotes, anchor/alias name preservation,
different format string scalars, integers and float etc. You can use its underlying technology
to easily get what you want, without mucking with representers:

  1. import sys
  2. import io
  3. import ruamel.yaml
  5. images = [
  6.    dict(name='nginx:latest', newName='12345678910.dkr.ecr.eu-west-1.amazonaws.com/nginx', newTag='latest'),
  7. ]
  8. chart = 'nginx'
  10. def data_as_literal_scalar(d):
  11.     """dump a data structure d and make it a literal scalar string for further dumping"""
  12.     yaml = ruamel.yaml.YAML()
  13.     yaml.indent(sequence=4, offset=2)  # this indents even the root sequence by 2 extra positions
  14.     buf = io.StringIO()
  15.     yaml.dump(d, buf)
  16.     v = ''.join([x[2:] for x in buf.getvalue().splitlines(True)])  # strip extra positions
  17.     return ruamel.yaml.scalarstring.LiteralScalarString(v)
  19. data = [dict(patch=data_as_literal_scalar([{
  20.                                    "op": "replace",
  21.                                    "path": "/spec/postRenderers",
  22.                                    "value": [
  23.                                        {
  24.                                            "kustomize": {
  25.                                                "images": images
  26.                                            }
  27.                                        }
  28.                                    ]
  29.                                  }]),
  30.     target={
  31.                 "kind": "HelmRelease",
  32.                 "name": chart,
  33.                 "namespace": chart
  34.             },
  35. )]
  37. yaml = ruamel.yaml.YAML()
  38. yaml.dump(data, sys.stdout)

which gives:

  1. - patch: |
  2. - op: replace
  3. path: /spec/postRenderers
  4. value:
  5. - kustomize:
  6. images:
  7. - name: nginx:latest
  8. newName: 12345678910.dkr.ecr.eu-west-1.amazonaws.com/nginx
  9. newTag: latest
  10. target:
  11. kind: HelmRelease
  12. name: nginx
  13. namespace: nginx

huangapple
  • 本文由 发表于 2023年3月21日 00:41:43
  • 转载请务必保留本文链接：https://go.coder-hub.com/75793013.html
  • python
  • pyyaml
  • yaml
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定