可以通过编程方式从已实例化的类生成一个.pyi文件以进行自动完成吗?

huangapple go评论127阅读模式
英文:

Is it possible to programmatically generate a pyi file from an instantiated class (for autocomplete)?

问题

class MyClass:
def init(self, dictionary):
for k, v in dictionary.items():
setattr(self, k, v)

我正在尝试弄清楚如何为这个动态生成的类获得智能感知。大多数集成开发环境可以读取pyi文件来实现这种功能。

我不想手动编写pyi文件。

是否可以实例化这个类,并以编程方式将pyi文件写入磁盘?

mypy有stubgen工具,但我无法弄清楚是否可以以这种方式使用它。

我是否可以从mypy中导入stubgen,并以某种方式将MyClass(<some dict>)传递给它?

英文:

I'm creating a class from a dictionary like this:

  1. class MyClass:
  2. def __init__(self, dictionary):
  3. for k, v in dictionary.items():
  4. setattr(self, k, v)

I'm trying to figure out how I can get Intellisense for this dynamically generated class. Most IDEs can read pyi files for this sort of thing.

I don't want to write out a pyi file manually though.

Is it possible to instantiate this class and programmatically write a pyi file to disk from it?

mypy has the stubgen tool, but I can't figure out if it's possible to use it this way.

Can I import stubgen from mypy and feed it MyClass(&lt;some dict&gt;) somehow?

答案1

得分: 2

静态分析程序如stubgen不适用于分析动态填充的类,因为它们无法看到您完全形成的类的源代码以生成类的存根。您必须在运行源代码以填充实例属性之前在运行时生成存根。

假设您有一个动态填充的类,就像您的示例一样:

  1. class MyClass:
  2. def __init__(self, dictionary: dict[str, object]) -> None:
  3. k: str
  4. v: object
  5. for k, v in dictionary.items():
  6. setattr(self, k, v)

并且您将此字典传递给构造函数:

  1. import statistics
  2. instance: MyClass = MyClass({"a": 1, "b": "my_string", "distribution": statistics.NormalDist(0.0, 1.0)})

并且您希望这作为输出:

  1. import statistics
  2. class MyClass:
  3. a: int
  4. b: str
  5. distribution: statistics.NormalDist
  6. def __init__(self, dictionary: dict[str, object]) -> None:
  7. ...

生成上述输出的最简单方法是通过钩入实例创建和初始化,以便不影响已存在于类上的__new____init__链式super调用。这可以通过元类的__call__方法完成:

  1. class _PostInitialisationMeta(type):
  2. """
  3. Metaclass for classes subject to dynamic stub generation
  4. """
  5. def __call__(
  6. cls, dictionary: dict[str, object], *args: object, **kwargs: object
  7. ) -> object:
  8. """
  9. Override instance creation and initialisation. Generate a string representing
  10. the class's stub definition suitable for a `.pyi` file.
  11. Parameters
  12. ----------
  13. dictionary
  14. Mapping from instance attribute names to attribute values
  15. *args
  16. **kwargs
  17. Other positional and keyword arguments to the class's `__new__` and
  18. `__init` methods
  19. Returns
  20. -------
  21. object
  22. Created instance
  23. """
  24. instance: object = super().__call__(dictionary, *args, **kwargs)
  25. <generate string here>
  26. return instance

然后,您可以解析类为抽象语法树,通过添加、删除或转换节点来修改树,然后取消解析转换后的树。下面是使用Python标准库的ast.NodeVisitor的一个可能的实现:

仅适用于Python 3.9+

  1. from __future__ import annotations
  2. import ast
  3. import inspect
  4. import typing as t
  5. # ... 其他代码 ...
  6. class _DynamicClassStubsGenerator(ast.NodeVisitor):
  7. """
  8. Generate and cache stubs for class instances whose instance variables are populated
  9. dynamically
  10. """
  11. @classmethod
  12. def cache_stub_for_dynamic_class(
  13. StubsGenerator, Class: type, dictionary: dict[str, object], /
  14. ) -> None:
  15. # ... 其他代码 ...

然后,您可以像往常一样运行您的类,然后检查存储在缓存_CLASS_TO_STUB_SOURCE_DICT中的内容:

  1. class MyClass(metaclass=_PostInitialisationMeta):
  2. def __init__(self, dictionary: dict[str, object]) -> None:
  3. k: str
  4. v: object
  5. for k, v in dictionary.items():
  6. setattr(self, k, v)
  7. >>> MyClass({"a": 1, "b": "my_string", "distribution": statistics.NormalDist(0.0, 1.0)})
  8. >>> src: str
  9. >>> for src in _CLASS_TO_STUB_SOURCE_DICT.values():
  10. ... print(src)
  11. ...
  12. import statistics
  13. class MyClass:
  14. a: int
  15. b: str
  16. distribution: statistics.NormalDist
  17. def __init__(self, dictionary: dict[str, object]) -> None:
  18. ...

在实际应用中,.pyi文件是基于每个模块的类型接口,因此上面的实现不会立即可用,因为它只适用于一个类。在将源代码写入.pyi文件之前,还必须对.pyi模块中的其他类型的节点进行更多处理,决定如何处理未注释的节点、重复的导入等等。这是stubgen可能会派上用场的地方 - 它可以分析模块的静态部分,您可以使用输出,编写一个ast.NodeTransformer来将输出转换为您动态生成的类。

英文:

Static analysis programs like stubgen are the wrong tool for analysing a class populated dynamically, because they can't see the source code of your fully-formed class to give you the stub of the class. You have to do the stub generation at runtime by running the source code to populate your instance attributes first.


Let's say that you have a dynamically-populated class, as in your example,

  1. class MyClass:
  2. def __init__(self, dictionary: dict[str, object]) -&gt; None:
  3. k: str
  4. v: object
  5. for k, v in dictionary.items():
  6. setattr(self, k, v)

and you pass in this dictionary to the constructor,

  1. import statistics
  2. instance: MyClass = MyClass({&quot;a&quot;: 1, &quot;b&quot;: &quot;my_string&quot;, &quot;distribution&quot;: statistics.NormalDist(0.0, 1.0)})

and you want this as your output:

  1. import statistics
  2. class MyClass:
  3. a: int
  4. b: str
  5. distribution: statistics.NormalDist
  6. def __init__(self, dictionary: dict[str, object]) -&gt; None:
  7. ...

The easiest way to generate the output above is to hook into instance creation and initialisation, so you don't affect whatever __new__ or __init__ chained super calls which already exist on your class. This can be done via a metaclass's __call__ method:

  1. class _PostInitialisationMeta(type):
  2. &quot;&quot;&quot;
  3. Metaclass for classes subject to dynamic stub generation
  4. &quot;&quot;&quot;
  5. def __call__(
  6. cls, dictionary: dict[str, object], *args: object, **kwargs: object
  7. ) -&gt; object:
  8. &quot;&quot;&quot;
  9. Override instance creation and initialisation. Generate a string representing
  10. the class&#39;s stub definition suitable for a `.pyi` file.
  11. Parameters
  12. ----------
  13. dictionary
  14. Mapping from instance attribute names to attribute values
  15. *args
  16. **kwargs
  17. Other positional and keyword arguments to the class&#39;s `__new__` and
  18. `__init__` methods
  19. Returns
  20. -------
  21. object
  22. Created instance
  23. &quot;&quot;&quot;
  24. instance: object = super().__call__(dictionary, *args, **kwargs)
  25. &lt;generate string here&gt;
  26. return instance

You can then parse the class into an abstract syntax tree, modify the tree by adding, removing, or transforming nodes, then unparse the transformed tree. Here's one possible implementation using the Python standard library's ast.NodeVisitor:

Python 3.9+ only

  1. from __future__ import annotations
  2. import ast
  3. import inspect
  4. import typing as t
  5. if t.TYPE_CHECKING:
  6. class _SupportsBodyStatements(t.Protocol):
  7. body: list[ast.stmt]
  8. _CLASS_TO_STUB_SOURCE_DICT: t.Final[dict[type, str]] = {}
  9. class _PostInitialisationMeta(type):
  10. &quot;&quot;&quot;
  11. Metaclass for classes subject to dynamic stub generation
  12. &quot;&quot;&quot;
  13. def __call__(
  14. cls, dictionary: dict[str, object], *args: object, **kwargs: object
  15. ) -&gt; object:
  16. &quot;&quot;&quot;
  17. Override instance creation and initialisation. The first time an instance of a
  18. class is created and initialised, cache a string representing the class&#39;s stub
  19. definition suitable for a `.pyi` file.
  20. Parameters
  21. ----------
  22. dictionary
  23. Mapping from instance attribute names to attribute values
  24. *args
  25. **kwargs
  26. Other positional and keyword arguments to the class&#39;s `__new__` and
  27. `__init__` methods
  28. Returns
  29. -------
  30. object
  31. Created instance
  32. &quot;&quot;&quot;
  33. instance: object = super().__call__(dictionary, *args, **kwargs)
  34. _DynamicClassStubsGenerator.cache_stub_for_dynamic_class(cls, dictionary)
  35. return instance
  36. def _remove_docstring(node: _SupportsBodyStatements, /) -&gt; None:
  37. &quot;&quot;&quot;
  38. Removes a docstring node if it exists in the given node&#39;s body
  39. &quot;&quot;&quot;
  40. first_node: ast.stmt = node.body[0]
  41. if (
  42. isinstance(first_node, ast.Expr)
  43. and isinstance(first_node.value, ast.Constant)
  44. and (type(first_node.value.value) is str)
  45. ):
  46. node.body.pop(0)
  47. def _replace_body_with_ellipsis(node: _SupportsBodyStatements, /) -&gt; None:
  48. &quot;&quot;&quot;
  49. Replaces the body of a given node with a single `...`
  50. &quot;&quot;&quot;
  51. node.body[:] = [ast.Expr(ast.Constant(value=...))]
  52. class _DynamicClassStubsGenerator(ast.NodeVisitor):
  53. &quot;&quot;&quot;
  54. Generate and cache stubs for class instances whose instance variables are populated
  55. dynamically
  56. &quot;&quot;&quot;
  57. @classmethod
  58. def cache_stub_for_dynamic_class(
  59. StubsGenerator, Class: type, dictionary: dict[str, object], /
  60. ) -&gt; None:
  61. # Disallow stubs generation if the stub source is already generated
  62. try:
  63. _CLASS_TO_STUB_SOURCE_DICT[Class]
  64. except KeyError:
  65. pass
  66. else:
  67. return
  68. # Get class&#39;s source code
  69. src: str = inspect.getsource(Class)
  70. module_tree: ast.Module = ast.parse(src)
  71. class_statement: ast.stmt = module_tree.body[0]
  72. assert isinstance(class_statement, ast.ClassDef)
  73. # Strip unnecessary details from class body
  74. stubs_generator: _DynamicClassStubsGenerator = StubsGenerator()
  75. stubs_generator.visit(module_tree)
  76. # Adds the following:
  77. # - annotated instance attributes on the class body
  78. # - import statements for non-builtins
  79. # --------------------------------------------------
  80. added_import_nodes: list[ast.stmt] = []
  81. added_class_nodes: list[ast.stmt] = []
  82. k: str
  83. v: object
  84. for k, v in dictionary.items():
  85. value_type: type = type(v)
  86. value_type_name: str = value_type.__qualname__
  87. value_type_module_name: str = value_type.__module__
  88. annotated_assignment_statement: ast.stmt = ast.parse(
  89. f&quot;{k}: {value_type_name}&quot;
  90. ).body[0]
  91. assert isinstance(annotated_assignment_statement, ast.AnnAssign)
  92. added_class_nodes.append(annotated_assignment_statement)
  93. if value_type_module_name != &quot;builtins&quot;:
  94. annotation_expression: ast.expr = (
  95. annotated_assignment_statement.annotation
  96. )
  97. assert isinstance(annotation_expression, ast.Name)
  98. annotation_expression.id = (
  99. f&quot;{value_type_module_name}.{annotation_expression.id}&quot;
  100. )
  101. added_import_nodes.append(
  102. ast.Import(names=[ast.alias(name=value_type_module_name)])
  103. )
  104. module_tree.body[:] = [*added_import_nodes, *module_tree.body]
  105. class_statement.body[:] = [*added_class_nodes, *class_statement.body]
  106. _CLASS_TO_STUB_SOURCE_DICT[Class] = ast.unparse(module_tree)
  107. def visit_ClassDef(self, node: ast.ClassDef) -&gt; None:
  108. _remove_docstring(node)
  109. node.keywords = [] # Clear metaclass and other keywords in class definition
  110. self.generic_visit(node)
  111. def visit_FunctionDef(self, node: ast.FunctionDef) -&gt; None:
  112. _replace_body_with_ellipsis(node)
  113. def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -&gt; None:
  114. _replace_body_with_ellipsis(node)

You can then run your class as usual, and then inspect what's stored in the cache _CLASS_TO_STUB_SOURCE_DICT:

  1. class MyClass(metaclass=_PostInitialisationMeta):
  2. def __init__(self, dictionary: dict[str, object]) -&gt; None:
  3. k: str
  4. v: object
  5. for k, v in dictionary.items():
  6. setattr(self, k, v)
  7. &gt;&gt;&gt; MyClass({&quot;a&quot;: 1, &quot;b&quot;: &quot;my_string&quot;, &quot;distribution&quot;: statistics.NormalDist(0.0, 1.0)})
  8. &gt;&gt;&gt; src: str
  9. &gt;&gt;&gt; for src in _CLASS_TO_STUB_SOURCE_DICT.values():
  10. ... print(src)
  11. ...
  12. import statistics
  13. class MyClass:
  14. a: int
  15. b: str
  16. distribution: statistics.NormalDist
  17. def __init__(self, dictionary: dict[str, object]) -&gt; None:
  18. ...

In practice, .pyi files form the type interfaces on a per-module basis, so the implementation above isn't immediately usable as it is only for a class. You also have to do much more processing with other kinds of nodes in your .pyi module, decide what to do with unannotated nodes, repeated imports, etc., before writing the source to a .pyi file. This is where stubgen may come in handy - it can analyse the static parts of your module, and you can take that output and write an ast.NodeTransformer to transform that output into the classes you've generated dynamically.

huangapple
  • 本文由 发表于 2023年2月10日 12:45:58
  • 转载请务必保留本文链接:https://go.coder-hub.com/75407048.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定