在pyspark中使用Params。

huangapple
go评论92阅读模式
英文:

Use of Params in pyspak

问题

在这个示例中，我试图将overrides作为一个Params对象使用，并希望它被用作字符串列表。

但是，我无法使用下面的代码分配它的值。

  1. class _AB(Params):
  3.     overrides = Param(Params._dummy(), "overrides", "Parameters for environment setup", typeConverter=TypeConverters.toListString)
  4.     
  5.     def __init__(self, *args):
  6.         super().__init__(*args)
  7.         self._setDefault(overrides=None)
  8.         
  10. class A(_AB):
  11.   @keyword_only
  12.   def __init__(self, overrides):
  13.     super().__init__()
  14.     kwargs = self._input_kwargs
  15.     self.setParams(**kwargs)
  17.   @keyword_only
  18.   def setParams(self, overrides: List[str]):
  19.       kwargs = self._input_kwargs
  20.       print(kwargs)
  21.       return self._set(**kwargs)
  23.   def c(self):
  24.     print(self.overrides.__dict__['typeConverter'].__dict__)
  25.     for i in self.overrides:
  26.        print(i)
  28. a = A(overrides=["dsfs", "Sdf"])
  29. a.c()

当我在函数c内打印它时，它给我一个空字典。
当我在函数c内打印它时，它给我一个错误：

  1. TypeError: 'Param' object is not iterable

我猜这是因为它无法将某个值分配给overrides变量。

英文:

In this example, I am trying to use overrides as a Params object and I want it to be used as a list of strings.

But I am not able to assign its value using the below code.

  1. class _AB(Params):
  3.     overrides = Param(Params._dummy(), "overrides", "Parameters for environment setup", typeConverter=TypeConverters.toListString)
  4.     
  5.     def __init__(self, *args):
  6.         super().__init__(*args)
  7.         self._setDefault(overrides=None)
  8.         
  10. class A(_AB):
  11.   @keyword_only
  12.   def __init__(self, overrides):
  13.     super().__init__()
  14.     kwargs = self._input_kwargs
  15.     self.setParams(**kwargs)
  17.   @keyword_only
  18.   def setParams(self, overrides: List[str]):
  19.       kwargs = self._input_kwargs
  20.       print(kwargs)
  21.       return self._set(**kwargs)
  23.   def c(self):
  24.     print(self.overrides.__dict__['typeConverter'].__dict__)
  25.     for i in self.overrides:
  26.        print(i)
  28. a = A(overrides=["dsfs", "Sdf"])
  29. a.c()

It gives me a blank dictionary when I print it inside function c.
It gives me an error:

  1. TypeError: 'Param' object is not iterable

I guess it's happening because it's not able to assign some value to overrides variable.

答案1

得分: 1

Param对象需要从一个获取函数中访问，否则将报告一些错误。我应该从一个获取函数中访问self.overrides。

例如：

  1. class _AB(Params):
  3.     overrides = Param(Params._dummy(), "overrides", "Overrides parameters for environment setup", typeConverter=TypeConverters.toListString)
  4.     
  5.     def __init__(self, *args):
  6.         super().__init__(*args)
  7.         self._setDefault(overrides=None)
  8.     def getoverrides(self):
  9.         return self.getOrDefault(self.overrides)
  11. class A(_AB):
  12.   @keyword_only
  13.   def __init__(self, overrides):
  14.     super().__init__()
  15.     kwargs = self._input_kwargs
  16.     self.setParams(**kwargs)
  18.   @keyword_only
  19.   def setParams(self, overrides: List[str]):
  20.       kwargs = self._input_kwargs
  21.       return self._set(**kwargs)
  23.   def c(self):
  24.     overrides = self.getoverrides()
  25.     for i in overrides:
  26.        print(i)
  28. a = A(overrides=["Alpha", "Beta"])
  29. a.c()

它会输出：

  1. Alpha
  2. Beta
英文:

Param object required to access from a get function else it will report some error. I should access self.overrides from a get function.

E.g:

  1. class _AB(Params):
  3.     overrides = Param(Params._dummy(), "overrides", "Overrides parameters for environment setup", typeConverter=TypeConverters.toListString)
  4.     
  5.     def __init__(self, *args):
  6.         super().__init__(*args)
  7.         self._setDefault(overrides=None)
  8.     def getoverrides(self):
  9.         return self.getOrDefault(self.overrides)
  11. class A(_AB):
  12.   @keyword_only
  13.   def __init__(self, overrides):
  14.     super().__init__()
  15.     kwargs = self._input_kwargs
  16.     self.setParams(**kwargs)
  18.   @keyword_only
  19.   def setParams(self, overrides: List[str]):
  20.       kwargs = self._input_kwargs
  21.       return self._set(**kwargs)
  23.   def c(self):
  24.     overrides = self.getoverrides()
  25.     for i in overrides:
  26.        print(i)
  28. a = A(overrides=["Alpha", "Beta"])
  29. a.c()

It gives:

  1. Alpha
  2. Beta

huangapple
  • 本文由 发表于 2023年7月11日 15:03:19
  • 转载请务必保留本文链接：https://go.coder-hub.com/76659423.html
  • pyspark
  • python
  • simpletransformers
  • transformer-model
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定