在pyspark中使用Params。

huangapple go评论54阅读模式
英文:

Use of Params in pyspak

问题

在这个示例中,我试图将overrides作为一个Params对象使用,并希望它被用作字符串列表。

但是,我无法使用下面的代码分配它的值。

class _AB(Params):

    overrides = Param(Params._dummy(), "overrides", "Parameters for environment setup", typeConverter=TypeConverters.toListString)
    
    def __init__(self, *args):
        super().__init__(*args)
        self._setDefault(overrides=None)
        

class A(_AB):
  @keyword_only
  def __init__(self, overrides):
    super().__init__()
    kwargs = self._input_kwargs
    self.setParams(**kwargs)

  @keyword_only
  def setParams(self, overrides: List[str]):
      kwargs = self._input_kwargs
      print(kwargs)
      return self._set(**kwargs)

  def c(self):
    print(self.overrides.__dict__['typeConverter'].__dict__)
    for i in self.overrides:
       print(i)

a = A(overrides=["dsfs", "Sdf"])
a.c()

当我在函数c内打印它时,它给我一个空字典。
当我在函数c内打印它时,它给我一个错误:

TypeError: 'Param' object is not iterable

我猜这是因为它无法将某个值分配给overrides变量。

英文:

In this example, I am trying to use overrides as a Params object and I want it to be used as a list of strings.

But I am not able to assign its value using the below code.

class _AB(Params):

    overrides = Param(Params._dummy(), "overrides", "Parameters for environment setup", typeConverter=TypeConverters.toListString)
    
    def __init__(self, *args):
        super().__init__(*args)
        self._setDefault(overrides=None)
        

class A(_AB):
  @keyword_only
  def __init__(self, overrides):
    super().__init__()
    kwargs = self._input_kwargs
    self.setParams(**kwargs)

  @keyword_only
  def setParams(self, overrides: List[str]):
      kwargs = self._input_kwargs
      print(kwargs)
      return self._set(**kwargs)

  def c(self):
    print(self.overrides.__dict__['typeConverter'].__dict__)
    for i in self.overrides:
       print(i)

a = A(overrides=["dsfs", "Sdf"])
a.c()

It gives me a blank dictionary when I print it inside function c.
It gives me an error:

TypeError: 'Param' object is not iterable

I guess it's happening because it's not able to assign some value to overrides variable.

答案1

得分: 1

Param对象需要从一个获取函数中访问,否则将报告一些错误。我应该从一个获取函数中访问self.overrides。

例如:

class _AB(Params):

    overrides = Param(Params._dummy(), "overrides", "Overrides parameters for environment setup", typeConverter=TypeConverters.toListString)
    
    def __init__(self, *args):
        super().__init__(*args)
        self._setDefault(overrides=None)
    def getoverrides(self):
        return self.getOrDefault(self.overrides)

class A(_AB):
  @keyword_only
  def __init__(self, overrides):
    super().__init__()
    kwargs = self._input_kwargs
    self.setParams(**kwargs)

  @keyword_only
  def setParams(self, overrides: List[str]):
      kwargs = self._input_kwargs
      return self._set(**kwargs)

  def c(self):
    overrides = self.getoverrides()
    for i in overrides:
       print(i)

a = A(overrides=["Alpha", "Beta"])
a.c()

它会输出:

Alpha
Beta
英文:

Param object required to access from a get function else it will report some error. I should access self.overrides from a get function.

E.g:

class _AB(Params):

    overrides = Param(Params._dummy(), "overrides", "Overrides parameters for environment setup", typeConverter=TypeConverters.toListString)
    
    def __init__(self, *args):
        super().__init__(*args)
        self._setDefault(overrides=None)
    def getoverrides(self):
        return self.getOrDefault(self.overrides)

class A(_AB):
  @keyword_only
  def __init__(self, overrides):
    super().__init__()
    kwargs = self._input_kwargs
    self.setParams(**kwargs)

  @keyword_only
  def setParams(self, overrides: List[str]):
      kwargs = self._input_kwargs
      return self._set(**kwargs)

  def c(self):
    overrides = self.getoverrides()
    for i in overrides:
       print(i)

a = A(overrides=["Alpha", "Beta"])
a.c()

It gives:

Alpha
Beta

huangapple
  • 本文由 发表于 2023年7月11日 15:03:19
  • 转载请务必保留本文链接:https://go.coder-hub.com/76659423.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定