How to avoid warning message when read BigQuery data to custom data type: Can't verify serialized elements of type BoundedSource

huangapple go评论66阅读模式
英文:

How to avoid warning message when read BigQuery data to custom data type: Can't verify serialized elements of type BoundedSource

问题

我在这里定义了一个自定义数据类型引用文档。
https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java#L127

并且使用以下代码从BigQuery中读取数据。
https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java#L375

警告信息:
无法验证类型为BoundedSource的序列化元素是否具有良定义的equals方法。这可能会在某些PipelineRunner上产生错误结果。

此消息出现在步骤TriggerIdCreation/Read(CreateSource)/Read(CreateSource)/Read(BoundedToUnboundedSourceAdapter)/StripIds.out0

我尝试为自定义数据类型类添加equals()方法,就像这样:

@Override
public boolean equals(Object object) {
    if (this == object) return true;
    if (object == null || getClass() != object.getClass()) return false;
    if (!super.equals(object)) return false;
    WeatherData that = (ErrorTelop) object;
    return Objects.equals(xxx, that.xxx) && Objects.equals(yyy, that.yyy);
}

但这并没有起到作用。

有没有人有想法如何避免这个警告?

英文:

I defined a custom data type reference the document here.
https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java#L127

And read data from BigQuery using the code below.
https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java#L375

Warning message:
Can't verify serialized elements of type BoundedSource have well defined equals method. This may produce incorrect results on some PipelineRunner.

This message occurs at step TriggerIdCreation/Read(CreateSource)/Read(CreateSource)/Read(BoundedToUnboundedSourceAdapter)/StripIds.out0

I tried to add equals() method to the custom data type class like this

    @Override
    public boolean equals(Object object) {
        if (this == object) return true;
        if (object == null || getClass() != object.getClass()) return false;
        if (!super.equals(object)) return false;
        WeatherData that = (ErrorTelop) object;
        return Objects.equals(xxx, that.xxx) &&
               Objects.equals(yyy, that.yyy);
    }

which does not help.

Any one have an idea to avoid this warning?

答案1

得分: 0

你所看到的警告似乎与你目前的操作无关。我认为警告是由 Apache Beam 自身的某些操作引发的。它实际上抱怨的是 BoundedSource,这是 Beam 内部的一种类型,而不是你的自定义类型。从我对代码的检查中看,这很可能与提到的 BoundedToUnboundedSourceAdapter 有关。

如果你的流水线正常工作,那么你可能可以忽略这个警告。如果你确实想通知某人,你可以联系Beam 用户或开发者列表。

英文:

The warning you're getting doesn't seem to be due to anything you're doing. I think the warning is coming from something Apache Beam itself is doing. The actual type that it's complaining about is BoundedSource, an internal Beam type, not your custom type, and from my looking through the code it's most likely related to the BoundedToUnboundedSourceAdapter mentioned there.

If your pipeline is working correctly, then you can probably ignore this. If you do want to alert someone, you could contact the Beam user or dev lists.

huangapple
  • 本文由 发表于 2020年9月2日 13:03:17
  • 转载请务必保留本文链接:https://go.coder-hub.com/63698980.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定