英文:
How to avoid warning message when read BigQuery data to custom data type: Can't verify serialized elements of type BoundedSource
问题
我在这里定义了一个自定义数据类型引用文档。
https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java#L127
并且使用以下代码从BigQuery中读取数据。
https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java#L375
警告信息：
无法验证类型为BoundedSource的序列化元素是否具有良定义的equals方法。这可能会在某些PipelineRunner上产生错误结果。
此消息出现在步骤TriggerIdCreation/Read(CreateSource)/Read(CreateSource)/Read(BoundedToUnboundedSourceAdapter)/StripIds.out0
我尝试为自定义数据类型类添加equals()方法，就像这样：
@Override
public boolean equals(Object object) {
if (this == object) return true;
if (object == null || getClass() != object.getClass()) return false;
if (!super.equals(object)) return false;
WeatherData that = (ErrorTelop) object;
return Objects.equals(xxx, that.xxx) && Objects.equals(yyy, that.yyy);
}
但这并没有起到作用。
有没有人有想法如何避免这个警告？
英文:
I defined a custom data type reference the document here.
https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java#L127
And read data from BigQuery using the code below.
https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java#L375
Warning message:
Can't verify serialized elements of type BoundedSource have well defined equals method. This may produce incorrect results on some PipelineRunner.
This message occurs at step TriggerIdCreation/Read(CreateSource)/Read(CreateSource)/Read(BoundedToUnboundedSourceAdapter)/StripIds.out0
I tried to add equals() method to the custom data type class like this
@Override
public boolean equals(Object object) {
if (this == object) return true;
if (object == null || getClass() != object.getClass()) return false;
if (!super.equals(object)) return false;
WeatherData that = (ErrorTelop) object;
return Objects.equals(xxx, that.xxx) &&
Objects.equals(yyy, that.yyy);
}
which does not help.
Any one have an idea to avoid this warning?
答案1
得分: 0
你所看到的警告似乎与你目前的操作无关。我认为警告是由 Apache Beam 自身的某些操作引发的。它实际上抱怨的是 BoundedSource，这是 Beam 内部的一种类型，而不是你的自定义类型。从我对代码的检查中看，这很可能与提到的 BoundedToUnboundedSourceAdapter 有关。
如果你的流水线正常工作，那么你可能可以忽略这个警告。如果你确实想通知某人，你可以联系Beam 用户或开发者列表。
英文:
The warning you're getting doesn't seem to be due to anything you're doing. I think the warning is coming from something Apache Beam itself is doing. The actual type that it's complaining about is BoundedSource, an internal Beam type, not your custom type, and from my looking through the code it's most likely related to the BoundedToUnboundedSourceAdapter mentioned there.
If your pipeline is working correctly, then you can probably ignore this. If you do want to alert someone, you could contact the Beam user or dev lists.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论