如何将 Avro 数据转换为日期

huangapple go评论72阅读模式
英文:

How to convert to avro data to date

问题

我有一个如下的 Avro 模式...

 "namespace": "example.avro",
 "type": "record",
 "name": "UserDate",
 "fields": [
     {"name": "name", "type": "string"},         
	 {"name": "date", "type": [{"type":"int","logicalType":"date"}, "null"]},
     {"name": "datenotnullable", "type": {"type":"string","logicalType":"date"}}	 
 ]

当我检索 Avro 数据时,我计划检测 logicalType,然后在 logicalType 为 "date" 时执行日期转换。

我该如何实现并完成这一步骤呢?

    DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>(schema);
    GenericRecord user = null;
    try(DataFileReader<GenericRecord> dataFileReader = new DataFileReader<GenericRecord>(file, 
      datumReader)){
    while (dataFileReader.hasNext()) {

    user = dataFileReader.next(user);
  }
}

从上述代码中获取的数据仍然是原始的基本类型。

我不确定如何使这些数据符合定义的 logicalType?

从阅读中我认为我需要以某种方式实现 Conversion,但不确定如何操作...

有任何帮助吗?

英文:

I have an avro schema as per below...

 &quot;namespace&quot;: &quot;example.avro&quot;,
 &quot;type&quot;: &quot;record&quot;,
 &quot;name&quot;: &quot;UserDate&quot;,
 &quot;fields&quot;: [
     {&quot;name&quot;: &quot;name&quot;, &quot;type&quot;: &quot;string&quot;},         
	 {&quot;name&quot;: &quot;date&quot;, &quot;type&quot;: [{&quot;type&quot;:&quot;int&quot;,&quot;logicalType&quot;:&quot;date&quot;}, &quot;null&quot;]},
     {&quot;name&quot;: &quot;datenotnullable&quot;, &quot;type&quot;: {&quot;type&quot;:&quot;string&quot;,&quot;logicalType&quot;:&quot;date&quot;}}	 
 ]

When I retrieve the avro data, I plan to detect the logicalType and then do the conversion to date if the logicalType is "date"

How do I go about and achieve this?

    DatumReader&lt;GenericRecord&gt; datumReader = new GenericDatumReader&lt;GenericRecord&gt;(schema);
    GenericRecord user = null;
    try(DataFileReader&lt;GenericRecord&gt; dataFileReader = new DataFileReader&lt;GenericRecord&gt;(file, 
      datumReader)){
    while (dataFileReader.hasNext()) {

    user = dataFileReader.next(user);
  }
}

The data coming from above code is still the original primitive type.

I am not sure how to have this data as per defined logicalType?

From reading, I think somehow I need to implement Conversion, but not sure how to do this...

Any help?

答案1

得分: 2

我在这里找到了答案:https://stackoverflow.com/a/52041154/3663854

根据下面的修改,我的代码运行起来了:

final GenericData genericData = new GenericData();
genericData.addLogicalTypeConversion(new MyTimestampConversion());
DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>(schema1, schema1, genericData);
//DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>(schema); //原始代码

File file2 = new File("long.avro");

GenericRecord user = null;

try(DataFileReader<GenericRecord> dataFileReader = new DataFileReader<GenericRecord>(file, datumReader)){
    while (dataFileReader.hasNext()) {
        user = dataFileReader.next(user);
    }
}

添加的转换部分:

public static class MyTimestampConversion extends Conversion<String> {
    public MyTimestampConversion() {
    }

    public Class<String> getConvertedType() {
        return String.class;
    }

    public String getLogicalTypeName() {
        return "timestamp-millis";
    }

    public String fromLong(Long millisFromEpoch, Schema schema, LogicalType type) {
        return (new DateTime(millisFromEpoch, DateTimeZone.UTC)).toString();
    }

    public Long toLong(String timestamp, Schema schema, LogicalType type) {
        return Long.valueOf(timestamp);
    }

    public String fromCharSequence(CharSequence value, Schema schema, LogicalType type) {
        return (new DateTime(value, DateTimeZone.UTC)).toString();
    }

    public CharSequence toCharSequence(String value, Schema schema, LogicalType type) {
        return value;
    }
}

结果(原始,不工作):

{"DateModified": 520171631042}
{"DateModified": 0}

结果(当前,工作):

{"DateModified": "1986-06-26T12:07:11.042Z"}
{"DateModified": "1970-01-01T00:00:00.000Z"}
英文:

I found the answer here https://stackoverflow.com/a/52041154/3663854

Modifying my code as per below, it work

    final GenericData genericData = new GenericData();
    genericData.addLogicalTypeConversion(new MyTimestampConversion());
    DatumReader&lt;GenericRecord&gt; datumReader = new GenericDatumReader&lt;GenericRecord&gt;(schema1, schema1, genericData);
    //DatumReader&lt;GenericRecord&gt; datumReader = new GenericDatumReader&lt;GenericRecord&gt;(schema); //original code
    
    File file2 = new File(&quot;long.avro&quot;);

    GenericRecord user = null;
    
    try(DataFileReader&lt;GenericRecord&gt; dataFileReader = new DataFileReader&lt;GenericRecord&gt;(file, datumReader)){
        while (dataFileReader.hasNext()) {

            user = dataFileReader.next(user);
        }
    }

Added Conversion

public static class MyTimestampConversion extends Conversion&lt;String&gt; {
    public MyTimestampConversion() {
    }

    public Class&lt;String&gt; getConvertedType() {
        return String.class;
    }

    public String getLogicalTypeName() {
        return &quot;timestamp-millis&quot;;
    }

    public String fromLong(Long millisFromEpoch, Schema schema, LogicalType type) {
        return (new DateTime(millisFromEpoch, DateTimeZone.UTC)).toString();
    }

    public Long toLong(String timestamp, Schema schema, LogicalType type) {
        return new Long(timestamp);
    }

    public String fromCharSequence(CharSequence value, Schema schema, LogicalType type) {
        return (new DateTime(value, DateTimeZone.UTC)).toString();
    }

    public CharSequence toCharSequence(String value, Schema schema, LogicalType type) {
        return value;
    }
}

Result (original, not working)

{&quot;DateModified&quot;: 520171631042}
{&quot;DateModified&quot;: 0}

Result (current, working)

{&quot;DateModified&quot;: &quot;1986-06-26T12:07:11.042Z&quot;}
{&quot;DateModified&quot;: &quot;1970-01-01T00:00:00.000Z&quot;}

huangapple
  • 本文由 发表于 2020年9月30日 15:38:45
  • 转载请务必保留本文链接:https://go.coder-hub.com/64132967.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定