如何将 Avro 数据转换为日期

huangapple
go评论98阅读模式
英文:

How to convert to avro data to date

问题

我有一个如下的 Avro 模式...

  1.  "namespace": "example.avro",
  2.  "type": "record",
  3.  "name": "UserDate",
  4.  "fields": [
  5.      {"name": "name", "type": "string"},         
  6. 	 {"name": "date", "type": [{"type":"int","logicalType":"date"}, "null"]},
  7.      {"name": "datenotnullable", "type": {"type":"string","logicalType":"date"}}	 
  8.  ]

当我检索 Avro 数据时，我计划检测 logicalType，然后在 logicalType 为 "date" 时执行日期转换。

我该如何实现并完成这一步骤呢？

  1.     DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>(schema);
  2.     GenericRecord user = null;
  3.     try(DataFileReader<GenericRecord> dataFileReader = new DataFileReader<GenericRecord>(file, 
  4.       datumReader)){
  5.     while (dataFileReader.hasNext()) {
  7.     user = dataFileReader.next(user);
  8.   }
  9. }

从上述代码中获取的数据仍然是原始的基本类型。

我不确定如何使这些数据符合定义的 logicalType？

从阅读中我认为我需要以某种方式实现 Conversion，但不确定如何操作...

有任何帮助吗？

英文:

I have an avro schema as per below...

  1.  &quot;namespace&quot;: &quot;example.avro&quot;,
  2.  &quot;type&quot;: &quot;record&quot;,
  3.  &quot;name&quot;: &quot;UserDate&quot;,
  4.  &quot;fields&quot;: [
  5.      {&quot;name&quot;: &quot;name&quot;, &quot;type&quot;: &quot;string&quot;},         
  6. 	 {&quot;name&quot;: &quot;date&quot;, &quot;type&quot;: [{&quot;type&quot;:&quot;int&quot;,&quot;logicalType&quot;:&quot;date&quot;}, &quot;null&quot;]},
  7.      {&quot;name&quot;: &quot;datenotnullable&quot;, &quot;type&quot;: {&quot;type&quot;:&quot;string&quot;,&quot;logicalType&quot;:&quot;date&quot;}}	 
  8.  ]

When I retrieve the avro data, I plan to detect the logicalType and then do the conversion to date if the logicalType is "date"

How do I go about and achieve this?

  1.     DatumReader&lt;GenericRecord&gt; datumReader = new GenericDatumReader&lt;GenericRecord&gt;(schema);
  2.     GenericRecord user = null;
  3.     try(DataFileReader&lt;GenericRecord&gt; dataFileReader = new DataFileReader&lt;GenericRecord&gt;(file, 
  4.       datumReader)){
  5.     while (dataFileReader.hasNext()) {
  7.     user = dataFileReader.next(user);
  8.   }
  9. }

The data coming from above code is still the original primitive type.

I am not sure how to have this data as per defined logicalType?

From reading, I think somehow I need to implement Conversion, but not sure how to do this...

Any help?

答案1

得分: 2

我在这里找到了答案：https://stackoverflow.com/a/52041154/3663854

根据下面的修改，我的代码运行起来了：

  1. final GenericData genericData = new GenericData();
  2. genericData.addLogicalTypeConversion(new MyTimestampConversion());
  3. DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>(schema1, schema1, genericData);
  4. //DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>(schema); //原始代码
  6. File file2 = new File("long.avro");
  8. GenericRecord user = null;
  10. try(DataFileReader<GenericRecord> dataFileReader = new DataFileReader<GenericRecord>(file, datumReader)){
  11.     while (dataFileReader.hasNext()) {
  12.         user = dataFileReader.next(user);
  13.     }
  14. }

添加的转换部分：

  1. public static class MyTimestampConversion extends Conversion<String> {
  2.     public MyTimestampConversion() {
  3.     }
  5.     public Class<String> getConvertedType() {
  6.         return String.class;
  7.     }
  9.     public String getLogicalTypeName() {
  10.         return "timestamp-millis";
  11.     }
  13.     public String fromLong(Long millisFromEpoch, Schema schema, LogicalType type) {
  14.         return (new DateTime(millisFromEpoch, DateTimeZone.UTC)).toString();
  15.     }
  17.     public Long toLong(String timestamp, Schema schema, LogicalType type) {
  18.         return Long.valueOf(timestamp);
  19.     }
  21.     public String fromCharSequence(CharSequence value, Schema schema, LogicalType type) {
  22.         return (new DateTime(value, DateTimeZone.UTC)).toString();
  23.     }
  25.     public CharSequence toCharSequence(String value, Schema schema, LogicalType type) {
  26.         return value;
  27.     }
  28. }

结果（原始，不工作）：

  1. {"DateModified": 520171631042}
  2. {"DateModified": 0}

结果（当前，工作）：

  1. {"DateModified": "1986-06-26T12:07:11.042Z"}
  2. {"DateModified": "1970-01-01T00:00:00.000Z"}
英文:

I found the answer here https://stackoverflow.com/a/52041154/3663854

Modifying my code as per below, it work

  1.     final GenericData genericData = new GenericData();
  2.     genericData.addLogicalTypeConversion(new MyTimestampConversion());
  3.     DatumReader&lt;GenericRecord&gt; datumReader = new GenericDatumReader&lt;GenericRecord&gt;(schema1, schema1, genericData);
  4.     //DatumReader&lt;GenericRecord&gt; datumReader = new GenericDatumReader&lt;GenericRecord&gt;(schema); //original code
  5.     
  6.     File file2 = new File(&quot;long.avro&quot;);
  8.     GenericRecord user = null;
  9.     
  10.     try(DataFileReader&lt;GenericRecord&gt; dataFileReader = new DataFileReader&lt;GenericRecord&gt;(file, datumReader)){
  11.         while (dataFileReader.hasNext()) {
  13.             user = dataFileReader.next(user);
  14.         }
  15.     }

Added Conversion

  1. public static class MyTimestampConversion extends Conversion&lt;String&gt; {
  2.     public MyTimestampConversion() {
  3.     }
  5.     public Class&lt;String&gt; getConvertedType() {
  6.         return String.class;
  7.     }
  9.     public String getLogicalTypeName() {
  10.         return &quot;timestamp-millis&quot;;
  11.     }
  13.     public String fromLong(Long millisFromEpoch, Schema schema, LogicalType type) {
  14.         return (new DateTime(millisFromEpoch, DateTimeZone.UTC)).toString();
  15.     }
  17.     public Long toLong(String timestamp, Schema schema, LogicalType type) {
  18.         return new Long(timestamp);
  19.     }
  21.     public String fromCharSequence(CharSequence value, Schema schema, LogicalType type) {
  22.         return (new DateTime(value, DateTimeZone.UTC)).toString();
  23.     }
  25.     public CharSequence toCharSequence(String value, Schema schema, LogicalType type) {
  26.         return value;
  27.     }
  28. }

Result (original, not working)

  1. {&quot;DateModified&quot;: 520171631042}
  2. {&quot;DateModified&quot;: 0}

Result (current, working)

  1. {&quot;DateModified&quot;: &quot;1986-06-26T12:07:11.042Z&quot;}
  2. {&quot;DateModified&quot;: &quot;1970-01-01T00:00:00.000Z&quot;}

huangapple
  • 本文由 发表于 2020年9月30日 15:38:45
  • 转载请务必保留本文链接：https://go.coder-hub.com/64132967.html
  • avro
  • java
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定