英文:
How to convert to avro data to date
问题
我有一个如下的 Avro 模式...
"namespace": "example.avro",
"type": "record",
"name": "UserDate",
"fields": [
{"name": "name", "type": "string"},
{"name": "date", "type": [{"type":"int","logicalType":"date"}, "null"]},
{"name": "datenotnullable", "type": {"type":"string","logicalType":"date"}}
]
当我检索 Avro 数据时,我计划检测 logicalType,然后在 logicalType 为 "date" 时执行日期转换。
我该如何实现并完成这一步骤呢?
DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>(schema);
GenericRecord user = null;
try(DataFileReader<GenericRecord> dataFileReader = new DataFileReader<GenericRecord>(file,
datumReader)){
while (dataFileReader.hasNext()) {
user = dataFileReader.next(user);
}
}
从上述代码中获取的数据仍然是原始的基本类型。
我不确定如何使这些数据符合定义的 logicalType?
从阅读中我认为我需要以某种方式实现 Conversion,但不确定如何操作...
有任何帮助吗?
英文:
I have an avro schema as per below...
"namespace": "example.avro",
"type": "record",
"name": "UserDate",
"fields": [
{"name": "name", "type": "string"},
{"name": "date", "type": [{"type":"int","logicalType":"date"}, "null"]},
{"name": "datenotnullable", "type": {"type":"string","logicalType":"date"}}
]
When I retrieve the avro data, I plan to detect the logicalType and then do the conversion to date if the logicalType is "date"
How do I go about and achieve this?
DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>(schema);
GenericRecord user = null;
try(DataFileReader<GenericRecord> dataFileReader = new DataFileReader<GenericRecord>(file,
datumReader)){
while (dataFileReader.hasNext()) {
user = dataFileReader.next(user);
}
}
The data coming from above code is still the original primitive type.
I am not sure how to have this data as per defined logicalType?
From reading, I think somehow I need to implement Conversion, but not sure how to do this...
Any help?
答案1
得分: 2
我在这里找到了答案:https://stackoverflow.com/a/52041154/3663854
根据下面的修改,我的代码运行起来了:
final GenericData genericData = new GenericData();
genericData.addLogicalTypeConversion(new MyTimestampConversion());
DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>(schema1, schema1, genericData);
//DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>(schema); //原始代码
File file2 = new File("long.avro");
GenericRecord user = null;
try(DataFileReader<GenericRecord> dataFileReader = new DataFileReader<GenericRecord>(file, datumReader)){
while (dataFileReader.hasNext()) {
user = dataFileReader.next(user);
}
}
添加的转换部分:
public static class MyTimestampConversion extends Conversion<String> {
public MyTimestampConversion() {
}
public Class<String> getConvertedType() {
return String.class;
}
public String getLogicalTypeName() {
return "timestamp-millis";
}
public String fromLong(Long millisFromEpoch, Schema schema, LogicalType type) {
return (new DateTime(millisFromEpoch, DateTimeZone.UTC)).toString();
}
public Long toLong(String timestamp, Schema schema, LogicalType type) {
return Long.valueOf(timestamp);
}
public String fromCharSequence(CharSequence value, Schema schema, LogicalType type) {
return (new DateTime(value, DateTimeZone.UTC)).toString();
}
public CharSequence toCharSequence(String value, Schema schema, LogicalType type) {
return value;
}
}
结果(原始,不工作):
{"DateModified": 520171631042}
{"DateModified": 0}
结果(当前,工作):
{"DateModified": "1986-06-26T12:07:11.042Z"}
{"DateModified": "1970-01-01T00:00:00.000Z"}
英文:
I found the answer here https://stackoverflow.com/a/52041154/3663854
Modifying my code as per below, it work
final GenericData genericData = new GenericData();
genericData.addLogicalTypeConversion(new MyTimestampConversion());
DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>(schema1, schema1, genericData);
//DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>(schema); //original code
File file2 = new File("long.avro");
GenericRecord user = null;
try(DataFileReader<GenericRecord> dataFileReader = new DataFileReader<GenericRecord>(file, datumReader)){
while (dataFileReader.hasNext()) {
user = dataFileReader.next(user);
}
}
Added Conversion
public static class MyTimestampConversion extends Conversion<String> {
public MyTimestampConversion() {
}
public Class<String> getConvertedType() {
return String.class;
}
public String getLogicalTypeName() {
return "timestamp-millis";
}
public String fromLong(Long millisFromEpoch, Schema schema, LogicalType type) {
return (new DateTime(millisFromEpoch, DateTimeZone.UTC)).toString();
}
public Long toLong(String timestamp, Schema schema, LogicalType type) {
return new Long(timestamp);
}
public String fromCharSequence(CharSequence value, Schema schema, LogicalType type) {
return (new DateTime(value, DateTimeZone.UTC)).toString();
}
public CharSequence toCharSequence(String value, Schema schema, LogicalType type) {
return value;
}
}
Result (original, not working)
{"DateModified": 520171631042}
{"DateModified": 0}
Result (current, working)
{"DateModified": "1986-06-26T12:07:11.042Z"}
{"DateModified": "1970-01-01T00:00:00.000Z"}
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论