问题

Here is the translated code portion:

我有这个 JSON 文件

```json
{"created_at":"2022-01-02 12:17:43.399 UTC","updated_at":"2022-01-02 12:17:43.399 UTC"}

尝试将其读取为

read_df = spark \
            .read \
            .option("timestampFormat", "yyyy-MM-dd HH:mm:ss.SSS 'UTC'") \
            .option("inferSchema", "true") \
            .json(path)

但是推断的模式会返回给我

root
 |-- created_at: string (nullable = true)
 |-- updated_at: string (nullable = true)

我尝试通过 withColumn("timestamp",to_timestamp(col("created_at"), "yyyy-MM-dd HH:mm:ss.SSS 'UTC'")) 强制转换它，并且它有效。

我不想提供模式，而是让它自动推断，因为我有不同模式的不同文件，并且想要重复使用读取函数。

我不确定出了什么问题。

Spark 版本：3.3.2

<details>
<summary>英文:</summary>

I have this json file

```json
{&quot;created_at&quot;:&quot;2022-01-02 12:17:43.399 UTC&quot;,&quot;updated_at&quot;:&quot;2022-01-02 12:17:43.399 UTC&quot;}

Trying to read it as

read_df = spark \
            .read \
            .option("timestampFormat", "yyyy-MM-dd HH:mm:ss.SSS 'UTC'") \
            .option("inferSchema", "true") \
            .json(path)

but the inferred schema gives me back

root
 |-- created_at: string (nullable = true)
 |-- updated_at: string (nullable = true)

I've tried to forced it via withColumn("timestamp",to_timestamp(col("created_at"), "yyyy-MM-dd HH:mm:ss.SSS 'UTC'")) and it works.

I don't want to provide myself the schema but let infer it because I have different files with different schemas and want to re-use the function for reading.

I'm not sure what's wrong.

Spark versionL 3.3.2

答案1

得分: 4

时间戳的推断必须显式启用（文档，代码)：

自版本3.0.1起，默认情况下已禁用时间戳类型推断。将JSON选项inferTimestamp设置为true以启用此类类型推断。

read_df = spark \
            .read \
            .option("timestampFormat", "yyyy-MM-dd HH:mm:ss.SSS 'UTC'") \
            .option("inferSchema", "true") \
            .option("inferTimestamp", "true") \
            .json(path)

返回两个时间戳列。

read_df = spark \
            .read \
            .option("timestampFormat", "yyyy-MM-dd HH:mm:ss.SSS 'UTC'") \
            .option("inferSchema", "true") \
            .option("inferTimestamp", "true") \
            .json(path)