遇到在Snowflake中合并现有表和S3阶段Parquet文件时的问题。

huangapple go评论54阅读模式
英文:

Facing issue while merging existing table in snowflake with s3 stage parquet file

问题

我尝试执行以下脚本:

MERGE INTO data_stage.test_temp4 ex USING (
SELECT
$1:device_attr_hk,
$1:applicationruntime_raw,
$1:devicefamily_raw,
$1:deviceprofile_raw,
$1:manufacturer_raw,
$1:model_raw,
$1:application_runtime_os,
$1:device_profile,
$1:device_family,
$1:device_platform,
$1:manufacturer,
$1:model,
$1:device_type,
$1:creation_timestamp,
$1:change_hkey
FROM @dss.test_plus.test_plus_stage/dev/sub_solutions/test/ (file_format => 'parquet', PATTERN => '.*.parquet')
) ne ON ex.device_attr_hk = ne.device_attr_hk WHEN MATCHED THEN
UPDATE SET device_attr_hk = ne.device_attr_hk, applicationruntime_raw = ne.applicationruntime_raw, devicefamily_raw = ne.devicefamily_raw, deviceprofile_raw = ne.deviceprofile_raw, manufacturer_raw = ne.manufacturer_raw, model_raw = ne.model_raw, application_runtime_os = ne.application_runtime_os, device_profile = ne.device_profile, device_family = ne.device_family, device_platform = ne.device_platform, manufacturer = ne.manufacturer,  model = ne.model,  device_platform = ne.device_platform ,  manufacturer = ne.manufacturer,  model = ne.model,  device_type = ne.device_type,  creation_timestamp = ne.creation_timestamp,  change_hkey = ne.change_hkey WHEN NOT MATCHED THEN
INSERT
(device_attr_hk, applicationruntime_raw, devicefamily_raw, deviceprofile_raw, manufacturer_raw, model_raw, application_runtime_os, device_profile, device_family, device_platform, manufacturer, model, device_type, creation_timestamp, change_hkey) VALUES
(ne.device_attr_hk, ne.applicationruntime_raw, ne.devicefamily_raw, ne.deviceprofile_raw, ne.manufacturer_raw, ne.model_raw, ne.application_runtime_os, ne.device_profile, ne.device_family, ne.device_platform, ne.manufacturer, ne.model, ne.device_type, ne.creation_timestamp, ne.change_hkey);
commit;

但是我遇到了错误:SQL编译错误:文件格式'PARQUET' 不存在或未授权。

注意: 文件已经存在于stage: @dss.test_plus.test_plus_stage/dev/sub_solutions/test/,而且我能够访问它。

我怀疑为什么我无法访问此文件并进行合并操作,脚本中是否存在问题。

英文:

I tried to execute below script:

MERGE INTO  data_stage.test_temp4  ex  USING (
SELECT
$1:device_attr_hk,
$1:applicationruntime_raw,
$1:devicefamily_raw,
$1:deviceprofile_raw,
$1:manufacturer_raw,
$1:model_raw,
$1:application_runtime_os,
$1:device_profile,
$1:device_family,
$1:device_platform,
$1:manufacturer,
$1:model,
$1:device_type,
$1:creation_timestamp,
$1:change_hkey
FROM  @dss.test_plus.test_plus_stage/dev/sub_solutions/test/  (file_format => 'parquet', PATTERN => '.*.parquet')

) ne ON ex.device_attr_hk = ne.device_attr_hk WHEN MATCHED THEN
UPDATE SET device_attr_hk = ne.device_attr_hk, applicationruntime_raw = ne.applicationruntime_raw, devicefamily_raw = ne.devicefamily_raw, deviceprofile_raw = ne.deviceprofile_raw, manufacturer_raw = ne.manufacturer_raw, model_raw = ne.model_raw, application_runtime_os = ne.application_runtime_os, device_profile = ne.device_profile, device_family = ne.device_family, device_platform = ne.device_platform, manufacturer = ne.manufacturer,  model = ne.model,  device_platform = ne.device_platform ,  manufacturer = ne.manufacturer,  model = ne.model,  device_type = ne.device_type,  creation_timestamp = ne.creation_timestamp,  change_hkey = ne.change_hkey WHEN NOT MATCHED THEN
INSERT
(device_attr_hk, applicationruntime_raw, devicefamily_raw, deviceprofile_raw, manufacturer_raw, model_raw, application_runtime_os, device_profile, device_family, device_platform, manufacturer, model, device_type, creation_timestamp, change_hkey) VALUES
(ne.device_attr_hk, ne.applicationruntime_raw, ne.devicefamily_raw, ne.deviceprofile_raw, ne.manufacturer_raw, ne.model_raw, ne.application_runtime_os, ne.device_profile, ne.device_family, ne.device_platform, ne.manufacturer, ne.model, ne.device_type, ne.creation_timestamp, ne.change_hkey);
commit;

But I am getting error: SQL compilation error: File format 'PARQUET' does not exist or not authorized.

Note: file already avialable on stage: @dss.test_plus.test_plus_stage/dev/sub_solutions/test/ And I have able to access it.

I have doubt , why I am not able to access this file an merge, is there any issue in script that written.

答案1

得分: 1

你之所以收到此错误是因为您尝试使用名称为'parquet'的file_format,但该文件格式不存在。

我建议您首先创建一个命名的file_format,并在命令中使用其名称。

步骤1. 创建一个命名的文件格式,例如:

CREATE OR REPLACE FILE FORMAT my_parquet_format
  TYPE = PARQUET;

步骤2. 更改脚本中包含FROM的行如下:

FROM  @dss.test_plus.test_plus_stage/dev/sub_solutions/test/  (file_format =>'my_parquet_format', PATTERN =>'*.parquet')

然后您可以再次尝试。

英文:

You are getting this error because you try using the file_format with the name 'parquet' which does not exist

I suggest you create a named file_format first and use its name in your command.

Step 1. Create a named file format, e.g.:

CREATE OR REPLACE FILE FORMAT my_parquet_format
  TYPE = PARQUET;

Step 2. Change the line with FROM in your script as follows:

FROM  @dss.test_plus.test_plus_stage/dev/sub_solutions/test/  (file_format => 'my_parquet_format', PATTERN => '.*.parquet')

Then you can try again.

huangapple
  • 本文由 发表于 2023年3月15日 20:31:15
  • 转载请务必保留本文链接:https://go.coder-hub.com/75744727.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定