使用Kusto从存储中摄取多个文件

huangapple go评论56阅读模式
英文:

Ingest multiple files from storage using Kusto

问题

我使用.export 命令 导出了约 6000 万行到存储,如下所示:

.export async to json (h@"https://contoso.blob.core.windows.net/oversub/adx/VmCounterFiveMinutes/2023-05-20;storage-account-access-key") <|
contosoEntity

我从查询中删除了查询的一部分,但导出正常。导出了 192 个文件,共计 62374652 行

我还需要将数据导入到另一个 Kusto 群集,如何使用.ingest 命令从存储中摄取所有存储中特定文件夹中的 .json 文件呢?

如果有其他更好的方案用于将数据摄取回另一个 Kusto 群集,我将很高兴知道。

我尝试过但失败了的方法:

1-

.ingest into table VmCounterFiveMinuteRoleInstanceCentralBondTable_2 (h@'https://contoso.blob.core.windows.net/oversub/adx/VmCounterFiveMinutes/2023-05-20/*.json;storage-account-access-key')

2-

.ingest into table VmCounterFiveMinuteRoleInstanceCentralBondTable_2 (h@'https://contoso.blob.core.windows.net/oversub/adx/VmCounterFiveMinutes/2023-05-20/*.ext;storage-account-access-key')
    with (format='json')
英文:

I exported about 60 millions rows using the .export command to a storage as my KQL script below:

.export async to json (h@"https://contoso.blob.core.windows.net/oversub/adx/VmCounterFiveMinutes/2023-05-20;storage-account-access-key") <|
contosoEntity

I removed sentity parts from my query, but the exportation works fine. It exported 192 file(s), total of 62374652 rows.

I also need to import into another Kusto cluster, how can I use the .ingest command from storage to ingest all .json files from a specific folder from storage?

If there any other better solution to ingest back to another Kusto cluster, I will be glad to know.

What I tried and failed:

1-

.ingest into table VmCounterFiveMinuteRoleInstanceCentralBondTable_2 (h@'https://contoso.blob.core.windows.net/oversub/adx/VmCounterFiveMinutes/2023-05-20/*.json;storage-account-access-key')

2-

.ingest into table VmCounterFiveMinuteRoleInstanceCentralBondTable_2 (h@'https://contoso.blob.core.windows.net/oversub/adx/VmCounterFiveMinutes/2023-05-20/*.ext;storage-account-access-key')
    with (format='json')

答案1

得分: 1

你可以使用名为"lightingest"的命令行工具,或在Kusto Web资源管理器中使用OneClick向导工具

英文:

You can use the command line tool called "lightingest" or use the OneClick wizard tool in Kusto Web explorer

答案2

得分: 0

你在运行(不建议使用的)直接导入命令(.ingest)时无法指定通配符(*)路径。

你可以使用LightIngest导入向导来导入已导出的文件(如果需要,可以筛选 blob 名称前缀)。

英文:

you can't specify a wildcarded (*) path when running the (unrecommended) direct ingestion command (.ingest).

you could use LightIngest or the Ingestion wizard to ingest the files you've exported (with filters on blob name prefix, if required)

huangapple
  • 本文由 发表于 2023年5月25日 01:51:46
  • 转载请务必保留本文链接:https://go.coder-hub.com/76326225.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定