如何导出具有特定行数的Bigquery表格

huangapple go评论137阅读模式
英文:

How can export Bigquery table with spesific row count

问题

I've exported bigquery table as a csv file into a bucket using this:

EXPORT DATA
 OPTIONS (
  uri = 'gs://export/test/*.csv',
  format = 'csv'
  overwrite = true
)
AS (
select * from table 
);

And It is exported as several files on the bucket, that's nice beacuse I use these files for some computanioal processes and It provides parallelism. But still file size are kinda big and I want to spesifty a limit for row count for each file. Is there any way to do that?

Is there a solution for file size limit in gcp doc: https://cloud.google.com/bigquery/docs/exporting-data#control_the_exported_file_size but it requires temp table to partion process

英文:

I've exported bigquery table as a csv file into a bucket using this:

EXPORT DATA
 OPTIONS (
  uri = 'gs://export/test/*.csv',
  format = 'csv'
  overwrite = true
)
AS (
select * from table 
);

And It is exported as several files on the bucket, that's nice beacuse I use these files for some computanioal processes and It provides parallelism. But still file size are kinda big and I want to spesifty a limit for row count for each file. Is there any way to do that?

Is there a solution for file size limit in gcp doc: https://cloud.google.com/bigquery/docs/exporting-data#control_the_exported_file_size but it requires temp table to partion process

答案1

得分: 1

如@guillaume blaquiere在评论中提到的:

您可以省略temps表,但这会更昂贵。关键在于对每个“分区”(或块)执行一个“导出数据”语句。可以在分区表上执行,也可以即兴执行。随您的意愿。

将答案发布为“社区共享”,因为这是最佳实践,有助于以后可能遇到此用例的社区。

随时编辑此答案以获取附加信息。

英文:

As @guillaume blaquiere mentioned in the comments:

You can omit the temps table but it will be more expensive. The trick here is to perform an export data statement for each "partition" (or chunk). Do it on a partitioned table, or do it on the fly. As you wish.

Posting the answer as community wiki as this is the BEST PRACTICE and for the benefit of the community that might encounter this use case in the future.

Feel free to edit this answer for additional information.

huangapple
  • 本文由 发表于 2023年3月7日 14:39:44
  • 转载请务必保留本文链接:https://go.coder-hub.com/75658699.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定