英文:
How can export Bigquery table with spesific row count
问题
I've exported bigquery table as a csv file into a bucket using this:
EXPORT DATA
OPTIONS (
uri = 'gs://export/test/*.csv',
format = 'csv'
overwrite = true
)
AS (
select * from table
);
And It is exported as several files on the bucket, that's nice beacuse I use these files for some computanioal processes and It provides parallelism. But still file size are kinda big and I want to spesifty a limit for row count for each file. Is there any way to do that?
Is there a solution for file size limit in gcp doc: https://cloud.google.com/bigquery/docs/exporting-data#control_the_exported_file_size but it requires temp table to partion process
英文:
I've exported bigquery table as a csv file into a bucket using this:
EXPORT DATA
OPTIONS (
uri = 'gs://export/test/*.csv',
format = 'csv'
overwrite = true
)
AS (
select * from table
);
And It is exported as several files on the bucket, that's nice beacuse I use these files for some computanioal processes and It provides parallelism. But still file size are kinda big and I want to spesifty a limit for row count for each file. Is there any way to do that?
Is there a solution for file size limit in gcp doc: https://cloud.google.com/bigquery/docs/exporting-data#control_the_exported_file_size but it requires temp table to partion process
答案1
得分: 1
如@guillaume blaquiere在评论中提到的:
您可以省略temps表,但这会更昂贵。关键在于对每个“分区”(或块)执行一个“导出数据”语句。可以在分区表上执行,也可以即兴执行。随您的意愿。
将答案发布为“社区共享”,因为这是最佳实践,有助于以后可能遇到此用例的社区。
随时编辑此答案以获取附加信息。
英文:
As @guillaume blaquiere mentioned in the comments:
You can omit the temps table but it will be more expensive. The trick here is to perform an export data
statement for each "partition" (or chunk). Do it on a partitioned table, or do it on the fly. As you wish.
Posting the answer as community wiki as this is the BEST PRACTICE and for the benefit of the community that might encounter this use case in the future.
Feel free to edit this answer for additional information.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论