英文:
How can I efficiently copy over 10,000 sharded tables between GCP projects in Big Query?
问题
我目前正在为我的公司进行迁移项目，其中我的任务是从一个GCP项目迁移数据到另一个BigQuery项目中。我正在为进行迁移的公司仍在使用分片表，并且他们希望保持这种格式。这意味着我需要将一个项目中一个数据集中的超过10,000个表的选定部分复制到我正在迁移的另一个项目中的数据集中。我对GCP环境相对较新，想知道是否有人能够就我如何最优地完成这项任务给予建议。这些分片表分布在6个不同的表之间。
目前我考虑的方法是在BigQuery中编写一个动态查询，利用数据集架构生成一堆Cloud Shell复制命令。思路是将此查询的结果导出到Google表格，然后分批将这些命令复制到Cloud Shell中。
英文:
I am currently working on a migration project for my company where I am tasked to migrate data from one GCP project to another in Big Query. The company that I am doing the migraion for is still using sharded tables and they want to keep it in that format.This means that I need to copy a selection of over 10 000 tables from a dataset in one project over to a dataset in the other project that I am migrating towards. I am relatively new to the GCP environemt and was wondering if anyone is able to give me advice on optimal ways that I can accomplish this. The sharded tables are split between 6 diffrent tables.
Currently the approach that I am considering is writing a dynamic query in big query to use the dataset schema to generate a bunch of cloud shell copy commands. The idea is to export the results of this query to google sheets and then copy these commands into cloud shell in batches.
答案1
得分: 0
你可以从源项目将数据集1复制到目标项目。
英文:
You can copy the dataset from your source project to the target
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论