合并多个BigTable并查询它们

huangapple go评论36阅读模式
英文:

Union multiple BigTables and query them

问题

我们有一个产品,其中我们客户的数据保存在GCP BigTable实例中。每个客户都有自己的数据表,但所有表都位于同一个BigTable实例中。

现在我们的新要求是要有一个超级用户,能够查看所有客户的数据,并能够以一种“联合”方式查询它...一次性查询所有表。所以我正在尝试找到一种解决方法,不改变客户数据保存的方式,也不复制数据。
每个客户存储的数据量大约为6TB,同一个BigTable实例中可能会有10-20个客户。

谢谢!

英文:

We have a product where the data of our customers saved in GCP BigTable instance. Each customer has it's own table for it's own data, but all the tables located in the same instance of BigTable.

Now our new requirement, is to have a Superuser that able to see the data from all customers and able to query it in kind of "union" way...query all tables at once. So I'm trying to find a way how to solve it, without changing the way how customers data is saved and without duplicating the data.
The amount of data that is stored for each customer is about 6TB, and there might be 10-20 customers on the same instance of BigTable.

Thank you!

答案1

得分: 1

由于 Bigtable 的读请求使用流来获取请求的行的内容。建议多行读取时,使用客户端库而不是直接的 API 调用,该库使用 ADC 进行身份验证。因此,应为表设置的查看器角色应允许您读取数据。

您可以尝试使用 SQL 传统关键字 UNION ALL 来组合多个表的选择结果,并使用行键来获取所需的数据,并使用连接来避免基于表列和结构在模式中设置的数据重复。请查看以下示例:

SELECT * FROM `table1`
 UNION ALL 
SELECT * FROM `table2` 

值得注意的是,从您当前拥有的大小的表中读取会影响查询性能,并且在长期内也不可行。相反,您可以创建视图,并进行最新的读取或检查数据,可以查阅官方文档以了解如何优化云 Bigtable 数据服务的使用。

英文:

As the read requests for Bigtable use the stream to get back the contents of the requested rows.This use would be multiple rows can and reads ,it is recommended that you make use of Client libraries instead of direct API calls,which use the ADC for authentication.So the viewer roles set for your user on the tables should allow you to read data.

You may try and use the SQL legacy keyword UNION ALL to combine the select results from multiple tables and fetch the data as you need with the row keys and use the joins to avoid the duplication of data based on the table columns and structures you have currently set up in the schema.Please find the below example

SELECT * FROM `table1`
 UNION ALL 
SELECT * FROM `table2` 

It is to be noted that the reads from the tables of the size you currently have would have an impact on the query performance and would also not be feasible at a long term.Instead you can create views and make the latest reads or check the data there check this official documentation for optimized use of database service for cloud big table.

huangapple
  • 本文由 发表于 2023年2月27日 00:17:51
  • 转载请务必保留本文链接:https://go.coder-hub.com/75573328.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定