英文:
Load data into clickhouse from other clickhouse servers
问题
我需要从其他ClickHouse服务器加载数据到ClickHouse服务器。
我有四个ClickHouse服务器，每秒大约有25000次插入（总共100000次/秒）。我想从所有这些数据库服务器加载数据到第五个服务器。
我已经研究了像URL或JDBC桥这样的表引擎，但我想知道有哪些选项，用于连续的推送/拉取或定期加载。由于文件大小和额外工作的原因，使用文件是不可行的。数据库每天都在忙碌。有哪些可能的解决方案。提前感谢。
英文:
I need to load data into ClickHouse server from other ClickHouse servers.
I have four ClickHouse servers which have 25000 inserts/server/second approx. (100000 inserts/sec). I want to load data from all these database servers to a fifth server.
I have looked into table engines like URL or JDBC bridge but I was wondering what options do I have, for continuous push / pull or scheduled loads. Using files is not possible because of the large size and extra work. DBs are busy 24x7. What are the possible solutions.
Thanks in advance.
答案1
得分: 1
我会在那四台服务器上创建引擎为Distributed的表，并指向第五台服务器，使用remote_servers集群描述。并且使用Materialized View来拦截/复制插入操作。
英文:
I would create tables with engine=Distributed on those 4 servers and point to it 5-th server, using remote_servers cluster description. And using Materialized View will intercept/duplicate inserts.
答案2
得分: 1
@Denny Crane 在GitHub上回答了不同的问题，这更容易理解。
https://github.com/ClickHouse/ClickHouse/issues/15295#issuecomment-698960325
MaterializedView 是一个插入触发器。
> 它从插入获取数据。
>
> 它从表中不读取（选择）。
>
> 必须在发生插入的服务器上创建 MaterializedView。您可以将 MaterializedView 的输出重定向到远程服务器。创建表 xxx 作为 remote(); 创建 Materialized View .... 到 xxx 作为 select .....。
使用 remote：
https://clickhouse.com/docs/en/sql-reference/table-functions/remote
再次感谢。
英文:
@Denny Crane has answered to different question on github that's easier to understand.
https://github.com/ClickHouse/ClickHouse/issues/15295#issuecomment-698960325
MaterializedView is an insert trigger.
> It gets data from INSERT.
>
> It never reads (selects) from table.
>
> You must create MaterializedView at a server where inserts happen. You
> can re-route MaterializedView output to remote server. create table
> xxx as remote(); create Materialized View .... to xxx as select .....
Use remote :
https://clickhouse.com/docs/en/sql-reference/table-functions/remote
Thanks again.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论