英文:
How to send data from multiple event hubs to ADX?
问题
我在Azure的多个区域部署了我的服务。在所有区域中，我都有一个事件中心和ADX集群（包括数据库和表格）。
假设我的服务在X和Y两个区域中。当用户发出请求时，它会由最近的区域进行处理。请求还包含一个名为“origin-region”的属性。所以，如果用户位于区域X，并且他的请求包含属性“origin-region”设置为区域Y，那么此请求将由区域X的服务处理，然后其数据将被发送到区域X的事件中心。
最终，我想要从区域X的事件中心获取数据，并将其存储到区域Y的ADX表中。
对于这个用例，最佳设计应该是什么？
我考虑创建多个表格（比如说region-x-table、region-y-table等）在不同区域的每个表格中。并且从事件中心到所有区域的ADX表格之间建立一对多的映射。
基本上，每个区域的region-x-table将摄取位于区域X事件中心中的数据。来自X事件中心的数据将只摄取到请求属性“orgin-region”中存在的那个区域的ADX表中。
根据请求的“origin-region”的值，区域X的事件中心数据将被摄取到区域X的ADX表(region-x-table)或区域Y的ADX表(region-x-table)中。
因此，在每个区域中会有1个事件中心和n个表格（region-x-table、region-y-table等）。
是否有更好的方法来处理这个问题？
我还考虑过只在ADX中创建一个表格，并通过检查数据模式从多个区域的事件中心摄取数据，但这是不可能的，因为一个表格只能从一个事件中心摄取数据。
英文:
I have my service deployed in multiple regions in azure. In all the regions I have an event hub and ADX cluster(database and table).
Suppose my service is in two regions X and Y. When user request comes it is processed by the nearest region. The request also contains one property of origin-region. So if user is present in region X and his request contains the property "origin-region" set as region Y, this request will be processed by region X's service and then the data of it will be sent to region X's event hub.
Finally I want this data from region X's event hub to Y's ADX table.
What should be the best design of this use case?
I was thinking about creating multiple tables (lets say region-x-table, region-y-table and so on) in each of the different regions. And there will be one to many mapping from event hub to adx table of all the regions.
Basically, region-x-table of any region will ingest the data present in the region X's event hub. And the data from X's event hub will be only ingested in that region's ADX table which is present in the request's property "orgin-region".
Region X's event hub's data will be ingested to either region X's ADX table(region-x-table) or region Y's ADX table(region-x-table) based on the value of "origin-region" of the request.
So in each region there will be 1 event hub, and n tables(region-x-table, region-y-table,....) for each region.
Is there any better approach to this?
I also thought of creating only one table in ADX and ingest data from multiple region's event hub by checking the data schema but it is not possible as one table can only ingest data from one event hub.
答案1
得分: 1
将数据从多个事件中心发送到ADX可以通过为每个事件中心创建数据连接并配置连接设置到事件中心、事件中心命名空间和消费者组来完成。
创建每个地区的多个表，并将来自相应事件中心的数据摄入到各自的表中也是一个不错的方法。
这还允许更多地区的灵活性。
这将确保根据请求的origin-region属性将数据摄入到正确的表中。
另一种方法
使用 Azure Stream Analytics 处理来自事件中心的数据，并根据origin-region属性将其发送到适当的ADX表中。
这涉及为每个地区创建 Stream Analytics 作业，并配置它以从相应事件中心读取数据并写入适当的ADX表中。
有关更多信息，请参考 Create an Event Hubs data connection 和 Use one-click ingestion data from Event Hub into Azure Synapse Data Explorer.。
英文:
>Sending data from multiple event hubs to ADX can be done by creating a data connection for each event hub and configuring the connection settings to event hub, event hub namespace and consumer groups.
- Creating multiple tables in each region and ingesting data from the corresponding event hub into the respective table is also a good approach.
- It also allows more flexibility for more regions.
And this will ensure that the data is ingested into the correct table based on the origin-region property of the request.
Another approach
Using Azure Stream Analytics to process the data from the event hubs and send it to the appropriate ADX table based on the origin-region property.
This involves in creation of Stream Analytics job for each region and configuring it to read from the corresponding event hub and write to the appropriate ADX table.
For more information refer to Create an Event Hubs data connection and Use one-click ingestion data from Event Hub into Azure Synapse Data Explorer.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论