RabbitMQ Shovel到Azure Event Hub或Service Bus

huangapple
go评论64阅读模式
英文:

RabbitMQ Shovel to Azure Event Hub or Service Bus

问题

我希望有人已经使这项工作生效。
我正在尝试连接RabbitMQ犁头到Azure事件中心或服务总线。
我在MS上找到了两个教程（非常简短的设置），但都导致了犁头错误。
在示例中，对事件中心或服务总线的请求成功。

我猜测以下教程中可能缺少某些内容，或者某些组件已更改并引入了不兼容的更改：

从RabbitMQ导入数据到Azure数据资源管理器通过事件中心
https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/importing-data-from-rabbitmq-into-azure-data-explorer-via-event/ba-p/3777688
犁头错误

AMQP允许直接连接经纪人，例如使用路由器（如Apache Qpid Dispatch Router）或经纪人本机的“犁头”之一，如RabbitMQ。
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-amqp-overview

如何将服务总线与RabbitMQ集成
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-integrate-with-rabbitmq

版本：
RabbitMQ 3.10.7，Erlang 25.0

案例和步骤：
添加新的Azure服务总线命名空间
rabbitmq01
基本
公共端点
创建我们的Azure服务总线队列
from-rabbitmq01

启用RabbitMQ犁头插件

'rabbitmq-plugins enable rabbitmq_shovel_management'

在Rabbitmq中创建队列
telemetry01

连接RabbitMQ到Azure服务总线
添加SAS策略，立即管理，仅进行发送测试。
rabbitmq-shovel01
主要连接字符串

连接到AMQP的连接字符串
https://red-mushroom-0f7446a0f.azurestaticapps.net/
amqps://rabbitmq-shovel01:SAS-KEY@rabbitmq01.servicebus.windows.net:5671/?sasl=plain

TNC

'Test-NetConnection -ComputerName rabbitmq01.servicebus.windows.net -Port 5671 = True'

使用上述属性制作动态犁头。
犁头完全按照教程完成
犁头状态只是启动，永远不会运行。

日志：

'2023-07-08 19:09:56.385000+02:00 [error] <0.742.0> supervisor: {<0.742.0>,amqp10_client_connection_sup}
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0> errorContext: start_error
2023-07-08 19:09:56.385000+02:00 [error] <0.742.0> reason: {badmatch,{error,closed}}'

[...]
'2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> ** Stacktrace =
2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> ** [{amqp10_client_frame_reader,init,1,
2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> [{file,"amqp10_client_frame_reader.erl"},
2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> {line,109}]},
2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> {gen_statem,init_it,6,[{file,"gen_statem.erl"},{line,1001}]},
2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]'

[...]
'2023-07-08 19:09:56.386000+02:00 [error] <0.688.0> Shovel 'rmq-2-sb' could not connect to destination
2023-07-08 19:09:56.387000+02:00 [error] <0.687.0> supervisor: {<0.687.0>,rabbit_shovel_dyn_worker_sup}'

犁头到总线的请求成功

已测试

  1. 我尝试更改服务总线中的TLS版本。
  2. 启用了rabbitmq_amqp1_0插件，https://github.com/rabbitmq/rabbitmq-amqp1.0。
  3. 在URI中尝试了许多组合。在测试https://red-mushroom-0f7446a0f.azurestaticapps.net/和URL编码后。
  4. Azure ServiceBus可以使用AMQP 1.0协议。自从版本3.7以来，RabbitMQ支持使用AMQP 1.0的犁头，其中源或目标（或两者）都使用AMQP 1.0。https://gist.github.com/kjnilsson/159c643fb34604f8ea20be336109261b

我也向RabbitMQ社区发送了一封邮件。
提前感谢您

2023年7月11日更新
新的RabbitMQ 3.12.1和Erlang 26.0

今天我进行了两个教程的新测试。对于事件中心，我完全按照其进行。犁头错误是：
'2023-07-11 10:01:57.638000+02:00 [error] <0.2006.0> supervisor: {<0.2006.0>,amqp10_client_connection_sup}'
'2023-07-11 10:01:57.638000+02:00 [error] <0.2006.0> errorContext: start_error'
'2023-07-11 10:01:57.638000+02:00 [error] <0.2006.0&gt

英文:

I am hoping someone has got this to work.
I am the trying to connect RabbitMQ shovel to Azure Event hub or Service Bus.
I found two tutorials on MS, (very short to set up) but both results in the shovel error.
The request to Event hub or Service bus is success in the example.

My guess is that something is missing from the below tutorials or that some components have been changed and breaking changes has been introduced:

Importing Data from RabbitMQ into Azure Data Explorer via Event Hubs
https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/importing-data-from-rabbitmq-into-azure-data-explorer-via-event/ba-p/3777688
Shovel error

AMQP allows for interconnecting connecting brokers directly, for instance using routers like Apache Qpid Dispatch Router or broker-native "shovels" like the one of RabbitMQ.
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-amqp-overview

How to integrate Service Bus with RabbitMQ
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-integrate-with-rabbitmq

Version:
RabbitMQ 3.10.7, Erlang 25.0

Case and steps:
Adding a new Azure Service Bus Namespace
rabbitmq01
basic
public endpoint
Creating our Azure Service Bus Queue
from-rabbitmq01

Enabling the RabbitMQ Shovel Plugin

rabbitmq-plugins enable rabbitmq_shovel_management

Created queue in Rabbitmq
telemetry01

Connecting RabbitMQ to Azure Service Bus
Add SAS Policy, Manage now, tested with just send also.
rabbitmq-shovel01
Primary Connection String

Connection String to AMQP
https://red-mushroom-0f7446a0f.azurestaticapps.net/
amqps://rabbitmq-shovel01:SAS-KEY@rabbitmq01.servicebus.windows.net:5671/?sasl=plain

TNC

Test-NetConnection -ComputerName rabbitmq01.servicebus.windows.net -Port 5671 = True

Make the dynamic shovel with above properties.
Shovel was done exactly like the tutorial
The shovel status is just starting, never goes to running.

Log:

2023-07-08 19:09:56.385000+02:00 [error] &lt;0.742.0&gt; supervisor: {&lt;0.742.0&gt;,amqp10_client_connection_sup}
2023-07-08 19:09:56.385000+02:00 [error] &lt;0.742.0&gt; errorContext: start_error
2023-07-08 19:09:56.385000+02:00 [error] &lt;0.742.0&gt; reason: {badmatch,{error,closed}}
[...]
2023-07-08 19:09:56.386000+02:00 [error] &lt;0.744.0&gt; ** Stacktrace =
2023-07-08 19:09:56.386000+02:00 [error] &lt;0.744.0&gt; ** [{amqp10_client_frame_reader,init,1,
2023-07-08 19:09:56.386000+02:00 [error] &lt;0.744.0&gt; [{file,&quot;amqp10_client_frame_reader.erl&quot;},
2023-07-08 19:09:56.386000+02:00 [error] &lt;0.744.0&gt; {line,109}]},
2023-07-08 19:09:56.386000+02:00 [error] &lt;0.744.0&gt; {gen_statem,init_it,6,[{file,&quot;gen_statem.erl&quot;},{line,1001}]},
2023-07-08 19:09:56.386000+02:00 [error] &lt;0.744.0&gt; {proc_lib,init_p_do_apply,3,[{file,&quot;proc_lib.erl&quot;},{line,240}]}]
[...]
2023-07-08 19:09:56.386000+02:00 [error] &lt;0.688.0&gt; Shovel &#39;rmq-2-sb&#39; could not connect to destination
2023-07-08 19:09:56.387000+02:00 [error] &lt;0.687.0&gt; supervisor: {&lt;0.687.0&gt;,rabbit_shovel_dyn_worker_sup}

Shovel to Bus requests success

Tested

  1. I have tried to change TLS version in Service bus.
  2. Enabled rabbitmq_amqp1_0 plugin, https://github.com/rabbitmq/rabbitmq-amqp1.0.
  3. Tested a lot of combinations with the URI. After testing https://red-mushroom-0f7446a0f.azurestaticapps.net/ and URL enconding.
  4. Azure ServiceBus can be used with the AMQP 1.0 protocol. Since version 3.7 RabbitMQ supports shovels where either the source or destination (or both) uses AMQP 1.0. https://gist.github.com/kjnilsson/159c643fb34604f8ea20be336109261b

I have sent a mail to RabbitMQ community also.
Thank you in advance

11.07.2023 Update
New RabbitMQ 3.12.1 and Erlang 26.0

Today I did a new test with both tutorials. For the Event hub i followed it exactly. The shovel error is:
2023-07-11 10:01:57.638000+02:00 [error] &lt;0.2006.0&gt; supervisor: {&lt;0.2006.0&gt;,amqp10_client_connection_sup}
2023-07-11 10:01:57.638000+02:00 [error] &lt;0.2006.0&gt; errorContext: start_error
2023-07-11 10:01:57.638000+02:00 [error] &lt;0.2006.0&gt; reason: {options,incompatible,[{verify,verify_peer},{cacerts,undefined}]}
[...]
2023-07-11 10:09:27.788000+02:00 [error] &lt;0.2476.0&gt; Shovel &#39;shovel-07&#39; could not connect to destination

For the Service bus it is the same.

First I tried with shovel Source AMQP 0.9.1, Destination, AMQP 1.0.
:
2023-07-11 10:44:41.042000+02:00 [error] &lt;0.3627.0&gt; supervisor: {&lt;0.3627.0&gt;,amqp10_client_connection_sup}
2023-07-11 10:44:41.042000+02:00 [error] &lt;0.3627.0&gt; errorContext: start_error
2023-07-11 10:44:41.042000+02:00 [error] &lt;0.3627.0&gt; reason: {options,incompatible,[{verify,verify_peer},{cacerts,undefined}]}
[...]
2023-07-11 10:44:41.043000+02:00 [error] &lt;0.3610.0&gt; Shovel &#39;shovel_01&#39; could not connect to destination

Then I tried shovel Source AMQP 1.0, Destination, AMQP 1.0.

And as expected:
2023-07-11 11:08:18.155000+02:00 [error] &lt;0.8999.0&gt; amqp1_0_plugin_not_enabled

Enabled plugin amqp1_0

2023-07-11 11:14:58.438000+02:00 [error] &lt;0.9613.0&gt; supervisor: {&lt;0.9613.0&gt;,amqp10_client_connection_sup}
2023-07-11 11:14:58.438000+02:00 [error] &lt;0.9613.0&gt; errorContext: start_error
2023-07-11 11:14:58.438000+02:00 [error] &lt;0.9613.0&gt; reason: {options,incompatible,[{verify,verify_peer},{cacerts,undefined}]}
[...]
2023-07-11 11:14:58.438000+02:00 [error] &lt;0.9582.0&gt; Shovel &#39;shovel_03&#39; could not connect to destination

After update to: RabbitMQ 3.12.1 and Erlang 26.0
.So it has do to with TLS, and that was not a step in any of the tutorials.
This is gonna take time RabbitMQ Shovel到Azure Event Hub或Service Bus Any help is appreciated

I got in touch with MS, as per delivered test results, they did not find a issue with the test(s) done. TNC images validated connectivity. Further troubleshooting will be done by support. Looking forward to that. Collaboration is key.

答案1

得分: 0

  • 启用所需的插件：

rabbitmq-plugins enable rabbitmq_shovel_management
rabbitmq-plugins enable rabbitmq_shovel
rabbitmq-plugins enable rabbitmq_management

RabbitMQ Shovel到Azure Event Hub或Service Bus

RabbitMQ Shovel到Azure Event Hub或Service Bus

RabbitMQ Shovel到Azure Event Hub或Service Bus

RabbitMQ Shovel到Azure Event Hub或Service Bus

RabbitMQ Shovel到Azure Event Hub或Service Bus

RabbitMQ Shovel到Azure Event Hub或Service Bus

RabbitMQ Shovel到Azure Event Hub或Service Bus

英文:

Sample RabbitMQ Shovel to Azure Service Bus:

  • Enable the required

rabbitmq-plugins enable rabbitmq_shovel_management
rabbitmq-plugins enable rabbitmq_shovel

rabbitmq-plugins enable rabbitmq_management

RabbitMQ Shovel到Azure Event Hub或Service Bus

RabbitMQ Shovel到Azure Event Hub或Service Bus

RabbitMQ Shovel到Azure Event Hub或Service Bus

RabbitMQ Shovel到Azure Event Hub或Service Bus

RabbitMQ Shovel到Azure Event Hub或Service Bus

RabbitMQ Shovel到Azure Event Hub或Service Bus

RabbitMQ Shovel到Azure Event Hub或Service Bus

答案2

得分: 0

我联系了微软，根据提供的测试结果，他们没有发现测试中存在问题。TNC 图像已验证连接。进一步的故障排除将由支持部门进行。期待着那一天。合作是关键。

英文:

I got in touch with MS, as per delivered test results, they did not find a issue with the test(s) done. TNC images validated connectivity. Further troubleshooting will be done by support. Looking forward to that. Collaboration is key.

答案3

得分: 0

https://www.erlang.org/blog/otp-26-highlights/
Erlang/OTP 26 亮点
2023年5月16日
[...]
Erlang/OTP 26 亮点
在 OTP 26 中，verify 选项的默认值现在是 verify_peer，而不再是 verify_none。
主机验证需要使用 cacerts 或 cacertsfile 选项提供受信任的 CA 证书。
因此，如果使用空选项列表进行连接尝试，将在 OTP 26 中失败：

                [{verify,verify_peer},{cacerts,undefined}]}}```

cacerts 选项的默认值是 undefined，与 {verify,verify_peer} 选项不兼容。
为了使连接成功，建议的方法是使用 cacerts 选项提供用于验证的 CA 证书。

这是 2023 年 5 月 16 日之后的重大更改：

```amqps://rabbitmq-shovel:StringOfRandomChars@rabbitmq.servicebus.windows.net:5671/?sasl=plain```

应该与以下格式类似：

```amqps://rabbitmq-shovel:StringOfRandomChars@rabbitmq.servicebus.windows.net:5671/?cacertfile=/path/to/cacert_root_certificate_servicebus.pem&amp;verify=verify_peer（某种方式？？）```

因此，https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-integrate-with-rabbitmq 必须更新以提供新的信息，而工具 https://red-mushroom-0f7446a0f.azurestaticapps.net/ 也必须生成带有 SSL 相关内容的新 URI。我相信 Azure Service Bus 的公共根证书也必须可获取。请提供输入？

2023 年 7 月 15 日更新
https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/importing-data-from-rabbitmq-into-azure-data-explorer-via-event/ba-p/3777688
教程版本：RabbitMQ 3.11.7 Erlang 25.2。

已测试与 RabbitMQ 3.11.10 Erlang 25.2。

RabbitMQ 日志：

2023-07-15 10:38:00.787000+02:00 [warning] <0.1427.0> 描述：“由于未启用证书路径验证，因此无法验证服务器的真实性”
2023-07-15 10:38:00.787000+02:00 [warning] <0.1427.0> 原因：“要启用此功能，需要选项 {verify, verify_peer} 和选项 'cacertfile' 或 'cacerts' 中的一个。”
2023-07-15 10:38:00.787000+02:00 [warning] <0.1427.0>
2023-07-15 10:38:01.394000+02:00 [error] <0.1408.0> Shovel 'rmq2eh' 在虚拟主机 '/' 中停止，原因：{outbound_link_detached,
2023-07-15 10:38:01.394000+02:00 [error] <0.1408.0> {'v1_0.error',
2023-07-15 10:38:01.394000+02:00 [error] <0.1408.0> {symbol,
2023-07-15 10:38:01.394000+02:00 [error] <0.1408.0> <<"amqp:unauthorized-access">>},

[...] 2023-07-15 10:49:09.143000+02:00 主机名 =>
2023-07-15 10:49:09.143000+02:00 <<"xxxxxxxx.servicebus.windows.net">>,


教程必须更新为支持最高版本的 RabbitMQ 和 Erlang，以便新用户知道可以使用哪个版本以及有哪些依赖关系。如果可以找到的话，我将使用与中心教程中使用的相同的 RabbitMQ 版本进行一次测试。此测试使用的 Erlang 版本相同。

更新版本（找不到 3.11.7 版）：
RabbitMQ 3.11.0 Erlang 25.2 结果：

2023-07-15 11:46:16.886000+02:00 [error] <0.2202.0> Shovel 'rmq2ev114' 在虚拟主机 '/' 中停止，原因：{outbound_link_detached,
2023-07-15 11:46:16.886000+02:00 [error] <0.2202.0> {'v1_0.error',
2023-07-15 11:46:16.886000+02:00 [error] <0.2202.0> {symbol,
2023-07-15 11:46:16.886000+02:00 [error] <0.2202.0> <<"amqp:unauthorized-access">>},
[...]
2023-07-15 11:46:16.887000+02:00 [warning] <0.2220.0> 主机名 =>
2023-07-15 11:46:16.887000+02:00 [warning] <0.2220.0> <<"xxxxx.servicebus.windows.net">>,
[...]
2023-07-15 11:46:21.895000+02:00 [warning] <0.2254.0> 描述：“由于未启用证书路径验证，因此无法验证服务器的真实性”
2023-07-15 11:46:21.895000+02:00 [warning] <0.2254.0> 原因：“要启用此功能，需要选项 {verify, verify_peer} 和选项 'cacertfile' 或 'cacerts' 中的一个。”


2023 年 7 月 15 日 18:11 现在终于可以工作了。
现在它适用于 Azure Event Hub 和 Azure Service bus，使用 TLS 和 Azure 根证书的新 URI，并

<details>
<summary>英文:</summary>

https://www.erlang.org/blog/otp-26-highlights/
Erlang/OTP 26 Highlights
May 16, 2023
[...]
Erlang/OTP 26 Highlights
In OTP 26, the default value for the verify option is now verify_peer instead of verify_none. 
Host verification requires trusted CA certificates to be supplied using one of the options cacerts or cacertsfile. 
Therefore, a connection attempt with an empty option list will fail in OTP 26:

    {error,{options,incompatible,
                    [{verify,verify_peer},{cacerts,undefined}]}}

The default value for the cacerts option is undefined, which is not compatible with the {verify,verify_peer} option.
To make the connection succeed, the recommended way is to use the cacerts option to supply CA certificates to be used for verifying.

This is the breaking change (I would believe) after 16.05.2023:

    amqps://rabbitmq-shovel:StringOfRandomChars@rabbitmq.servicebus.windows.net:5671/?sasl=plain

Is must be on format similar to this I would belive:

    amqps://rabbitmq-shovel:StringOfRandomChars@rabbitmq.servicebus.windows.net:5671/?cacertfile=/path/to/cacert_root_certificate_servicebus.pem&amp;verify=verify_peer (something??)

So https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-integrate-with-rabbitmq must be updated with new information and the tool https://red-mushroom-0f7446a0f.azurestaticapps.net/ must also generate new URI with ssl stuff, and I would belive the public root certificate for Azure Service Bus must be obtainable. Input in this?

15.07.2023 update
https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/importing-data-from-rabbitmq-into-azure-data-explorer-via-event/ba-p/3777688
Tutorial version: RabbitMQ 3.11.7 Erlang 25.2.

Tested with RabbitMQ 3.11.10 Erlang 25.2.

RabbitMQ log:

    2023-07-15 10:38:00.787000+02:00 [warning] &lt;0.1427.0&gt; Description: &quot;Server authenticity is not verified since certificate path validation is not enabled&quot;
    2023-07-15 10:38:00.787000+02:00 [warning] &lt;0.1427.0&gt;      Reason: &quot;The option {verify, verify_peer} and one of the options &#39;cacertfile&#39; or &#39;cacerts&#39; are required to enable this.&quot;
    2023-07-15 10:38:00.787000+02:00 [warning] &lt;0.1427.0&gt; 
    2023-07-15 10:38:01.394000+02:00 [error] &lt;0.1408.0&gt; Shovel &#39;rmq2eh&#39; in virtual host &#39;/&#39; is stopping, reason: {outbound_link_detached,
    2023-07-15 10:38:01.394000+02:00 [error] &lt;0.1408.0&gt;                                                           {&#39;v1_0.error&#39;,
    2023-07-15 10:38:01.394000+02:00 [error] &lt;0.1408.0&gt;                                                            {symbol,
    2023-07-15 10:38:01.394000+02:00 [error] &lt;0.1408.0&gt;                                                             &lt;&lt;&quot;amqp:unauthorized-access&quot;&gt;&gt;},

     [...] 2023-07-15 10:49:09.143000+02:00 hostname =&gt;
    2023-07-15 10:49:09.143000+02:00 &lt;&lt;&quot;xxxxxxxx.servicebus.windows.net&quot;&gt;&gt;,

The tutorial(s) must be update with highest RabbitMQ and Erlang version it supports, so that new user know what version(s) they can use and what are the dependency&#39;s.. I will do one test with same RabbitMQ version used in the hub tutorial if I can find it. The Erlang version used for this test is the same.

Update versions (could not find 3.11.7):
RabbitMQ 3.11.0 Erlang 25.2 result:

    2023-07-15 11:46:16.886000+02:00 [error] &lt;0.2202.0&gt; Shovel &#39;rmq2ev114&#39; in virtual host &#39;/&#39; is stopping, reason: {outbound_link_detached,
    2023-07-15 11:46:16.886000+02:00 [error] &lt;0.2202.0&gt;                                                              {&#39;v1_0.error&#39;,
    2023-07-15 11:46:16.886000+02:00 [error] &lt;0.2202.0&gt;                                                               {symbol,
    2023-07-15 11:46:16.886000+02:00 [error] &lt;0.2202.0&gt;                                                                &lt;&lt;&quot;amqp:unauthorized-access&quot;&gt;&gt;},
    [...]
    2023-07-15 11:46:16.887000+02:00 [warning] &lt;0.2220.0&gt;                                                                                    hostname =&gt;
    2023-07-15 11:46:16.887000+02:00 [warning] &lt;0.2220.0&gt;                                                                                     &lt;&lt;&quot;xxxxx.servicebus.windows.net&quot;&gt;&gt;,
    [...]
    2023-07-15 11:46:21.895000+02:00 [warning] &lt;0.2254.0&gt; Description: &quot;Server authenticity is not verified since certificate path validation is not enabled&quot;
    2023-07-15 11:46:21.895000+02:00 [warning] &lt;0.2254.0&gt;      Reason: &quot;The option {verify, verify_peer} and one of the options &#39;cacertfile&#39; or &#39;cacerts&#39; are required to enable this.&quot;

15.07.2023 18:11 It works now finally.
It now works for Azure Event Hub and for Azure Service bus, new URI with TLS and Azure Root certificate and had to add an extra root policy also to the namespace for each service, not just the sub SAS
https://github.com/spawnmarvel/test/blob/main/it_works.jpg

I am out, I am done, my life is back.





</details>

huangapple
  • 本文由 发表于 2023年7月10日 16:54:46
  • 转载请务必保留本文链接：https://go.coder-hub.com/76652187.html
  • azure
  • azure-eventhub
  • rabbitmq
  • servicebus
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定