Prometheus Alertmanager 处理顺序

huangapple go评论85阅读模式
英文:

Prometheus alertmanager order of handling

问题

我目前正在重新设计 Alert Manager 的现有警报配置。只有一个小问题,我不完全理解 Alert Manager。

假设我有以下配置:

routes:
  - match:
      severity: "warning"
    receiver: "hipchat-teamX"

  - match_re:
      application: "(foo|bar)"
    receiver: "hipchat-teamX"

  - match_re:
      application: "(barfoo.*)"
    receiver: "hipchat-teamY"

然后,Prometheus 触发了一个具有以下值的警报:application: "barfooOne"severity: "warning"

警报将发送给哪个接收器?为什么?

Alert Manager 是否处理上下结构?

何时决定声明路由(routes)的新子级?

英文:

I'm currently redesigning an existing alert configuration of alert manager. There's only a small issue, I don't understand alert manager fully.

Assume I have the following configuration;

routes:

match:
  severity:"warning"
receiver: "hipchat-teamX"

match_re:
  application:"(foo|bar)"
receiver: "hipchat-teamX"

match_re:
  application:"(barfoo.*)"
receiver: "hipchat-teamY"

Then an alert is firing from Prometheus with the following values, application:"barfooOne"
severity:"warning"

To which receiver will the alert send? And why?

Does alert manager handle an up down structure?

And when do you decide to declare a new child of the route (routes)?

答案1

得分: 3

我在GitHub的开发者的帮助下找到了一个解决方案。

你可以将配置的处理方式类比为带有AND和OR声明的if语句。

Alertmanager将配置应用于一个上下、左右的原则。其中上下类似于常规if语句中使用的OR语句。

当你从左到右进行比较时,可以将其与if命令中的AND语句进行比较。

此外,你应该记住,当第一个语句匹配,并且没有声明任何"AND"部分时,alertmanager会将警报推送给接收器,而不会进一步查找其他匹配项。因此,在上面的示例中,如果prometheus触发了具有以下标签的警报{severity: "warning", application: "barfoo"},则警报将发送到hipchat-teamX。

请注意,application的标签值与hipchat-teamY的路由匹配。那么为什么它不发送到那个路由呢?答案并不难理解,因为第一个匹配成功,并且没有声明子路由,所以它将停止查找其他匹配项。

然而,如果alertmanager具有以下代码的配置,警报将被发送到接收器"hipchat-teamY"。

routes:
  - match:
      severity: "warning"
    receiver: "hipchat-teamX"
      
  - routes:
    - match_re:
        application: "(barfoo.*)"
      receiver: "hipchat-teamY"

  - match_re:
      application: "(foo|bar)"
    receiver: "hipchat-teamX"

  - match_re:
      application: "(barfoo.*)"
    receiver: "hipchat-teamY"

为了使其更加高级,假设prometheus发送了具有以下标签的新警报{severity: "critical", application: "barfoo"}。根据以下配置,警报将路由到hipchat-teamX。

而对于具有标签{severity: "critical", application: "foo"}的警报,将发送到hipchat-teamY。

routes:
  - match:
      severity: "warning"
    receiver: "hipchat-teamX"
      
  - routes:
    - match_re:
        application: "(barfoo.*)"
      receiver: "hipchat-teamY"

    - match_re:
        application: "(foo|bar)"
      receiver: "hipchat-teamX"

  - match:
      severity: "critical"
    receiver: "hipchat-teamY"
      
  - routes: 
    - match_re:
        application: "(barfoo.*)"
      receiver: "hipchat-teamX"

    - match_re:
        application: "(foo|bar)"
      receiver: "hipchat-teamY"

我希望这个解释能帮助其他遇到相同问题或疑问的人。

英文:

I found an solution with some help of the developers at GitHub.

You may see the processing of the config in the following way, similar to an if statement with AND and OR declarations.

Alertmanager applies the config to a up-down, left-right principle. Where the up-down is similar to the OR statement you will use in a regular if statement.

When you're going from left to right, you could compare it with the AND statement in if commands.

Further you should keep in mind that when the first statement matches, and there are no 'AND' parts declared, alertmanager pushes the alert to the receiver without looking further for any other match. So in the example above, if prometheus fires an alert with the following labels {severity: "warning", application: "barfoo"}, the alert is sent to hipchat-teamX.

Note the following, the label value of application matches the route for hipchat-teamY. So why it isn't sent to that route? The answer is not that diffucult, it's because the first one matches, and there are no subroutes declared, so it will stop looking for any other match.

However, if the alertmanager has config like the code below, the alert will be sent to the receiver 'hipchat-teamY'.

routes:

match:
  severity:"warning"
receiver: "hipchat-teamX"
  
  routes:
  match_re:
    application:"(barfoo.*)"
  receiver: "hipchat-teamY"

match_re:
  application:"(foo|bar)"
receiver: "hipchat-teamX"

match_re:
  application:"(barfoo.*)"
receiver: "hipchat-teamY"

To make it a bit more advanced, asume prometheus sends a new alert with the following labels {severity: "critical", application: "barfoo"}. With the following config, the alert is routed to hipchat-teamX.

And with the labels {severity: "critical", application: "foo"}, the alert is sent to hipchat-teamY.

routes:

match:
  severity:"warning"
receiver: "hipchat-teamX"
  
  routes:
  match_re:
    application:"(barfoo.*)"
  receiver: "hipchat-teamY"

  match_re:
    application:"(foo|bar)"
  receiver: "hipchat-teamX"

match:
  severity:"critical"
receiver: "hipchat-teamY"
  
  routes: 
  match_re:
    application:"(barfoo.*)"
  receiver: "hipchat-teamX"

  match_re:
    application:"(foo|bar)"
  receiver: "hipchat-teamY"

I hope this explanation will help others who are experiencing the same issue or question.

答案2

得分: 1

我也是新手,所以只能根据我的理解来回答,根据你的情况,第二个接收者将会收到警报,因为默认情况下 continuefalse

英文:

I'm new as well so I can only go off of my understand, which is that in your scenario, the 2nd receiver will receive the alert because continue is false by default.

huangapple
  • 本文由 发表于 2017年6月17日 15:31:36
  • 转载请务必保留本文链接:https://go.coder-hub.com/44601971.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定