英文:
Monitor Multiple Airflow instances
问题
我目前有一个任务，需要监视在多个客户服务器上运行的完全独立的Airflow实例。它们都有类似的DAG。我需要合并来自所有这些实例的指标并对其进行监视。请问是否有人可以提出一种方法？
我尝试使用Prometheus与多个StatsD导出器和Grafana仪表板，但似乎无法使其正常工作。
英文:
I currently have task a task to monitor completely independent airflow instances running across multiple customer servers. All of them have similar DAGs. I have to combine metrics from all these instances and monitor them. Can anyone please suggest a approach for it?
I tried to use prometheus with multiple statsd exporter and grafana dashboards but I can't seem to get it working.
答案1
得分: 1
最佳方法是使用标签来区分不同实例的指标，但不幸的是，Airflow 不会为 Prometheus 推送标签，只有 DataDog 会这样做，而且从 2.6.0 版本开始，InfluxDB 也会支持（参见此链接）。
如果您可以使用其中一个服务，您可以使用相同的stats-prefix，并添加statsd-tags 来区分不同的 Airflow 实例：
# 实例 1:
statsd-datadog-tags=instance:1
# 实例 2:
statsd-datadog-tags=instance:2
然后，您可以将这些指标放在同一个监控/仪表板上，并按instance名称分组和过滤它们。
另外，在 2.6.0 版本中，我们将支持从指标名称中删除变量，并将其作为标签发送，这样您就可以为所有您的 DAGs/任务创建一个仪表板，并按标签分组以更好地监视它们（参见此链接）。
英文:
The best approach is using tags to differentiate between the metrics of the different instances, but unfortunately, Airflow doesn't push tags for prometheus, it does only for datadog and it will do soon (starting from 2.6.0) for InfluxDB.
If you can use one of these services, you can use the same stats-prefix, and add statsd-tags to separate between the different airflow instances:
# instance 1:
statsd-datadog-tags=instance:1
# instance 2:
statsd-datadog-tags=instance:2
Then you will be able to put the metrics on the same monitor/dashboard, and group and filter them by instance name.
Also, in 2.6.0 we will support removing the variables from the metrics names and sending them as tags, so you will be able to create one dashboard for all your dags/tasks and group by tags to better monitor them.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论