解密 Docker 容器名称在 Grafana 中使用 Loki 和 Promtail

huangapple go评论68阅读模式
英文:

Decrypting Docker Container Names in Grafana with Loki and Promtail

问题

我的任务是配置Docker日志监控,涉及约70个容器。将所有日志合并到一个面板中在压力测试和监视Docker应用程序时并不理想。我希望能够在面板中选择一个Docker容器应用程序并查看相应的日志。我已经实现了这一点,但问题是容器名称是加密的,我需要解密它们。

我的问题是:如何使用Loki数据源和Promtail解密容器名称?

此外,我注意到Prometheus数据源显示了已解密的容器名称,但Loki数据源没有。以下是我为两者的配置:

另外,我正在像普通应用程序一样运行Loki和Promtail,并配置了systemd守护程序。

Loki:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s

schema_config:
  configs:
  - from: 2021-03-08
    store: boltdb
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 24h

storage_config:
  boltdb:
    directory: /tmp/loki/index

Promtail:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: cadvisor
  static_configs:
  - targets: ['localhost:8080/containers/']
    labels:
      job: cadvisor
      __path__: /containers
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log
- job_name: containers
  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs
      __path__: /var/lib/docker/containers/*/*log

  pipeline_stages:
  - json:
      expressions:
        output: log
        stream: stream
        attrs:
  - json:
      expressions:
        tag:
      source: attrs
  - regex:
      expression: (?P<container_name>(?:[^|]*[^|]))
      source: tag
  - timestamp:
      format: RFC3339Nano
      source: time
  - labels:
      # tag:
      stream:
      container_name:
  - output:
      source: output

此外,我了解到loki-driver-docker插件,但我尝试运行它后没有任何变化,而且文档很糟糕,只是说运行它,几乎没有关于配置或如何检查它是否正常工作的信息。

所以,我很乐意听取任何想法和建议。论坛是我解决这个问题的最后机会。

更新:当前情况如下:
解密 Docker 容器名称在 Grafana 中使用 Loki 和 Promtail

我希望它能像cadvisor中那样解密:
解密 Docker 容器名称在 Grafana 中使用 Loki 和 Promtail

英文:

My task is to configure Docker log monitoring for around 70 containers. Consolidating all logs into a single panel is not ideal, especially during stress tests and monitoring Docker applications. I want to be able to select a Docker container application in a panel and see the corresponding logs. I have achieved this, but the problem is that the container names are encrypted, and I need to decrypt them.

My question is: How can I decrypt the container names using the Loki data source and Promtail?

Additionally, I noticed that the Prometheus data source displays decrypted container names, but the Loki data source does not. Here are my configurations for both:

Also, I am running Loki and Promtail like normal application with configured systemd daemon.
Loki:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096


ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s

schema_config:
  configs:
  - from: 2021-03-08
    store: boltdb
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 24h

storage_config:
  boltdb:
    directory: /tmp/loki/index

Promtail:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: cadvisor
  static_configs:
  - targets: [&#39;localhost:8080/containers/&#39;]
    labels:
      job: cadvisonr
      __path__: /containers
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

- job_name: containers
  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs
      __path__: /var/lib/docker/containers/*/*log

  pipeline_stages:
  - json:
      expressions:
        output: log
        stream: stream
        attrs:
  - json:
      expressions:
        tag:
      source: attrs
  - regex:
      expression: (?P&lt;container_name&gt;(?:[^|]*[^|]))
      source: tag
  - timestamp:
      format: RFC3339Nano
      source: time
  - labels:
      # tag:
      stream:
      container_name:
  - output:
      source: output

Also, I know about loki-driver-docker plugin, but I tried to run it and nothing changes for me, and documentation is terrible, it says just run it and almost nothing about configuration or how to check if it is working correct.

So, I would be glad to hear any ideas and suggestions. Forum is the last chance for me to solve this problem.

UPD: Currently, it looks like this:
解密 Docker 容器名称在 Grafana 中使用 Loki 和 Promtail

And I want it to be decrypted like in cadvisor:
解密 Docker 容器名称在 Grafana 中使用 Loki 和 Promtail

答案1

得分: 5

它的工作原理如下:

  • promtail具有后端<something>_configs
  • 每个后端略有不同
  • loki收集日志行
  • loki中的每行日志都有“标签”
  • promtail后端可以将一些内部信息转换为“标签”并发送到loki
  • promtail文件后端只能看到文件,所以static_config只能看到文件
  • promtail Docker后端连接到Docker守护程序,并且实际上了解这些Docker容器的某些信息

Promtail配置可能如下所示:

  - job_name: "docker"
    docker_sd_configs:
      - host: "unix:///var/run/docker.sock"
        refresh_interval: "1s"
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        target_label: "container_name"
      - source_labels: ['__meta_docker_container_id']
        target_label: "container_id"
      # 等等。

然后在grafana-loki中,您可以使用LogQL语言过滤日志,例如:

logcli '{container_name="something"}'
英文:

It works like this:

  • promtail has backends &lt;something&gt;_configs
  • every backend is slightly different
  • loki collects lines of logs
  • every line of logs in loki has "labels"
  • promtail backends can convert some internal info to "labels" and send that loki
  • promtail file backend only sees files, so static_config only sees files
  • promtail docker backend connects to docker deamon and actually knows something about those dockers

Promtail config may look like this:

  - job_name: &quot;docker&quot;
    docker_sd_configs:
      - host: &quot;unix:///var/run/docker.sock&quot;
        refresh_interval: &quot;1s&quot;
    relabel_configs:
      - source_labels: [&#39;__meta_docker_container_name&#39;]
        target_label: &quot;container_name&quot;
      - source_labels: [&#39;__meta_docker_container_id&#39;]
        target_label: &quot;container_id&quot;
      # etc.

Then in grafana-loki you can filter the logs using LogQL language, like:

logcli &#39;{container_name=&quot;something&quot;}&#39;

huangapple
  • 本文由 发表于 2023年3月31日 18:19:21
  • 转载请务必保留本文链接:https://go.coder-hub.com/75897396.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定