如何配置Grafana Agent?

huangapple go评论78阅读模式
英文:

How to configure Grafana Agent?

问题

我想使用Grafana Agent将指标发送到Prometheus。我在静态模式下安装了Grafana Agent(版本0.34),启动了它,并进行了配置,但是Prometheus(版本2.24.0)没有接收到来自Grafana Agent的任何指标。

配置

我使用了示例配置,请参见集成。我只将localhost替换为运行Prometheus的机器的IP地址。

metrics:
  wal_directory: /tmp/wal
  global:
    remote_write:
      - url: http://172.31.11.242:9009/api/v1/prom/push

integrations:
  agent:
    enabled: true

日志

...
Jul 13 11:51:37 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:37.559384828Z caller=dedupe.go:112 agent=prometheus instance=b9394761ac53f9217ed8581af212bcd2 component=remote level=inf
o remote_name=b93947-21c28b url=http://172.31.11.242:9009/api/v1/prom/push msg="Done replaying WAL" duration=17.612896148s
Jul 13 11:51:39 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:39.981665238Z caller=dedupe.go:112 agent=prometheus instance=b9394761ac53f9217ed8581af212bcd2 component=remote level=war
n remote_name=b93947-21c28b url=http://172.31.11.242:9009/api/v1/prom/push msg="Failed to send batch, retrying" err="Post \"http://172.31.11.242:9009/api/v1/prom/push\": dial tcp 172.31.11.242:90
09: connect: connection refused"
...

研究

  • 我检查了运行Prometheus的机器上的端口,没有打开的9009端口:
$ sudo netstat -tulpn | grep 9009
  • 我阅读了启动Prometheus

    Prometheus应该启动起来。您还应该能够在http://localhost:9090上浏览到有关自身的状态页面。

    因此,Prometheus的默认端口是9090。我将9009替换为9090,但仍然没有收到任何指标。

  • 我阅读了HTTP API

    当前稳定的HTTP API 可以在Prometheus服务器上的/api/v1下访问。

    因此,API位于/api/v1,而不是/api/。我将/api/prom/push替换为/api/v1/prom/push,但仍然没有收到任何指标。

问题

如何配置Grafana Agent将指标发送到Prometheus?

英文:

I want to use Grafana Agent to send metrics to Prometheus. I installed Grafana Agent (version 0.34) in static mode, started it and configured it, but Prometheus (version 2.24.0) doesn't receive any metrics from Grafana Agent.

Configuration

I use the example configuration, see Integrations. I only replaced localhost with the IP address of the machine running Prometheus.

metrics:
  wal_directory: /tmp/wal
  global:
    remote_write:
      - url: http://172.31.11.242:9009/api/prom/push

integrations:
  agent:
    enabled: true

Logs

Jul 13 11:51:19 prometheus systemd[1]: Started Monitoring system and forwarder.
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.934501653Z caller=server.go:191 level=info msg="server listening on addresses" http=127.0.0.1:9090 grpc=127.0.0.1:909
1 http_tls_enabled=false grpc_tls_enabled=false
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.935000668Z caller=node.go:85 level=info agent=prometheus component=cluster msg="applying config"
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.935158879Z caller=remote.go:180 level=info agent=prometheus component=cluster msg="not watching the KV, none set"
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.936021874Z caller=zapadapter.go:78 level=info component=traces msg="Traces Logger Initialized"
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.939769201Z caller=reporter.go:103 level=info msg="running usage stats reporter"
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.942906407Z caller=wal.go:211 level=info agent=prometheus instance=b9394761ac53f9217ed8581af212bcd2 msg="replaying WAL
, this may take a while" dir=/tmp/wal/b9394761ac53f9217ed8581af212bcd2/wal
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.943101908Z caller=wal.go:260 level=info agent=prometheus instance=b9394761ac53f9217ed8581af212bcd2 msg="WAL segment l
oaded" segment=0 maxSegment=2
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.945060553Z caller=wal.go:260 level=info agent=prometheus instance=b9394761ac53f9217ed8581af212bcd2 msg="WAL segment l
oaded" segment=1 maxSegment=2
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.945229786Z caller=wal.go:260 level=info agent=prometheus instance=b9394761ac53f9217ed8581af212bcd2 msg="WAL segment l
oaded" segment=2 maxSegment=2
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.946371012Z caller=dedupe.go:112 agent=prometheus instance=b9394761ac53f9217ed8581af212bcd2 component=remote level=inf
o remote_name=b93947-21c28b url=http://172.31.11.242:9009/api/prom/push msg="Starting WAL watcher" queue=b93947-21c28b
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.946418916Z caller=dedupe.go:112 agent=prometheus instance=b9394761ac53f9217ed8581af212bcd2 component=remote level=inf
o remote_name=b93947-21c28b url=http://172.31.11.242:9009/api/prom/push msg="Starting scraped metadata watcher"
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.946532386Z caller=dedupe.go:112 agent=prometheus instance=b9394761ac53f9217ed8581af212bcd2 component=remote level=inf
o remote_name=b93947-21c28b url=http://172.31.11.242:9009/api/prom/push msg="Replaying WAL" queue=b93947-21c28b
Jul 13 11:51:37 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:37.559384828Z caller=dedupe.go:112 agent=prometheus instance=b9394761ac53f9217ed8581af212bcd2 component=remote level=inf
o remote_name=b93947-21c28b url=http://172.31.11.242:9009/api/prom/push msg="Done replaying WAL" duration=17.612896148s
Jul 13 11:51:39 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:39.981665238Z caller=dedupe.go:112 agent=prometheus instance=b9394761ac53f9217ed8581af212bcd2 component=remote level=war
n remote_name=b93947-21c28b url=http://172.31.11.242:9009/api/prom/push msg="Failed to send batch, retrying" err="Post \"http://172.31.11.242:9009/api/prom/push\": dial tcp 172.31.11.242:90
09: connect: connection refused"

Research

  • I checked the port on the machine running Prometheus and there is no open port 9009:

       $ sudo netstat -tulpn | grep 9009
    
  • I read Starting Prometheus:
    > Prometheus should start up. You should also be able to browse to a status page about itself at http://localhost:9090.

    So the default port of Prometheus is 9090. I replaced 9009 with 9090, but I still get no metrics.

  • I read HTTP API:
    > The current stable HTTP API is reachable under /api/v1 on a Prometheus server.

    So the API is at /api/v1, not at /api/. I replaced /api/prom/push with /api/v1/prom/push, but I still get no metrics.

Question

How to configure Grafana Agent to send metrics to Prometheus?

答案1

得分: 1

我阅读了Remote Write Receiver

> Prometheus可以配置为Prometheus远程写入协议的接收器。这不被视为摄取样本的有效方式。请小心在特定的低容量用例中使用它。它不适用于替代通过抓取进行摄取并将Prometheus变成基于推送的指标收集系统。

> 通过设置--web.enable-remote-write-receiver来启用远程写入接收器。启用后,远程写入接收器的端点是/api/v1/write。在此处找到更多详细信息。

> 在v2.33中新增

因此,我不得不升级Prometheus并更改其服务配置。还必须更改Grafana Agent的配置中的URL。

Prometheus的服务配置

[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.console.templates=/opt/prometheus/consoles \
  --web.console.libraries=/opt/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.route-prefix=/ \
  --web.enable-remote-write-receiver
SyslogIdentifier=prometheus
Restart=always

[Install]
WantedBy=multi-user.target

Grafana Agent的配置

metrics:
  wal_directory: /tmp/wal
  global:
    remote_write:
      - url: http://172.31.11.242:9090/api/v1/write

integrations:
  agent:
    enabled: true
英文:

I read Remote Write Receiver:

> Prometheus can be configured as a receiver for the Prometheus remote write protocol. This is not considered an efficient way of ingesting samples. Use it with caution for specific low-volume use cases. It is not suitable for replacing the ingestion via scraping and turning Prometheus into a push-based metrics collection system.
>
> Enable the remote write receiver by setting --web.enable-remote-write-receiver. When enabled, the remote write receiver endpoint is /api/v1/write. Find more details here.
>
> New in v2.33

So I had to upgrade Prometheus and change its service configuration. Also I had to change the URL in Grafana Agent's configuration.

Prometheus' service configuration

[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.console.templates=/opt/prometheus/consoles \
  --web.console.libraries=/opt/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.route-prefix=/ \
  --web.enable-remote-write-receiver
SyslogIdentifier=prometheus
Restart=always

[Install]
WantedBy=multi-user.target

Grafana Agent's configuration

metrics:
  wal_directory: /tmp/wal
  global:
    remote_write:
      - url: http://172.31.11.242:9090/api/v1/write

integrations:
  agent:
    enabled: true

huangapple
  • 本文由 发表于 2023年7月13日 20:09:01
  • 转载请务必保留本文链接:https://go.coder-hub.com/76679217.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定