query-exporter在Docker容器中无法工作。

huangapple
go评论177阅读模式
英文:

query-exporter in Docker container not working

问题

我正在尝试在Docker容器中运行query-exporter。在开发人员的建议下，我已经在docker中启用了IPv6，方法是将以下内容添加到我的docker daemon.json文件中并重新启动：

  1. {
  2.   "experimental": true,
  3.   "ip6tables": true
  4. }

我正在使用以下docker-compose文件：

  1. version: "3.3"
  3. services:
  4.   prometheus:
  5.     container_name: prometheus
  6.     image: prom/prometheus
  7.     restart: always
  8.     volumes:
  9.       - ./prometheus:/etc/prometheus/
  10.       - prometheus_data:/prometheus
  11.     command:
  12.       - '--config.file=/etc/prometheus/prometheus.yml'
  13.       - '--storage.tsdb.path=/prometheus'
  14.       - '--web.console.libraries=/usr/share/prometheus/console_libraries'
  15.       - '--web.console.templates=/usr/share/prometheus/consoles'
  16.     ports:
  17.       - 9090:9090
  18.     networks:
  19.       - prom_app_net
  21.   # 其他服务...
  23. volumes:
  24.   # 定义卷...
  25. networks:
  26.   prom_app_net:
  27.   slurm:
  28.     enable_ipv6: true
  29.     ipam:
  30.       config:
  31.         - subnet: 2001:0DB8::/112

然后在slurmctld容器上安装了query-exporter，并使用以下config.yaml文件运行它：

  1. databases:
  2.   db1:
  3.     dsn: sqlite:////test.db
  4.     connect-sql:
  5.       - PRAGMA application_id = 123
  6.       - PRAGMA auto_vacuum = 1
  7.     labels:
  8.       region: us1
  9.       app: app1
  11. metrics:
  12.   metric1:
  13.     type: gauge
  14.     description: A sample gauge
  16. queries:
  17.   query1:
  18.     interval: 5
  19.     databases: [db1]
  20.     metrics: [metric1]
  21.     sql: SELECT random() / 1000000000000000 AS metric1

但是它没有工作 - Prometheus将目标列为离线。但容器设置似乎没问题，因为如果我运行以下测试导出器：

  1. from prometheus_client import start_http_server, Summary
  2. import random
  3. import time
  5. # 创建一个用于跟踪时间和请求的指标。
  6. REQUEST_TIME = Summary('request_processing_seconds', 'Time spent processing request')
  8. # 使用指标装饰函数。
  9. @REQUEST_TIME.time()
  10. def process_request(t):
  11.     """一个需要一些时间的虚拟函数。"""
  12.     time.sleep(t)
  14. if __name__ == '__main__':
  15.     # 启动服务器以公开指标。
  16.     start_http_server(8082)
  17.     # 生成一些请求。
  18.     while True:
  19.         process_request(random.random())

Prometheus可以成功连接到目标。

根据更新信息，问题可能是query-exporter尝试绑定IPv6，而测试的test_query.py使用IPv4在端口8082上。您可以尝试将Prometheus指向8082/tcp -> [::]:8082，但如何实现取决于您的具体配置。

英文:

I am trying to get query-exporter to run in a Docker container. With advice from the developer I have enabled IPv6 in docker by putting:

  1. {
  2.   "experimental": true,
  3.   "ip6tables": true
  4. }

in my docker daemon.json and restarted.

I am using the following docker-compose file:

  1. version: "3.3"
  3. services:
  4.   prometheus:
  5.     container_name: prometheus
  6.     image: prom/prometheus
  7.     restart: always
  8.     volumes:
  9.       - ./prometheus:/etc/prometheus/
  10.       - prometheus_data:/prometheus
  11.     command:
  12.       - '--config.file=/etc/prometheus/prometheus.yml'
  13.       - '--storage.tsdb.path=/prometheus'
  14.       - '--web.console.libraries=/usr/share/prometheus/console_libraries'
  15.       - '--web.console.templates=/usr/share/prometheus/consoles'
  16.     ports:
  17.       - 9090:9090
  18.     networks:
  19.       - prom_app_net
  20.     
  22.   grafana:
  23.     container_name: grafana
  24.     image: grafana/grafana
  25.     user: '472'
  26.     restart: always
  27.     environment:
  28.       GF_INSTALL_PLUGINS: 'grafana-clock-panel,grafana-simple-json-datasource'
  29.     volumes:
  30.       - grafana_data:/var/lib/grafana
  31.       - ./grafana/provisioning/:/etc/grafana/provisioning/
  32.       - './grafana/grafana.ini:/etc/grafana/grafana.ini'
  33.     env_file:
  34.       - ./grafana/.env_grafana
  35.     ports:
  36.       - 3000:3000
  37.     depends_on:
  38.       - prometheus
  39.     networks:
  40.       - prom_app_net
  43.   mysql:
  44.     image: mariadb:10.10
  45.     hostname: mysql
  46.     container_name: mysql
  47.     environment:
  48.       MYSQL_RANDOM_ROOT_PASSWORD: "yes"
  49.       MYSQL_DATABASE: slurm_acct_db
  50.       MYSQL_USER: slurm
  51.       MYSQL_PASSWORD: password
  52.     volumes:
  53.       - var_lib_mysql:/var/lib/mysql
  54.     networks:
  55.       - slurm
  56. #    network_mode: host
  59.   slurmdbd:
  60.     image: prom-slurm-cluster:${IMAGE_TAG:-21.08.6}
  61.     build:
  62.       context: .
  63.       args:
  64.         SLURM_TAG: ${SLURM_TAG:-slurm-21-08-6-1}
  65.     command: ["slurmdbd"]
  66.     container_name: slurmdbd
  67.     hostname: slurmdbd
  68.     volumes:
  69.       - etc_munge:/etc/munge
  70.       - etc_slurm:/etc/slurm
  71.       - var_log_slurm:/var/log/slurm
  72.       - cgroups:/sys/fs/cgroup:ro
  73.     expose:
  74.       - "6819"
  75.     ports:
  76.       - "6819:6819"
  77.     depends_on:
  78.       - mysql
  79.     privileged: true
  80.     cgroup: host
  81.     networks:
  82.       - slurm
  83.     #network_mode: host
  85.   slurmctld:
  86.     image: prom-slurm-cluster:${IMAGE_TAG:-21.08.6}
  87.     command: ["slurmctld"] 
  88.     container_name: slurmctld
  89.     hostname: slurmctld
  90.     volumes:
  91.       - etc_munge:/etc/munge
  92.       - etc_slurm:/etc/slurm
  93.       - slurm_jobdir:/data
  94.       - var_log_slurm:/var/log/slurm
  95.       - etc_prometheus:/etc/prometheus
  96.       - /sys/fs/cgroup:/sys/fs/cgroup:rw
  97.     expose:
  98.       - "6817"
  99.       - "8080"
  100.       - "8081"
  101.       - "8082/tcp"
  102.     ports:
  103.       - 8080:8080
  104.       - 8081:8081
  105.       - 8082:8082/tcp
  106.     depends_on:
  107.       - "slurmdbd"
  108.     privileged: true
  109.     cgroup: host
  111.     #network_mode: host
  112.     networks:
  113.       - slurm
  115.   c1:
  116.     image: prom-slurm-cluster:${IMAGE_TAG:-21.08.6}
  117.     command: ["slurmd"]
  118.     hostname: c1
  119.     container_name: c1
  120.     volumes:
  121.       - etc_munge:/etc/munge
  122.       - etc_slurm:/etc/slurm
  123.       - slurm_jobdir:/data
  124.       - var_log_slurm:/var/log/slurm
  125.       - cgroups:/sys/fs/cgroup:ro
  126.     expose:
  127.       - "6818"
  128.     depends_on:
  129.       - "slurmctld"
  130.     privileged: true
  131.     cgroup: host 
  132.     #network_mode: host
  133.     networks:
  134.       - slurm
  135.     
  137.   c2:
  138.     image: prom-slurm-cluster:${IMAGE_TAG:-21.08.6}
  139.     command: ["slurmd"]
  140.     hostname: c2
  141.     container_name: c2
  142.     volumes:
  143.       - etc_munge:/etc/munge
  144.       - etc_slurm:/etc/slurm
  145.       - slurm_jobdir:/data
  146.       - var_log_slurm:/var/log/slurm
  147.       - cgroups:/sys/fs/cgroup:ro
  148.     expose:
  149.       - "6818"
  150.       - "22"
  151.     depends_on:
  152.       - "slurmctld"
  153.     privileged: true
  154.     cgroup: host
  155.     networks:
  156.       - slurm
  157.     #network_mode: host
  162. volumes:
  163.   etc_munge:
  164.   etc_slurm:
  165.   slurm_jobdir:
  166.   var_lib_mysql:
  167.   var_log_slurm:
  168.   grafana_data:
  169.   prometheus_data:
  170.   cgroups: 
  171.   etc_prometheus:
  173. networks:
  174.   prom_app_net:
  175.   slurm:
  176.     enable_ipv6: true
  177.     ipam:
  178.       config: 
  179.         - subnet: 2001:0DB8::/112

Then installed query-exporter on the slurmctld container and run it with the following config.yaml:

  1. databases:
  2.   db1:
  3.     dsn: sqlite:////test.db
  4.     connect-sql:
  5.       - PRAGMA application_id = 123
  6.       - PRAGMA auto_vacuum = 1
  7.     labels:
  8.       region: us1
  9.       app: app1
  12. metrics:
  13.   metric1:
  14.     type: gauge
  15.     description: A sample gauge
  18. queries:
  19.   query1:
  20.     interval: 5
  21.     databases: [db1]
  22.     metrics: [metric1]
  23.     sql: SELECT random() / 1000000000000000 AS metric1

But it is not working - prometheus lists the target as being down:

query-exporter在Docker容器中无法工作。

But the container set-up seems to be fine as if I run the following test exporter:

  1. from prometheus_client import start_http_server, Summary
  2. import random
  3. import time
  5. # Create a metric to track time spent and requests made.
  6. REQUEST_TIME = Summary('request_processing_seconds', 'Time spent processing request')
  8. # Decorate function with metric.
  9. @REQUEST_TIME.time()
  10. def process_request(t):
  11.     """A dummy function that takes some time."""
  12.     time.sleep(t)
  14. if __name__ == '__main__':
  15.     # Start up the server to expose the metrics.
  16.     start_http_server(8082)
  17.     # Generate some requests.
  18.     while True:
  19.         process_request(random.random())

Prometheus can connect to the target fine:

query-exporter在Docker容器中无法工作。

Can anyone see what the problem could be?

Thanks!

Update

I run query-exporter by hand on the slurmctld container. There isnt anything in the container logs about query-exporter:

  1. 2023-07-10 10:11:37 ---> Starting the MUNGE Authentication service (munged) ...
  2. 2023-07-10 10:11:37 ---> Waiting for slurmdbd to become active before starting slurmctld ...
  3. 2023-07-10 10:11:37 -- slurmdbd is not available.  Sleeping ...
  4. 2023-07-10 10:11:39 -- slurmdbd is now active ...
  5. 2023-07-10 10:11:39 ---> starting systemd ...

I think th etest_query.py that works is using IPv4 on port 8082, while the query exporter is trying to bind IPv6.

docker port slurmctld gives:

  1. 8080/tcp -> 0.0.0.0:8080
  2. 8080/tcp -> [::]:8080
  3. 8081/tcp -> 0.0.0.0:8081
  4. 8081/tcp -> [::]:8081
  5. 8082/tcp -> 0.0.0.0:8082
  6. 8082/tcp -> [::]:8082

I guess i need to pint prometheus at 8082/tcp -> [::]:8082 when the query-exporter runs, but I'm not sure how to do it.

答案1

得分: 0

使用 query-exporter config.yaml -H 0.0.0.0 -p 8082 运行可以使其工作。

英文:

Running with query-exporter config.yaml -H 0.0.0.0 -p 8082 gets it to work.

huangapple
  • 本文由 发表于 2023年7月10日 17:57:43
  • 转载请务必保留本文链接：https://go.coder-hub.com/76652641.html
  • docker
  • docker-compose
  • prometheus
  • python
  • sqlite
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定