为什么Filebeat每天都创建一个新的数据流?

huangapple go评论63阅读模式
英文:

Why is filebeat creating a new datastream everyday?

问题

一个非常简单的Filebeat配置。

该配置创建以下内容:

  • 一个名为:voucher-app-logs-8.7.0-2023.04.19 的数据流
  • 一个名为:.ds-voucher-app-logs-8.7.0-2023.04.19-2023.04.19-000001 的索引
  • 每天创建一个新的数据流

我的问题:

  • 为什么每天都要创建一个新的数据流?
  • 如何更改数据流的名称?
  • 如何仅为这些索引模式创建一个数据流,而不是每天都创建一个新的数据流?
英文:

A very simple filebeat

filebeat:
  inputs:
  - type: filestream
    id: vouchers-logs-stream
    paths:
      - /path/to/logs/*.log
    json:
      keys_under_root: true
      add_error_key: true
      overwrite_keys: true
      message_key: message
    parsers:
      - ndjson:
          target: ""
          add_error_key: true
output:
  elasticsearch:
    hosts: [ "..." ]
    username: "..."
    password: "..."
    index: voucher-app-logs-%{[agent.version]}-%{+yyyy.MM.dd}
setup:
  template:
    name: "voucher-app-logs"
    pattern: "voucher-app-logs*"
    overwrite: false
  ilm:
    enabled: true
    policy_name: "voucher-app-logs-lifecycle-policy"

This configuration creates the following:

  • A data stream called: voucher-app-logs-8.7.0-2023.04.19
  • A index called
    .ds-voucher-app-logs-8.7.0-2023.04.19-2023.04.19-000001
  • And at each day, a new datastream is created

My questions:

  • Why create a new datastream everyday?
  • How to change the name of the datastream?
  • How to create only one datastream for these index patterns instead of
    creating one new ds every day?

I have read the docs and found nothing about that. Perhaps I missed something?

答案1

得分: 1

Tldr;

Beats 正在按照指示执行 ^^

在输出部分:

output:
  elasticsearch:
    hosts: [ "..." ]
    username: "..."
    password: "..."
    index: voucher-app-logs-%{[agent.version]}-%{+yyyy.MM.dd}
                                                 ^^^^^^^^^^^
                                                 这将每天更改

所以每天 Beat 会寻找尚不存在的数据流。并将创建一个。

Solution

删除数据是您的选择吗?

output:
  elasticsearch:
    hosts: [ "..." ]
    username: "..."
    password: "..."
    index: voucher-app-logs-%{[agent.version]}
英文:

Tldr;

Beats is doing what he is told to ^^

In the output section:

output:
  elasticsearch:
    hosts: [ "..." ]
    username: "..."
    password: "..."
    index: voucher-app-logs-%{[agent.version]}-%{+yyyy.MM.dd}
                                                 ^^^^^^^^^^^
                                                 This will change every day

So everyday the beat is going to look for a datastream that does not exist yet. And will create one.

Solution

Is removing the data an option for you ?

output:
  elasticsearch:
    hosts: [ "..." ]
    username: "..."
    password: "..."
    index: voucher-app-logs-%{[agent.version]}

huangapple
  • 本文由 发表于 2023年4月20日 01:31:54
  • 转载请务必保留本文链接:https://go.coder-hub.com/76057330.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定