如何从Fluent-Bit日志中删除“timestamp”日期?

huangapple go评论94阅读模式
英文:

How to remove "timestamp" date from the Fluent-Bit logs?

问题

I'm testing Fluent-bit for my local k8s cluster which has a CRI runtime interface and I'm sending logs to a slack channel. But the problem is that Fluent-Bit is assigning a "timestamp" in the log and I'm not able to remove it. Maybe someone knows a solution?

Here is the ConfigMap of my Fluent-Bit:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging1
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         2
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020    
 
    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-syslog.conf
 
  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*
        Parser            cri
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On    

  filter-kubernetes.conf: |

  output-syslog.conf: |
    [OUTPUT]
        Name               slack
        Match              *
        webhook            [LINK]    
 

  parsers.conf: |
    [PARSER]
        Name          cc
        Format        regex
        Format        cri
        Regex         ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key      time
        Time_Format   %Y-%m-%dT%H:%M:%S.%L%z    

Also here is the raw log coming from my app:

2023-04-12T16:09:02.016483996Z stderr F 10.244.0.1 - - [12/Apr/2023 16:09:02] "GET / HTTP/1.1" 200 -

And this is the log that is sent to Slack:

["timestamp": 1681315742.016981904, {"log"=>"2023-04-12T16:09:02.016483996Z stderr F 10.244.0.1 - - [12/Apr/2023 16:09:02] "GET / HTTP/1.1" 200 -"}]

I've used different Filters and Parsers. Currently, my Fluent-bit is using the latest image.

英文:

I'm testing Fluent-bit for my local k8s cluster which has a CRI runtime interface and I'm sending logs to a slack channel. But the problem is that Fluent-Bit is assigning a "timestamp" in the log and I'm not able to remove it. Maybe someone knows a solution?

Here is the ConfigMap of my Fluent-Bit:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging1
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         2
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020
 
    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-syslog.conf
 
  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*
        Parser            cri
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On

  filter-kubernetes.conf: |

  output-syslog.conf: |
    [OUTPUT]
        Name               slack
        Match              *
        webhook            [LINK]
        
 
  parsers.conf: |
    [PARSER]
        Name          cc
        Format        regex
        Format        cri
        Regex         ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key      time
        Time_Format   %Y-%m-%dT%H:%M:%S.%L%z

Also here is the raw log coming from my app:

2023-04-12T16:09:02.016483996Z stderr F 10.244.0.1 - - [12/Apr/2023 16:09:02] "GET / HTTP/1.1" 200 -

And this is the log that is sent to Slack:

["timestamp": 1681315742.016981904, {"log"=>"2023-04-12T16:09:02.016483996Z stderr F 10.244.0.1 - - [12/Apr/2023 16:09:02] "GET / HTTP/1.1" 200 -"}]

I've used different Filters and Parsers. Currently my Fluent-bit is using the latest image

答案1

得分: 1

The parser cri does not exist in your configuration, therefore the files are not parsed correctly and you receive "2023-04-12T16:09:02.016483996Z stderr F " as part of your message log. Just use the official parsers.conf provided by fluent-bit or fix your typos (Name: cri, not cc, Format is regex).

A valid snippet would be:

[PARSER]

http://rubular.com/r/tjUt3Awgg4

Name cri
Format regex
Regex ^(?

英文:

The parser cri does not exists in your configuration, therefore the files are not parsed correctly and you receive "2023-04-12T16:09:02.016483996Z stderr F " as part of your message log. Just use the official parsers.conf provided by fluent-bit or fix your typos (Name: cri, not cc, Format is regex).

A valid snipped would be:

[PARSER]
    # http://rubular.com/r/tjUt3Awgg4
    Name cri
    Format regex
    Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z

huangapple
  • 本文由 发表于 2023年4月13日 20:09:04
  • 转载请务必保留本文链接:https://go.coder-hub.com/76005246.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定