英文:
How to remove "timestamp" date from the Fluent-Bit logs?
问题
I'm testing Fluent-bit for my local k8s cluster which has a CRI runtime interface and I'm sending logs to a slack channel. But the problem is that Fluent-Bit is assigning a "timestamp" in the log and I'm not able to remove it. Maybe someone knows a solution?
Here is the ConfigMap of my Fluent-Bit:
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging1
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         2
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020    
 
    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-syslog.conf
 
  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*
        Parser            cri
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On    
  filter-kubernetes.conf: |
  output-syslog.conf: |
    [OUTPUT]
        Name               slack
        Match              *
        webhook            [LINK]    
 
  parsers.conf: |
    [PARSER]
        Name          cc
        Format        regex
        Format        cri
        Regex         ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key      time
        Time_Format   %Y-%m-%dT%H:%M:%S.%L%z    
Also here is the raw log coming from my app:
2023-04-12T16:09:02.016483996Z stderr F 10.244.0.1 - - [12/Apr/2023 16:09:02] "GET / HTTP/1.1" 200 -
And this is the log that is sent to Slack:
["timestamp": 1681315742.016981904, {"log"=>"2023-04-12T16:09:02.016483996Z stderr F 10.244.0.1 - - [12/Apr/2023 16:09:02] "GET / HTTP/1.1" 200 -"}]
I've used different Filters and Parsers. Currently, my Fluent-bit is using the latest image.
英文:
I'm testing Fluent-bit for my local k8s cluster which has a CRI runtime interface and I'm sending logs to a slack channel. But the problem is that Fluent-Bit is assigning a "timestamp" in the log and I'm not able to remove it. Maybe someone knows a solution?
Here is the ConfigMap of my Fluent-Bit:
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging1
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         2
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020
 
    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-syslog.conf
 
  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*
        Parser            cri
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
  filter-kubernetes.conf: |
  output-syslog.conf: |
    [OUTPUT]
        Name               slack
        Match              *
        webhook            [LINK]
        
 
  parsers.conf: |
    [PARSER]
        Name          cc
        Format        regex
        Format        cri
        Regex         ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key      time
        Time_Format   %Y-%m-%dT%H:%M:%S.%L%z
Also here is the raw log coming from my app:
2023-04-12T16:09:02.016483996Z stderr F 10.244.0.1 - - [12/Apr/2023 16:09:02] "GET / HTTP/1.1" 200 -
And this is the log that is sent to Slack:
["timestamp": 1681315742.016981904, {"log"=>"2023-04-12T16:09:02.016483996Z stderr F 10.244.0.1 - - [12/Apr/2023 16:09:02] "GET / HTTP/1.1" 200 -"}]
I've used different Filters and Parsers. Currently my Fluent-bit is using the latest image
答案1
得分: 1
The parser cri does not exist in your configuration, therefore the files are not parsed correctly and you receive "2023-04-12T16:09:02.016483996Z stderr F " as part of your message log. Just use the official parsers.conf provided by fluent-bit or fix your typos (Name: cri, not cc, Format is regex).
A valid snippet would be:
[PARSER]
http://rubular.com/r/tjUt3Awgg4
Name cri
Format regex
Regex ^(?
英文:
The parser cri does not exists in your configuration, therefore the files are not parsed correctly and you receive "2023-04-12T16:09:02.016483996Z stderr F " as part of your message log. Just use the official parsers.conf provided by fluent-bit or fix your typos (Name: cri, not cc, Format is regex).
A valid snipped would be:
[PARSER]
    # http://rubular.com/r/tjUt3Awgg4
    Name cri
    Format regex
    Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。


评论