Elasticsearch/Filebeat _meta/fields.yml not loading into kibana.

huangapple go评论68阅读模式
英文:

Elasticsearch/Filebeat _meta/fields.yml not loading into kibana

问题

我被分配到我们产品的一个Filebeat插件。我编写了一个IngestPipeline,并且默认映射在各种方式上不幸地,所以我想通过_meta/fields.yml来改变这个。我根据以下配置了yml文件:
https://www.elastic.co/guide/en/beats/devguide/7.8/filebeat-modules-devguide.html#_metafields_yml_2
https://www.elastic.co/guide/en/beats/devguide/current/event-fields-yml.html
https://github.com/elastic/kibana/issues/82273

我的yml文件看起来像这样:

  • 键:pac
    标题:"pac"
    描述:缺少描述
    pac模块
    字段:

    • 名称:pac.log
      类型:组
      描述:缺少描述
      字段:

      • 名称:deo
        类型:组
        描述:缺少描述
        字段:

        • 名称:duration
          类型:长整型
          描述:deo的持续时间
        • 名称:category
          类型:关键词
          描述:deo的类别
        • 名称:owner
          类型:关键词
          描述:deo的用户
        • 名称:version
          类型:浮点数
          描述:deo的版本
        • 名称:name
          类型:关键词
          描述:deo的名称
        • 名称:reference
          类型:双精度
          描述:deo的参考编号
        • 名称:state
          类型:关键词
          描述:deo的状态
        • 名称:status
          类型:关键词
          描述:deo的状态
        • 名称:trigger
          类型:组
          描述:缺少描述
          字段:

          • 名称:category
            类型:关键词
            描述:deo-trigger的类别
          • 名称:name
            类型:文本
            描述:deo-trigger的名称
          • 名称:path
            类型:文本
            描述:触发器属性的全文本
          • 名称:provider
            类型:关键词
            描述:deo-trigger的供应商
        • 名称:wiring
          类型:组
          描述:缺少描述
          字段:

          • 名称:async
            类型:布尔值
            描述:deo是否异步连接
          • 名称:deoId
            类型:关键词
            描述:deo的识别号
          • 名称:execute
            类型:布尔值
            描述:deo是否执行
          • 名称:owner
            类型:关键词
            描述:deo的用户
          • 名称:shared
            类型:布尔值
            描述:deo是否共享
          • 名称:stopOnError
            类型:布尔值
            描述:deo是否在错误时停止
    • 名称:do
      类型:组
      描述:缺少描述
      字段:

      • 名称:name
        类型:关键词
        描述:do任务的名称
      • 名称:state
        类型:组
        描述:缺少描述
        字段:

        • 名称:from
          类型:关键词
          描述:do任务使用的状态
        • 名称:to
          类型:关键词
          描述:do任务切换到的状态
    • 名称:esa
      类型:组
      描述:缺少描述
      字段:

      • 名称:connection
        类型:关键词
        描述:ESA的连接状态
      • 名称:name
        类型:关键词
        描述:ESA的名称
      • 名称:state
        类型:组
        描述:缺少描述
        字段:

        • 名称:from
          类型:关键词
          描述:建立ESA连接的状态
        • 名称:to
          类型:关键词
          描述:ESA连接建立的状态
    • 名称:monitor
      类型:组
      描述:缺少描述
      字段:

      • 名称:heap
        类型:组
        描述:缺少描述
        字段:

        • 名称:bytes
          类型:长整型
          描述:堆的已使用字节数
        • 名称:pct
          类型:浮点数
          描述:最大可用字节数的百分比
      • 名称:heapgc
        类型:组
英文:

I'm assigned to a filebeat plugin of our product. I wrote a IngestPipeline and de default mapping is in various ways unfortunately so i wanted to change this with the _meta/fields.yml.
I configured the yml file accordingly to:
https://www.elastic.co/guide/en/beats/devguide/7.8/filebeat-modules-devguide.html#_metafields_yml_2
https://www.elastic.co/guide/en/beats/devguide/current/event-fields-yml.html
https://github.com/elastic/kibana/issues/82273

My yml file looks like this:

- key: pac
  title: "pac"
  description: Description missing
    pac Module
  fields:
    - name: pac.log
      type: group
      description: Description missing
      fields:
        - name: deo
          type: group
          description: Description missing
          fields:
            - name: duration
              type: long
              description: Duration of the deo
            - name: category
              type: keyword
              description: Category of the deo
            - name: owner
              type: keyword
              description: User of the deo
            - name: version
              type: float
              description: Version of the deo
            - name: name
              type: keyword
              description: Name of the deo
            - name: reference
              type: double
              description: Referencenumber of the deo
            - name: state
              type: keyword
              description: State of the deo
            - name: status
              type: keyword
              description: Status of the deo
            - name: trigger
              type: group
              description: Description missing
              fields:
                - name: category
                  type: keyword
                  description: Category of the deo-trigger
                - name: name
                  type: text
                  description: Name of the deo-trigger
                - name: path
                  type: text
                  description: Full-Text of the trigger properties
                - name: provider
                  type: keyword
                  description: Supplier of the deo-trigger
            - name: wiring
              type: group
              description: Description missing
              fields:
                - name: async
                  type: boolean
                  description: If deos wired asynchronously
                - name: deoId
                  type: keyword
                  description: Identification number of the deo
                - name: execute
                  type: boolean
                  description: If deo is executed or not
                - name: owner
                  type: keyword
                  description: User of the deo
                - name: shared
                  type: boolean
                  description: Deo was shared
                - name: stopOnError
                  type: boolean
                  description: If deo stopped on Error
        - name: do
          type: group
          description: Description missing
          fields:
            - name: name
              type: keyword
              description: Name of the do task
            - name: state
              type: group
              description: Description missing
              fields:
                - name: from
                  type: keyword
                  description: State from which the do task was used
                - name: to
                  type: keyword
                  description: State to which the do task was switched
        - name: esa
          type: group
          description: Description missing
          fields:
            - name: connection
              type: keyword
              description: Connection status of the ESA
            - name: name
              type: keyword
              description: Name of the ESA
            - name: state
              type: group
              description: Description missing
              fields:
                - name: from
                  type: keyword
                  description: State from which the ESA Connection was established
                - name: to
                  type: keyword
                  description: State to which the ESA Connection was established
        - name: monitor
          type: group
          description: Description missing
          fields:
            - name: heap
              type: group
              description: Description missing
              fields:
                - name: bytes
                  type: long
                  description: Used bytes of the heap
                - name: pct
                  type: float
                  description: Percentage of the maximum available bytes
            - name: heapgc
              type: group
              description: Description missing
              fields:
                - name: bytes
                  type: long
                  description: Used bytes of the heapgc
                - name: pct
                  type: float
                  description: Percentage of the maximum available bytes
        - name: service
          type: group
          description: Description missing
          fields:
            - name: class
              type: keyword
              description: Class of the service
            - name: duration
              type: long
              description: How long the service call took
            - name: name
              type: keyword
              description: Name of the service call
            - name: operation
              type: keyword
              description: Operation type of the service call
            - name: success
              type: boolean
              description: Was the service call successfull or not
        - name: system
          type: group
          description: Description missing
          fields:
            - name: category
              type: keyword
              description: Category of the System
            - name: priority
              type: keyword
              description: Priority of the System
            - name: monitor
              type: group
              description: Description missing
              fields:
                - name: cpu
                  type: group
                  description: Description missing
                  fields:
                    - name: pct
                      type: float
                      description: Percentage of the cpu usage
        - name: wiring
          type: text
          description: Description missing
        - name: meta
          type: text
          description: Description missing
        - name: tags
          type: keyword
          description: Description missing
        - name: timestamp
          type: date
          description: Description missing
        - name: level
          type: keyword
          description: Description missing
        - name: logger
          type: keyword
          description:  Description missing

I would like to have the field types as configured in the fields.yml file. But at first no mapping is available and if i start to send logs the mapping for my fields is created automatically with unfortunate types.

答案1

得分: 0

yaml文件按照上面的描述正常工作。我的问题是,我继承项目的同事没有使用filebeat构建脚本。因此,上传的fields.yml文件没有包含我的字段。我在filebeat.yml配置文件中重新配置了fields.yaml路径,现在它可以工作了。

英文:

The yaml file works as it is in the description above. My Problem was that my collegue from whom i inherited the project didn't use the filebeat build script. So the uploaded fields.yml file wasn't extended by my fields. I reconfigured the fields.yaml path inside the filebeat.yml configuration file and now it works.

huangapple
  • 本文由 发表于 2023年2月8日 20:50:32
  • 转载请务必保留本文链接:https://go.coder-hub.com/75386057.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定