所有消息都会收到一个“用户级通知”

huangapple go评论48阅读模式
英文:

All messages receive a "user level notice"

问题

我正在尝试解析来自我的网络设备的消息,这些消息以类似以下格式发送:

<30>Feb 14 11:33:59 wireless: ath0     Sending auth to xx:xx:xx:xx:xx:xx. Status: The request has been declined due to MAC ACL (52).
<190>Feb 14 11:01:29 CCR00 user admin logged out from xx.xx.xx.xx via winbox
<134>2023 Feb 14 11:00:33 ZTE command-log:An alarm 36609 level notification occurred at 11:00:33 02/14/2023 CET sent by MCP GponRm notify: <gpon-onu_1/1/1:1> SubType:1 Pos:1 ONU Uni lan los. restore\n on \n

使用这个logstash.conf文件:

input {
	beats {
		port => 5044
	}

	tcp {
		port => 50000
	}
  udp {
		port => 50000
	}
}

## 在这里添加您的过滤器/Logstash插件配置

filter {

      grok {	
        match => { 
		"message" => "^(?:<%{POSINT:syslog_pri}>)?%{GREEDYDATA:message_payload}"
	   }
	 }	 
      syslog_pri {	  	  
      }
      mutate {
	remove_field => [ "@version" , "message" ]
	 }
}


output {
	stdout {}
	elasticsearch {
		hosts => "elasticsearch:9200"
		user => "logstash_internal"
		password => "${LOGSTASH_INTERNAL_PASSWORD}"
	}
}

这导致了以下输出:

{
  "@timestamp": [
    "2023-02-14T10:38:59.228Z"
  ],
  "data_stream.dataset": [
    "generic"
  ],
  "data_stream.namespace": [
    "default"
  ],
  "data_stream.type": [
    "logs"
  ],
  "event.original": [
    "<14> Feb 14 11:38:59 UBNT BOXSERV[boxs Req]: boxs.c(691) 55381193 %% Error 17 occurred reading thermal sensor 2 data\n\u0000"
  ],
  "host.ip": [
    "10.125.132.10"
  ],
  "log.syslog.facility.code": [
    1
  ],
  "log.syslog.facility.name": [
    "user-level"
  ],
  "log.syslog.severity.code": [
    5
  ],
  "log.syslog.severity.name": [
    "notice"
  ],
  "message_payload": [
    " Feb 14 11:38:59 UBNT[boxs Req]: boxs.c(691) 55381193 %% Error 17 occurred reading thermal sensor 2 data\n\u0000"
  ],
  "syslog_pri": [
    "14"
  ],
  "_id": "UzmBT4YBAZPdbqc4m_IB",
  "_index": ".ds-logs-generic-default-2023.02.04-000001",
  "_score": null
}

这基本上是令人满意的,但我希望log.syslog.facility.namelog.syslog.severity.name字段由syslog_pri过滤器处理,将输入的&lt;14&gt;转换为secur/authAlert,但无论syslog消息的<PRI>部分包含什么,我始终得到默认的user-levelnotice

是否有人可以提供建议,或者如果语法错误,可能修复我的.conf文件?

非常感谢!我已经正确配置了Logstash来接收日志并将它们发送到Elasticsearch,但grok/syslog_pri并没有产生预期的结果。

英文:

Im trying to parse a message from my network devices which send messages in format similar to

&lt;30&gt;Feb 14 11:33:59 wireless: ath0     Sending auth to xx:xx:xx:xx:xx:xx. Status: The request has been declined due to MAC ACL (52).\n
&lt;190&gt;Feb 14 11:01:29 CCR00 user admin logged out from xx.xx.xx.xx via winbox
&lt;134&gt;2023 Feb 14 11:00:33 ZTE command-log:An alarm 36609 level notification occurred at 11:00:33 02/14/2023 CET sent by MCP GponRm notify: &lt;gpon-onu_1/1/1:1&gt; SubType:1 Pos:1 ONU Uni lan los. restore\n on  \n

using this logstash.conf file

input {
	beats {
		port =&gt; 5044
	}

	tcp {
		port =&gt; 50000
	}
  udp {
		port =&gt; 50000
	}
}

## Add your filters / logstash plugins configuration here

filter {

      grok {	
        match =&gt; { 
		&quot;message&quot; =&gt; &quot;^(?:&lt;%{POSINT:syslog_pri}&gt;)?%{GREEDYDATA:message_payload}&quot;
	   }
	 }	 
      syslog_pri {	  
         }
      mutate {
	remove_field =&gt; [ &quot;@version&quot; , &quot;message&quot; ]
	 }
}


output {
	stdout {}
	elasticsearch {
		hosts =&gt; &quot;elasticsearch:9200&quot;
		user =&gt; &quot;logstash_internal&quot;
		password =&gt; &quot;${LOGSTASH_INTERNAL_PASSWORD}&quot;
	}
}

which results in this output

{
  &quot;@timestamp&quot;: [
    &quot;2023-02-14T10:38:59.228Z&quot;
  ],
  &quot;data_stream.dataset&quot;: [
    &quot;generic&quot;
  ],
  &quot;data_stream.namespace&quot;: [
    &quot;default&quot;
  ],
  &quot;data_stream.type&quot;: [
    &quot;logs&quot;
  ],
  &quot;event.original&quot;: [
    &quot;&lt;14&gt; Feb 14 11:38:59 UBNT BOXSERV[boxs Req]: boxs.c(691) 55381193 %% Error 17 occurred reading thermal sensor 2 data\n\u0000&quot;
  ],
  &quot;host.ip&quot;: [
    &quot;10.125.132.10&quot;
  ],
  &quot;log.syslog.facility.code&quot;: [
    1
  ],
  &quot;log.syslog.facility.name&quot;: [
    &quot;user-level&quot;
  ],
  &quot;log.syslog.severity.code&quot;: [
    5
  ],
  &quot;log.syslog.severity.name&quot;: [
    &quot;notice&quot;
  ],
  &quot;message_payload&quot;: [
    &quot; Feb 14 11:38:59 UBNT[boxs Req]: boxs.c(691) 55381193 %% Error 17 occurred reading thermal sensor 2 data\n\u0000&quot;
  ],
  &quot;syslog_pri&quot;: [
    &quot;14&quot;
  ],
  &quot;_id&quot;: &quot;UzmBT4YBAZPdbqc4m_IB&quot;,
  &quot;_index&quot;: &quot;.ds-logs-generic-default-2023.02.04-000001&quot;,
  &quot;_score&quot;: null
}

which is mostly satisfactory, but i would expect the
log.syslog.facility.name
and
log.syslog.severity.name
fields to be processed by the
syslog_pri
filter
with imput of
&lt;14&gt;
to result into
secur/auth
and
Alert
recpectively,

but i keep getting the default user-level notice for all my messages, no matter what the <PRI> part of the syslog message contains

anyone could advise and maybe fix my .conf syntax, if its wrong?

thank you very much!

i have logstash configured properly to receive logs and send them to elastics, but the grok/syslog_pri doesnt yield expected results

答案1

得分: 0

syslog_pri过滤器设置[log][syslog][facility][code]表明已启用ECS兼容性1。因此,如果您没有在syslog_pri过滤器上设置syslog_pri_field_name选项,它将尝试解析[log][syslog]priority。如果该字段不存在,则将解析默认值13,即用户级/注意3

英文:

The fact that the syslog_pri filter is setting [log][syslog][facility][code] shows that it has ECS compatibility enabled. As a result, if you do not set the syslog_pri_field_name option on the syslog_pri filter, it will try to parse [log][syslog][priority]. If that field does not exist then it will parse the default value of 13, which is user-level/notice.

答案2

得分: 0

谢谢您的回答,我已根据提供的建议调整了代码,并且现在它表现如预期。

我将稍微调整消息以满足我的需求,但这超出了本问题的范围。

非常感谢!

英文:

thank you for the answer, i have adjusted the code by the given advice

filter {

  grok {	
    match =&gt; {  		&quot;message&quot; =&gt; &quot;^(?:&lt;%{POSINT:syslog_code}&gt;)?%{GREEDYDATA:message_payload}&quot;
   } 	 }	 
  syslog_pri { 		syslog_pri_field_name =&gt; &quot;syslog_code&quot;	  
     }
  mutate { 	remove_field =&gt; [ &quot;@version&quot; , &quot;message&quot; ] 	 } }

and now it behaves as intended

                  &quot;event&quot; =&gt; {
        &quot;original&quot; =&gt; &quot;&lt;30&gt;Feb 15 18:41:04 dnsmasq-dhcp[960]: DHCPACK(eth0) 10.0.0.165 xx:xx:xx:xx:xx CZ\n&quot;
    },
         &quot;@timestamp&quot; =&gt; 2023-02-15T17:41:04.977038615Z,
    &quot;message_payload&quot; =&gt; &quot;Feb 15 18:41:04 dnsmasq-dhcp[960]: DHCPACK(eth0) 10.0.0.165 xx:xx:xx:xx:xx CZ\n&quot;,
                &quot;log&quot; =&gt; {
        &quot;syslog&quot; =&gt; {
            &quot;severity&quot; =&gt; {
                &quot;code&quot; =&gt; 6,
                &quot;name&quot; =&gt; &quot;informational&quot;
            },
            &quot;facility&quot; =&gt; {
                &quot;code&quot; =&gt; 3,
                &quot;name&quot; =&gt; &quot;daemon&quot;
            }

}
    },
        &quot;syslog_code&quot; =&gt; &quot;30&quot;,
               &quot;host&quot; =&gt; {
        &quot;ip&quot; =&gt; &quot;xx.xx.xx.xx&quot;
    } }

i will adjust the message a bit to fit my needs,
but that is out of the scope of this question

thank you very much!

huangapple
  • 本文由 发表于 2023年2月14日 18:45:20
  • 转载请务必保留本文链接:https://go.coder-hub.com/75446717.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定