英文:
fluentd: Not able to send syslogs to Elasticsearch using fluent-plugin-elasticsearch
问题
我正在为我的Linux和Windows机器在RHEL8虚拟机上使用Fluentd和Elasticsearch设置日志转发。以下是我已安装的内容：
- RHEL8 - VMware虚拟机
- Fluentd v1.0
- td-agent 4.5.0 fluentd 1.16.1
- fluent-plugin-elasticsearch-5.3.0
我能够将日志重定向到td-agent.log文件，但无法将日志转发到Elasticsearch。
安装了fluent-plugin-elasticsearch插件后，当我启动td-agent时，它不断生成以下日志：
2023-06-14 04:45:26 -0500 [info]: #0 init worker0 logger path="\"/var/log/td-agent/td-agent.log\" rotate_age=nil rotate_size=nil
2023-06-14 04:45:26 -0500 [info]: adding match pattern="td.." type="tdlog"
2023-06-14 04:45:26 -0500 [warn]: #0 [output_td] Use different plugin for secondary. Check the plugin works with primary like secondary_file primary="Fluent::Plugin::TreasureDataLogOutput" secondary="Fluent::Plugin::FileOutput"
2023-06-14 04:45:26 -0500 [info]: adding match pattern="debug." type="stdout"
2023-06-14 04:45:26 -0500 [info]: adding match pattern="system." type="elasticsearch"
2023-06-14 04:45:26 -0500 [error]: #0 unexpected error error_class=Elastic::Transport::Transport::Error error="EOFError (EOFError)"
...（以下省略）
这是我的td-agent.conf配置，用于Syslog转发：
<source>
@type syslog
port 5140
tag system
</source>
<match system.**>
@type elasticsearch
host localhost
port 9200
logstash_format true
</match>
我的目标是将日志转发到Elasticsearch，并使用Kibana仪表板进行过滤和分析。
我已经尝试使用fluentd-elasticsearch插件的较低版本，但那时代理根本没有启动。我还尝试在另一台非硬化的RHEL7机器上使用相同的配置。
英文:
I am setting up log forwarding for my Linux & Windows machines on a RHEL8 VM using Fluentd and Elasticsearch. This is what I have installed:
- RHEL8 - VMWare VM
- Fluentd v1.0
- td-agent 4.5.0 fluentd 1.16.1
- fluent-plugin-elasticsearch-5.3.0
I am able to redirect the logs to the td-agent.log file but not able to forward the logs to Elasticsearch.
After installing the fluent-plugin-elasticsearch plugin, when I turn on the td-agent, it continuously generates these logs:
2023-06-14 04:45:26 -0500 [info]: #0 init worker0 logger path="/var/log/td-agent/td-agent.log" rotate_age=nil rotate_size=nil
2023-06-14 04:45:26 -0500 [info]: adding match pattern="td.." type="tdlog"
2023-06-14 04:45:26 -0500 [warn]: #0 [output_td] Use different plugin for secondary. Check the plugin works with primary like secondary_file primary="Fluent::Plugin::TreasureDataLogOutput" secondary="Fluent::Plugin::FileOutput"
2023-06-14 04:45:26 -0500 [info]: adding match pattern="debug." type="stdout"
2023-06-14 04:45:26 -0500 [info]: adding match pattern="system." type="elasticsearch"
2023-06-14 04:45:26 -0500 [error]: #0 unexpected error error_class=Elastic::Transport::Transport::Error error="EOFError (EOFError)"
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:324:in rescue in perform_request' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:285:in perform_request'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/http/faraday.rb:36:in perform_request' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elastic-transport-8.2.1/lib/elastic/transport/client.rb:176:in perform_request'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elasticsearch-8.7.0/lib/elasticsearch.rb:71:in method_missing' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elasticsearch-api-8.7.0/lib/elasticsearch/api/actions/info.rb:41:in info'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:498:in detect_es_major_version' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:489:in block in handle_last_seen_es_major_version'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/elasticsearch_index_template.rb:56:in retry_operate' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:486:in handle_last_seen_es_major_version'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:338:in configure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:187:in configure'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:132:in add_match' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:74:in block in configure'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in each' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in configure'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:149:in configure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:105:in configure'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:80:in run_configure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:616:in block in run_worker'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:962:in main_process' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:608:in run_worker'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/command/fluentd.rb:372:in <top (required)>' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/bin/fluentd:15:in <top (required)>'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/bin/fluentd:23:in load' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/bin/fluentd:23:in
2023-06-14 04:45:26 -0500 [error]: Worker 0 exited unexpectedly with status 1
This is my td-agent.conf configuration for syslog forwarding:
<source>
@type syslog
port 5140
tag system
</source>
<match system.**>
@type elasticsearch
host localhost
port 9200
logstash_format true
</match>
My objective is to forward the logs to Elasticsearch and use Kibana dashboard to filter and analyze.
I have already tried using a lower version on the fluentd-elasticsearch plugin but at that time the agent did not start at all. Have tried the same conf in another non-hardened RHEL7 machine.
答案1
得分: 1
我在另一个帖子中找到了解决方案：
<https://stackoverflow.com/questions/35921195/curl-52-empty-reply-from-server-timeout-when-querying-elastiscsearch>
导致我查找此帖子的原因是 Curl 输出：
# curl -i -X --HEAD http://127.0.0.1:9200/
curl: (52) Empty reply from server
在 /etc/elasticsearch/elasticsearch.yml2 中将 Xpack 安全参数从 true 更改为 false：
xpack.security.enabled: false
重新启动 ES 服务，td-agent 日志中的错误消失了。
英文:
I found the solution in a different thread:
<https://stackoverflow.com/questions/35921195/curl-52-empty-reply-from-server-timeout-when-querying-elastiscsearch>
What made me search this thread was the Curl output:
# curl -i -X --HEAD http://127.0.0.1:9200/
curl: (52) Empty reply from server
Changed Xpack security parameter in /etc/elasticsearch/elasticsearch.yml2 from true to false:
xpack.security.enabled: false
Restarted ES service and the errors in td-agent logs were gone.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论