英文:
Query not executing in Elasticsearch input plugin for Logstash
问题
我是新手，尝试通过elasticsearch输入插件在logstash中执行简单的查询。我已经解决了一些初始错误，现在只剩下一些显著的警告，但是没有从查询中获取任何输出。
我已经验证了查询在elasticsearch本身中可以正常工作，并且输出在使用生成器的情况下也正常工作。以下是配置，其中一些私人信息已被替换。
input {
elasticsearch {
id => "es_input_plugin"
user => myuser
password => mypassword
hosts => myhost
ca_file => "C:\devsetup\logstash-8.6.0\rootca3.crt"
ssl => true
index => "log-com-ms-ldap--*"
query => '{
"query": {
"range": {
"@timestamp": {
"gte": "now-1d/d"
}
}
},
"size": 1
}'
docinfo => true
docinfo_target => "[@metadata][doc]"
}
}
output {
stdout {
codec => rubydebug
}
email {
to => myemail
from => mysystem
subject => 'Success'
body => 'Success'
address => myaddress
via => 'smtp'
}
}
以下是运行文件时获得的输出。
C:\devsetup\logstash-8.6.0\bin>logstash.bat -f C:\Users\ElamR\Documents\elasticinput.conf
"Using bundled JDK: C:\devsetup\logstash-8.6.0\jdk\bin\java.exe"
Sending Logstash logs to C:/devsetup/logstash-8.6.0/logs which is now configured via log4j2.properties
[2023-02-23T13:01:56,964][INFO ][logstash.runner ] Log4j configuration path used is: C:\devsetup\logstash-8.6.0\config\log4j2.properties
[2023-02-23T13:01:56,973][WARN ][logstash.runner ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2023-02-23T13:01:56,975][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.6.0", "jruby.version"=>"jruby 9.3.8.0 (2.6.8) 2022-09-13 98d69c9461 OpenJDK 64-Bit Server VM 17.0.5+8 on 17.0.5+8 +indy +jit [x86_64-mswin32"]}
[2023-02-23T13:01:56,980][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED}
[2023-02-23T13:01:57,019][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-02-23T13:01:58,272][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-02-23T13:01:58,700][INFO ][org.reflections.Reflections] Reflections took 180 ms to scan 1 urls, producing 127 keys and 444 values
[2023-02-23T13:02:01,422][INFO ][logstash.javapipeline ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2023-02-23T13:02:02,095][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["C:/Users/ElamR/Documents/elasticinput.conf"], :thread=>"#<Thread:0x6f11e638@C:/devsetup/logstash-8.6.0/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}
[2023-02-23T13:02:02,797][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.7}
[2023-02-23T13:02:04,827][INFO ][logstash.inputs.elasticsearch][main] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2023-02-23T13:02:04,831][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-02-23T13:02:04,849][INFO
<details>
<summary>英文:</summary>
I'm new to the stack, and am trying to execute simple queries in logstash via the elasticsearch input plugin. I have worked through some initial errors and now have only a couple of notable warnings, but am not getting any output from the query.
I've verified that the query works properly in elasticsearch itself and that the output functions properly using by using a generator. Here is the config, with some private info swapped out.
input {
elasticsearch {
id => "es_input_plugin"
user => myuser
password => mypassword
hosts => myhost
ca_file => "C:\devsetup\logstash-8.6.0\rootca3.crt"
ssl => true
index => "log-com-ms-ldap--*"
query => '{
"query": {
"range": {
"@timestamp": {
"gte": "now-1d/d"
}
}
},
"size": 1
}'
docinfo => true
docinfo_target => "[@metadata][doc]"
}
}
output {
stdout {
codec => rubydebug
}
email {
to => myemail
from => mysystem
subject => 'Success'
body => 'Success'
address => myaddress
via => 'smtp'
}
}
And here is the output I get when running the file.
C:\devsetup\logstash-8.6.0\bin>logstash.bat -f C:\Users\ElamR\Documents\elasticinput.conf
"Using bundled JDK: C:\devsetup\logstash-8.6.0\jdk\bin\java.exe"
Sending Logstash logs to C:/devsetup/logstash-8.6.0/logs which is now configured via log4j2.properties
[2023-02-23T13:01:56,964][INFO ][logstash.runner ] Log4j configuration path used is: C:\devsetup\logstash-8.6.0\config\log4j2.properties
[2023-02-23T13:01:56,973][WARN ][logstash.runner ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2023-02-23T13:01:56,975][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.6.0", "jruby.version"=>"jruby 9.3.8.0 (2.6.8) 2022-09-13 98d69c9461 OpenJDK 64-Bit Server VM 17.0.5+8 on 17.0.5+8 +indy +jit [x86_64-mswin32]"}
[2023-02-23T13:01:56,980][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2023-02-23T13:01:57,019][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-02-23T13:01:58,272][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-02-23T13:01:58,700][INFO ][org.reflections.Reflections] Reflections took 180 ms to scan 1 urls, producing 127 keys and 444 values
[2023-02-23T13:02:01,422][INFO ][logstash.javapipeline ] Pipeline main is configured with pipeline.ecs_compatibility: v8 setting. All plugins in this pipeline will default to ecs_compatibility => v8 unless explicitly configured otherwise.
[2023-02-23T13:02:02,095][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["C:/Users/ElamR/Documents/elasticinput.conf"], :thread=>"#<Thread:0x6f11e638@C:/devsetup/logstash-8.6.0/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}
[2023-02-23T13:02:02,797][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.7}
[2023-02-23T13:02:04,827][INFO ][logstash.inputs.elasticsearch][main] ECS compatibility is enabled but target option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the target option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2023-02-23T13:02:04,831][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-02-23T13:02:04,849][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2023-02-23T13:02:05,108][WARN ][logstash.inputs.elasticsearch][main][es_input_plugin] Ignoring clear_scroll exception {:message=>"[404] {"succeeded":true,"num_freed":0}", :exception=>Elasticsearch::Transport::Transport::Errors::NotFound}
[2023-02-23T13:02:05,215][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2023-02-23T13:02:05,356][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2023-02-23T13:02:05,362][INFO ][logstash.runner ] Logstash shut down.
My thoughts have been centered around the final warning, regarding clear_scroll exception, but I have yet to find any related references.
Any help would be greatly appreciated. Thanks
</details>
# 答案1
**得分**: 1
以下是翻译好的部分：
- Ensuring that the SSL certificate is checked by specifying the ssl certificate verification option.
确保通过指定SSL证书验证选项来检查SSL证书。
- Giving users the opportunity to provide the root certificate for SSL verification with the ssl certificate authorities setting
为用户提供提供根证书以进行SSL验证的机会，使用ssl证书颁发机构设置。
- A list of hosts is indicated by enclosing the hosts option in an array.
通过将主机选项封装在数组中来表示主机列表。
- To guarantee that the email addresses are handled as strings, you should enclose them in quotes.
为了确保电子邮件地址被视为字符串，您应该将它们括在引号中。
<details>
<summary>英文:</summary>
### Logstash Pipeline for Elasticsearch to Email
The principal modifications consist of:
- Ensuring that the SSL certificate is checked by specifying the ssl certificate verification option.
- Giving users the opportunity to provide the root certificate for SSL verification with the ssl certificate authorities setting
- A list of hosts is indicated by enclosing the hosts option in an array.
- To guarantee that the email addresses are handled as strings, you should enclose them in quotes.
Here is the updated pipeline
```bash
input {
elasticsearch {
id => "es_input_plugin"
user => "myuser"
password => "mypassword"
hosts => ["myhost"]
ssl_certificate_verification => true
ssl_certificate_authorities => ["C:\devsetup\logstash-8.6.0\rootca3.crt"]
ssl => true
index => "log-com-ms-ldap-*"
query => '{
"query": {
"range": {
"@timestamp": {
"gte": "now-1d/d"
}
}
},
"size": 1
}'
docinfo => true
docinfo_target => "[@metadata][doc]"
}
}
output {
stdout {
codec => rubydebug
}
email {
to => "myemail"
from => "mysystem"
subject => 'Success'
body => 'Success'
address => "myaddress"
via => 'smtp'
}
}
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论