英文:
Parsing traceId, spanId from log line using Opentelemetry logprocessor
问题
我们需要解析日志行中的属性，以便在后端（例如：ES）上进行查询。
我尝试使用支持 operators 的 otel filelog 接收器，但我不得不放弃它并将日志收集更改为 fluentbit，因为 filelog 不支持多行解析。
现在我面临的情况是，我必须在处理器阶段解析/操作日志。为此，我想使用 logtransform 处理器。
但不幸的是，它似乎不是 otel-contrib 发行版的一部分。
Otel collector contrib 版本：0.68.0
Otel collector contrib 配置
processor:
logstransform:
operators:
- type: regex_parser
id: trace_parser
regex: 'traceId=(?P<trace_id>\S*)''
parse_from: body
trace:
trace_id:
parse_from: attributes.trace_id
错误：
2023/01/05 17:22:19 collector server run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:
* error decoding 'processors': unknown processors type: "logstransform" for id: "logstransform" (valid values: [attributes groupbytrace k8sattributes metricstransform probabilistic_sampler tail_sampling batch memory_limiter transform cumulativetodelta routing servicegraph span groupbyattrs resourcedetection experimental_metricsgeneration redaction resource spanmetrics deltatorate filter])
日志行：
[2022-12-30 12:34:56] [INFO] [Hotel Management System] traceId=4bf92f3577b34da6a3ce929d0e0e4736 spanId=00f067aa0ba902b7 message="Guest John Doe checked into room 214 for a 3-night stay." guestName=John Doe roomNumber=214 stayLength=3 night
期望：
资源架构URL：https://opentelemetry.io/schemas/1.6.1
资源属性：
-> cloud.provider: Str(gcp)
-> cloud.account.id: Str(project-jiomarket-non-prod)
-> cloud.platform: Str(gcp_kubernetes_engine)
-> cloud.region: Str(asia-south1)
-> k8s.cluster.name: Str(cluster-central-alpha)
-> host.id: Str(6758479764707402031)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope
LogRecord #0
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-01-05 18:22:02.466838531 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Str([2023-01-05 12:34:56] [INFO] [Hotel Management System] traceId=4bf92f3577b34da6a3ce929d0e0e4736 spanId=00f067aa0ba902b7 message="Guest John Doe checked into room 214 for a 3-night stay." guestName=John Doe roomNumber=214 stayLength=3 night
)
Attributes:
-> fluent.tag: Str(kube.var.log.pods.newco_newcoshop-master-55c7f9d9f4-rmnqc_d02aa156-1671-4d58-b017-2e763b2d1683.newcoshop-master.test.log)
Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736
Span ID: 00f067aa0ba902b7
Flags: 0
{"kind": "exporter", "data_type": "logs", "name": "logging"}
任何指导将不胜感激。
谢谢
英文:
We need to parse attributes from log line so as to make them queryable on backends(eg: ES).
Now I tried using otel filelog receiver which supports stanza operators, but the reason I had to drop it and replace log collection to fluentbit because filelog doesn't support multiline parsing.
Now I have a situation that I'm left with parsing/manipulating log at processors stage. For the same I wanted to use logtransform processor
But unfortunately it seems it not part of otel-contrib distribution.
Otel collector contrib version: 0.68.0
Otel collector contrib config
processor:
logstransform:
operators:
- type: regex_parser
id: trace_parser
regex: 'traceId=(?P<trace_id>\S*)'
parse_from: body
trace:
trace_id:
parse_from: attributes.trace_id
Error:
2023/01/05 17:22:19 collector server run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:
* error decoding 'processors': unknown processors type: "logstransform" for id: "logstransform" (valid values: [attributes groupbytrace k8sattributes metricstransform probabilistic_sampler tail_sampling batch memory_limiter transform cumulativetodelta routing servicegraph span groupbyattrs resourcedetection experimental_metricsgeneration redaction resource spanmetrics deltatorate filter])
logline:
[2022-12-30 12:34:56] [INFO] [Hotel Management System] traceId=4bf92f3577b34da6a3ce929d0e0e4736 spanId=00f067aa0ba902b7 message="Guest John Doe checked into room 214 for a 3-night stay." guestName=John Doe roomNumber=214 stayLength=3 night
Expectation:
Resource SchemaURL: https://opentelemetry.io/schemas/1.6.1
Resource attributes:
-> cloud.provider: Str(gcp)
-> cloud.account.id: Str(project-jiomarket-non-prod)
-> cloud.platform: Str(gcp_kubernetes_engine)
-> cloud.region: Str(asia-south1)
-> k8s.cluster.name: Str(cluster-central-alpha)
-> host.id: Str(6758479764707402031)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope
LogRecord #0
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-01-05 18:22:02.466838531 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Str([2023-01-05 12:34:56] [INFO] [Hotel Management System] traceId=4bf92f3577b34da6a3ce929d0e0e4736 spanId=00f067aa0ba902b7 message="Guest John Doe checked into room 214 for a 3-night stay." guestName=John Doe roomNumber=214 stayLength=3 night
)
Attributes:
-> fluent.tag: Str(kube.var.log.pods.newco_newcoshop-master-55c7f9d9f4-rmnqc_d02aa156-1671-4d58-b017-2e763b2d1683.newcoshop-master.test.log)
Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736
Span ID: 00f067aa0ba902b7
Flags: 0
{"kind": "exporter", "data_type": "logs", "name": "logging"}
Any guidance would be appreciated.
Thanks
答案1
得分: 1
Indeed, it's currently not included in the latest builds of the distribution. I have opened a bug ticket for this issue.
英文:
Indeed, it's currently not included in the latest builds of the distribution. I have opened a bug ticket for this issue.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论