OpenTelemetry .Net Application – Metrics Collected Slowly by Collector

huangapple go评论62阅读模式
英文:

OpenTelemetry .Net Application - Metrics Collected Slowly by Collector

问题

我正在运行一个简单的POC,使用OpenTelemetry在.NET 6 Web API中。我试图在.NET应用程序中使用Otlp导出程序来发送指标到一个收集器。Prometheus可以正常抓取收集器暴露的端点。

然而,指标的初始导出需要很长时间。一旦开始运行,它似乎每60秒发送一次指标。我真的希望将它设置为更合理的5-15秒,以便更好地演示。

我尝试了简单的导出程序,但没有成功。我尝试设置批量导入程序的MaxQueueSize、ScheduledDelayMilliseconds和ExporterTimeoutMilliseconds非常低。

在.NET应用程序端启用ConsoleExporter时,指标似乎按预期的默认值10秒传送。因此,这可能是与收集器在呈现指标端点之前进行某种积累有关的问题?我没有在收集器中使用批处理处理器,所以我不确定为什么会这样做。

当我在API本身上使用PrometheusExporter /抓取端点时,我也没有遇到任何缓慢的问题。只有当我尝试使用Otel Exporter导出指标时才会出现。

有趣的是,我还将OTLP导出程序配置为跟踪,我几乎可以立即获得所有结果到收集器/ jaeger中。

我目前正在使用最新的OpenTelemetry稳定包(1.32.0)。我正在使用以下收集器的Docker镜像:
otel/opentelemetry-collector-contrib:0.42.0

代码可以在这里找到:
https://github.com/MetalHexx/OpenTelemetryDotNetPoc

缩写的启动代码:

services.AddOpenTelemetryMetrics(builder =>
{
     builder
         .AddMeter(App_Source)
         .SetResourceBuilder(resource!)
         .AddAspNetCoreInstrumentation()
         .AddOtlpExporter(options =>
         {
             options.Endpoint = new Uri("http://poc-collector:4319/v1/metrics");
             options.Protocol = OtlpExportProtocol.HttpProtobuf;
             options.ExportProcessorType = ExportProcessorType.Simple;
         });
});

缩写的收集器配置:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4319

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    namespace: poc 

  logging:
    loglevel: debug

service:
  telemetry:
    logs:
      level: "debug"
  extensions: []
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [logging, prometheus]
英文:

I am running a simple POC for using open telemetry in .net 6 web api. I am trying to use the Otlp exporter in the .net app to send metrics to a collector. Prometheus is scraping the endpoint that the collector exposes just fine.

However, the initial export of metrics takes a long time. It seems to send metrics every 60 seconds once it gets going. I really would prefer to set it to a more reasonable 5-15 seconds or so to make it demo better.

I tried the simple exporter with no luck. I tried to set the batch importer with a very low MaxQueueSize, ScheduledDelayMilliseconds and ExporterTimeoutMilliseconds.

When enabling a ConsoleExporter on the .net application side, the metrics seem to come through the expected default of 10 seconds. So it may be an issue with the collector doing some sort of accumulation before rendering an export on the metrics endpoint? I'm not using a batch processor in the collector, so I am not sure why it would do that.

I also didn't have any slowness issues when I used the PrometheusExporter / scrape endpoint on the API itself. It's only when I try to export metrics with the Otel Exporter.

What is interesting is that I also have the OTLP exporter configured for tracing and I get all my results into the collector / jaeger almost instantly.

I am currently using the latest Open Telemetry stable packages (1.32.0). I am using the following docker image for the collector:
otel/opentelemetry-collector-contrib:0.42.0

Code available here:
https://github.com/MetalHexx/OpenTelemetryDotNetPoc

Abbreviated startup code:

services.AddOpenTelemetryMetrics(builder =>
{
     builder
         .AddMeter(App_Source)
         .SetResourceBuilder(resource!)
         .AddAspNetCoreInstrumentation()
         .AddOtlpExporter(options =>
         {
             options.Endpoint = new Uri("http://poc-collector:4319/v1/metrics");
             options.Protocol = OtlpExportProtocol.HttpProtobuf;
             options.ExportProcessorType = ExportProcessorType.Simple;
         });
});

Abbreviated Collector config:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4319

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    namespace: poc 

  logging:
    loglevel: debug

service:
  telemetry:
    logs:
      level: "debug"
  extensions: []
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [logging, prometheus]

答案1

得分: 2

根据规范,我找到了解决方案,确实默认值为60秒。
解决方案是使用OTEL_METRIC_EXPORT_INTERVAL环境变量。

在.NET SDK中,您可以通过添加metricReaderOptions参数来指定它,然后可以设置导出间隔的毫秒数:

.AddOtlpExporter((exporterOptions, metricReaderOptions) =>
{
     exporterOptions.Endpoint = new Uri("http://poc-collector:4319/v1/metrics"); 
     exporterOptions.Protocol = OtlpExportProtocol.HttpProtobuf;
     exporterOptions.ExportProcessorType = ExportProcessorType.Simple;
metricReaderOptions.PeriodicExportingMetricReaderOptions.ExportIntervalMilliseconds = 5000;

})

Credit goes to utpilla on Github:
https://github.com/open-telemetry/opentelemetry-dotnet/issues/4026#issuecomment-1363411811

英文:

I found the solution. According to the specification, indeed the default is 60 seconds.
The solution is to use the OTEL_METRIC_EXPORT_INTERVAL environment variable.

In the .net SDK you can specify it by adding the metricReaderOptions parameter, then you can set the number of milliseconds for the export interval:

.AddOtlpExporter((exporterOptions, metricReaderOptions) =>
{
     exporterOptions.Endpoint = new Uri("http://poc-collector:4319/v1/metrics"); 
     exporterOptions.Protocol = OtlpExportProtocol.HttpProtobuf;
     exporterOptions.ExportProcessorType = ExportProcessorType.Simple;
metricReaderOptions.PeriodicExportingMetricReaderOptions.ExportIntervalMilliseconds = 5000;

})

Credit goes to utpilla on Github:
https://github.com/open-telemetry/opentelemetry-dotnet/issues/4026#issuecomment-1363411811

huangapple
  • 本文由 发表于 2023年2月24日 10:03:07
  • 转载请务必保留本文链接:https://go.coder-hub.com/75552005.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定