Log export failures when `IncludeScopes` is enabled for ASP.NET Core applications

[stevejgordon]

If Information level logs for System.Net.Http.HttpClient are enabled in an application, the following two logs are produced and captured in the console output:

info: System.Net.Http.HttpClient.Default.ClientHandler[101]
      Received HTTP response headers after 765.3231ms - 200
info: System.Net.Http.HttpClient.Default.LogicalHandler[101]
      End processing HTTP request after 830.0194ms - 200

However, an error is returned once the next batch of logs is exported.

fail: Elastic.OpenTelemetry[0]
      OTEL-SDK: [32868] Export failed for https://my-project-123abc.ingest.us-east-1.aws.elastic.cloud/opentelemetry.proto.collector.logs.v1.LogsService/Export. Message: Export failed due to unexpected status code.. Status(StatusCode="Unavailable", Detail="flush failed (400): {"error":{"type":"x_content_parse_exception","caused_by":{"type":"json_parse_exception"}}}").

Steps:

• Investigate if this occurs with the classic APM endpoint
• Investigate if this occurs without the Elastic logger defaults
• Investigate the server-side error to understand what specifically was in the JSON
• Investigate if this needs fixing in the upstream SDK, EDOT or on the server

[stevejgordon]

Initial investigation results:

This only affects the Managed OTLP endpoint and occurs when IncludeScopes on the OpenTelemetryLoggerOptions is set to true, the default for EDOT .NET.

Next steps:

• Investigate the server-side error to understand what specifically was in the JSON
• Investigate if this needs fixing in the upstream SDK, EDOT or on the server

[stevejgordon]

This can be reproduced even without IncludeScopes by adding an attribute with the same key more than once on a LogRecord. IncludeScopes highlights it because multiple scopes include the Kestrel RequestId`.

I will test with the local collector to see if the same occurs through that ingest path.

[stevejgordon]

The local collector has the same issue. The root cause is that the .NET SDK does not comply with the spec to deduplicate, and the elasticsearchexporter is also not deduping.

References:

• Duplicate Log Attribute Keys open-telemetry/opentelemetry-dotnet#4324
• Error indexing in elasticsearchexporter when duplicate attributes recieved for log records open-telemetry/opentelemetry-collector-contrib#39304

[stevejgordon]

The only immediate solution is to remove the IncludeScopes option in EDOT .NET until one of the upstream components resolves this.

[stevejgordon]

While the core issue of the SDK not being spec compliant with regards to de-duping remains, this is no longer an issue for our scenario since EDOT collector with the Elasticsearch exporter does deduplicate and therefore ensures that spans are not lost.

The next step is to validate this with a local sample application, document the de-duplication behaviour (first attribute wins) and determine if we can now go ahead and re-enable include scopes by default.