1. Common metadata across tracing, metrics, and logging to facilitate contextual correlation, including metadata passed between services as part of REST or RPC APIs; this is a critical element of service observability in the age of distributed, horizontally scaled systems

2. An optional unified data path for tracing, metrics, and logging to facilitate common tooling and signal routing to your observability backend

Adoption of metrics and tracing among developers to date has been relatively small. Further, the number of proprietary vendors and APIs (compared to adoption rate) is relatively large. As such, OpenTelemetry took a greenfield approach to developing new, vendor-agnostic APIs for tracing and metrics. In contrast, most developers have nearly 100% log coverage across their services. Moreover, logging is largely supported by a small number of vendor-agnostic, open-source logging libraries and associated APIs (e.g., Logback and ILogger). As such, OpenTelemetry’s approach to logging meets developers where they already are using hooks into existing, popular logging frameworks. In this way, developers can add OpenTelemetry as a log signal output without otherwise altering their code and investment in logging as an observability signal.

Notably, logging is the least mature of OTel supported observability signals. Depending on your service’s language, and your appetite for adventure, there exist several options for exporting logs from your services and applications and marrying them together in your observability backend.

The intent of this article is to explore the current state of the art of OpenTelemetry logging and to provide guidance on the available approaches with the following tenants in mind:

• Correlation of service logs with OTel-generated tracing where applicable
• Proper capture of exceptions
• Common context across tracing, metrics, and logging
• Support for slf4j key-value pairs (“structured logging”)
• Automatic attachment of metadata carried between services via OTel baggage
• Use of an Elastic^® Observability backend
• Consistent data fidelity in Elastic regardless of the approach taken

OpenTelemetry logging models

Three models currently exist for getting your application or service logs to Elastic with correlation to OTel tracing and baggage:

1. Output logs from your service (alongside traces and metrics) using an embedded OpenTelemetry Instrumentation library to Elastic via the OTLP protocol

2. Write logs from your service to a file scraped by the OpenTelemetry Collector, which then forwards to Elastic via the OTLP protocol

3. Write logs from your service to a file scraped by Elastic Agent (or Filebeat), which then forwards to Elastic via an Elastic-defined protocol

Note that (1), in contrast to (2) and (3), does not involve writing service logs to a file prior to ingestion into Elastic.

Logging vs. span events

It is worth noting that most APM systems, including OpenTelemetry, include provisions for span events. Like log statements, span events contain arbitrary, textual data. Additionally, span events automatically carry any custom attributes (e.g., a “user ID”) applied to the parent span, which can help with correlation and context. In this regard, it may be advantageous to translate some existing log statements (inside spans) to span events. As the name implies, of course, span events can only be emitted from within a span and thus are not intended to be a general purpose replacement for logging.

Unlike logging, span events do not pass through existing logging frameworks and therefore cannot (practically) be written to a log file. Further, span events are technically emitted as part of trace data and follow the same data path and signal routing as other trace data.

Polyfill appender

Some of the demos make use of a custom Logback “Polyfill appender” (inspired by OTel’s Logback MDC), which provides support for attaching slf4j key-value pairs to log messages for models (2) and (3).

Elastic Common Schema

For log messages to exhibit full fidelity within Elastic, they eventually need to be formatted in accordance with the Elastic Common Schema (ECS). In models (1) and (2), log messages remain formatted in OTel log semantics until ingested by the Elastic APM Server. The Elastic APM Server then translates OTel log semantics to ECS. In model (3), ECS is applied at the source.

Notably, OpenTelemetry recently adopted the Elastic Common Schema as its standard for semantic conventions going forward! As such, it is anticipated that current OTel log semantics will be updated to align with ECS.

Getting started

The included demos center around a “POJO” (no assumed framework) Java project. Java is arguably the most mature of OTel-supported languages, particularly with respect to logging options. Notably, this singular Java project was designed to support the three models of logging discussed here. In practice, you would only implement one of these models (and corresponding project dependencies).

The demos assume you have a working Docker environment and an Elastic Cloud instance.

1. git clone https://github.com/ty-elastic/otel-logging

2. Create an .env file at the root of otel-logging with the following (appropriately filled-in) environment variables:

# the service name
OTEL_SERVICE_NAME=app4

# Filebeat vars
ELASTIC_CLOUD_ID=(see https://www.elastic.co/guide/en/beats/metricbeat/current/configure-cloud-id.html)
ELASTIC_CLOUD_AUTH=(see https://www.elastic.co/guide/en/beats/metricbeat/current/configure-cloud-id.html)

# apm vars
ELASTIC_APM_SERVER_ENDPOINT=(address of your Elastic Cloud APM server... i.e., https://xyz123.apm.us-central1.gcp.cloud.es.io:443)
ELASTIC_APM_SERVER_SECRET=(see https://www.elastic.co/guide/en/apm/guide/current/secret-token.html)

3. Start up the demo with the desired model:

• If you want to demo logging via OTel APM Agent, run MODE=apm docker-compose up
• If you want to demo logging via OTel filelogreceiver, run MODE=filelogreceiver docker-compose up
• If you want to demo logging via Elastic filebeat, run MODE=filebeat docker-compose up

4. Validate incoming span and correlated log data in your Elastic Cloud instance

Model 1: Logging via OpenTelemetry instrumentation

This model aligns with the long-term goals of OpenTelemetry: integrated tracing, metrics, and logging (with common attributes) from your services via the OpenTelemetry Instrumentation libraries, without dependency on log files and scrappers.

In this model, your service generates log statements as it always has, using popular logging libraries (e.g., Logback for Java). OTel provides a “Southbound hook” to Logback via the OTel Logback Appender, which injects ServiceName, SpanID, TraceID, slf4j key-value pairs, and OTel baggage into log records and passes the composed records to the co-resident OpenTelemetry Instrumentation library. We further employ a custom LogRecordProcessor to add baggage to the log record as attributes.

The OTel instrumentation library then formats the log statements per the OTel logging spec and ships them via OTLP to either an OTel Collector for further routing and enrichment or directly to Elastic.

Notably, as language support improves, this model can and will be supported by runtime agent binding with auto-instrumentation where available (e.g., no code changes required for runtime languages).

One distinguishing advantage of this model, beyond the simplicity it affords, is the ability to more easily tie together attributes and tracing metadata directly with log statements. This inherently makes logging more useful in the context of other OTel-supported observability signals.

Architecture

Although not explicitly pictured, an OpenTelemetry Collector can be inserted in between the service and Elastic to facilitate additional enrichment and/or signal routing or duplication across observability backends.

Pros

• Simplified signal architecture and fewer “moving parts” (no files, disk utilization, or file rotation concerns)
• Aligns with long-term OTel vision
• Log statements can be (easily) decorated with OTel metadata
• No polyfill adapter required to support structured logging with slf4j
• No additional collectors/agents required
• Conversion to ECS happens within Elastic keeping log data vendor-agnostic until ingestion
• Common wireline protocol (OTLP) across tracing, metrics, and logs

Cons

• Not available (yet) in many OTel-supported languages
• No intermediate log file for ad-hoc, on-node debugging
• Immature (alpha/experimental) Unknown “glare” conditions, which could result in loss of log data if service exits prematurely or if the backend is unable to accept log data for an extended period of time

Demo

MODE=apm docker-compose up

Model 2: Logging via the OpenTelemetry Collector

Given the cons of Model 1, it may be advantageous to consider a model that continues to leverage an actual log file intermediary between your services and your observability backend. Such a model is possible using an OpenTelemetry Collector collocated with your services (e.g., on the same host), running the filelogreceiver to scrape service log files.

In this model, your service generates log statements as it always has, using popular logging libraries (e.g., Logback for Java). OTel provides a MDC Appender for Logback (Logback MDC), which adds SpanID, TraceID, and Baggage to the Logback MDC context.

Notably, no log record structure is assumed by the OTel filelogreceiver. In the example provided, we employ the logstash-logback-encoder to JSON-encode log messages. The logstash-logback-encoder will read the OTel SpanID, TraceID, and Baggage off the MDC context and encode it into the JSON structure. Notably, logstash-logback-encoder doesn’t explicitly support slf4j key-value pairs. It does, however, support Logback structured arguments, and thus I use the Polyfill Appender to convert slf4j key-value pairs to Logback structured arguments.

From there, we write the log lines to a log file. If you are using Kubernetes or other container orchestration in your environment, you would more typically write to stdout (console) and let the orchestration log driver write to and manage log files.

We then configure the OTel Collector to scrape this log file (using the filelogreceiver). Because no assumptions are made about the format of the log lines, you need to explicitly map fields from your log schema to the OTel log schema.

From there, the OTel Collector batches and ships the formatted log lines via OTLP to Elastic.

Architecture

Pros

• Easy to debug (you can manually read the intermediate log file)
• Inherent file-based FIFO buffer
• Less susceptible to “glare” conditions when service prematurely exits
• Conversion to ECS happens within Elastic keeping log data vendor-agnostic until ingestion
• Common wireline protocol (OTLP) across tracing, metrics, and logs

Cons

• All the headaches of file-based logging (rotation, disk overflow)
• Beta quality and not yet proven in the field
• No support for slf4j key-value pairs

Demo

MODE=filelogreceiver docker-compose up

Model 3: Logging via Elastic Agent (or Filebeat)

Although the second model described affords some resilience as a function of the backing file, the OTel Collector filelogreceiver module is still decidedly “beta” in quality. Because of the importance of logs as a debugging tool, today I generally recommend that customers continue to import logs into Elastic using the field-proven Elastic Agent or Filebeat scrappers. Elastic Agent and Filebeat have many years of field maturity under their collective belt. Further, it is often advantageous to deploy Elastic Agent anyway to capture the multitude of signals outside the purview of OpenTelemetry (e.g., deep Kubernetes and host metrics, security, etc.).

In this model, your service generates log statements as it always has, using popular logging libraries (e.g., Logback for Java). As with model 2, we employ OTel’s Logback MDC to add SpanID, TraceID, and Baggage to the Logback MDC context.

From there, we employ the Elastic ECS Encoder to encode log statements compliant to the Elastic Common Schema. The Elastic ECS Encoder will read the OTel SpanID, TraceID, and Baggage off the MDC context and encode it into the JSON structure. Similar to model 2, the Elastic ECS Encoder doesn’t support sl4f key-vair arguments. Curiously, the Elastic ECS encoder also doesn’t appear to support Logback structured arguments. Thus, within the Polyfill Appender, I add slf4j key-value pairs as MDC context. This is less than ideal, however, since MDC forces all values to be strings.

From there, we write the log lines to a log file. If you are using Kubernetes or other container orchestration in your environment, you would more typically write to stdout (console) and let the orchestration log driver write to and manage log files.We then configure Elastic Agent or Filebeat to scrape the log file. Notably, the Elastic ECS Encoder does not currently translate incoming OTel SpanID and TraceID variables on the MDC. Thus, we need to perform manual translation of these variables in the Filebeat (or Elastic Agent) configuration to map them to their ECS equivalent.

Architecture

Pros

• Robust and field-proven
• Easy to debug (you can manually read the intermediate log file)
• Inherent file-based FIFO buffer
• Less susceptible to “glare” conditions when service prematurely exits
• Native ECS format for easy manipulation in Elastic
• Fleet-managed via Elastic Agent

Cons

• All the headaches of file-based logging (rotation, disk overflow)
• No support for slf4j key-value pairs or Logback structured arguments
• Requires translation of OTel SpanID and TraceID in Filebeat config
• Disparate data paths for logs versus tracing and metrics
• Vendor-specific logging format

Demo

MODE=filebeat docker-compose up

Recommendations

For most customers, I currently recommend Model 3 — namely, write to logs in ECS format (with OTel SpanID, TraceID, and Baggage metadata) and collect them with an Elastic Agent installed on the node hosting the application or service. Elastic Agent (or Filebeat) today provides the most field-proven and robust means of capturing log files from applications and services with OpenTelemetry context.

Further, you can leverage this same Elastic Agent instance (ideally running in your Kubernetes daemonset) to collect rich and robust metrics and logs from Kubernetes and many other supported services via Elastic Integrations. Finally, Elastic Agent facilitates remote management via Fleet, avoiding bespoke configuration files.

Alternatively, for customers who either wish to keep their nodes vendor-neutral or use a consolidated signal routing system, I recommend Model 2, wherein an OpenTelemetry collector is used to scrape service log files. While workable and practiced by some early adopters in the field today, this model inherently carries some risk given the current beta nature of the OpenTelemetry filelogreceiver.

I generally do not recommend Model 1 given its limited language support, experimental/alpha status (the API could change), and current potential for data loss. That said, in time, with more language support and more thought to resilient designs, it has clear advantages both with regard to simplicity and richness of metadata.

Extracting more value from your logs

In contrast to tracing and metrics, most organizations have nearly 100% log coverage over their applications and services. This is an ideal beachhead upon which to build an application observability system. On the other hand, logs are notoriously noisy and unstructured; this is only amplified with the scale enabled by the hyperscalers and Kubernetes. Collecting log lines reliably is the easy part; making them useful at today’s scale is hard.

Given that logs are arguably the most challenging observability signal from which to extract value at scale, one should ideally give thoughtful consideration to a vendor’s support for logging in the context of other observability signals. Can they handle surges in log rates because of unexpected scale or an error or test scenario? Do they have the machine learning tool set to automatically recognize patterns in log lines, sort them into categories, and identify true anomalies? Can they provide cost-effective online searchability of logs over months or years without manual rehydration? Do they provide the tools to extract and analyze business KPIs buried in logs?

As an ardent and early supporter of OpenTelemetry, Elastic, of course, natively ingests OTel traces, metrics, and logs. And just like all logs coming into our system, logs coming from OTel-equipped sources avail themselves of our mature tooling and next-gen AI Ops technologies to enable you to extract their full value.Interested? Reach out to our pre-sales team to get started building with Elastic!

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

OpenTelemetry is quickly becoming the standard foundation for modern observability. Organizations want its open ecosystem, unified model, and vendor-neutral instrumentation—but moving a mature production environment to OTel is rarely straightforward.

Most teams already rely on battle-tested pipelines for logs, metrics, and security signals. They have dashboards tuned over years, operational practices built around existing data flows, and mission-critical systems where disruption simply isn’t an option.

This means the question isn’t "Why OpenTelemetry?" It’s "How do we get there without breaking what already works?"

Elastic Observability introduces a way to ingest telemetry without disrupting existing data and dashboards with Hybrid ingestion. Released in Elastic 9.2, its a low-friction way to adopt OTel receivers alongside existing native Elastic integrations-managed centrally through Fleet.

This hybrid approach offers one of the most pragmatic and operationally safe routes to OTel adoption available today.

The Challenge: Adopting OTel Without Disrupting the Present

For many organizations, the path to OTel adoption is complicated by realities such as:

• Established log pipelines powering critical alerting
• Legacy infrastructure that isn’t easily re-instrumented
• Existing dashboards and visualizations built on Elastic-native datasets
• Teams with different levels of OTel experience
• Risk constraints that make large changes difficult to roll out

Standardizing on OTel is the right long-term direction, but replacing everything at once is neither realistic nor desirable.

Teams need a way to bring OTel into their environment incrementally, while preserving continuity, reliability, and central governance.

Elastic Agent 9.2+: Hybrid Ingestion as a Bridge to the Future

Elastic Agent now supports two fully supported ingestion paths, both running inside the same unified agent:

1. Elastic-native integrations

Perfect for logs and host-level telemetry, with mature dashboards, alerts, and ECS mappings.

2. OpenTelemetry input integrations (OTel receivers)

Powered by upstream OTel Collector components, managed directly from Fleet.

And crucially:

You can use both, simultaneously, on the same agent.

This hybrid ingestion model allows teams to:

• Continue collecting logs using native Elastic integrations
• Begin collecting metrics or traces via OTel receivers
• Maintain full control through Fleet
• Introduce OTel exactly where and when it makes sense
• Avoid running parallel agents or duplicate pipelines

It’s a way to evolve—not replace—your observability strategy.

A Practical Example: Adding OTel Inputs While Keeping Your Existing Pipelines

Imagine a system where NGINX logs are already handled via Elastic-native integrations. These pipelines drive dashboards, audits, and critical alerts. Interrupting them isn’t an option.

At the same time, your platform team wants to standardize metrics and service telemetry using OpenTelemetry.

With Elastic Agent hybrid ingestion, both goals align:

1. Keep your existing log integration in Fleet
2. Add an OTel input integration (e.g., OTel nginxreceiver)
3. Fleet deploys both inside the same Elastic Agent
4. Deployment is done at scale across your infrastructure from a single management console
5. Logs and OTel metrics flow into Elasticsearch side-by-side

No re-instrumentation. No duplicate agents. No loss of historical visibility. No new tooling for operations. No external deployment tool.

Whether the component is a web server, reverse proxy, database, JVM runtime, or custom service already instrumented in OTel, the workflow is the same.

Why This Hybrid Approach Matters Strategically

Hybrid ingestion is not simply a technical capability—it’s an organizational enabler for OpenTelemetry transformation.

Incremental migration without downtime

Teams can begin adopting OTel at the exact pace they’re comfortable with. Existing collection signals remain stable. OTel metrics or logs are added progressively.

Fleet remains your single control plane

Fleet continues to manage:

• agent lifecycle
• policy management
• version upgrades
• diagnostics and monitoring

Even as OTel becomes part of your ingestion strategy.

Consistent semantics across teams

Adopting OTel receivers through EDOT helps harmonize telemetry models across microservices, infrastructure, and applications.

OTel becomes the shared language—Elastic becomes the scalable backend.

Future-proof flexibility

When the day comes that a team needs advanced OTel features, custom pipelines, custom processors, or additional exporters, they can build their own EDOT custom collector flavor and use it in their elastic-agent in hybrid mode.

This allows deep customization without abandoning the Elastic Agent runtime.

No vendor lock-in—full ecosystem alignment

Hybrid ingestion leverages upstream OpenTelemetry components directly. This reinforces the open, vendor-neutral ecosystem organizations prefer when standardizing observability across teams while being supported by Elastic.

What About Standalone Mode? (Advanced Use Cases)

While Fleet-managed hybrid ingestion will meet the needs of most users, Elastic Agent in hybrid mode also support standalone deployment with the same functions as the managed version.

• native integrations support
• full control over Otel receivers, processors, and exporters
• Elasticsearch output as the backend

This is particularly useful for platform teams testing advanced OTel deployments or building custom telemetry strategies.

But it remains optional—the managed experience is still the default path.

Conclusion: A Modern, Flexible Path Toward OpenTelemetry

Migrating to OpenTelemetry is a journey, not a switch. With hybrid ingestion, Elastic provides a realistic, scalable, and low-risk pathway for organizations that want to adopt OTel gradually while maintaining operational continuity.

Elastic Agent 9.2+ enables teams to:

• retain reliable log integrations
• introduce OTel inputs seamlessly
• manage everything from Fleet
• reduce complexity and operational overhead
• expand into OTel at the right pace
• stay aligned with open standards and best practices

It brings the best of both worlds—Elastic-native richness and OTel-standard flexibility—into a single agent and a unified operational model.

Hybrid isn’t a workaround. It’s the strategic bridge between where your observability platform is today and where it needs to go next.

Technical Walkthrough: Deploying Hybrid Elastic Agent + EDOT in Fleet

Before we close, let’s look at what this actually looks like in practice. Conceptual advantages are important, but many teams want to see how hybrid ingestion works when deployed through Fleet.

The example below walks through a simple, production-ready setup using Elastic Agent 9.2, combining a native integration and an OTel input integration inside a single agent, the same approach you can apply to any service across your environment.

Here is a step-by-step guide showing how to deploy Elastic Agent 9.2 in Fleet-managed hybrid mode, using the OTel nginxreceiver as one concrete example. This applies to any service with an OTel receiver (Redis, HAProxy, Kafka, JVM, etc.).

Requirements

• Elastic Stack 9.2+
• Elastic Agent 9.2+
• Fleet configured in Kibana
• A host running your workload (NGINX in this example)
• NGINX stub_status endpoint or any equivalent OTel metrics endpoint
• API key with ingest privileges

1. Create or Select an Agent Policy

1. In Kibana → Management → Fleet → Agent policies
2. Create a new policy: nginx-o11y
3. Enable system monitoring (recommended)
4. Save

2. Enroll Elastic Agent into the Policy

From the policy page:

1. Click Add agent
2. Choose your OS
3. Copy the installation command
4. Run:

sudo elastic-agent install \
  --url=<FLEET_URL> \
  --enrollment-token=<ENROLLMENT_TOKEN>

You should soon see the agent appear as Healthy in Fleet.

3. Add the Native Integration (Logs)

1. In Fleet, go to Integrations.
2. Search for NGINX.
3. Click Add NGINX.
4. Select your nginx-o11y policy.
5. Only enable log collection (access + error logs).
6. Save and deploy.

4. Validate Log Collection

1. In Kibana, go to Analytics → Discover and search for:

data_stream.dataset : "nginx.access" or "nginx.error"

2. Or open the built-in dashboard:

Analytics → Dashboards → [Logs Nginx] Access and error logs

5. Collecting NGINX Metrics via the OTel NGINX Receiver

Elastic Agent 9.2+ allows Fleet to deploy OTel input integrations.
This scenario uses the OpenTelemetry nginxreceiver through a Fleet-managed integration.

5.1. Install the NGINX OpenTelemetry Integration Content

1. In Kibana, go to Management → Fleet → Integrations.
2. Search for NGINX OpenTelemetry Assets.
3. Click Add Integration.

5.2. Install the NGINX OpenTelemetry Input Integration

1. In Kibana, go to Management → Fleet → Integrations.
2. Search for NGINX OpenTelemetry Input Package.
3. Click Add Integration.
4. Assign it to your agent nginx-o11y policy.

Provide the endpoint for the NGINX status page:

• Endpoint: http://localhost/status
• Collection interval: 10s

Click Add integration.

6. Validate OTel Metrics

1. Go to Analytics → Dashboards.
2. Open: [Metrics Nginx OTEL Overview] Dashboard

You should see metrics such as active connections, writes, reads, waiting, and request counts.

7. Closing thoughts

This example highlights how straightforward hybrid ingestion becomes with Elastic Agent 9.2. By combining native integrations and OTel receivers within a single, centrally managed policy, you gain the flexibility to adopt OpenTelemetry where it adds the most value without disrupting existing pipelines or introducing operational overhead.

Whether you extend this pattern to additional services, experiment with other OTel receivers, or scale it across your fleet, the deployment model remains consistent, repeatable, and production-ready.

For more information and other innovations Elastic Observability has made check out:

• Discover how Elastic is evolving data ingestion with OpenTelemetry

• Learn how OpAMP enables centralized configuration of OpenTelemetry SDKs

• Explore how Streams reshape AI-driven log investigation workflows

What is APM?

Application performance monitoring lets you see where your applications spend their time, what they are doing, what other applications or services they are calling, and what errors or exceptions they are encountering.

In addition, APM also lets you see history and trends for key performance indicators, such as latency and throughput, as well as transaction and dependency information:

Whether you're setting up alerts for SLA breaches, trying to gauge the impact of your latest release, or deciding where to make the next improvement, APM can help with your root-cause analysis to help improve your users' experience and drive your mean time to resolution (MTTR) toward zero.

Logical architecture

Elastic APM relies on the APM Integration inside Elastic Agent, which forwards application trace and metric data from applications instrumented with APM agents to an Elastic Observability cluster. Elastic APM supports multiple agent flavors:

• Native Elastic APM Agents, available for multiple languages, including Java, .NET, Go, Ruby, Python, Node.js, PHP, and client-side JavaScript
• Code instrumented with OpenTelemetry
• Code instrumented with OpenTracing
• Code instrumented with Jaeger

In this blog, we'll provide a quick example of how to instrument code with the native Elastic APM Python agent, but the overall steps are similar for other languages.

Please note that there is a strong distinction between the Elastic APM Agent and the Elastic Agent. These are very different components, as you can see in the diagram above, so it's important not to confuse them.

Install the Elastic Agent

The first step is to install the Elastic Agent. You either need Fleet installed first, or you can install the Elastic Agent standalone. Install the Elastic Agent somewhere by following this guide. This will give you an APM Integration endpoint you can hit. Note that this step is not necessary in Elastic Cloud, as we host the APM Integration for you. Check Elastic Agent is up by running:

curl <ELASTIC_AGENT_HOSTNAME>:8200

Instrumenting sample code with an Elastic APM agent

The instructions for the various language agents differ based on the programming language, but at a high level they have a similar flow. First, you add the dependency for the agent in the language's native spec, then you configure the agent to let it know how to find the APM Integration.

You can try out any flavor you'd like, but I am going to walk through the Python instructions using this Python example that I created.

Get the sample code (or use your own)

To get started, I clone the GitHub repository then change to the directory:

git clone https://github.com/davidgeorgehope/PythonElasticAPMExample
cd PythonElasticAPMExample

How to add the dependency

Adding the Elastic APM Dependency is simple — check the app.py file from the github repo and you will notice the following lines of code.

import elasticapm
from elasticapm import Client

app = Flask(__name__)
app.config["ELASTIC_APM"] = {    "SERVICE_NAME": os.environ.get("APM_SERVICE_NAME", "flask-app"),    "SECRET_TOKEN": os.environ.get("APM_SECRET_TOKEN", ""),    "SERVER_URL": os.environ.get("APM_SERVER_URL", "http://localhost:8200"),}
elasticapm.instrumentation.control.instrument()
client = Client(app.config["ELASTIC_APM"])

The Python library for Flask is capable of auto detecting transactions, but you can also start transactions in code as per the following, as we have done in this example:

@app.route("/")
def hello():
    client.begin_transaction('demo-transaction')
    client.end_transaction('demo-transaction', 'success')

Configure the agent

The agents need to send application trace data to the APM Integration, and to do this it has to be reachable. I configured the Elastic Agent to listen on my local host's IP, so anything in my subnet can send data to it. As you can see from the code below, we use docker-compose.yml to pass in the config via environment variables. Please edit these variables for your own Elastic installation.

# docker-compose.yml
version: "3.9"
services:
  flask_app:
    build: .
    ports:
      - "5001:5001"
    environment:
      - PORT=5001
      - APM_SERVICE_NAME=flask-app
      - APM_SECRET_TOKEN=your_secret_token
      - APM_SERVER_URL=http://host.docker.internal:8200

Some commentary on the above:

• service_name: If you leave this out it will just default to the application's name, but you can override that here.
• secret_token: Secret tokens allow you to authorize requests to the APM Server, but they require that the APM Server is set up with SSL/TLS and that a secret token has been set up. We're not using HTTPS between the agents and the APM Server, so we'll comment this one out.
• server_url: This is how the agent can reach the APM Integration inside Elastic Agent. Replace this with the name or IP of your host running Elastic Agent.

Now that the Elastic APM side of the configuration is done, we simply follow the steps from the README to start up.

docker-compose up --build -d

The build step will take several minutes.

You can navigate to the running sample application by visiting http://localhost:5001. There's not a lot to the sample, but it does generate some APM data. To generate a bit of a load, you can reload them a few times or run a quick little script:

#!/bin/bash
# load_test.sh
url="http://localhost:5001"
for i in {1..1000}
do
  curl -s -o /dev/null $url
  sleep 1
done

This will just reload the pages every second.

Back in Kibana, navigate back to the APM app (hamburger icon, then select APM ) and you should see our new flask-app service (I let mine run so it shows a bit more history):

The Service Overview page provides an at-a-glance summary of the health of a service in one place. If you're a developer or an SRE, this is the page that will help you answer questions like:

• How did a new deployment impact performance?
• What are the top impacted transactions?
• How does performance correlate with underlying infrastructure?

This view provides a list of all of the applications that have sent application trace data to Elastic APM in the specified period of time (in this case, the last 15 minutes). There are also sparklines showing mini graphs of latency, throughput, and error rate. Clicking on flask-app takes us to the service overview page, which shows the various transactions within the service (recall that my script is hitting the / endpoint, as seen in the Transactions section). We get bigger graphs for Latency , Throughput , Errors , and Error Rates.

When you're instrumenting real applications, under real load, you'll see a lot more connectivity (and errors!)

Clicking on a transaction in the transaction view, in this case, our sample app's demo-transaction transaction, we can see exactly what operations were called:

This includes detailed information about calls to external services, such as database queries:

What's next?

Now that you've got your Elastic Observability cluster up and running and collecting out-of-the-box application trace data, explore the public APIs for the languages that your applications are using, which allow you to take your APM data to the next level. The APIs allow you to add custom metadata, define business transactions, create custom spans, and more. You can find the public API specs for the various APM agents (such as Java, Ruby, Python, and more) on the APM agent documentation pages.

If you'd like to learn more about Elastic APM, check out our webinar on Elastic APM in the shift to cloud native to see other ways that Elastic APM can help you in your ecosystem.

If you decide that you'd rather have us host your observability cluster, you can sign up for a free trial of the Elasticsearch Service on Elastic Cloud and change your agents to point to your new cluster.

Originally published May 5, 2021; updated April 6, 2023.

“Anything that emits data can be instrumented and observed.”

That’s the mindset that started this little experiment.

The Curiosity Behind It

Over the years, I’ve worked closely with customers to design IT solutions that are scalable, secure, and cost-effective. I’ve partnered with them on their cloud migration and digital transformation journeys, enabling full-stack Observability across their cloud and on-premise systems.

One day, I found myself looking at the appliances in my own home — the dishwasher, washer, dryer, and refrigerator — and realized that they, too, were generating valuable data. What if I could observe them? What if the same principles that power enterprise telemetry could help me understand my home appliances — their patterns, behavior, and efficiency?

That curiosity became the seed for this experiment: IoT Observability at home, powered by OpenTelemetry - EDOT, and Agent Builder.

Building the IoT Observability Foundation

The idea was simple:

1. Treat every device as a data source.
2. Use OpenTelemetry to capture signals
3. Use EDOT (Elastic Distribution of OpenTelemetry) as a unified collector and exporter.
4. Send all data to an Elastic Serverless Observability cluster.
5. Layer Agent Builder on top, to talk to the data using natural language.

So now, my dishwasher, washer, dryer and refrigerator — all part of an Elastic-powered, home-scale telemetry pipeline.

Turning Signals into Stories

Technical overview: What does this system do?

I set up a system that connects my LG ThinQ smart appliances — washer, dryer, dishwasher, and refrigerator — to Home Assistant, turning everyday household devices into observable systems by sending metrics, logs, and traces to Elastic Cloud Serverless.

Key Capabilities:

✅ Natural language queries (Agent Builder)
✅ Real-time appliance state monitoring
✅ Anomaly detection
✅ Full stack observability stack

Architecture overview

The Aha Moment

What is Agent Builder?

Agent Builder provides an out-of-the-box conversational agent to allow you to immediately start chatting with any data in Elasticsearch (or from external sources through integrations) with a full experience built in Kibana and accessible via API. Developers can also customize their tools to search specific indexes or use ES|QL for business logic, relevance tuning, or personalization.

It has the ability to transform natural language into intuitive, piped, multi-step ES|QL, giving the agent the power to do analytical and hybrid semantic search. Finally, developers can compose custom Agents based on a set of user defined instructions and configurable set of available tools, and these Agents can be interacted with via chat in Kibana or via APIs, MCP and A2A.

Agent Builder creates a transformative experience, turning raw telemetry into an interactive dialogue. So instead of building complex queries manually, I can simply ask:

"Can you show me a report for all my appliances?" …and voilà, I get the insights right in Kibana.

Conclusion

This experiment reminded me that Observability isn’t limited to enterprise systems. Anything that emits data, whether it’s a Kubernetes pod or a coffee maker, has insights that can be uncovered. It could be any IoT devices for that matter, your data center thermostat, your office building badge scanners — all emit telemetry that can be valuable to ensuring safe and efficient operations. The same principles that help organizations gain visibility into production workloads can also bring insights, efficiency, and a sense of connection to the systems around us every day.

By combining OpenTelemetry (EDOT), Elastic Cloud Serverless, and Agent Builder, I realized how simple it can be to go from raw telemetry to conversation — turning metrics into meaning and data into dialogue.

This experiment showed me something simple yet profound: Observability is no longer just about dashboards and alerts; it’s about conversations. When data becomes conversational, insights become accessible to everyone — not just developers or SREs, but anyone curious enough to ask “why?”

Anything that emits data can be observed.

Now, with Agent Builder:

Anything that emits data can also answer back.

With AI, and agents, these gates are slowly becoming more sophisticated. More and more these gates are using a Model Context Protocol (MCP) server for this check. This is a newer, more cutting-edge "agentic" approach. It allows your CI/CD pipeline to act as an intelligent agent that "asks" your cluster for its health status before making a change.

A standard Kubernetes deployment workflow generally follows these high-level steps:

1. Verification Gate: Ensuring all automated testing has passed.

2. Artifact Creation: Building the Docker container.

3. Environment Gate: Verifying that the production Kubernetes environment, supporting infrastructure, and existing applications are healthy.

Kubernetes Deployment: Triggering the final release. Modern workflows often use GitOps tools like ArgoCD or Flux, where a simple image tag update in Docker Hub automatically synchronizes the cluster.

Kubernetes health checks can range from simple to complex depending on your Service Level Objectives (SLOs) and operational maturity. Typically, the primary goal is to ensure the cluster is healthy and not nearing a resource bottleneck. Common "red flag" metrics used in these gates include:

I will show you how you can use a CI/CD pipeline that integrates Observability AI Agents with GitHub Actions via an Model Context Protocol (MCP) server, creating automated pre-deployment health checks for Kubernetes clusters.

By introducing an observability checkpoint before deployment, we transform the pipeline into an intelligent system that:

• Queries real-time metrics from Kubernetes clusters

• Analyzes capacity using custom ESQL queries

• Makes autonomous decisions about deployment readiness

• Prevents failures proactively rather than reacting to them

• Provides actionable feedback to engineering teams

Here is the “architecture” of what is being deployed and how it works in this blog.

As you can see the flow uses Elastic Observability, which is storing and analyzing Kubernetes OpenTelemetry metrics from the opentelemetry-kube-stack-cluster-stats-collector (deployed via OpenTelemetry Operator).

Github Actions calls the Observability Kubernetes Agent, via the Elastic MCP server, which has tools that help check for some of the “red flag” issues identified in the table above.

Based on the results, Github Actions will either stop the process or continue to deploy the artifact via a trigger for ArgoCD.

The Observability Kubernetes Agent was built using Elastic’s Agent Builder capability, as well as some of the tools it uses. These are then exposed via the MCP server.

Hence the overall set of components used here include:

1. GitHub Actions: Orchestrates the build and deployment workflow

2. Elastic MCP Server: Serverless endpoint that exposes AI agents

3. Observability Kubernetes Agent: Custom agent with specialized ESQL tools

4. Kubernetes Cluster: Target deployment environment with metrics collection

5. ES|QL Query Tools: Precision queries for node and pod resource analysis

What Happens When a Kubernetes Health Check Fails in GitHub Actions?

How the Pipeline Blocks a Deployment Automatically

When the cluster exceeds capacity thresholds, the workflow automatically blocks deployment. In this scenario I didn’t load the cluster, but used a simple check of whether more than 25% of resources were being used to purposely stop the deployment.

The workflow shows:

• Build Docker Image (28s)

• Push to Docker Hub (5s)

• K8s Health via Elastic O11y K8s Agent (16s) - FAILED

• Deploy to otel-test Cluster - BLOCKED

Annotation: "Cluster has resource issues - blocking deployment"

What Does the AI Agent's Health Check Response Look Like?

The agent provides detailed analysis:

Step 1: Finding Kubernetes analysis agent...
Found agent: Observability Kubernetes Agent (kubernetes_analysis_agent)

Step 2: Querying cluster health...
Prompt: tell me if my cluster otel-test is using more than 25% memory or CPU on any of its nodes

Agent Response:
================================================================
Yes, your cluster "otel-test" has nodes and pods using more than 25% of resources.

++Node exceeding 25%:++
- ip-192-168-165-175.us-west-2.compute.internal
  - Memory: 36.44%
  - CPU: 7.99% (below threshold)

++All other nodes are below the 25% threshold++ for both CPU and memory.

While the query for pods doesn't show percentage values directly, the data indicates
normal resource usage patterns for the pods in your cluster, with none appearing to
consume excessive resources relative to their allocations.
================================================================

Cluster has resource issues - blocking deployment
Error: Process completed with exit code 1.

As you can see, a prompt was sent to the Observability Kubernetes Agent via MCP vs having to build some logic or call another script etc.

This single check prevented:

• A deployment that would have failed

• Wasted CI/CD minutes

• Potential service degradation

• Manual SRE intervention

What it provided:

• Provided actionable intelligence for capacity planning

How to Build a Kubernetes Health Check Agent in Elastic

Building the agent isn’t hard, Elastic’s AgentBuilder’s UI makes it easy to create it and have it running in minutes. 

How to Configure the Observability Kubernetes Agent

Other than naming the agent, you need to provide it with some instructions.

Custom Instructions:

# Agent Instructions

## Primary Role
You are a Kubernetes monitoring assistant that helps users analyze cluster performance
and resource utilization. Your primary goal is to provide clear, accurate information
about Kubernetes clusters using available data sources.

## Tool Selection Guidelines
1. When users ask about Kubernetes metrics, node performance, or cluster health:
   - Use ESQL tools for detailed analysis
   - Query metrics from kubeletstatsreceiver.otel-default

2. For alert-related queries:
   - Use the alerts tool to check active alerts

3. Always provide context about:
   - Time ranges queried
   - Cluster names
   - Resource thresholds

How to Write ES|QL Queries for Kubernetes Node and Pod Metrics

I created several tools that checked Node CPU and memory, pod CPU and memory, and OOM from pods. Additionally, the Observability Kubernetes Agent utilized a large portion of the OOTB tools like observability_alerts as part of its abilities. 

Here is an example of the node CPU and memory tool, which uses a simple ES|QL query against OpenTelemetry metrics to check the CPU and memory utilization in the cluster.

ES|QL Query:

FROM metrics-kubeletstatsreceiver.otel-default
| WHERE resource.attributes.k8s.cluster.name == ?cluster_name
  AND @timestamp > NOW() - 3 hours
| STATS
    avg_cpu_usage = AVG(metrics.k8s.node.cpu.usage),
    avg_memory_usage = AVG(metrics.k8s.node.memory.usage),
    avg_memory_available = AVG(metrics.k8s.node.memory.available),
    avg_memory_working_set = AVG(metrics.k8s.node.memory.working_set)
  BY resource.attributes.k8s.node.name
| EVAL
    cpu_usage_pct = avg_cpu_usage * 100,
    memory_usage_pct = (avg_memory_working_set / (avg_memory_working_set + avg_memory_available)) * 100
| SORT cpu_usage_pct DESC, memory_usage_pct DESC
| KEEP resource.attributes.k8s.node.name, cpu_usage_pct, memory_usage_pct
| LIMIT 100

Parameters:

• cluster_name (string): Name of the K8s cluster to analyze

How to Expose the Agent via the Elastic MCP Server

Once configured, the agent is automatically available via Elastic's MCP server running in your Observability project. The MCP server provides a standardized interface that any MCP-compatible client can query.

MCP Endpoint: https://your-elastic-project.elastic.cloud/mcp

Authentication: Uses Elastic API keys for secure access

Why Agentic CI/CD Matters for Kubernetes Operations

Agentic CI/CD represents an evolution in proactive deployment strategies. By integrating Elastic Observability AI agents with GitHub Actions via MCP, we've created a system that:

Prevents failures before they happen Provides real-time cluster health insights Makes data-driven deployment decisions Reduces operational burden on SRE teams Improves overall deployment reliability

This approach is at the cutting edge of modern CI/CD practices. While traditional pipelines focus solely on the "Build-Push-Deploy" cycle, agentic pipelines introduce automated pre-deployment guardrails using observability data, transforming your CI/CD infrastructure into an intelligent agent that actively protects production environments.

Resources and Next Steps

Sign up for Elastic Cloud Serverless and try this out with your pipeline.

Documentation

• Elastic Agent Builder

• ESQL Query Language

• Model Context Protocol

• GitHub Actions Workflows

Elastic’s GitHub Copilot Extension aims to combine the capabilities of GitHub Copilot and Elastic AI Assistant for Observability. This could enable developers to access critical insights from Elastic AI Assistant from GitHub Copilot Chat on GitHub.com, Visual Studio, GitHub.com, Visual Studio, and VS Code - places where they write their code.

Developers will be able ask questions such as

• What errors are active?
• What’s the latest stacktrace for my application?
• What caused a slowdown in the application after the last push to the dev environment?
• How to write an ES|QL for query that my app will send to Elasticsearch?
• What runbook from Github has been loaded into Elasticsearch and is related to the issue I’m investigating And many more!

Watch Jeff's PoC Demo@Microsoft Build 2024

Elastic AI Assistant surfaced in GitHub Copilot Chat from our Extension (Proof of Concept)

What is the Elastic AI Assistant for Observability

The Elastic Observability AI Assistant for Observability, a user-centric tool, is a game-changer in providing contextual insights and streamlining troubleshooting within the Elastic Observability environment. By harnessing generative AI capabilities, the assistant offers open prompts that decipher error messages and propose remediation actions. It adopts a Retrieval-Augmented Generation (RAG) approach to fetch the most pertinent internal information, such as APM traces, log messages, SLOs, GitHub issues, runbooks, and more. This contextual assistance is a huge leap forward for Site Reliability Engineers (SREs) and operations teams, offering immediate, relevant solutions to issues based on existing documentation and resources, boosting developer productivity.

For more information on setting up and using the AI Assistant for Observability check out the blog Getting started with the Elastic AI Assistant for Observability and Microsoft Azure OpenAI. Additionally, learn how Elastic Observability AI Assistant uses RAG to help analyze application issues with GitHub issues.

One unique feature of the AI Assistant is its API support. This allows you to take advantage of all the capabilities provided by the Elastic AI Assistant, and integrate them right into your workflow.

What is a GitHub Copilot Extension

GitHub Copilot Extensions, a new addition to GitHub Copilot, revolutionizes the developer experience by integrating a diverse array of tools and services directly into the developer's workflow. These unique extensions, crafted by partners, enable developers to interact with various services and tools using natural language within their Integrated Development Environment (IDE) or GitHub.com. This integration eliminates the need for context-switching, allowing developers to maintain their flow state, troubleshoot issues, and deploy solutions with unparalleled efficiency. These extensions will be accessible through GitHub Copilot Chat in the GitHub Marketplace, with options for organizations to create private extensions tailored to their internal tooling.

What’s next

We are participating in the Github Limited Beta Program as a partner and exploring the possibility of bringing Elastic GitHub Copilot Extension to the GitHub Marketplace. We are excited to unlock insights from Elastic Observability to GitHub Copilot users side by side to the code behind those services. Stay tuned!

Resources:

• Getting Started with Elastic AI Assistant for Observability with Azure OpenAI
• The Elastic AI Assistant for Observability escapes Kibana!
• Elastic Observability AI Assistant uses RAG to help analyze application issues with GitHub issues
• Troubleshooting with Elastic AI Assistant using your organization's runbooks
• The AI Assistant Observability documentation
• GitHub Copilot Extensions Blog Announcement
• ES|QL documentation

Modern customer‑facing applications, whether e‑commerce sites, streaming platforms, or API gateways, run on fleets of microservices and cloud resources. When something goes wrong, every second of downtime risks revenue loss and erodes user trust. Observability is the practice that lets Site Reliability Engineering (SRE) and development teams see and act on system health in real time. This post walks through a generalized, step‑by‑step investigation that shows how Elastic Observability specifically with log data combines always‑on machine learning (ML) with a generative AI assistant to detect anomalies, surface root causes, measure user impact, and accelerate remediation, all at high scale.

Anomaly Detection

A production environment is ingesting millions of log lines per minute. Elastic’s AIOps jobs continuously profile normal log throughput and content without any manual rules. When log volume or message structure deviates beyond learned baselines, the platform automatically fires a high‑fidelity anomaly alert. Because the models are unsupervised, they adapt to changing traffic patterns and flag both sudden spikes (e.g., 10× error surge) and rare new log categories.

In addition to looking directly for Log Spikes, Elastic trains seasonal/univariant models to predict expected event counts per bucket and applies statistical tests to classify outliers. Simultaneously, log categorization clusters similar messages with cosine similarity on token embeddings, making it trivial to identify a previously unseen error string.

Investigating Alerts: Automated Pattern Analysis

Clicking the alert reveals more than a timestamp. Elastic’s ML job already correlates the spike with the dominant new log pattern ERROR 1114 (HY000): table "orders" is full and surfaces example lines. Instead of grep‑driven hunting, engineers get an immediate hypothesis about what subsystem is failing and why.

If deeper context is needed, the builtin Elastic AI Assistant can be invoked directly from the alert. Thanks to Retrieval‑Augmented Generation (RAG) over your telemetry, the assistant explains the anomaly in plain language, references the exact log events, and proposes next steps without hallucinating.

AI‑Assisted Root Cause Verification

From within the same chat, you might ask, “Using lens create a single graph of all http response status codes =400 from logs-nginx.access-default over the last 3 hours..”  The assistant translates that intent into an ES|QL aggregation, retrieves the data, and renders a bar chart with no DSL knowledge required. If there are a number of errors with a status code above 400, you’ve validated that end‑users are impacted.

Global Impact Analysis with Enriched Logs

Structured log enrichment (e.g., GeoIP, user ID, service tags) lets the assistant answer business questions on the fly. A query like “What are the top 10 source.geo.country_name with http.response.status.code>=400 over the last 3 hours. Use logs-nginx.access-default. Provide counts for each country name.” surfaces whether the incident is regional or global.

Quantifying Business Impact

Technical metrics alone rarely sway executives. Suppose historical data shows the application normally processes $1,000 in transactions per minute. The assistant can combine that baseline with real‑time failure counts to estimate revenue loss. Presenting financial impact alongside error graphs sharpens prioritization and justifies extraordinary remediation steps.

Pinpointing Infrastructure & Ownership

Every log is automatically enriched with Kubernetes, cloud, and custom metadata. A single question “Which pod and cluster emit the ‘table full’ error, and who owns it?” returns the full information about the pod, namespace and owner as shown below.

Immediate, accurate routing replaces frantic Slack threads, cutting minutes (or hours) off of downtime.

Some of the magic happening here is because we can put instructions in the Elastic AI Assistants knowledge base to guide the AI assistant. For example this simple entry in the knowledge base is what allows the assistant to populate the response in the previous screenshot.

If asked about Kubernetes pod, namespace, cluster, location, or owner run the "query" tool.
1. Use the index `logs-mysql.error-default` unless another log location is specified.
2. Include the following fields in the query:
   - Pod: `agent.name`
   - Namespace: `data\_stream.namespace`
   - Cluster Name: `orchestrator.cluster.name`
   - Cloud Provider: `cloud.provider`
   - Region: `cloud.region`
   - Availability Zone: `cloud.availability\_zone`
   - Owner: `cloud.account.id`
3. Use the ES|QL query format:
   esql
   FROM logs-mysql.error-default
   | KEEP agent.name, data\_stream.namespace, orchestrator.cluster.name, cloud.provider, cloud.region, cloud.availability\_zone, cloud.account.id
   
4. Ensure the query is executed within the appropriate time range and context. 

Leveraging Institutional Knowledge with RAG

Elastic can index runbooks, GitHub issues, and wikis alongside telemetry. Asking “Find documentation on fixing a full orders table” retrieves and summarizes a prior runbook that details archiving old rows and adding a partition. Grounding remediation in proven procedures avoids guesswork and accelerates fixes.

Automated Communication & Documentation

Good incident response includes timely stakeholder updates. A prompt such as “Draft an incident update email with root cause, impact, and next steps” lets the assistant assemble a structured message and send it via the alerting framework’s email or Slack connector complete with dashboard links and next‑update timelines. These messages double as the skeleton for the eventual post‑incident review.

Again as before, some of the magic happening here is because we can put instructions in the Elastic AI Assistants knowledge base to guide the AI assistant. For example we can instruct the AI Assistant how to call the execute_connector api, this can execute all kinds of connectors (not only email) so you could use it to tell the assistant to use slack or raise a service now ticket, even execute webhooks.

Here are specific instructions to send an email. Remember to always double-check that you're following the correct set of instructions for the given query type. Provide clear, concise, and accurate information in your response.

## Email Instructions

If the user's query requires sending an email:
1. Use the `Elastic-Cloud-SMTP` connector with ID `elastic-cloud-email`.
2. Prepare the email parameters:
   - Recipient email address(es) in the `to` field (array of strings)
   - Subject in the `subject` field (string)
   - Email body in the `message` field (string)
3. Include
   - Details for the alert along with a link to the alert
   - Root cause analysis
   - Revenue impact
   - Remediation recommendations
   - Link to GitHub issue
   - All relevant information from this conversation
   - Link to the Business Health Dashboard
4. Send the email immediately. Do not ask the user for confirmation.
5. Execute the connector using this format:
   
   execute_connector(
     id="elastic-cloud-email",
     params={
       "to": ["recipient@example.com"],
       "subject": "Your Email Subject",
       "message": "Your email content here."
     }
   )
   
6. Check the response and confirm if the email was sent successfully.

Conclusion & Key Takeaways

Elastic Observability's combination of unsupervised ML, schema-aware data ingestion, and a context-rich RAG powered AI assistant enables teams to transform incident response from reactive firefighting into proactive, data-driven operations. By automatically detecting anomalies, correlating patterns, and providing contextual insights, teams can:

• Preserve revenue by quantifying business impact in real-time and prioritizing accordingly
• Scale expertise by embedding institutional knowledge into RAG-powered recommendations
• Improve continuously through automated documentation that feeds back into the knowledge base

The key is to collect logs broadly, maintain a unified observability store, and let ML and AI handle the heavy lifting. The payoff isn't just reduced downtime, it's the transformation of incident response from a source of organizational stress into a competitive advantage.

Try out this exact scenario and get hands in with this Elastic Logging Workshop: https://play.instruqt.com/elastic/invite/rx4yvknhpfci

This guide explains the key considerations for full-stack monitoring AI web agents, exploring both best practices and practical examples using OpenLit, OpenTelemetry and Elastic. Specifically we'll cover monitoring an example web travel planner located in this example repo.

Why is AI agent observability different?

The aim of traditional monitoring is to detect and alert on failures, performance issues, inefficiencies and resource bottlenecks. Monitoring AI agents still adheres to this common goal, but there are several differences that must be considered:

• AI models are probabilistic, meaning that the same input can lead to different outputs. This makes it hard to define and monitor success based on a single correct answer.
• AI systems can appear to function correctly on the surface, but their outputs may be suspect, incorrect, or biased without a way to immediately detect it. Telemetry must therefore be able to capture hidden capabilities such as tool call executions for SREs to scrutinize.
• The dynamic and evolving nature of LLMs can mean that their behavior can change dramatically between updates and versions due to changes in data, embeddings, or prompts. This means monitoring and pre-production evaluation when upgrading is vitally important for performance continuity.
• Models are black boxes. For this reason it's often difficult to understand why an AI made a particular decision. This makes troubleshooting harder compared to systems with clear, explicit logic.
• Beyond traditional metrics, AI output must be monitored for issues like hallucinations (generating false information), toxicity, and bias, which can damage user trust and lead to reputational harm.
• Contextual performance of an AI system can vary greatly depending on the context, including user interaction. Capturing user prompts and telemetry helps establish a complete picture of system performance.
• From a security perspective, AI agents can be vulnerable to adversarial attacks including data poisoning and obfuscation. Monitoring for unusual behavioral patterns and prompts is crucial to detect and mitigate these threats.

For these reasons the metrics and tracing that SREs capture and investigate will differ.

AI agent monitoring in practice

Let's apply these concepts by instrumenting an actual AI agent and capturing telemetry. Here we shall be using the TypeScript SDK of OpenLit, an open-source library that generates OpenTelemetry signals from LLM interactions in JavaScript applications. Specifically we shall instrument a simple web travel planner agent, available here that uses LLMs to generate travel recommendations based on user prompts and information from various tools. OpenLit works well for this type of project due to its TypeScript SDK and built in capabilities for capturing LLM interactions, tool calls, and generating evaluation and guardrail metrics.

The architecture diagram shows the key components:

The concepts and best practices discussed in this article can be applied to any AI agent regardless of the specific monitoring tools used. Many vendors have AI monitoring capabilities. Alternative open source technologies are also available for agentic monitoring, including LangSmith, OpenLLMetry, or indeed manual instrumentation using OpenTelemetry SDKs and the AI semantic conventions.

Prerequisites

This project requires that the following prerequisites are met:

• Active Elastic cluster (Cloud, Serverless or self-managed)
• OpenAI, Azure OpenAI, or compatible LLM provider API key
• Node.js 18+ with npm or yarn
• OTLP-compatible endpoint (Elastic Managed OTLP endpoint or OTel collector)

The following environment variables should be set:

• OTEL_ENDPOINT: Your Elastic OTLP endpoint or OTel collector URL
• OPENAI_API_KEY: API key for the evaluation/guardrail LLM
• OPENAI_ENDPOINT: Optional custom base URL for OpenAI-compatible providers

Basic instrumentation

Often DevOps engineers and SREs start with automatic instrumentation to obtain basic telemetry. This is possible with the OpenLit Python SDK. However with TypeScript we manually have to add our configuration to the AI entrypoint (here api/chat/route.ts).

First we install the dependency using our favourite package manager:

npm install openlit

Then we add the OpenLit configuration to our entrypoint:

import openlit from "openlit";

// Other imports omitted for brevity 

// Allow streaming responses up to 30 seconds to address typically longer responses from LLMs
export const maxDuration = 30;

openlit.init({
  applicationName: "ai-travel-agent", // akin to OTEL resource name
  environment: "development",
  otlpEndpoint: process.env.OTEL_ENDPOINT, // OTLP compatible endpoint (Elastic ingest or OTel collector)
  disableBatch: true, // batching disabled for demo purposes - not recommended for production use
});

// Post request handler
export async function POST(req: Request) {
   // AI logic omitted for brevity - see full code in repo
}

This instrumentation will automatically generate OpenTelemetry traces for all LLM interactions, including tool calls, and send them to the specified OTLP endpoint. Note that for production rather than demo usage, environment should be set to production and batching should not be disabled to ensure optimal network usage and protect the OTel backend.

Let's discuss the key telemetry signals that are generated in subsequent sections.

Inputs

The first rule of debugging AI agents is simple: if you don't capture the prompt, you can't reproduce the problem. Unlike traditional applications where inputs are predictable request parameters, AI agents consume free-form user prompts that can trigger wildly different behaviors based on subtle phrasing changes. OpenLit automatically captures system prompts and all user messages as structured attributes on your traces, giving you the exact input that caused your agent to hallucinate, loop, or fail.

The full conversation needs to be available to SREs to understand the context of failures and performance issues and identify patterns in the inputs that may be causing issues. However these inputs are also useful for improving agent behavior, and can be used as testing messages to evaluate model performance and test enhancements once sanitized for identifiable attributes such as PII.

Beyond prompts, we still need comprehensive logging, specifically capturing full stack traces emitted by our applications. This is crucial for diagnosing issues that may arise from the underlying infrastructure or codebase, rather than the AI model itself. For example, the below error sent to Elastic shows a simple fetch error. We must not forget that traditional errors can still occur in AI applications, and capturing them is essential.

Tracing

Traces are essential for understanding the flow of requests through your AI agent, especially when it comes to tool calls. Generally traces are a hierarchy of spans, which themselves are a single, timed unit representing a specific operation, such as a database query or an HTTP handler. In AI systems they also represent tool calls made by the LLM, along with the API calls and data retrieval steps performed within the tool execution.

Visualizing tool calling patterns is important in validating pre-production systems as well as monitoring production systems for several reasons:

1. It helps us evaluate the tool calling capabilities of different models. LLMs make the choice of which tools to use based on the user prompt, system instructions and the tool metadata (such as name and description). By visualizing the tool calling patterns we can understand whether the model is correctly interpreting the tool metadata and making appropriate calls based on the prompt.
2. It allows us to identify inefficient or erroneous tool calling patterns. For example, if we see a pattern of repeated calls to the same tool with similar inputs, it may indicate that the model is stuck in a loop or not effectively utilizing the tools. Or if a single tool is being called where we would expect multiple tools to be called, it may indicate that the model is not correctly connecting the prompt or system instructions require said tool(s).
3. Commonly occurring tool-calling patterns can also be identified to optimize the available tools. For example, if the location and weather tools are frequently called together, it may make sense to combine them into a single tool that provides both pieces of information in one call.

With the above configuration, we can see the traces for each tool call, as illustrated in the below example:

Metrics

While tracing is essential for understanding the flow of requests and tool calls, metrics are crucial for monitoring the overall health and performance of your system, agentic or not. When considering metrics many think solely of cost and total token usage. While both are important, they are not the only metrics that matter.

Through the example above OpenLit automatically generates key metrics that can be used to evaluate agent performance, such as request latency, error rates, cost and token usage, which can be visualized in Elastic to identify trends and anomalies. Token usage specifically can be split by input, output and reasoning token counts, helping us identify optimization opportunities at key stages in the generation cycle. For example, an increase in input token counts may indicate a significant increase in context length that can be optimized via prompt and context engineering techniques.

It's also important to make sure that metrics capture traditional performance metrics such as CPU, memory and request counts to caches, traditional databases and vector databases. This helps us identify whether performance issues are being caused by the AI model itself or by underlying infrastructure problems. Alerting for spikes in key measures such as large token usage increases or request volumes would also be considered best practice.

Evaluation

AI evaluation refers to the process of assessing the performance of AI models and the quality of the responses they generate. This involves monitoring various metrics and signals to ensure that the AI system is functioning as intended, providing accurate outputs, and not exhibiting undesirable behaviors such as hallucinations, toxic behavior or bias. While evaluation is considered as a pre-production activity to test and validate an agentic system, it's also important to continue monitoring these signals in production to identify issues over time.

There are several different evaluation methodologies that we can use. OpenLit makes use of AI as a Judge. This involves using an LLM to evaluate the quality of the output generated by another LLM based on a set of criteria. An example of traditional evaluation is depicted below:

When considering evaluation from a monitoring viewpoint, it's important to identify hallucinations, bias, toxicity and potential injection issues in production. Hallucinations, bias and toxic responses expose us to reputational risk and loss of user trust. Out of the box, OpenLit identifies the following issues, calculates a score and provides an explanation of the issue:

These issues can be identified using the below code:

import openlit from "openlit";

// Other imports omitted for brevity

// Allow streaming responses up to 30 seconds to address typically longer responses from LLMs
export const maxDuration = 30;

// Tools and Azure configuration omitted for brevity

openlit.init({
  applicationName: "ai-travel-agent",
  environment: "development",
  otlpEndpoint: process.env.OTEL_ENDPOINT,
  disableBatch: true,
});

// Choose one of the following approaches:
// Option 1: enable all available evaluations
const evalsAll = openlit.evals.All({
  provider: "openai",
  collectMetrics: true, // Ensures evaluations are exported to Elastic
  apiKey: process.env.OPENAI_API_KEY,
  baseUrl: process.env.OPENAI_ENDPOINT
});

// Option 2: enable specific evaluations with custom configuration
const evalsHallucination = openlit.evals.Hallucination({
  provider: "openai",
  collectMetrics: true,
  apiKey: process.env.OPENAI_API_KEY,
  baseUrl: process.env.OPENAI_ENDPOINT
});

// Post request handler
export async function POST(req: Request) {
  const { messages, id } = await req.json();

  try {
    const convertedMessages = await convertToModelMessages(messages);
    const prompt = `You are a helpful assistant that returns travel itineraries...`;

    const result = streamText({
      model: azure("gpt-4o"),
      system: prompt,
      messages: convertedMessages,
      stopWhen: stepCountIs(2),
      tools,
      experimental_telemetry: { isEnabled: true },
      onFinish: async ({ text, steps }) => {
        // Concatenate tool results and content as full evaluation context
        const toolResults = steps.flatMap((step) => {
          return step.content
            .filter((content) => content.type == "tool-result")
            .map((c) => {
              return JSON.stringify(c.output);
            });
        });

        // Measure evaluation
        const evalResults = await evalsAll.measure({
          prompt: prompt,
          contexts: convertedMessages
            .map((m) => {
              return m.content.toString();
            })
            .concat(toolResults),
          text: text,
        });
        console.log(`Evals results: ${evalResults}`);
      },
    });

    // Return data stream to allow the useChat hook to handle the results as they are streamed through for a better user experience
    return result.toUIMessageStreamResponse();
  } catch (e) {
    console.error(e);
    return new NextResponse(
      "Unable to generate a plan. Please try again later!"
    );
  }
}

By using the collectMetrics option, the evaluation results are automatically exported as metrics to Elastic, allowing us to monitor the quality of our AI agent's outputs over time and identify trends or issues that may arise in production. The evaluation results can also be used to trigger alerts or automated responses if certain thresholds or SLOs are breached, such as a high evaluation score sustained for several minutes, increased number of hallucinations detected over time, or triggering of a toxic result.

The advantage of using LLMs to evaluate results is that they help identify issues and inaccuracies quickly compared to leveraging manual quality checks. However, this methodology does have limitations, specifically:

1. Increased cost and latency due to the additional requests to an LLM to evaluate results. This can be mitigated by using a smaller, cheaper model for evaluation, by only evaluating a sample of responses, or using cached responses to reduce the number of LLM calls for similar questions.
2. LLM evaluations are prone to biases. Specifically, Zheng et al. cite in their 2023 paper that LLM evaluations are subject to:

• Positional bias, where an LLM prefers responses where the answer is located in a specific position in the response, and may miss correct answers located elsewhere in the reply.
• Self-enhancement bias, where LLMs show preference for responses they have generated compared to other models. This can be a consideration if you wish to use cheaper, or self-hosted models for evaluation.
• Verbosity bias, where they prefer more expansive responses over succinct replies.

Guardrail monitoring

In addition to assessing the quality of responses, we must also be monitoring for dangerous or irrelevant responses that could be harmful to users. The quality of in-built protections within models are patchy and model dependent. Research from several research papers, including Anthropic in their 2025 agentic misalignment paper, show that in some cases models can resort to malicious behaviors and the model bypassing company policies and moral expectations.

Guardrail detection in monitoring tools allows us to identify risky responses generated by AI agents, such as generating harmful content, engaging in inappropriate interactions, or performing injection actions to try and hack into systems or elicit confidential information. Using OpenLit as our example, we are able to monitor for breaches of the following guardrail types:

These shields can be set up using OpenLit as per the below code:

import openlit from "openlit";

// Other imports omitted for brevity

// Allow streaming responses up to 30 seconds to address typically longer responses from LLMs
export const maxDuration = 30;

// Tools and Azure configuration omitted for brevity

openlit.init({
  applicationName: "ai-travel-agent",
  environment: "development",
  otlpEndpoint: process.env.OTEL_ENDPOINT,
  disableBatch: true,
});

// Choose one of the following approaches:
// Option 1: enable all available guardrails
const guardsAll = openlit.guard.All({
  provider: "openai",
  collectMetrics: true,
  apiKey: process.env.OPENAI_API_KEY,
  baseUrl: process.env.OPENAI_ENDPOINT,
  validTopics: ["travel", "culture"],
  invalidTopics: ["finance", "software engineering"],
});

// Option 2: enables specific guardrail types (for example, prompt injection detection)
const guardsPromptInjection = openlit.guard.PromptInjection({
  provider: "openai",
  collectMetrics: true, // Ensures guardrail breaches are exported to Elastic
  apiKey: process.env.OPENAI_API_KEY,
  baseUrl: process.env.OPENAI_ENDPOINT
});

// Post request handler
export async function POST(req: Request) {
  const { messages, id } = await req.json();

  try {
    const convertedMessages = await convertToModelMessages(messages);
    const prompt = `You are a helpful assistant that returns travel itineraries...`;

    const result = streamText({
      model: azure("gpt-4o"),
      system: prompt,
      messages: convertedMessages,
      stopWhen: stepCountIs(2),
      tools,
      experimental_telemetry: { isEnabled: true },
      onFinish: async ({ text, steps }) => {
        const guardrailResult = await guardsAll.detect(text);
        console.log(`Guardrail results: ${guardrailResult}`);
      },
    });

    // Return data stream to allow the useChat hook to handle the results as they are streamed through for a better user experience
    return result.toUIMessageStreamResponse();
  } catch (e) {
    console.error(e);
    return new NextResponse(
      "Unable to generate a plan. Please try again later!"
    );
  }
}

In the event of a guardrail breach, metrics containing detail of the breach are sent to Elastic, similar to the following:

Of course we can leverage dashboards to visualize trends of guardrail breaches, including metrics such as volumes by category, as shown in the below example (with the corresponding NDJSON available here):

We can also perform action on these breaches to notify relevant teams via alerts triggered via the available alerting tools. These should be triggered based on the severity of the detected issue, as well as the classification, for example mentions of violence, illegal themes, or injection attacks may trigger immediately compared to minor inaccuracies. The guardrail breaches can also be used to trigger automated responses, such as blocking the response from being sent to the user, or providing a warning message to the user that their request has been flagged for review, triggering a human-in-the-loop response for the relevant teams.

Conclusion

AI agents are becoming more autonomous, more powerful, and more unpredictable. For this reason, it's important to introduce monitoring telemetry as early as possible in the development process and in organizational cultures. This article helps you understand how monitoring AI agents is different and how to do it using OpenLit to generate OpenTelemetry signals to send to Elastic. Check out the code here and start monitoring your AI agents in production.

Developer resources:

• Observing AI Agents Example
• OpenLit SDK Documentation
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena | Zheng et al. 2023
• Agentic Misalignment: How LLMs could be insider threats | Anthropic

Managing the Hidden Complexity of Message-Driven Architectures

Amazon MQ is a managed message broker service for Apache ActiveMQ Classic and RabbitMQ that manages the setup, operation, and maintenance of message brokers. Messaging systems like RabbitMQ, managed by Amazon MQ, are pivotal in modern decoupled, event-driven applications. By serving as an intermediary between services, RabbitMQ facilitates asynchronous communication through message queuing, routing, and reliable delivery, making it an ideal fit for microservices, real-time pipelines, and event-driven architectures. However, this flexibility introduces operational challenges, such as retries, processing delays, consumer failures, and queue backlogs, which can gradually impact downstream performance and system reliability.

With Elastic’s Amazon MQ integration, users gain deep visibility into message flow patterns, queue performance, and consumer health. This integration allows for the proactive detection of bottlenecks, helps optimize system behaviour, and ensures reliable message delivery at scale.

In this blog, we'll dive into the operational challenges of RabbitMQ in modern architectures, while also examining the common gaps and strategies for overcoming them.

Why Observability for RabbitMQ on Amazon MQ Matters?

RabbitMQ brokers are integral to distributed systems, handling tasks ranging from order processing to payment workflows and notification delivery. Any disruption can cascade into significant downstream issues. Observability into RabbitMQ helps answer critical operational questions like:​

• Is CPU and memory utilization increasing over time?
• What are the trends in the message publish rate, message confirmation rate?
• Are consumers failing to acknowledge messages?
• Which queues are experiencing abnormal growth?
• Are there an increasing number of messages being dead-lettered over time?

Enhanced Observability with Amazon MQ Integration

Elastic provides a dedicated Amazon MQ integration for RabbitMQ that utilizes Amazon CloudWatch metrics and logs to deliver comprehensive observability data. This integration enables the ingestion of metrics related to connections, nodes, queues, exchanges, and system logs.

By deploying Elastic Agent with this integration, the users can monitor:​

• Queue performance and Dead-letter queue (DLQ) metrics include total message count (MessageCount.max), messages ready for delivery (MessageReadyCount.max), and unacknowledged messages (MessageUnacknowledgedCount.max). MessageCount.max metric tracks the total number of messages in a queue, including those that have been dead-lettered, and monitoring this over time can help identify trends in message accumulation, which may suggest issues leading to dead-lettering.
• Consumer behaviour through metrics like consumer count (ConsumerCount.max) and acknowledgement rate (AckRate.max), which help identify underperforming consumers or potential backlogs.
• Messaging throughput by tracking publish (PublishRate.max), confirm (ConfirmRate.max), and acknowledgement rates in real time. These are crucial for understanding application messaging patterns and flow.
• Broker and node-level health, including memory usage (RabbitMQMemUsed.max), CPU utilization (SystemCpuUtilization.max), disk availability (RabbitMQDiskFree.min), and file descriptor usage (RabbitMQFdUsed.max). These indicators are essential for diagnosing resource saturation and avoiding service disruption.

Integrating Amazon MQ Metrics into Elastic Observability

Elastic's Amazon MQ integration facilitates the ingestion of CloudWatch metrics and logs into Elastic Observability, delivering near real-time insights into RabbitMQ. The prebuilt Amazon MQ dashboard visualizes this data, providing a centralized view of broker health, messaging activity, and resource usage, helping users quickly detect and resolve issues. Elastic's alerting for Observability enables proactive notifications based on custom conditions, while its SLO capabilities allow users to define and track key performance targets, strengthening system reliability and service commitments. 

Elastic brings together logs and metrics from Amazon MQ alongside data from a wide range of other services and applications, whether running in AWS, on-premises, or across multi-cloud environments, offering unified observability from a single platform.

Prerequisites

To follow along, ensure you have:

• An account on Elastic Cloud and a deployed stack in AWS (see instructions here). Ensure you are using version 8.16.5 or higher. Alternatively, you can use Elastic Cloud Serverless, a fully managed solution that eliminates infrastructure management, automatically scales based on usage, and lets you focus entirely on extracting value from your data.
• An AWS account with permissions to pull the necessary data from AWS. See details in our documentation.

Architecture

Tracing Audit Flows from RabbitMQ to AWS Lambda

Consider a financial audit trail use case, where every user action, such as a funds transfer, is published to RabbitMQ. A Python-based AWS Lambda function consumes these messages, deduplicates them using the id field, and logs structured audit events for downstream analysis.

Sample payload sent through RabbitMQ:

{
  "id": "txn-849302",
  "type": "audit",
  "payload": {
    "user_id": "u-10245",
    "event": "funds.transfer",
    "amount": 1200.75,
    "currency": "USD",
    "timestamp": "T14:20:15Z",
    "ip": "192.168.0.8",
    "location": "New York, USA"
  }
}

You can now correlate message publishing activity from RabbitMQ with AWS Lambda invocation logs, track processing latency, and configure alerts for conditions like drops in consumer throughput or an unexpected surge in RabbitMQ queue depth.

AWS Lambda Function: Processing RabbitMQ Messages

This Python-based AWS Lambda function processes audit events received from RabbitMQ. It deduplicates messages based on the id field and logs structured event data for downstream analysis or compliance. Save the code below in a file named app.py.

import json
import logging
import base64
# Configure logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
# In-memory set to track processed message IDs for deduplication
processed_ids = set()
def lambda_handler(event, context):
    logger.info("Lambda triggered by RabbitMQ event")
    if 'rmqMessagesByQueue' not in event:
        logger.warning("Invalid event: missing 'rmqMessagesByQueue'")
        return {'statusCode': 400, 'body': 'Invalid RabbitMQ event'}
    for queue_name, messages in event['rmqMessagesByQueue'].items():
        logger.info(f"Processing queue: {queue_name}, Messages count: {len(messages)}")
        for msg in messages:
            try:
                raw_data = msg['data']
                decoded_json = base64.b64decode(raw_data).decode('utf-8')
                message = json.loads(decoded_json)
                logger.info(f"Decoded message: {json.dumps(message)}")
                message_id = message.get('id')
                if not message_id:
                    logger.warning("Message missing 'id', skipping.")
                    continue
                if message_id in processed_ids:
                    logger.warning(f"Duplicate message detected: {message_id}")
                    continue
                payload = message.get('payload', {})
                logger.info(f"Processing message ID: {message_id}")
                logger.info(f"Event Type: {message.get('type')}")
                logger.info(f"User ID: {payload.get('user_id')}")
                logger.info(f"Event: {payload.get('event')}")
                logger.info(f"Amount: {payload.get('amount')} {payload.get('currency')}")
                logger.info(f"Timestamp: {payload.get('timestamp')}")
                logger.info(f"IP Address: {payload.get('ip')}")
                logger.info(f"Location: {payload.get('location')}")
                processed_ids.add(message_id)
            except Exception as e:
                logger.error(f"Error processing message: {str(e)}")
    return {'statusCode': 200, 'body': 'Messages processed successfully'}

Setting up AWS Secrets Manager

To securely store and manage your RabbitMQ credentials, use AWS Secrets Manager.​

1. Create a New Secret:

   • Navigate to the AWS Secrets Manager console.
   • Choose Store a new secret.
   • Select Other type of secret.
   • Enter the following key-value pairs:
     • username: Your RabbitMQ username
     • password: Your RabbitMQ password

2. Configure the Secret:

   • Provide a meaningful name, such as RabbitMQAccess.
   • Optionally, add tags and set rotation if needed.​

3. Store the Secret:

   • Review the settings and store the secret. Note the ARN of the secret you have created.

Setting up Amazon MQ for RabbitMQ

To get started with RabbitMQ on Amazon MQ, follow these steps to set up your broker.

• Open the Amazon MQ console.
• Create a new broker with the RabbitMQ engine.
• Choose your preferred deployment option—single-instance or clustered
• Use the same username and password that you previously stored in AWS Secrets Manager.
• Under Additional settings, enable CloudWatch Logs for observability.
• Configure access and security settings, ensuring that the broker is accessible to your AWS Lambda function.

• After the broker is created, note the following important details:

  • ARN of the RabbitMQ broker.
  • RabbitMQ web console URL.

• You’ll need the RabbitMQ log group ARN to set up Elastic’s Amazon MQ integration for RabbitMQ. Follow these steps to locate it:

  • Go to the General – Enabled Logs section of the broker. 
  • Copy the CloudWatch log group ARN.

Create a RabbitMQ Queue

Now that the RabbitMQ broker is configured, use the management console to create a queue where messages will be published.

• Access the RabbitMQ management console using the web console URL.
• Create a new queue (example: myQueue) to receive messages.

Build and deploy the AWS Lambda function

In this section, we'll set up the Lambda function using AWS SAM, add the message processing logic, and deploy it to AWS. This Lambda function will be responsible for consuming messages from RabbitMQ and logging audit events.

Before continuing, make sure you have completed the following prerequisites.

• AWS SAM prerequisites

• Install the AWS SAM CLI

Next, follow the steps outlined below to continue with the setup.

1. In your command line, run the command sam init from a directory of your choice.
2. The AWS SAM CLI will walk you through the setup.
   • Select AWS Quick Start Templates.
   • Choose the Hello World Example 
   • Use the Python runtime and zip package type.
   • Proceed with the default options.
   • Name your application as sample-rabbitmq-app.
   • The AWS SAM CLI downloads your starting template and creates the application project directory structure.
3. From your command line, move to the newly created sample-rabbitmq-app directory.
   • Replace the content of the hello_world/app.py file with the lambda function code for rabbitmq message processing.
   • In the template.yaml file, use the values mentioned below to update the file content.

     Resources: SampleRabbitMQApp:   Type: AWS::Serverless::Function   Properties:     CodeUri: hello_world/     Description: A starter AWS Lambda function.     MemorySize: 128     Timeout: 3     Handler: app.lambda_handler     Runtime: python3.10     PackageType: Zip     Policies:       - Statement:           - Effect: Allow             Resource: '*'             Action:               - mq:DescribeBroker               - secretsmanager:GetSecretValue               - ec2:CreateNetworkInterface               - ec2:DescribeNetworkInterfaces               - ec2:DescribeVpcs               - ec2:DeleteNetworkInterface               - ec2:DescribeSubnets               - ec2:DescribeSecurityGroups     Events:       MQEvent:         Type: MQ         Properties:           Broker: <ARN of the Broker>           Queues:             - myQueue           SourceAccessConfigurations:             - Type: BASIC_AUTH               URI: <ARN of the secret>
     

4. Run the command sam deploy --guided and wait for the confirmation message. This deploys all of the resources.

Sending Audit Events to RabbitMQ and Triggering Lambda

To test the end-to-end setup, simulate the flow by publishing audit event data into RabbitMQ using its web UI. Once the message is sent, it triggers the Lambda function. 

1. Navigate to the Amazon MQ console and select your newly created broker.

2. Locate and open the Rabbit web console URL
   

3. Under the Queues and Streams tab, select the target queue (example: myQueue).

4. Enter the message payload, and click Publish message to send it to the queue.
   Here’s a sample payload published via RabbitMQ:

   {
     "id": "txn-849302",
     "type": "audit",
     "payload": {
       "user_id": "u-10245",
       "event": "funds.transfer",
       "amount": 1200.75,
       "currency": "USD",
       "timestamp": "T14:20:15Z",
       "ip": "192.168.0.8",
       "location": "New York, USA"
     }
   }
   

5. Navigate to the AWS Lambda function created earlier.

6. Under the Monitor tab, click View CloudWatch logs.

7. Check the latest log stream to confirm that the Lambda was triggered by Amazon MQ and that the message was processed successfully.

Configuring Amazon MQ integration for Metrics and Logs collection

Elastic’s Amazon MQ integration simplifies the collection of logs and metrics from RabbitMQ brokers managed by Amazon MQ. Logs are ingested via Amazon CloudWatch Logs, while metrics are fetched from the specified AWS region at a defined interval.

Elastic provides a default configuration for metrics collection. You can accept these defaults or adjust settings such as the Collection Period to better fit your needs.

To enable the collection of logs:

1. Navigate to the Amazon MQ console and select the newly created broker.
2. Click the Logs hyperlink under the General – Enabled Logs section to open the detailed log settings page.
3. From this page, copy the CloudWatch log group ARN.
4. In Elastic, set up the Amazon MQ integration and paste the CloudWatch log group ARN.
5. Accept Defaults or Customize Settings – Elastic provides a default configuration for logs collection. You can accept these defaults or adjust settings such as collection intervals to better fit your needs.

Visualizing RabbitMQ Workloads with the Pre-Built Amazon MQ Dashboard

You can access the RabbitMQ dashboard by:

1. Navigate to the Dashboard Menu – Select the Dashboard menu option in Elastic and search for [Amazon MQ] RabbitMQ Overview to open the dashboard.

2. Navigate to the Integrations Menu – Open the Integrations menu in Elastic, select Amazon MQ, go to the Assets tab, and choose [Amazon MQ] RabbitMQ Overview from the dashboard assets

The Amazon MQ RabbitMQ dashboard in the Elastic integration delivers a comprehensive overview of broker health and messaging activity. It provides real-time insights into broker resource utilization, queue and topic performance, connection trends, and messaging throughput. The dashboard helps users track system behaviour, detect performance bottlenecks, and ensure reliable message delivery across distributed applications.

Broker Metrics

This section provides a centralised view of the overall health and performance of the RabbitMQ broker on Amazon MQ. The visualizations highlights the number of configured exchanges and queues, active broker connections, producers, consumers, and total messages in flight. System-level metrics such as CPU utilization, memory consumption, and free disk space help assess whether the broker has sufficient resources to handle current workloads.

Message flow metrics such as publish rate, confirmation rate, and acknowledgement rate are displayed to provide visibility into how messages are processed through the broker. Monitoring trends in these values helps detect message delivery issues, throughput degradation, or potential saturation of the broker under load.

Node Metrics

Node-level visibility helps identify resource imbalances across nodes in clustered RabbitMQ setups. This section includes per-node CPU usage, memory consumption, and available disk space, offering insight into the underlying infrastructure's ability to support broker operations.

Queue Metrics

Queue-specific insights are critical for understanding message delivery patterns and backlog conditions. This section details total messages, ready messages, and unacknowledged messages, segmented by broker, virtual host, and queue.

By observing how these counts change over time, users can identify slow consumers, message build-ups, or delivery issues that may affect application performance or lead to dropped messages under pressure.

Logs

This section displays log level, process ID, and raw message content. These logs provide immediate visibility into events such as connection failures, resource thresholds being hit, or unexpected queue behaviors.

Detecting Queue Backlogs with Alerting Rules

Elastic’s alert framework allows you to define rules that monitor critical RabbitMQ metrics and automatically trigger actions when specific thresholds are breached. 

Alert: Queue Backlog (Message Ready or Unacknowledged Messages)

This alert helps detect queue backlog in Amazon MQ by evaluating two metrics

• MessageUnacknowledgedCount.max and
• MessageReadyCount.max. 

The alert is triggered if either condition persists for more than 10 minutes:

• MessageUnacknowledgedCount.max exceeds 5,000
• MessageReadyCount.max exceeds 7,000

These thresholds should be adjusted based on typical message volume and consumer throughput. Sustained high values can indicate that consumers are not keeping up or message delivery pipelines are congested, potentially causing delays or dropped messages. Sustained high values may result in processing delays or dropped messages if not addressed.

Tracking Resource Utilization to Maintain RabbitMQ Performance

Elastic’s Service-level objectives (SLOs) capabilities allow you to define and monitor performance targets using key indicators like latency, availability, and error rates. Once configured, Elastic continuously evaluates these SLOs in real time, offering intuitive dashboards, alerts for threshold violations, and insights into error budget consumption. This enables teams to stay ahead of issues, ensuring service reliability and consistent performance.

SLO: Node Resource Health (CPU, Memory, Disk)

This SLO focuses on ensuring RabbitMQ brokers and nodes have sufficient resources to process messages without performance degradation. It tracks CPU, memory, and disk usage across RabbitMQ brokers and nodes to prevent resource exhaustion that could lead to service interruptions.

Target thresholds:

• SystemCpuUtilization.max remains below 85% for 99% of the time.
• RabbitMQMemUsed.max remains below 80% of RabbitMQMemLimit.max for 99% of the time.
• RabbitMQDiskFree.min remains above 25% of RabbitMQDiskFreeLimit.max for 99% of the time.

Sustained high values in CPU or memory usage can signal resource contention, which may result in slower message processing or downtime. Low disk availability may cause the broker to stop accepting messages, risking message loss. These thresholds are designed to catch early signs of resource saturation and ensure smooth, uninterrupted message flow across RabbitMQ deployments.

Conclusion

As RabbitMQ-based messaging architectures scale and become more complex, the need for in-depth visibility into system performance and potential issues deepens. Elastic’s Amazon MQ integration brings that visibility front and center—helping you go beyond basic health checks to understand real-time messaging throughput, queue backlog trends, and resource saturation across your brokers and consumers.

By leveraging the prebuilt dashboards, configuring alerts and SLOs, you can proactively detect anomalies, fine-tune consumer performance, and ensure reliable delivery across your event-driven applications.

The Elastic AI Assistant significantly enhances the process, not only in identifying but also in resolving issues. This is further enhanced by Elastic’s new Service Level Objective (SLO) capability, allowing you to streamline your entire site reliability engineering (SRE) process from detecting potential issues to enhancing the overall customer experience.

In this blog, we will demonstrate how you, as an SRE, can detect issues in a service equipped with OpenTelemetry. We will explore problem identification using Elastic APM, Elastic’s AIOps capabilities, and the Elastic AI Assistant.

We will illustrate this using the OpenTelemetry demo, with a feature flag (cartService) that is activated.

Our walkthrough will encompass two scenarios:

1. When the SLO for cart service becomes noncompliant, we will analyze the error through Elastic APM. The Elastic AI Assistant will assist by providing a runbook and a GitHub issue to facilitate issue analysis.

2. Should the SLO for the cart service be noncompliant, we will examine the trace that indicates a high failure rate. We will employ AIOps for failure correlation and the AI Assistant to analyze logs and Kubernetes metrics directly from the Assistant.

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up the configuration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).

• We used the OpenTelemetry Demo. Directions for using Elastic with OpenTelemetry Demo are here.

• Additionally you will need to connect your AI Assistant to your favorite LLM. We used Azure OpenAI GPT-4.

• We also ran the OpenTelemetry Demo on Kubernetes, specifically on GKE.

SLO noncompliance

Elastic APM recently released the SLO (Service Level Objectives) feature in 8.12. This feature enables setting measurable performance targets for services, such as availability, latency, traffic, errors, and saturation or define your own. Key components include:

• Defining and monitoring SLIs (Service Level Indicators)

• Monitoring error budgets indicating permissible performance shortfalls

• Alerting on burn rates showing error budget consumption

We set up two SLOs for cart service:

• Availability SLO , which monitors its availability by ensuring that transactions succeed. We set up the feature flag in the OpenTelemetry application, which generates an error for EmptyCart transactions 10% of the time.

• Latency SLO to ensure transactions are not going below a specific latency, which will reduce customer experiences.

Because of the OTel cartservice feature flag, the availability SLO is triggered, and within the SLO details, we see that over a seven-day period the availability is well below our target of 99.9, at 95.5. Additionally all the error budget that was available is also exhausted.

With SLO, you can easily identify when issues with customer experience occur, or when potential issues with services arise before they become potentially worse.

Scenario 1: Analyzing APM trace and logs with AI Assistant

Once the SLO is found as non-compliant, we can dive into cart service to investigate in Elastic APM. The following walks through the set of steps you can take in Elastic APM and how to use the AI Assistant to analyze the issue:

From the video, we can see that once in APM, we took the following steps.

1. Investigated the trace EmptyCart, which was experiencing larger than normal failure rates.

2. The trace showed a significant number of failures, which also resulted in slightly larger latency.

3. We used AIOps failure correlation to identify the potential component causing the failure, which correlated to a field value of FailedPrecondition.

4. While filtering on that value and reviewing the logs, we still couldn’t understand what this meant.

5. This is where you can use Elastic’s AI Assistant to further your understanding of the issue.

AI Assistant helped us analyze the following:

1. It helped us understand what the log message meant and that it was related to the Redis connection failure issue.

2. Because we couldn’t connect to Redis, we asked the AI Assistant to give us the metrics for the Redis Kubernetes pods.

3. We learned there were two pods for Redis from the logs over the last two hours.

4. However, we also learned that the memory of one seems to be increasing.

5. It seems that Redis restarted (hence the second pod), and with this information we could dive deeper into what happened to Redis.

You can see how quickly we could correlate a significant amount of information, logs, metrics, and traces through the AI Assistant and Elastic’s APM capabilities. We didn’t have to go through multiple screens to hunt down information.

Scenario 2: Analyzing APM error with AI Assistant

Once the SLO is found as noncompliant, we can dive into cart service to investigate in Elastic APM. The following walks through the set of steps you can take in Elastic APM and use the AI Assistant to analyze the issue:

From the video, we can see that once in APM, we took the following steps:

1. We noticed a specific error for the APM service.

2. We investigated this in the error tab, and while we see it’s an issue with connection to Redis, we still need more information.

3. The AI Assistant helps us understand the stacktrace and provides some potential causes for the error and ways to diagnose and resolve it.

4. We also asked it for a runbook, created by our SRE team, which gives us steps to work through this particular issue.

But as you can see, AI Assistant provides us not only with information about the error message but also how to diagnose it and potentially resolve it with an internal runbook.

Achieving operational excellence, optimal performance, and reliability

We’ve shown how an OpenTelemetry instrumented application (OTel demo) can be analyzed using Elastic’s features, especially the AI Assistant coupled with Elastic APM, AIOps, and the latest SLO features. Elastic significantly streamlines the process of identifying and resolving issues within your applications.

Through our detailed walkthrough of two distinct scenarios, we have seen how Elastic APM and the AI Assistant can efficiently analyze and address noncompliance with SLOs in a cart service. The ability to quickly correlate information, logs, metrics, and traces through these tools not only saves time but also enhances the overall effectiveness of the troubleshooting process.

The use of Elastic's AI Assistant in these scenarios underscores the value of integrating advanced AI capabilities into operational workflows. It goes beyond simple error analysis, offering insights into potential causes and providing actionable solutions, sometimes even with customized runbooks. This integration of technology fundamentally changes how SREs approach problem-solving, making the process more efficient and less reliant on manual investigation.

Overall, the advancements in Elastic’s APM, AIOps capabilities, and the AI Assistant, particularly in handling OpenTelemetry data, represent a significant step forward in operational excellence. These tools enable SREs to not only react swiftly to emerging issues but also proactively manage and optimize the performance and reliability of their services, thereby ensuring an enhanced customer experience.

Try it out

Existing Elastic Cloud customers can access many of these features directly from the Elastic Cloud console. Not taking advantage of Elastic on cloud? Start a free trial.

• Build better Service Level Objectives (SLOs) from logs and metrics
• Elastic Observability 8.12: GA for AI Assistant, SLO, and Mobile APM support
• Native Observability support in Elastic Observability
• Context-aware insights using the Elastic AI Assistant for Observability

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

Elastic Cloud provides a powerful solution to meet these challenges. Its scalable, high-performance platform enables organizations to ingest and analyze all data types efficiently (from transactional data to customers’ personal information to claims data), delivering actionable insights that empower fraud prevention teams to detect anomalies and stop fraud before it occurs. From identifying unusual spending patterns to uncovering hidden threats, Elastic Cloud offers the speed and flexibility needed to safeguard assets in an increasingly digital economy.

In this blog, we’ll walk you through how Elastic Cloud can be used to identify fraud within credit card transactions—a key area of focus due to the high volume of data and the significant potential for fraudulent activity.

We’ll use a Node.js code example to generate an example set of credit card transactions. The generated transactions include a data anomaly similar to an anomaly that might occur as a result of fraudulent activity known as “Card Testing”, which is when a malicious actor tests to see if stolen credit card data can be used to make fraudulent transactions. We’ll then import the credit card transactions into an Elastic Cloud index and use Elastic Observability’s Anomaly Detection feature to analyze the transactions to detect potential signs of “Card Testing”.

Performing fraud detection with Elastic Cloud

Generate example credit card transactions

Begin the process by using a terminal on your local computer to run a Node.js code example that will generate some example credit card transaction data.

Within your terminal window, run the following git clone command to clone the Github repository containing the Node.js code example:

git clone https://github.com/elastic/observability-examples

Run the following cd command to change directory to the code example folder:

cd observability-examples/anomaly-detection

Run the following npm install command to install the code example’s dependencies:

npm install

Enter the following node command to run the code example which will generate a JSON file named transactions.ndjson containing 1000 example credit card transactions:

node generate-transactions.js 

Now that we've got some credit card transaction data, we can import the transactions into Elastic Cloud to analyze the data.

Import transactions data into an Elastic Cloud index

We’ll start the import process in Elastic Cloud. Create an Elastic Serverless project in which we can import and analyze the transaction data. Click Create project.

Click Next in the Elastic for Observability project type tile.

Click Create project.

Click Continue.

Select the Application tile.

Enter the text “Upload” into the search box.

Select the Upload a file tile.

Click Select or drag and drop a file.

Select the transactions.ndjson file on your local computer that was created from running the Node.js code example in a previous step.

Click Import.

Enter an Index name and click Import.

You’ll see a confirmation when the import process completes and the new index is successfully created.

Use Anomaly Detection to analyze credit card transactions

Anomaly Detection is a powerful tool that can analyze your data to find unusual patterns that would otherwise be difficult, if not impossible, to manually uncover. Now that we've got transaction data loaded into an index, let's use anomaly detection to analyze it. Click Machine learning in the navigation menu.

Select Anomaly Detection Jobs

Click Create anomaly detection job.

Select the Index containing the imported transactions as the data source of the anomaly detection job.

As mentioned above, one form of credit card fraud is called “Card Testing” where a malicious actor tests a batch of credit cards to determine if they are still valid.

We can analyze the transaction data in our index to detect fraudulent “Card Testing” by using the anomaly detection Population wizard. Select the Population wizard tile.

Click Use full data.

Click Next.

Click the Population field selector and select IPAddress.

Click the Add metric option.

Select Count(Event rate) as the metric to be added.

Click Next.

Enter a Job ID and click Next.

Click Next.

Click Create job.

Once the job completes, click View results.

You should see that an anomaly has been detected. It looks like a specific IP Address has been identified performing an exceedingly high number of transactions with multiple credit cards on a single day.

You can click the red highlighted segments in the timeline to see more details to assist you with evaluating possible remediation actions to implement.

In just a few steps, we were able to create a machine learning job that grouped all the transactions by the IP address that sent them and identified slices of time where one IP sent an unusually large number of requests compared to other IPs. Our fraudster!

Take the next step in fraud prevention

Fraud detection is an ongoing battle for organizations across industries, and the stakes are higher than ever. As digital payments, insurance claims, and online banking continue to dominate, the need for robust, real-time solutions to detect and prevent fraud is critical. In this blog, we demonstrated how Elastic Cloud empowers organizations to address this challenge effectively.

By using Elastic Cloud’s powerful capabilities, we ingested and analyzed a dataset of credit card transactions to detect potential fraudulent activity, such as “Card Testing.” From ingesting data into an Elastic index to leveraging machine learning-powered anomaly detection, this step-by-step process highlighted how Elastic Cloud can uncover hidden patterns and provide actionable insights to fraud prevention teams.

This example is just the beginning of what Elastic Cloud can do. Its scalable architecture, flexible tools, and powerful analytics make it an invaluable asset for any organization looking to protect their customers and assets from fraud. Whether it's detecting unusual spending patterns, identifying compromised accounts, or monitoring large-scale operations, Elastic Cloud provides the speed, precision, and efficiency financial services organizations need to stay one step ahead of fraudsters.

As fraud continues to evolve, so must the tools we use to combat it. Elastic Cloud gives you the power to meet these challenges head-on, enabling your institution to provide a safer, more secure experience for your customers.

Ready to explore more? View a guided tour of all the steps in this blog post or create an Elastic Serverless Observability project and start analyzing your data for anomalies today.

Related resources:

• Overview: Accelerate fraud detection and prevention with Elastic
• Blog: AI-powered fraud detection: Protecting financial services with Elastic
• Blog: Fraud in financial services: Leaning on generative AI to protect a rapidly expanding attack surface

As a senior software engineer working at Elastic®, I have been on the first line of support for anything related to Beats or Elastic Agent running on Kubernetes and Cloud Native integrations like Nginx ingress controller.

During my experience, I have seen all sorts of issues. Users have very different requirements. But at some point during their experience, most of them encounter a very common problem with Elasticsearch: index mapping exceptions.

How mappings work

Like any other document-based NoSQL database, Elasticsearch doesn’t force you to provide the document schema (called index mapping or simply mapping) upfront. If you provide a mapping, it will use it. Otherwise, it will infer one from the first document or any subsequent documents that contain new fields.

In reality, the situation is not black and white. You can also provide a partial mapping that covers only some of the fields, like the most common fields, and leave Elasticsearch to figure out the mapping of all the other fields during ingestion with Dynamic Mapping.

What happens when data is malformed?

No matter if you specified a mapping upfront or if Elasticsearch inferred one automatically, Elasticsearch will drop an entire document with just one field that doesn't match the mapping of an index and return an error instead. This is not much different from what happens with other SQL databases or NoSQL data stores with inferred schemas. The reason for this behavior is to prevent malformed data and exceptions at query time.

A problem arises if a user doesn't look at the ingestion logs and misses those errors. They might never figure out that something went wrong, or even worse, Elasticsearch might stop ingesting data entirely if all the subsequent documents are malformed.

The above situation sounds very catastrophic, but it's entirely possible since I have seen it many times when on-call for support or on discuss.elastic.co. The situation is even more likely to happen if you have user-generated documents, so you don't have full control over the quality of your data.

Luckily, there is a setting that not many people know about in Elasticsearch that solves the exact problems above. This field has been there since Elasticsearch 2.0. We are talking ancient history here since the latest version of the stack at the time of writing is Elastic Stack 8.9.0.

Let's now dive into how to use this Elasticsearch feature.

A toy use case

To make it easier to interact with Elasticsearch, I am going to use Kibana® Dev Tools in this tutorial.

The following examples are taken from the official documentation on ignore_malformed. I am here to expand on those examples by providing a few more details about what happens behind the scenes and on how to search for ignored fields. We are going to use the index name my-index, but feel free to change that to whatever you like.

First, we want to create an index mapping with two fields called number_one and number_two. Both fields have type integer, but only one of them has _ ignore_malformed _ set to true, and the other one inherits the default value ignore_malformed: false instead.

PUT my-index
{
  "mappings": {
    "properties": {
      "number_one": {
        "type": "integer",
        "ignore_malformed": true
      },
      "number_two": {
        "type": "integer"
      }
    }
  }
}

If the mentioned index didn’t exist before and the previous command ran successfully, you should get the following result:

{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "my-index"
}

To double-check that the above mapping has been created correctly, we can query the newly created index with the command:

GET my-index/_mapping

You should get the following result:

{
  "my-index": {
    "mappings": {
      "properties": {
        "number_one": {
          "type": "integer",
          "ignore_malformed": true
        },
        "number_two": {
          "type": "integer"
        }
      }
    }
  }
}

Now we can ingest two sample documents — both invalid:

PUT my-index/_doc/1
{
  "text":       "Some text value",
  "number_one": "foo"
}

PUT my-index/_doc/2
{
  "text":       "Some text value",
  "number_two": "foo"
}

The document with id=1 is correctly ingested, while the document with id=2 fails with the following error. The difference between those two documents is in which field we are trying to ingest a sample string “foo” instead of an integer.

{
  "error": {
    "root_cause": [
      {
        "type": "document_parsing_exception",
        "reason": "[3:17] failed to parse field [number_two] of type [integer] in document with id '2'. Preview of field's value: 'foo'"
      }
    ],
    "type": "document_parsing_exception",
    "reason": "[3:17] failed to parse field [number_two] of type [integer] in document with id '2'. Preview of field's value: 'foo'",
    "caused_by": {
      "type": "number_format_exception",
      "reason": "For input string: \"foo\""
    }
  },
  "status": 400
}

Depending on the client used for ingesting your documents, you might get different errors or warnings, but logically the problem is the same. The entire document is not ingested because part of it doesn’t conform with the index mapping. There are too many possible error messages to name, but suffice it to say that malformed data is quite a common problem. And we need a better way to handle it.

Now that at least one document has been ingested, you can try searching with the following query:

GET my-index/_search
{
  "fields": [
    "*"
  ]
}

Here, the parameter fields is required to show the values of those fields that have been ignored. More on this later.

From the result, you can see that only the first document (with id=1) has been ingested correctly while the second document (with id=2) has been completely dropped.

{
  "took": 14,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": null,
    "hits": [
      {
        "_index": "my-index",
        "_id": "1",
        "_score": null,
        "_ignored": ["number_one"],
        "_source": {
          "text": "Some text value",
          "number_one": "foo"
        },
        "fields": {
          "text": ["Some text value"],
          "text.keyword": ["Some text value"]
        },
        "ignored_field_values": {
          "number_one": ["foo"]
        },
        "sort": ["1"]
      }
    ]
  }
}

From the above JSON response, you will notice some things, such as:

• A new field called _ _ignored _ of type array with the list of all fields that have been ignored while ingesting documents
• A new field called _ ignored_field_values _ with a dictionary of ignored fields and their values
• The field called __ source _ contains the original document unmodified. This is especially useful if you want to fix the problems with the mapping later.
• The field called _ text _ was not present in the original mapping, but it is now included since Elasticsearch automatically inferred the type of this field. In fact, if you try to query the mapping of the index _ my-index _ again via the command:

GET my-index/_mapping

You should get this result:

{
  "my-index": {
    "mappings": {
      "properties": {
        "number_one": {
          "type": "integer",
          "ignore_malformed": true
        },
        "number_two": {
          "type": "integer"
        },
        "text": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
  }
}

Finally, if you ingest some valid documents like the following command:

PUT my-index/_doc/3
{
  "text":       "Some text value",
  "number_two": 10
}

You can check how many documents have at least one ignored field with the following Exists query:

GET my-index/_search
{
  "query": {
    "exists": {
      "field": "_ignored"
    }
  }
}

You can also see that out of the two documents ingested (with id=1 and id=3) only the document with id=1 contains an ignored field.

{
  "took": 193,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "my-index",
        "_id": "1",
        "_score": 1,
        "_ignored": ["number_one"],
        "_source": {
          "text": "Some text value",
          "number_one": "foo"
        }
      }
    ]
  }
}

Alternatively, you can search for all documents that have a specific field being ignored with this Terms query:

GET my-index/_search
{
  "query": {
    "terms": {
      "_ignored": [ "number_one"]
    }
  }
}

The result, in this case, will be the same as the previous one since we only managed to ingest a single document with that exact single field ignored.

Conclusion

Because we are a big fan of this flag, we've enabled _ ignore_malformed _ by default for all Elastic integrations and in the default index template for logs data streams as of 8.9.0. More information can be found in the official documentation for ignore_malformed.

And since I am personally working on this feature, I can reassure you that it is a game changer.

You can start by setting _ ignore_malformed _ on any cluster manually before Elastic Stack 8.9.0. Or you can use the defaults that we set for you starting from Elastic Stack 8.9.0.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In an era where APIs serve as the backbone of modern applications, having the means to maintain visibility and control over these vital components is absolutely essential. In this blog post, we dive deep into the comprehensive observability solution offered by Elastic^®, ensuring real-time visibility, advanced analytics, and actionable insights, empowering you to fine-tune your API Gateway for optimal performance.

For application owners and developers, this integration stands as a beacon of empowerment. Elastic's meticulous orchestration of the seamless merging of metrics, logs, and traces, built upon the robust ELK Stack foundation, equips them with potent real-time monitoring and analysis tools. These tools facilitate precise performance optimization and swift issue resolution, all within a secure and dependable environment.

With Elastic's AWS API Gateway integration, application owners and developers unlock the capability to proactively identify and resolve problems, fine-tune resource utilization, and provide extraordinary digital experiences to their users.

Architecture

Why the AWS API Gateway integration matters

API Gateway now serves as the foundation of contemporary application development, simplifying the process of creating and overseeing APIs on a large scale. Yet, monitoring and troubleshooting these API endpoints can be challenging. With the new AWS API Gateway integration introduced by Elastic, you can gain the following:

• Unprecedented visibility: Monitor your API Gateway endpoints' performance, error rates, and usage metrics in real time. Get a comprehensive view of your APIs' health and performance.
• Log analysis: Dive deep into API Gateway logs with ease. Our integration enables you to collect and analyze logs for HTTP, REST, and Websocket API types, helping you troubleshoot issues and gain valuable insights.
• Rapid issue resolution: Identify and resolve issues in your API Gateway workflows faster than ever. Elastic Observability's powerful search and analytics tools help you pinpoint problems with ease.
• Alerting and notifications: Set up custom alerts based on API Gateway metrics and logs. Receive notifications when performance thresholds are breached, ensuring that you can take action promptly.
• Optimized costs: Visualize resource usage and performance metrics for your API Gateway deployments. Use these insights to optimize resource allocation and reduce operational costs.
• Custom dashboards: Create customized dashboards and visualizations tailored to your API Gateway monitoring needs. Stay in control with real-time data and actionable insights.
• Effortless integration: Seamlessly connect your AWS API Gateway to our observability solution. Our intuitive setup process ensures a smooth integration experience.
• Scalability: Whether you have a handful of APIs or a complex API Gateway landscape, our observability solution scales to meet your needs. Grow confidently as your API infrastructure expands.

How to get started

Getting started with the AWS API Gateway integration in Elastic Observability is seamless. Here's a quick overview of the steps:

Prerequisites and configurations

If you intend to follow the steps outlined in this blog post, there are a few prerequisites and configurations that you should have in place beforehand.

1. You will need an account on Elastic Cloud and a deployed stack and agent. Instructions for deploying a stack on AWS can be found here. This is necessary for AWS API Gateway logging and analysis.

2. You will also need an AWS account with the necessary permissions to pull data from AWS. Details on the required permissions can be found in our documentation.

3. You can monitor API execution by using CloudWatch, which collects and processes raw data from API Gateway into readable, near-real-time metrics and logs. Details on the required steps to enable logging can be found here.

Step 1. Create an account with Elastic

Create an account on Elastic Cloud by following the steps provided.

Step 2. Add integration

• Log in to your Elastic Cloud deployment.

• Click on Add integrations. You will be navigated to a catalog of supported integrations.

• Search and select AWS API Gateway.

Step 3. Configure integration

• Click on the Add AWS API Gateway button and provide the required details.
• If this is your first time adding an AWS integration, you’ll need to configure and enroll the Elastic Agent on an AWS instance.

• Then complete the “Configure integration” form, providing all the necessary information required for agents to collect the AWS API Gateway metrics and associated CloudWatch logs. Multiple AWS credential methods are supported, including access keys, temporary security credentials, and IAM role ARN. Please see the IAM security and access documentation for more details. You can choose to collect API Gateway metrics, API Gateway logs via S3, or API Gateway logs via CloudWatch.
• Click on the Save and continue button at the bottom of the page.

Step 4. Analyze and monitor

Explore the data using the out-of-the-box dashboards available for the integration. Select Discover from the Elastic Cloud top-level menu.

Or, create custom dashboards, set up alerts, and gain actionable insights into your API Gateway service performance.

Here are key monitoring metrics collected through this integration across Rest APIs, HTTP APIs, and Websocket APIs:

• 4XXError – The number of client-side errors captured in a given period
• 5XXError – The number of server-side errors captured in a given period
• CacheHitCount – The number of requests served from the API cache in a given period
• CacheMissCount – The number of requests served from the backend in a given period, when API caching is enabled
• Count – The total number of API requests in a given period
• IntegrationLatency – The time between when API Gateway relays a request to the backend and when it receives a response from the backend
• Latency – The time between when API Gateway receives a request from a client and when it returns a response to the client — the latency includes the integration latency and other API Gateway overhead
• DataProcessed – The amount of data processed in bytes
• ConnectCount – The number of messages sent to the $connect route integration
  MessageCount – The number of messages sent to the WebSocket API, either from or to the client

Conclusion

The native integration of AWS API Gateway into Elastic Observability marks a significant advancement in streamlining the monitoring and management of your APIs. With this integration, you gain access to a wealth of insights, real-time visibility, and powerful analytics tools, empowering you to optimize your API performance, enhance security, and troubleshoot with ease. Don't miss out on this opportunity to take your API management to the next level, ensuring your digital assets operate at their best, all while providing a seamless experience for your users. Embrace this integration, and stay at the forefront of API observability in the ever-evolving world of digital technology.

Visit our documentation to learn more about Elastic Observability and the AWS API Gateway integration, or contact our sales team to get started!

Start a free trial today

Start your own 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world. Your AWS Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with AWS.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elastic integrations are designed to simplify observability by providing tools to ingest application data, process it through Ingest pipelines, and deliver prebuilt dashboards for visualization. With OpenTelemetry support, data collection and processing will transition to the OpenTelemetry Collector, while dashboards will need to adopt the OpenTelemetry data structure.

From a Log to an Integration

Although the concept of an OpenTelemetry Integration has not yet been officially defined, we envision it as a structured collection of artifacts that enables users to start monitoring an application from scratch. Each artifact has a specific role; for example, an OpenTelemetry Collector configuration file, which must be integrated into the main Collector setup. This bundled configuration instructs the Collector on how to gather and process data from the relevant application.

In the OpenTelemetry Collector, data collection is handled by the receivers component. Some receivers are tailored for specific applications, such as Kafka or MySQL, while others are designed to support general data collection methods. The specialized receivers combine data gathering and transformation within a single component. For the more generic receivers, however, additional components are needed to refine and transform the incoming data into a more application-specific format. Let’s take a look at how we can build an integration for monitoring a Nginx Ingress Controller.

The Ingress Nginx is an Ingress controller for Kubernetes, using NGINX as a reverse proxy and load balancer. Widely adopted, it plays a crucial role in directing external traffic into Kubernetes services, making its usage, performance and health essential to observe. How can we start observing the external requests done to our Ingress controller? Fortunately, the NGINX Ingress Controller generates a structured log entry for each processed request. This structured format ensures that each log entry follows a consistent structure, making it straightforward to parse and generate consistent output.

log_format upstreaminfo '$remote_addr - $remote_user [$time_local]
	"$request" ' '$status $body_bytes_sent "$http_referer" "$http_user_agent" '
	'$request_length $request_time [$proxy_upstream_name]
	[$proxy_alternative_upstream_name] $upstream_addr ' '$upstream_response_length
	$upstream_response_time $upstream_status $req_id';

All the field's definition can be found here.

The OpenTelemetry Contrib Collector does not include a receiver capable of reading and parsing all fields in an NGINX Ingress log. There are two primary reasons for this:

• Application Diversity: The landscape of applications is vast, with each generating logs in unique formats. Developing and maintaining a dedicated receiver for every application would be resource-intensive and difficult to scale.
• Data Source Flexibility: Receivers are typically designed to collect data from a specific source, like an HTTP endpoint. However, in some cases, we may want to parse logs from an alternate source, such as an NGINX Ingress log file stored in an AWS S3 bucket.

These challenges can be addressed by combining receivers and processors. Receivers handle the collection of raw data, while processors can extract specific values when a known data structure is detected. Do we need a dedicated processor to parse NGINX logs? Not necessarily. The transform processor can handle this by modifying telemetry data according to a specified configuration. This configuration is written in the OpenTelemetry Transformation Language (OTTL), a language for transforming open telemetry data based on the OpenTelemetry Collector Processing Exploration.

The concept of processors in OpenTelemetry is quite similar to the Ingest pipeline strategy currently used in Elastic integrations. The main challenge, therefore, lies in migrating Ingest pipeline configurations to OpenTelemetry Collector configurations. For a deeper dive into the challenges of such migrations, check out this article.

For reference, you can view the current Elastic NGINX Ingress Controller Ingest pipeline configuration in the following link: Elastic NGINX Ingress Controller Ingest Pipeline.

Let’s start with the data collection. By default, the NGINX Ingress Controller logs to stdout, and Kubernetes captures and stores these logs in a file. Assuming that the OpenTelemetry Collector running the following configuration has access to the Kubernetes Pod logs, we can use the filelog receiver to read the controller logs:

receivers:
  filelog/nginx:
    include_file_path: true
    include: [/var/log/pods/*nginx-ingress-nginx-controller*/controller/*.log]
    operators:
      - id: container-parser
        type: container

This configuration is designed to exclusively read the controller's pod logs, focusing on their default file path within a Kubernetes node. Furthermore, since the Ingress controller does not inherently have access to its associated Kubernetes metadata, the container-parser operator has been implemented to bridge this gap. This operator appends Kubernetes-specific attributes, such as k8s.pod.name and k8s.namespace.name, based solely on information available from the filename. For a detailed overview of the container-parser operator, see the following OpenTelemetry blog post.

Avoiding duplicated logs

The configuration outlined in this blog is designed for Kubernetes environments, where the collector runs as a Kubernetes Pod. In such setups, handling Pod restarts properly is crucial. By default, the filelog receiver reads the entire content of log files on startup. This behavior can lead to duplicate log entries being reprocessed and sent through the pipeline if the collector Pod is restarted.

To make the configuration resilient to restarts, you can use a storage extension to track file offsets. These offsets allow the filelog receiver to resume reading from the last processed position in the log file after a restart. Below is an example of how to add a file storage extension and update the filelog receiver configuration to store the offsets in a file:

extensions:
  file_storage:
  directory: /var/lib/otelcol

receivers:
  filelog/nginx:
    storage: file_storage
    ...

Important: The /var/lib/otelcol directory must be mounted as part of a Kubernetes persistent volume to ensure the stored offsets persist across Pod restarts.

Data transformation with OpenTelemetry processors

Now it’s time to parse the structured log fields and transform them into queryable OpenTelemetry fields. Initially, we considered using regular expressions with the extract_patterns function available in the OpenTelemetry Transformation Language (OTTL). However, Elastic recently contributed a new OTTL function, ExtractGrokPatterns, based on Grok—a regular expression dialect that supports reusable, aliased expressions. The function’s underlying library Elastic Go-Grok ships with numerous predefined grok patterns that simplify working with pattern matching, like %NUMBER that will match any number type; "123", "456.789", "-0.123".

Each Ingress Controller log entry begins with the client's source IP address (which may be a single IP or a list of IPs) and the username provided via Basic authentication, represented as “$remote_addr - $remote_user”. The Grok IP alias can be used to parse either an IPv4 or IPv6 address from the remote_addr field, while the %GREEDYDATA alias can capture the remote_user value.

For example, the following OTTL configuration will transform an unstructured body message to a structured one with two fields:

• Parses a single IP address and assign it to the source.address key.
• Delimited by a “-”, captures the optional value of the authenticated username in the user.name key.

transform/parse_nginx_ingress_access/log:
  log_statements:
    - context: log
      statements:
        - set(body, ExtractGrokPatterns(body, "%{IP:source.address} - (-|%{GREEDYDATA:user.name})", true))

The screenshot below illustrates the transformation process, showing the original input data alongside the resulting structured format (diff):

In real-world scenarios, NGINX Ingress Controller logs may begin with a list of IP addresses or, at times, a domain name. These variations can be handled with an extended Grok pattern. Similarly, we can use Grok to parse an HTTP UserAgent and URL strings, but additional OTTL functions, such as URL or UserAgent, are required to extract meaningful data from these fields.

The complete configuration is available in the documentation for Elastic’s OpenTelemetry NGINX Ingress Controller integration: Integration Documentation.

Usage

The Elastic OpenTelemetry NGINX Ingress Controller is currently on Technical preview. To access it, you must enable the "Display beta integrations" toggle in the Integrations menu within Kibana.

By installing the Elastic OpenTelemetry NGINX Ingress Controller integration, a couple of dashboards will become available in your Kibana profile. One of these dashboards provides insights into access events for the controller, displaying information such as HTTP response status codes over time, request volume per URL, distribution of incoming requests by browser, top requested pages, and more. The screenshot below shows the NGINX Ingress Controller Access Logs dashboard, displaying data from a controller routing requests to an OpenTelemetry Demo deployment:

The second dashboard focuses on errors within the Nginx Ingress controller, highlighting the volume of error events generated over time:

To start gathering and processing controller logs, we recommend incorporating the OpenTelemetry Collector pipeline outlined in the integration’s documentation into your collector configuration: Integration Documentation. Keep in mind that this configuration requires access to the Kubernetes node's Pods logs, typically stored in /var/log/pods/*. To ensure proper access, we recommend deploying the OpenTelemetry Collector as a daemonset in Kubernetes, as this deployment type allows the collector to access the necessary log directory on each node.

The OpenTelemetry Collector configuration service pipeline should include a similar configuration:

service:
  extensions: [file_storage]
  pipelines:
    logs/nginx_ingress_controller:
      receivers:
        - filelog
      processors:
        - transform/parse_nginx_ingress_access/log
        - transform/parse_nginx_ingress_error/log
        - resourcedetection/system
      exporters:
        - elasticsearch

Adding GeoIP Metadata

As an optional enhancement, the OpenTelemetry Collector GeoIP processor can be configured and added to the pipeline to enrich each NGINX Ingress Controller log with geographical attributes, such as the request’s originating country, region, and city, enabling geo maps in Kibana to visualize traffic distribution and geographic patterns.

While the OpenTelemetry GeoIP processor is similar to Elastic's GeoIP processor, it requires users to provide their own local GeoLite2 database. The following configuration extends the Integration’s configuration to include the GeoIP processor with a MaxMind's database.

processors:
  geoip:
    context: record
    providers:
      maxmind:
        database_path: /tmp/GeoLite2-City.mmdb

service:
  extensions: [file_storage]
  pipelines:
    logs/nginx_ingress_controller:
      receivers:
        - filelog
      processors:
        - transform/parse_nginx_ingress_access/log
        - transform/parse_nginx_ingress_error/log
        - resourcedetection/system
        - geoip
      exporters:
        - elasticsearch

Sample Kibana Map with the OpenTelemetry Nginx Ingress Controller integration:

Next steps

OpenTelemetry Log Event

A closer look at the OTTL integration’s statements reveals that the raw log message is replaced by the parsed fields. In other words, the configuration transforms the body log field* from a string into a structured map of key-value pairs, as seen in “set(body, ExtractGrokPatterns(body,...)”. This approach is based on treating each NGINX Ingress Controller log entry as an OpenTelemetry Event—a specialized type of LogRecord. Events are OpenTelemetry’s standardized semantic formatting for LogRecords, containing an “event.name” attribute which defines the structure of the body field. An NGINX Ingress Controller log record aligns well with the OpenTelemetry Event data model. It follows a structured format and clearly distinguishes between two event types: access logs and error logs. There is an ongoing PR to incorporate the NGINX Ingress controller log into the OpenTelemetry semantic convention: https://github.com/open-telemetry/semantic-conventions/pull/982

Operating system breakdown

Each controller log contains the source UserAgent, from which the integration extracts the browser that originated the request. This information is valuable for understanding user access patterns, as it provides insights into the types of browsers commonly interacting with your services. Additionally, an ongoing pull request into OTTL aims to extend this functionality by extracting operating system (OS) details as well, providing even deeper insights into the environments interacting with the NGINX Ingress Controller.

Configuration encapsulation

Setting up the configuration for the NGINX Ingress Controller integration can be somewhat tedious, as it involves adding several complex processor configurations to the existing collector pipelines. This process can quickly become cumbersome, especially for non-expert users or in cases where the collector configuration is already quite complex. In an ideal scenario, users would simply reference a pre-defined integration configuration, and the collector would automatically "unwrap" all the necessary components into the corresponding pipelines. This would significantly simplify the setup process, making it more accessible and reducing the risk of misconfigurations. To address this, there is a RFC (Request for Comments) proposing support for shareable, modular configurations within the OpenTelemetry Collector. This feature would allow users to easily collect signals from specific services or applications by referencing modular configurations, streamlining the setup and enhancing usability for complex scenarios.

*The OpenTelemetry community is currently discussing whether structured body-extracted information should be stored in the attributes or body field. For details, see this ongoing issue.

This product includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com

DevOps engineers continuously optimize software delivery, while SRE teams act as the stewards of application reliability, scalability, and top-tier performance. The challenge? These teams require a cutting-edge observability solution, one that encompasses full-stack insights, empowering them to rapidly manage, monitor, and rectify potential disruptions before they culminate into operational challenges.

Observability in our modern distributed software ecosystem goes beyond mere monitoring — it demands limitless data collection, precision in processing, and the correlation of this data into actionable insights. However, the road to achieving this holistic view is paved with obstacles, from navigating version incompatibilities to wrestling with restrictive proprietary code.

Enter OpenTelemetry (OTel), with the following benefits for those who adopt it:

• Escape vendor constraints with OTel, freeing yourself from vendor lock-in and ensuring top-notch observability.
• See the harmony of unified logs, metrics, and traces come together to provide a complete system view.
• Improve your application oversight through richer and enhanced instrumentations.
• Embrace the benefits of backward compatibility to protect your prior instrumentation investments.
• Embark on the OpenTelemetry journey with an easy learning curve, simplifying onboarding and scalability.
• Rely on a proven, future-ready standard to boost your confidence in every investment.

In this blog, we will explore how you can use automatic instrumentation in your Go application using Docker, without the need to refactor any part of your application code. We will use an application called Elastiflix, which helps highlight auto-instrumentation in a simple way.

Application, prerequisites, and config

The application that we use for this blog is called Elastiflix, a movie-streaming application. It consists of several micro-services written in .NET, NodeJS, Go, and Python.

Before we instrument our sample application, we will first need to understand how Elastic can receive the telemetry data.

All of Elastic Observability’s APM capabilities are available with OTel data. Some of these include:

• Service maps
• Service details (latency, throughput, failed transactions)
• Dependencies between services, distributed tracing
• Transactions (traces)
• Machine learning (ML) correlations
• Log correlation

In addition to Elastic’s APM and a unified view of the telemetry data, you will also be able to use Elastic’s powerful machine learning capabilities to reduce the analysis, and alerting to help reduce MTTR.

Prerequisites

• An Elastic Cloud account — sign up now.
• A clone of the Elastiflix demo application, or your own Go application
• Basic understanding of Docker — potentially install Docker Desktop
• Basic understanding of Go

View the example source code

The full source code, including the Dockerfile used in this blog, can be found on GitHub.

The following steps will show you how to instrument this application and run it on the command line or in Docker. If you are interested in a more complete OTel example, take a look at the docker-compose file here, which will bring up the full project.

Step-by-step guide

Step 0. Log in to your Elastic Cloud account

This blog assumes you have an Elastic Cloud account — if not, follow the instructions to get started on Elastic Cloud.

Step 1. Run the Docker Image with auto-instrumentation

We are going to use automatic instrumentation with the Go service from the Elastiflix demo application.

We will be using the following service from Elastiflix:

Elastiflix/go-favorite

Per the OpenTelemetry Automatic Instrumentation for Go documentation, you will configure the application to be auto-instrumented using docker-compose.

As specified in the OTEL Go documentation, we will use environment variables and pass in the configuration values to enable it to connect with Elastic Observability’s APM server.

Because Elastic accepts OTLP natively, we just need to provide the Endpoint and authentication where the OTEL Exporter needs to send the data, as well as some other environment variables.

Getting Elastic Cloud variables You can copy the endpoints and token from Kibana under the path /app/apm/onboarding?agent=openTelemetry.

You will need to copy the following environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT
OTEL_EXPORTER_OTLP_HEADERS

Update the docker-compose.yml file at the top of the Elastiflix repository, adding a go-auto service and updating the favorite-go one:

  favorite-go:
    build: go-favorite/.
    image: docker.elastic.co/demos/workshop/observability/elastiflix-go-favorite:${ELASTIC_VERSION}-${BUILD_NUMBER}
    depends_on:
      - redis
    networks:
      - app-network
    ports:
      - "5001:5000"
    environment:
      - REDIS_HOST=redis
      - TOGGLE_SERVICE_DELAY=${TOGGLE_SERVICE_DELAY:-0}
      - TOGGLE_CANARY_DELAY=${TOGGLE_CANARY_DELAY:-0}
      - TOGGLE_CANARY_FAILURE=${TOGGLE_CANARY_FAILURE:-0}
    volumes:
      - favorite-go:/app
  go-auto:
    image: otel/autoinstrumentation-go
    privileged: true
    pid: "host"
    networks:
      - app-network
    environment:
      OTEL_EXPORTER_OTLP_ENDPOINT: "REPLACE WITH OTEL_EXPORTER_OTLP_ENDPOINT"
      OTEL_EXPORTER_OTLP_HEADERS: "REPLACE WITH OTEL_EXPORTER_OTLP_HEADERS"
      OTEL_GO_AUTO_TARGET_EXE: "/app/main"
      OTEL_SERVICE_NAME: "go-favorite"
      OTEL_PROPAGATORS: "tracecontext,baggage"
    volumes:
      - favorite-go:/app
      - /proc:/host/proc

And, at the bottom of the file:

volumes:
  favorite-go:
networks:
  app-network:
    driver: bridge

Finally, in the configuration for the main node app, you will want to tell Elastiflix to call the Go favorites app by replacing the line:

environment:
  - API_ENDPOINT_FAVORITES=favorite-java:5000

with:

environment:
  - API_ENDPOINT_FAVORITES=favorite-go:5000

Step 3: Explore traces and logs in Elastic APM

Once you have this up and running, you can ping the endpoint for your instrumented service (in our case, this is /favorites), and you should see the app appear in Elastic APM, as shown below:

It will begin by tracking throughput and latency critical metrics for SREs to pay attention to.

Digging in, we can see an overview of all our Transactions.

And look at specific transactions:

This gives you complete visibility across metrics, and traces!

Summary

With this Dockerfile, you've transformed your simple Go application into one that's automatically instrumented with OpenTelemetry. This will aid greatly in understanding application performance, tracing errors, and gaining insights into how users interact with your software.

Remember, observability is a crucial aspect of modern application development, especially in distributed systems. With tools like OpenTelemetry, understanding complex systems becomes a tad bit easier.

In this blog, we discussed the following:

• How to auto-instrument Go with OpenTelemetry.
• Using standard commands in a Docker file, auto-instrumentation was done efficiently and without adding code in multiple places enabling manageability.
• Using OpenTelemetry and its support for multiple languages, DevOps and SRE teams can auto-instrument their applications with ease gaining immediate insights into the health of the entire application stack and reduce mean time to resolution (MTTR).

Since Elastic can support a mix of methods for ingesting data, whether it be using auto-instrumentation of open-source OpenTelemetry or manual instrumentation with its native APM agents, you can plan your migration to OTel by focusing on a few applications first and then using OpenTelemety across your applications later on in a manner that best fits your business needs.

Developer resources:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation
• Go: Auto-instrumentation Manual-instrumentation
• Best practices for instrumenting OpenTelemetry

General configuration and use case resources:

• Independence with OpenTelemetry on Elastic
• Modern observability and security on Kubernetes with Elastic and OpenTelemetry
• 3 models for logging with OpenTelemetry and Elastic
• Adding free and open Elastic APM as part of your Elastic Observability deployment
• Capturing custom metrics through OpenTelemetry API in code with Elastic
• Future-proof your observability platform with OpenTelemetry and Elastic
• Elastic Observability: Built for open technologies like Kubernetes, OpenTelemetry, Prometheus, Istio, and more

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the auto-instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all._

Observability across the entire stack of modern distributed applications requires data collection, processing, and correlation often in the form of dashboards. Ingesting all system data requires installing agents across stacks, frameworks, and providers — a process that can be challenging and time-consuming for teams who have to deal with version changes, compatibility issues, and proprietary code that doesn't scale as systems change.

Thanks to OpenTelemetry (OTel), DevOps and SRE teams now have a standard way to collect and send data that doesn't rely on proprietary code and have a large support community reducing vendor lock-in.

In a previous blog, we also reviewed how to use the OpenTelemetry demo and connect it to Elastic^®, as well as some of Elastic’s capabilities with OpenTelemetry and Kubernetes.

In this blog, we will show how to use automatic instrumentation for OpenTelemetry with the Node.js service of our application called Elastiflix, which helps highlight auto-instrumentation in a simple way.

The beauty of this is that there is no need for the otel-collector! This setup enables you to slowly and easily migrate an application to OTel with Elastic according to a timeline that best fits your business.

Application, prerequisites, and config

The application that we use for this blog is called Elastiflix, a movie streaming application. It consists of several micro-services written in .NET, NodeJS, Go, and Python.

Before we instrument our sample application, we will first need to understand how Elastic can receive the telemetry data.

All of Elastic Observability’s APM capabilities are available with OTel data. Some of these include:

• Service maps
• Service details (latency, throughput, failed transactions)
• Dependencies between services, distributed tracing
• Transactions (traces)
• Machine learning (ML) correlations
• Log correlation

In addition to Elastic’s APM and a unified view of the telemetry data, you will also be able to use Elastic’s powerful machine learning capabilities to reduce the analysis, and alerting to help reduce MTTR.

Prerequisites

• An Elastic Cloud account — sign up now
• A clone of the Elastiflix demo application, or your own Node.js application
• Basic understanding of Docker — potentially install Docker Desktop
• Basic understanding of Node.js

View the example source code

The full source code, including the Dockerfile used in this blog, can be found on GitHub. The repository also contains the same application without instrumentation. This allows you to compare each file and see the differences.

The following steps will show you how to instrument this application and run it on the command line or in Docker. If you are interested in a more complete OTel example, take a look at the docker-compose file here, which will bring up the full project.

Step-by-step guide

Step 0. Log in to your Elastic Cloud account

This blog assumes you have an Elastic Cloud account — if not, follow the instructions to get started on Elastic Cloud.

Step 1. Configure auto-instrumentation for the Node.js Service

We are going to use automatic instrumentation with Node.js service from the Elastiflix demo application.

We will be using the following service from Elastiflix:

Elastiflix/node-server-otel-manual

Per the OpenTelemetry JavaScript documentation and @open-telemetry/auto-instrumentions-node documentation, you will simply install the appropriate node packages using npm.

npm install --save @opentelemetry/api
npm install --save @opentelemetry/auto-instrumentations-node

If you are running the Node.js service on the command line, then here is how you can run auto-instrument with Node.js.

node --require '@opentelemetry/auto-instrumentations-node/register' app.js

For our application, we do this as part of the Dockerfile.

Dockerfile

FROM node:14

WORKDIR /app

COPY ["package.json", "./"]
RUN ls
RUN npm install --production
COPY . .

RUN npm install --save @opentelemetry/api
RUN npm install --save @opentelemetry/auto-instrumentations-node


EXPOSE 3001

CMD ["node", "--require", "@opentelemetry/auto-instrumentations-node/register", "index.js"]

Step 2. Running the Docker image with environment variables

As specified in the OTEL documentation, we will use environment variables and pass in the configuration values to enable it to connect with Elastic Observability’s APM server.

Because Elastic accepts OTLP natively, we just need to provide the Endpoint and authentication where the OTEL Exporter needs to send the data, as well as some other environment variables.

Getting Elastic Cloud variables
You can copy the endpoints and token from Kibana^® under the path /app/home#/tutorial/apm.

You will need to copy the following environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT
OTEL_EXPORTER_OTLP_HEADERS

Build the image

docker build -t  node-otel-auto-image .

Run the image

docker run \
       -e OTEL_EXPORTER_OTLP_ENDPOINT="<REPLACE WITH OTEL_EXPORTER_OTLP_ENDPOINT>" \
       -e OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <REPLACE WITH TOKEN>" \
       -e OTEL_RESOURCE_ATTRIBUTES="service.version=1.0,deployment.environment=production" \
       -e OTEL_SERVICE_NAME="node-server-otel-auto" \
       -p 3001:3001 \
       node-server-otel-auto

You can now issue a few requests in order to generate trace data. Note that these requests are expected to return an error, as this service relies on some downstream services that you may not have running on your machine.

curl localhost:3001/api/login
curl localhost:3001/api/favorites

# or alternatively issue a request every second

while true; do curl "localhost:3001/api/favorites"; sleep 1; done;

Step 3: Explore traces, metrics, and logs in Elastic APM

Exploring the Services section in Elastic APM, you’ll see the Node service displayed.

Clicking on the node-server-otel-auto service, you can see that it is ingesting telemetry data using OpenTelemetry.

Summary

In this blog, we discussed the following:

• How to auto-instrument Node.js with OpenTelemetry
• Using standard commands in a Dockerfile, auto-instrumentation was done efficiently and without adding code in multiple places enabling manageability

Since Elastic can support a mix of methods for ingesting data, whether it be using auto-instrumentation of open-source OpenTelemetry or manual instrumentation with its native APM agents, you can plan your migration to OTel by focusing on a few applications first and then using OpenTelemety across your applications later on in a manner that best fits your business needs.

Developer resources:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation
• Go: Manual-instrumentation
• Best practices for instrumenting OpenTelemetry

General configuration and use case resources:

• Independence with OpenTelemetry on Elastic
• Modern observability and security on Kubernetes with Elastic and OpenTelemetry
• 3 models for logging with OpenTelemetry and Elastic
• Adding free and open Elastic APM as part of your Elastic Observability deployment
• Capturing custom metrics through OpenTelemetry API in code with Elastic
• Future-proof your observability platform with OpenTelemetry and Elastic
• Elastic Observability: Built for open technologies like Kubernetes, OpenTelemetry, Prometheus, Istio, and more

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the auto-instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

DevOps engineers continuously optimize software delivery, while SRE teams act as the stewards of application reliability, scalability, and top-tier performance. The challenge? These teams require a cutting-edge observability solution, one that encompasses full-stack insights, empowering them to rapidly manage, monitor, and rectify potential disruptions before they culminate into operational challenges.

Observability in our modern distributed software ecosystem goes beyond mere monitoring — it demands limitless data collection, precision in processing, and the correlation of this data into actionable insights. However, the road to achieving this holistic view is paved with obstacles, from navigating version incompatibilities to wrestling with restrictive proprietary code.

Enter OpenTelemetry (OTel), with the following benefits for those who adopt it:

• Escape vendor constraints with OTel, freeing yourself from vendor lock-in and ensuring top-notch observability.
• See the harmony of unified logs, metrics, and traces come together to provide a complete system view.
• Improve your application oversight through richer and enhanced instrumentations.
• Embrace the benefits of backward compatibility to protect your prior instrumentation investments.
• Embark on the OpenTelemetry journey with an easy learning curve, simplifying onboarding and scalability.
• Rely on a proven, future-ready standard to boost your confidence in every investment.

In this blog, we will explore how you can use automatic instrumentation in your Java application using Docker, without the need to refactor any part of your application code. We will use an application called Elastiflix, which helps highlight auto-instrumentation in a simple way.

Application, prerequisites, and config

The application that we use for this blog is called Elastiflix, a movie-streaming application. It consists of several micro-services written in .NET, NodeJS, Go, and Python.

Before we instrument our sample application, we will first need to understand how Elastic can receive the telemetry data.

All of Elastic Observability’s APM capabilities are available with OTel data. Some of these include:

• Service maps
• Service details (latency, throughput, failed transactions)
• Dependencies between services, distributed tracing
• Transactions (traces)
• Machine learning (ML) correlations
• Log correlation

In addition to Elastic’s APM and a unified view of the telemetry data, you will also be able to use Elastic’s powerful machine learning capabilities to reduce the analysis, and alerting to help reduce MTTR.

Prerequisites

• An Elastic Cloud account — sign up now.
• A clone of the Elastiflix demo application, or your own Java application
• Basic understanding of Docker — potentially install Docker Desktop
• Basic understanding of Java

View the example source code

The full source code, including the Dockerfile used in this blog, can be found on GitHub. The repository also contains the same application without instrumentation. This allows you to compare each file and see the differences.

The following steps will show you how to instrument this application and run it on the command line or in Docker. If you are interested in a more complete OTel example, take a look at the docker-compose file here, which will bring up the full project.

Step-by-step guide

Step 0. Log in to your Elastic Cloud account

This blog assumes you have an Elastic Cloud account — if not, follow the instructions to get started on Elastic Cloud.

Step 1. Configure auto-instrumentation for the Java service

We are going to use automatic instrumentation with Java service from the Elastiflix demo application.

We will be using the following service from Elastiflix:

Elastiflix/java-favorite-otel-auto

Per the OpenTelemetry Automatic Instrumentation for Java documentation and documentation, you will simply install the appropriate Java packages.

Create a local OTel directory to download the OpenTelemetry Java agent. Download opentelemetry-javaagent.jar.

>mkdir /otel

>curl -L https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar –output /otel/opentelemetry-javaagent.jar

If you are going to run the service on the command line, then you can use the following command:

java -javaagent:/otel/opentelemetry-javaagent.jar \
-jar /usr/src/app/target/favorite-0.0.1-SNAPSHOT.jar --server.port=5000

For our application, we will do this as part of the Dockerfile.

Dockerfile

Start with a base image containing Java runtime
FROM maven:3.8.2-openjdk-17-slim as build

# Make port 8080 available to the world outside this container
EXPOSE 5000

# Change to the app directory
WORKDIR /usr/src/app

# Copy the local code to the container
COPY . .

# Build the application
RUN mvn clean install

USER root
RUN apt-get update && apt-get install -y zip curl
RUN mkdir /otel
RUN curl -L -o /otel/opentelemetry-javaagent.jar https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v1.28.0/opentelemetry-javaagent.jar

COPY start.sh /start.sh
RUN chmod +x /start.sh

ENTRYPOINT ["/start.sh"]

Step 2. Running the Docker Image with environment variables

As specified in the OTEL Java documentation, we will use environment variables and pass in the configuration values to enable it to connect with Elastic Observability’s APM server.

Because Elastic accepts OTLP natively, we just need to provide the Endpoint and authentication where the OTEL Exporter needs to send the data, as well as some other environment variables.

Getting Elastic Cloud variables
You can copy the endpoints and token from Kibana under the path /app/home#/tutorial/apm.

You will need to copy the following environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT
OTEL_EXPORTER_OTLP_HEADERS

Build the Docker image

docker build -t java-otel-auto-image .

Run the Docker image

docker run \
       -e OTEL_EXPORTER_OTLP_ENDPOINT="REPLACE WITH OTEL_EXPORTER_OTLP_ENDPOINT" \
       -e ELASTIC_APM_SECRET_TOKEN="REPLACE WITH THE BIT AFTER Authorization=Bearer " \
       -e OTEL_RESOURCE_ATTRIBUTES="service.version=1.0,deployment.environment=production" \
       -e OTEL_SERVICE_NAME="java-favorite-otel-auto" \
       -p 5000:5000 \
       java-otel-auto-image

You can now issue a few requests in order to generate trace data. Note that these requests are expected to return an error, as this service relies on a connection to Redis that you don’t currently have running. As mentioned before, you can find a more complete example using docker-compose here.

curl localhost:5000/favorites

# or alternatively issue a request every second

while true; do curl "localhost:5000/favorites"; sleep 1; done;

Step 3: Explore traces and logs in Elastic APM

Once you have this up and running, you can ping the endpoint for your instrumented service (in our case, this is /favorites), and you should see the app appear in Elastic APM, as shown below:

It will begin by tracking throughput and latency critical metrics for SREs to pay attention to.

Digging in, we can see an overview of all our Transactions.

And look at specific transactions:

Click on Logs, and we see that logs are also brought over. The OTel Agent will automatically bring in logs and correlate them with traces for you:

This gives you complete visibility across logs, metrics, and traces!

Basic concepts: How APM works with Java

Before we continue, let's first understand a few basic concepts and terms.

• Java Agent: This is a tool that can be used to instrument (or modify) the bytecode of class files in the Java Virtual Machine (JVM). Java agents are used for many purposes like performance monitoring, logging, security, and more.
• Bytecode: This is the intermediary code generated by the Java compiler from your Java source code. This code is interpreted or compiled on the fly by the JVM to produce machine code that can be executed.
• Byte Buddy: Byte Buddy is a code generation and manipulation library for Java. It is used to create, modify, or adapt Java classes at runtime. In the context of a Java Agent, Byte Buddy provides a powerful and flexible way to modify bytecode. Both the Elastic APM Agent and the OpenTelemetry Agent use Byte Buddy under the covers.

Now, let's talk about how automatic instrumentation works with Byte Buddy:

Automatic instrumentation is the process by which an agent modifies the bytecode of your application's classes, often to insert monitoring code. The agent doesn't modify the source code directly, but rather the bytecode that is loaded into the JVM. This is done while the JVM is loading the classes, so the modifications are in effect during runtime.

Here's a simplified explanation of the process:

1. Start the JVM with the agent: When starting your Java application, you specify the Java agent with the -javaagent command line option. This instructs the JVM to load your agent before the main method of your application is invoked. At this point, the agent has the opportunity to set up class transformers.

2. Register a class file transformer with Byte Buddy: Your agent will register a class file transformer with Byte Buddy. A transformer is a piece of code that is invoked every time a class is loaded into the JVM. This transformer receives the bytecode of the class, and it can modify this bytecode before the class is actually used.

3. Transform the bytecode: When your transformer is invoked, it will use Byte Buddy's API to modify the bytecode. Byte Buddy allows you to specify your transformations in a high-level, expressive way rather than manually writing complex bytecode. For example, you could specify a certain class and method within that class that you want to instrument and provide an "interceptor" that will add new behavior to that method.

4. Use the transformed classes: Once the agent has set up its transformers, the JVM continues to load classes as usual. Each time a class is loaded, your transformers are invoked, allowing them to modify the bytecode. Your application then uses these transformed classes as if they were the original ones, but they now have the extra behavior that you've injected through your interceptor.

In essence, automatic instrumentation with Byte Buddy is about modifying the behavior of your Java classes at runtime, without needing to alter the source code directly. This is especially useful for cross-cutting concerns like logging, monitoring, or security, as it allows you to centralize this code in your Java Agent, rather than scattering it throughout your application.

Summary

With this Dockerfile, you've transformed your simple Java application into one that's automatically instrumented with OpenTelemetry. This will aid greatly in understanding application performance, tracing errors, and gaining insights into how users interact with your software.

Remember, observability is a crucial aspect of modern application development, especially in distributed systems. With tools like OpenTelemetry, understanding complex systems becomes a tad bit easier.

In this blog, we discussed the following:

• How to auto-instrument Java with OpenTelemetry.
• Using standard commands in a Docker file, auto-instrumentation was done efficiently and without adding code in multiple places enabling manageability.
• Using OpenTelemetry and its support for multiple languages, DevOps and SRE teams can auto-instrument their applications with ease gaining immediate insights into the health of the entire application stack and reduce mean time to resolution (MTTR).

Since Elastic can support a mix of methods for ingesting data, whether it be using auto-instrumentation of open-source OpenTelemetry or manual instrumentation with its native APM agents, you can plan your migration to OTel by focusing on a few applications first and then using OpenTelemety across your applications later on in a manner that best fits your business needs.

Developer resources:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation
• Go: Manual-instrumentation
• Best practices for instrumenting OpenTelemetry

General configuration and use case resources:

• Independence with OpenTelemetry on Elastic
• Modern observability and security on Kubernetes with Elastic and OpenTelemetry
• 3 models for logging with OpenTelemetry and Elastic
• Adding free and open Elastic APM as part of your Elastic Observability deployment
• Capturing custom metrics through OpenTelemetry API in code with Elastic
• Future-proof your observability platform with OpenTelemetry and Elastic
• Elastic Observability: Built for open technologies like Kubernetes, OpenTelemetry, Prometheus, Istio, and more

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the auto-instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

DevOps engineers continuously optimize software delivery, while SRE teams act as the stewards of application reliability, scalability, and top-tier performance. The challenge? These teams require a cutting-edge observability solution, one that encompasses full-stack insights, empowering them to rapidly manage, monitor, and rectify potential disruptions before they culminate into operational challenges.

Observability in our modern distributed software ecosystem goes beyond mere monitoring — it demands limitless data collection, precision in processing, and the correlation of this data into actionable insights. However, the road to achieving this holistic view is paved with obstacles, from navigating version incompatibilities to wrestling with restrictive proprietary code.

Enter OpenTelemetry (OTel), with the following benefits for those who adopt it:

• Escape vendor constraints with OTel, freeing yourself from vendor lock-in and ensuring top-notch observability.
• See the harmony of unified logs, metrics, and traces come together to provide a complete system view.
• Improve your application oversight through richer and enhanced instrumentations.
• Embrace the benefits of backward compatibility to protect your prior instrumentation investments.
• Embark on the OpenTelemetry journey with an easy learning curve, simplifying onboarding and scalability.
• Rely on a proven, future-ready standard to boost your confidence in every investment.
• Explore manual instrumentation, enabling customized data collection to fit your unique needs.
• Ensure monitoring consistency across layers with a standardized observability data framework.
• Decouple development from operations, driving peak efficiency for both.

Given this context, OpenTelemetry emerges as an unmatched observability solution for cloud-native software, seamlessly enabling tracing, monitoring, and debugging. One of its strengths is the ability to auto-instrument applications, allowing developers the luxury of collecting invaluable telemetry without delving into code modifications.

In this post, we will dive into the methodology to instrument a .NET application using Docker, blending the best of both worlds: powerful observability without the code hassles.

What's covered?

• How APM works with .NET using CLR Profiler functionality
• Creating a Docker image for a .NET application with the OpenTelemetry instrumentation baked in
• Installing and running the OpenTelemetry .NET Profiler for automatic instrumentation

How APM works with .NET using CLR Profiler functionality

Before we delve into the details, let's clear up some confusion around .NET Profilers and CPU Profilers like Elastic^®’s Universal Profiling tool — we don’t want to get these two things mixed up, as they have very different purposes.

When discussing profiling tools, especially in the context of .NET, it's not uncommon to encounter confusion between a ".NET profiler" and a "CPU profiler." Though both are used to diagnose and optimize applications, they serve different primary purposes and operate at different levels. Let's clarify the distinction:

.NET Profiler

1. Scope: Specifically targets .NET applications. It is designed to work with the .NET runtime (i.e., the Common Language Runtime (CLR)).

2. Functionality:

3. Use cases:

CPU Profiler

1. Scope: More general than a .NET profiler. It can profile any application, irrespective of the language or runtime, as long as it runs on the CPU being profiled.

2. Functionality:

3. Use cases:

While both .NET profilers and CPU profilers aid in optimizing and diagnosing application performance, their approach and depth differ. A .NET profiler offers deep insights specifically into the .NET ecosystem, allowing for fine-grained analysis and instrumentation. In contrast, a CPU profiler provides a broader view, focusing on CPU usage patterns across any application, regardless of its development platform.

It's worth noting that for comprehensive profiling of a .NET application, you might use both: the .NET profiler to understand code-level behaviors specific to .NET and the CPU profiler to get an overview of CPU resource utilization.

Now that we've cleared that up, let's focus on the .NET Profiler, which we are discussing in this blog for automatic instrumentation of .NET applications. First, let's familiarize ourselves with some foundational concepts and terminologies relevant to a .NET Profiler:

• CLR (Common Language Runtime): CLR is a core component of the .NET framework, acting as the execution engine for .NET apps. It provides key services like memory management, exception handling, and type safety.
• Profiler API:.NET provides a set of APIs for profiling applications. These APIs let tools and developers monitor or manipulate .NET applications during runtime.
• IL (Intermediate Language): After compiling, .NET source code turns into IL, a low-level, platform-agnostic representation. This IL code is then compiled just-in-time (JIT) into machine code by the CLR during application execution.
• JIT compilation: JIT stands for just-in-time. In .NET, the CLR compiles IL to native code just before its execution.

Now, let's explore how automatic instrumentation works using CLR Profiler.

Automatic instrumentation in .NET, much like Java's bytecode instrumentation, revolves around modifying the behavior of your application's methods during runtime, without changing the actual source code.

Here’s a step-by-step breakdown:

1. Attach the profiler: When launching your .NET application, you'll have to specify to load the profiler. The CLR checks for the presence of a profiler by reading environment variables. If it finds one, the CLR initializes the profiler before any user code is executed.

2. Use Profiler API to monitor events: The Profiler API allows a profiler to monitor various events. For instance, method JIT compilation events can be tracked. When a method is about to be JIT compiled, the profiler gets notified.

3. Manipulate IL code: Upon getting notified of a JIT compilation, the profiler can manipulate the IL code of the method. Using the Profiler API, the profiler can insert, delete, or replace IL instructions. This is analogous to how Java agents modify bytecode. For example, if you want to measure a method's execution time, you'd modify the IL to insert calls to start and stop a timer at the beginning and end of the method, respectively.

4. Execution of transformed code: Once the IL has been modified, the JIT compiler will translate it into machine code. The application will then execute this machine code, which includes the additions made by the profiler.

5. Gather and report data: The added instrumentation can collect various data, such as method execution times or call counts. This data can then be relayed to an application performance management (APM) tool, which can provide insights, visualizations, and alerts based on the data.

In essence, automatic instrumentation with CLR Profiler is about modifying the behavior of your .NET methods at runtime. This is invaluable for monitoring, diagnosing, and fine-tuning the performance of .NET applications without intruding on the application's actual source code.

Prerequisites

• A basic understanding of Docker and .NET
• Elastic Cloud
• Docker installed on your machine (we recommend docker desktop)

View the example source code

The full source code, including the Dockerfile used in this blog, can be found on GitHub. The repository also contains the same application without instrumentation. This allows you to compare each file and see the differences.

The following steps will show you how to instrument this application and run it on the command line or in Docker. If you are interested in a more complete OTel example, take a look at the docker-compose file here, which will bring up the full project.

Step-by-step guide

This blog assumes you have an Elastic Cloud account — if not, follow the instructions to get started on Elastic Cloud.

Step 1. Base image setup

Start with the .NET runtime image for the base layer of our Dockerfile:

FROM ${ARCH}mcr.microsoft.com/dotnet/aspnet:7.0. AS base
WORKDIR /app
EXPOSE 8000

Here, we're setting up the application's runtime environment.

Step 2. Building the .NET application

This feature of Docker is just the best. Here, we compile our .NET application using the SDK image. In the bad old days, we used to build on a different platform and then put the compiled code into the Docker container. This way, we are much more confident our build will replicate from a developer’s desktop and into production by using Docker all the way through.

FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:8.0-preview AS build
ARG TARGETPLATFORM

WORKDIR /src
COPY ["login.csproj", "./"]
RUN dotnet restore "./login.csproj"
COPY . .
WORKDIR "/src/."
RUN dotnet build "login.csproj" -c Release -o /app/build

This section ensures that our .NET code is properly restored and compiled.

Step 3. Publishing the application

Once built, we'll publish the app:

FROM build AS publish
RUN dotnet publish "login.csproj" -c Release -o /app/publish

Step 4. Preparing the final image

Now, let's set up the final runtime image:

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish

Step 5. Installing OpenTelemetry

We'll install dependencies and download the OpenTelemetry auto-instrumentation script:

RUN apt-get update && apt-get install -y zip curl
RUN mkdir /otel
RUN curl -L -o /otel/otel-dotnet-install.sh https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/releases/download/v0.7.0/otel-dotnet-auto-install.sh
RUN chmod +x /otel/otel-dotnet-install.sh

Step 6. Configure OpenTelemetry

Designate where OpenTelemetry should reside and execute the installation script. Note that the ENV OTEL_DOTNET_AUTO_HOME is required as the script looks for it:

ENV OTEL_DOTNET_AUTO_HOME=/otel
RUN /bin/bash /otel/otel-dotnet-install.sh

Step 7. Additional configuration

Make sure the auto-instrumentation and platform detection scripts are executable and run the platform detection script.

COPY platform-detection.sh /otel/
RUN chmod +x /otel/instrument.sh
RUN chmod +x /otel/platform-detection.sh && /otel/platform-detection.sh

This platform detection script will check if the Docker build is for ARM64 and implement a workaround to get the OpenTelemetry instrumentation to work on MacOS. If you happen to be running locally on MacOS M1 or M2 processors, you will be grateful for this script.

Step 8. Entry point setup

Lastly, set the Docker image's entry point to both source the OpenTelemetry instrumentation, which sets up the environment variables required to bootstrap the .NET Profiler, and then we start our .NET application:

ENTRYPOINT ["/bin/bash", "-c", "source /otel/instrument.sh && dotnet login.dll"]

Step 9. Running the Docker image with environment variables

To build and run the Docker image, you'd typically follow these steps:

Build the Docker image

First, you'd want to build the Docker image from your Dockerfile. Let's assume the Dockerfile is in the current directory, and you'd like to name/tag your image dotnet-login-otel-image.

docker build -t dotnet-login-otel-image .

Run the Docker image

After building the image, you'd run it with the specified environment variables. For this, the docker run command is used with the -e flag for each environment variable.

docker run \
       -e OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer ${ELASTIC_APM_SECRET_TOKEN}" \
       -e OTEL_EXPORTER_OTLP_ENDPOINT="${ELASTIC_APM_SERVER_URL}" \
       -e OTEL_METRICS_EXPORTER="otlp" \
       -e OTEL_RESOURCE_ATTRIBUTES="service.version=1.0,deployment.environment=production" \
       -e OTEL_SERVICE_NAME="dotnet-login-otel-auto" \
       -e OTEL_TRACES_EXPORTER="otlp" \
       dotnet-login-otel-image

Make sure that ${ELASTIC_APM_SECRET_TOKEN} and ${ELASTIC_APM_SERVER_URL} are set in your shell environment, and replace them with their actual values from the cloud as shown below.
Getting Elastic Cloud variables

You can copy the endpoints and token from Kibana^® under the path /app/home#/tutorial/apm.

You can also use an environment file with docker run --env-file to make the command less verbose if you have multiple environment variables.

Once you have this up and running, you can ping the endpoint for your instrumented service (in our case, this is /login), and you should see the app appear in Elastic APM, as shown below:

It will begin by tracking throughput and latency critical metrics for SREs to pay attention to.

Digging in, we can see an overview of all our Transactions.

And look at specific transactions:

There is clearly an outlier here, where one transaction took over 200ms. This is likely to be due to the .NET CLR warming up. Click on Logs , and we see that logs are also brought over. The OTel Agent will automatically bring in logs and correlate them with traces for you:

Wrapping up

With this Dockerfile, you've transformed your simple .NET application into one that's automatically instrumented with OpenTelemetry. This will aid greatly in understanding application performance, tracing errors, and gaining insights into how users interact with your software.

Remember, observability is a crucial aspect of modern application development, especially in distributed systems. With tools like OpenTelemetry, understanding complex systems becomes a tad bit easier.

In this blog, we discussed the following:

• How to auto-instrument .NET with OpenTelemetry.
• Using standard commands in a Docker file, auto-instrumentation was done efficiently and without adding code in multiple places enabling manageability.
• Using OpenTelemetry and its support for multiple languages, DevOps and SRE teams can auto-instrument their applications with ease gaining immediate insights into the health of the entire application stack and reduce mean time to resolution (MTTR).

Since Elastic can support a mix of methods for ingesting data, whether it be using auto-instrumentation of open-source OpenTelemetry or manual instrumentation with its native APM agents, you can plan your migration to OTel by focusing on a few applications first and then using OpenTelemety across your applications later on in a manner that best fits your business needs.

Developer resources:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation
• Go: Manual-instrumentation
• Best practices for instrumenting OpenTelemetry

General configuration and use case resources:

• Independence with OpenTelemetry on Elastic
• Modern observability and security on Kubernetes with Elastic and OpenTelemetry
• 3 models for logging with OpenTelemetry and Elastic
• Adding free and open Elastic APM as part of your Elastic Observability deployment
• Capturing custom metrics through OpenTelemetry API in code with Elastic
• Future-proof your observability platform with OpenTelemetry and Elastic
• Elastic Observability: Built for open technologies like Kubernetes, OpenTelemetry, Prometheus, Istio, and more

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the auto-instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Observability across the entire stack of modern distributed applications requires data collection, processing, and correlation often in the form of dashboards. Ingesting all system data requires installing agents across stacks, frameworks, and providers — a process that can be challenging and time-consuming for teams who have to deal with version changes, compatibility issues, and proprietary code that doesn't scale as systems change.

Thanks to OpenTelemetry (OTel), DevOps and SRE teams now have a standard way to collect and send data that doesn't rely on proprietary code and has a large support community reducing vendor lock-in.

In a previous blog, we also reviewed how to use the OpenTelemetry demo and connect it to Elastic^®, as well as some of Elastic’s capabilities with OpenTelemetry visualizations and Kubernetes.

In this blog, we will show how to use automatic instrumentation for OpenTelemetry with the Python service of our application called Elastiflix, which helps highlight auto-instrumentation in a simple way.

The beauty of this is that there is no need for the otel-collector! This setup enables you to slowly and easily migrate an application to OTel with Elastic according to a timeline that best fits your business.

Application, prerequisites, and config

The application that we use for this blog is called Elastiflix, a movie-streaming application. It consists of several micro-services written in .NET, NodeJS, Go, and Python.

Before we instrument our sample application, we will first need to understand how Elastic can receive the telemetry data.

All of Elastic Observability’s APM capabilities are available with OTel data. Some of these include:

• Service maps
• Service details (latency, throughput, failed transactions)
• Dependencies between services, distributed tracing
• Transactions (traces)
• Machine learning (ML) correlations
• Log correlation

In addition to Elastic’s APM and a unified view of the telemetry data, you will also be able to use Elastic’s powerful machine learning capabilities to reduce the analysis, and alerting to help reduce MTTR.

Prerequisites

• An Elastic Cloud account — sign up now
• A clone of the Elastiflix demo application, or your own Python application
• Basic understanding of Docker — potentially install Docker Desktop
• Basic understanding of Python

View the example source code

The full source code, including the Dockerfile used in this blog, can be found on GitHub. The repository also contains the same application without instrumentation. This allows you to compare each file and see the differences.

The following steps will show you how to instrument this application and run it on the command line or in Docker. If you are interested in a more complete OTel example, take a look at the docker-compose file here, which will bring up the full project.

Step-by-step guide

Step 0. Log in to your Elastic Cloud account

This blog assumes you have an Elastic Cloud account — if not, follow the instructions to get started on Elastic Cloud.

Step 1. Configure auto-instrumentation for the Python Service

We are going to use automatic instrumentation with Python service from the Elastiflix demo application.

We will be using the following service from Elastiflix:

Elastiflix/python-favorite-otel-auto

Per the OpenTelemetry Automatic Instrumentation for Python documentation, you will simply install the appropriate Python packages using pip install.

>pip install opentelemetry-distro \
	opentelemetry-exporter-otlp

>opentelemetry-bootstrap -a install

If you are running the Python service on the command line, then you can use the following command:

opentelemetry-instrument python main.py

For our application, we do this as part of the Dockerfile.

Dockerfile

FROM python:3.9-slim as base

# get packages
COPY requirements.txt .
RUN pip install -r requirements.txt
WORKDIR /favoriteservice

#install opentelemetry packages
RUN pip install opentelemetry-distro \
	opentelemetry-exporter-otlp

RUN opentelemetry-bootstrap -a install

# Add the application
COPY . .

EXPOSE 5000
ENTRYPOINT [ "opentelemetry-instrument", "python", "main.py"]

Step 2. Running the Docker image with environment variables

As specified in the OTEL Python documentation, we will use environment variables and pass in the configuration values to enable it to connect with Elastic Observability’s APM server.

Because Elastic accepts OTLP natively, we just need to provide the Endpoint and authentication where the OTEL Exporter needs to send the data, as well as some other environment variables.

Getting Elastic Cloud variables
You can copy the endpoints and token from Kibana^® under the path /app/home#/tutorial/apm.

You will need to copy the following environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT
OTEL_EXPORTER_OTLP_HEADERS

Build the image

docker build -t  python-otel-auto-image .

Run the image

docker run \
       -e OTEL_EXPORTER_OTLP_ENDPOINT="<REPLACE WITH OTEL_EXPORTER_OTLP_ENDPOINT>" \
       -e OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20<REPLACE WITH TOKEN>" \
       -e OTEL_RESOURCE_ATTRIBUTES="service.version=1.0,deployment.environment=production" \
       -e OTEL_SERVICE_NAME="python-favorite-otel-auto" \
       -p 5001:5001 \
       python-otel-auto-image

Important: Note that the “OTEL_EXPORTER_OTLP_HEADERS” variable has the whitespace after Bearer escaped as “%20” — this is a requirement for Python.

You can now issue a few requests in order to generate trace data. Note that these requests are expected to return an error, as this service relies on a connection to Redis that you don’t currently have running. As mentioned before, you can find a more complete example using docker-compose here.

curl localhost:5000/favorites

# or alternatively issue a request every second

while true; do curl "localhost:5000/favorites"; sleep 1; done;

Step 3: Explore traces, metrics, and logs in Elastic APM

Exploring the Services section in Elastic APM, you’ll see the Python service displayed.

Clicking on the python-favorite-otel-auto service , you can see that it is ingesting telemetry data using OpenTelemetry.

In this blog, we discussed the following:

• How to auto-instrument Python with OpenTelemetry
• Using standard commands in a Dockerfile, auto-instrumentation was done efficiently and without adding code in multiple places

Since Elastic can support a mix of methods for ingesting data, whether it be using auto-instrumentation of open-source OpenTelemetry or manual instrumentation with its native APM agents, you can plan your migration to OTel by focusing on a few applications first and then using OpenTelemety across your applications later on in a manner that best fits your business needs.

Developer resources:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry
• Python: Auto-instrumentation, Manual-instrumentation
• Java: Auto-instrumentation, Manual-instrumentation
• Node.js: Auto-instrumentation, Manual-instrumentation
• .NET: Auto-instrumentation, Manual-instrumentation
• Go: Manual-instrumentation
• Best practices for instrumenting OpenTelemetry

General configuration and use case resources:

• Independence with OpenTelemetry on Elastic
• Modern observability and security on Kubernetes with Elastic and OpenTelemetry
• 3 models for logging with OpenTelemetry and Elastic
• Adding free and open Elastic APM as part of your Elastic Observability deployment
• Capturing custom metrics through OpenTelemetry API in code with Elastic
• Future-proof your observability platform with OpenTelemetry and Elastic
• Elastic Observability: Built for open technologies like Kubernetes, OpenTelemetry, Prometheus, Istio, and more

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the auto-instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

To close this gap, we built an automated pipeline that moves beyond simple monitoring. By automating the discovery and investigation phases, we have shifted the focus of the engineer from "what happened?" to "is this fix correct?"

The Bottleneck in the Feedback Loop

In a high-velocity engineering environment, the path from deployment to resolution involves several distinct stages: Ship, Monitor, Triage, Identify, Fix, and Review/Deploy.

Velocity typically stalls during triage and identification. While catastrophic failures are reported immediately, smaller errors—intermittent UI glitches or failed background tasks—often go unreported. This dependency on manual reporting creates an inflated time to resolution; by the time a report is filed and routed, the issue may have already impacted the fleet for days.

By automating discovery and investigation, even these "paper cut" bugs are quantified before they accumulate into significant technical debt. The goal is to ensure that by the time a developer enters the cycle to write a fix, the detective work is already complete.

Discovery: Automated Log Clustering

The first challenge in this process is signal-to-noise. In a massive production environment, creating a ticket for every error event is unmanageable.

Instead of analyzing individual log lines, we automate the triage process using ES|QL's CATEGORIZE grouping function. CATEGORIZE clusters text messages into groups of similarly formatted values, turning unstructured telemetry into a prioritized backlog of distinct error patterns.

For example, a query like the following runs on a rolling window across all Kibana error logs:

FROM kibana-server-logs
| WHERE log.level == "ERROR"
    AND @timestamp >= NOW() - 7 days
| STATS count = COUNT() BY category = CATEGORIZE(message)
| SORT count DESC

The result is a table of regex-like categories and their occurrence counts:

A category like TypeError Cannot read properties of undefined reading document with 1,200+ hits over the past week tells us there is a real, recurring defect worth investigating. A category like Connection error spread uniformly across the fleet is more likely infrastructure noise.

The output is used to automatically file prioritized issues in a backlog, each enriched with the category, its regex, the occurrence count, and deep links into the raw telemetry. This automation ensures the feedback loop no longer waits for a user report to trigger an investigation; the discovery is proactive and immediate. These prioritized clusters then serve as the direct input for our autonomous investigation agent.

Investigation: The Automated Detective

Once an error pattern is identified, the pipeline moves to the identification phase. We deployed an AI agent to run a complete investigation of the issue. Navigating a codebase of Kibana's complexity is a significant time sink; the agent accelerates this by correlating information across the stack using ES|QL (Elasticsearch Query Language).

Protocol-Driven Investigation

It is important to distinguish this agent from a traditional automation script. The agent does not follow a hardcoded state machine; instead, it is provided with a protocol that outlines investigation goals and available tools.

The protocol prescribes a phased approach: understand the error, analyze its distribution, correlate with other data sources, find the source, and report. Each phase is described in terms of goals, not commands. The following excerpt shows how the protocol defines the first investigation step:

### Phase 1: Understand the Error
- Review the pre-extracted error details from the backlog issue
- Check for similar/overlapping error backlog issues (include closed!)
  - the categorization is often imperfect; closed issues may have
    valuable context about fixes
- Query for error overview statistics
- Get sample error messages to understand the actual content

The agent is also provided with an ES|QL reference guide and a library of query templates. Here is one of the templates for analyzing version distribution (a common first step to determine whether an error is a regression):

FROM logging-*:cluster-kibana-*
| WHERE @timestamp >= NOW() - 4 hours
    AND log.level == "ERROR"
    AND message : "TypeError Cannot read properties"
| STATS
    error_count = COUNT(*),
    deployments = COUNT_DISTINCT(ece.deployment)
  BY `docker.container.labels.org.label-schema.version`
| SORT error_count DESC

Because the agent has the autonomy to choose which tools to call—and in what order—based on the results of previous queries, it can adapt its strategy to the specific error. It might decide to skip proxy analysis if the telemetry suggests a background task failure, or it might dive deep into git history if ES|QL reveals the bug only exists on a specific version. This flexibility allows it to navigate the nuance of a massive codebase without requiring a pre-defined path for every possible failure mode.

Lessons Learned: Query Discipline

Direct LLM access to production clusters requires tactical constraints to manage costs and performance. We codified several requirements into the investigation workflow to ensure efficiency:

• Query Budgets: The agent is restricted to ~15-20 queries per investigation, forcing it to form a hypothesis before data-retrieval.

• The 4-Hour Rule: The agent starts with a small time window (the most recent 1-4 hours) to leverage caches and reduce compute costs.

• Optimal Operators: The agent prefers equality filters and the MATCH (:) operator over LIKE or regex, which can make queries 50-1000× faster.

• Fail-Fast Timeouts: Every query has a strict timeout, requiring the agent to refine its filters rather than retrying expensive operations.

Source Code Contextualization

To complete the identification phase, the agent correlates telemetry with the git history and source files. It uses the stack trace and log patterns to narrow its search, parsing through potential code matches faster than a manual search. By identifying the specific line of code producing the error and checking recent PRs, the agent links a production symptom directly to its technical root cause.

Real-World Case Study: The Streams UI Crash

The value of this autonomous investigation is best illustrated by the rare edge cases it uncovers. In one instance, the clustering system surfaced a sporadic pattern:

.*?TypeError.+?Cannot.+?read.+?properties.+?of.+?undefined.+?reading.+?document.*?

A human might have dismissed this as generic telemetry noise, but the agent's investigation revealed a reproducible race condition in the Streams UI:

1. Quantification: Using ES|QL, the agent analyzed the error distribution and identified the specific application context (Streams) and the relevant loggers.

2. Code Analysis: It identified a logic error in processor_outcome_preview.tsx. The code was indexing into an array (originalSamples[currentDoc.index].document) without verifying the element existed.

3. Root Cause: The agent realized that when a user changed filters while a row was expanded, the currentDoc.index became stale before the next render cleared it.

4. Outcome: The agent provided a suggested fix (guarding the access) and recommended a regression test around filter changes during row expansion.

This case highlights the economic scale of autonomous triage. Sifting through thousands of "noisy" logs to find the few that represent real, fixable UI crashes is a non-starter for senior engineers. Agents process this volume at a fraction of the cost, acting as a high-fidelity filter that ensures human time is only spent on verified, actionable issues.

The Future of Engineering Velocity

Automating triaging and identification is the first step. We are currently layering in the ability to pass these findings to a coding agent for draft Pull Requests. Beyond production errors, we are also investigating agentic exploratory testing to stress-test features during the pre-release phase and catch bugs before they ever reach a user.

This autonomous layer is complementary to, not a replacement for, classic quality gates. Unit tests, API-level checks, and UI integration tests remain the primary defense. Our approach provides a safety net for the failures that inevitably bypass these gates in a complex environment, ensuring they are addressed with the same rigor as pre-release bugs.

As we move toward a more agent-driven development process, the ability to rapidly validate that changes are safe and to control overall quality is the primary bottleneck for engineering velocity. While code generation itself is becoming a commodity, the "reasoning" required to verify that a change is both correct and safe remains the most critical hurdle. By focusing our automation on the discovery and root-cause analysis of failures, we ensure that our engineering teams can scale their impact without being buried by the operational weight of maintaining quality. The goal is to build a system that can understand, diagnose, and eventually fix itself.

For more information on Elastic and its observability capabilities, check out Elastic Observability. You can also sign up for a free trial to try it out yourself.

Our goal was to transition to an automated, adaptive approach capable of handling both log parsing (field extraction) and log partitioning (source identification). We hypothesized that Large Language Models (LLMs), with their inherent understanding of code syntax and semantic patterns, could automate these tasks with minimal human intervention.

We are happy to announce that this feature is already available in Streams!

Dataset Description

We chose a Loghub collection of logs for PoC purposes. For our investigation, we selected representative samples from the following key areas:

• Distributed systems: We used the HDFS (Hadoop Distributed File System) and Spark datasets. These contain a mix of info, debug, and error messages typical of big data platforms.
• Server & web applications: Logs from Apache web servers and OpenSSH provided a valuable source of access, error, and security-relevant events. These are critical for monitoring web traffic and detecting potential threats.
• Operating systems: We included logs from Linux and Windows. These datasets represent the common, semi-structured system-level events that operations teams encounter daily.
• Mobile systems: To ensure our model could handle logs from mobile environments, we included the Android dataset. These logs are often verbose and capture a wide range of application and system-level activities on mobile devices.
• Supercomputers: To test performance on high-performance computing (HPC) environments, we incorporated the BGL (Blue Gene/L) dataset, which features highly structured logs with specific domain terminology.

A key advantage of the Loghub collection is that the logs are largely unsanitized and unlabeled, mirroring a noisy live production environment with microservice architecture.

Log examples:

[Sun Dec 04 20:34:21 2005] [notice] jk2_init() Found child 2008 in scoreboard slot 6
[Sun Dec 04 20:34:25 2005] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties
[Mon Dec 05 11:06:51 2005] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties
17/06/09 20:10:58 INFO output.FileOutputCommitter: Saved output of task 'attempt_201706092018_0024_m_000083_1138' to hdfs://10.10.34.11:9000/pjhe/test/1/_temporary/0/task_201706092018_0024_m_000083
17/06/09 20:10:58 INFO mapred.SparkHadoopMapRedUtil: attempt_201706092018_0024_m_000083_1138: Committed

In addition, we created a Kubernetes cluster with a typical web application + database set up to mine extra logs in the most common domain.

Example of common log fields: timestamp, log level (INFO, WARN, ERROR), source, message.

Few-Shot Log Parsing with an LLM

Our first set of experiments focused on a fundamental question: Can an LLM reliably identify key fields and generate consistent parsing rules to extract them?

We asked a model to analyse raw log samples and generate log parsing rules in regular expression (regex) and Grok formats. Our results showed that this approach has a lot of potential, but also significant implementation challenges.

High Confidence & Context Awareness

Initial results were promising. The LLM demonstrated a strong ability to generate parsing rules that matched the provided few-shot examples with high confidence. Besides simple pattern matching, the model showed a capacity for log understanding —it could correctly identify and name the log source (e.g., health tracking app, Nginx web app, Mongo database).

The "Goldilocks" Dilemma of Input Samples

Our experiments quickly surfaced a significant lack of robustness because of extreme sensitivity to the input sample. The model's performance fluctuates wildly based on the specific log examples included in the prompt. We observed a log similarity problem where the log sample needs to include just diverse enough logs:

• Too homogeneous (overfitting): If the input logs are too similar, the LLM tends to overspecify. It treats variable data—such as specific Java class names in a stack trace—as static parts of the template. This results in brittle rules that cover a tiny ratio of logs and extract unusable fields.
• Too heterogeneous (confusion): Conversely, if the sample contains significant formatting variance—or worse, "trash logs" like progress bars, memory tables, or ASCII art—the model struggles to find a common denominator. It often resorts to generating complex, broken regexes or lazily over-generalizing the entire line into a single message blob field.

The Context Window Constraint

We also encountered a context window bottleneck. When input logs were long, heterogeneous, or rich in extractable fields, the model's output often deteriorated, becoming "messy" or too long to fit into the output context window. Naturally, chunking helps in this case. By splitting logs using character-based and entity-based delimiters, we could help the model focus on extracting the main fields without being overwhelmed by noise.

The consistency & standardization gap

Even when the model successfully generated rules, we noted slight inconsistencies:

• Service naming variations: The model proposes different names for the same entity (e.g., labeling the source as "Spark," "Apache Spark," and "Spark Log Analytics" in different runs).
• Field naming variations: Field names lacked standardization (e.g., id vs. service.id vs. device.id). We normalized names using a standardized Elastic field naming.
• Resolution variance: The resolution of the field extraction varied depending on how similar the input logs were to one another.

Log Format Fingerprint

To address the challenge of log similarity, we introduce a high-performance heuristic: log format fingerprint (LFF).

Instead of feeding raw, noisy logs directly into an LLM, we first apply a deterministic transformation to reveal the underlying structure of each message. This pre-processing step abstracts away variable data, generating a simplified "fingerprint" that allows us to group related logs.

The mapping logic is simple to ensure speed and consistency:

1. Digit abstraction: Any sequence of digits (0-9) is replaced by a single ‘0’.
2. Text abstraction: Any sequence of alphabetical characters with whitespace is replaced by a single ‘a’.
3. Whitespace normalization: All sequences of whitespace (spaces, tabs, newlines) are collapsed into a single space.
4. Symbol preservation: Punctuation and special characters (e.g., :, [, ], /) are preserved, as they are often the strongest indicators of log structure.

We introduce the log mapping approach. The basic mapping patterns include the following:

Digits 0-9 of any length -> to ‘0.’

• Text (alphabetical characters with spaces) of any length -> to ‘a’.
• White spaces, tabs, and new lines -> to a single space.
• Let's look at an example of how this mapping allows us to transform the logs.

As a result, we obtain the following log masks:

Notice the fingerprints of the first two logs. Despite different timestamps, source classes, and message content, their prefixes (0/0/0 0:0:0 a a.a:) are identical. This structural alignment allows us to automatically bucket these logs into the same cluster.

The third log, however, produces a completely divergent fingerprint (0-0-0...). This allows us to algorithmically separate it from the first group before we ever invoke an LLM.

Bonus Part: Instant Implementation with ES|QL

It’s as easy as passing this query in Discover.

FROM loghub |
EVAL pattern = REPLACE(REPLACE(REPLACE(REPLACE(raw_message, "[ \t\n]+", " "), "[A-Za-z]+", "a"), "[0-9]+", "0"), "a( a)+", "a") |
STATS total_count = COUNT(), ratio = COUNT() / 2000.0, datasources=VALUES(filename), example=TOP(raw_message, 3, "desc") BY SUBSTRING(pattern, 0, 15) |
SORT total_count DESC |
LIMIT 100

Query breakdown:

FROM loghub: Targets our index containing the raw log data.

EVAL pattern = …: The core mapping logic. We chain REPLACE functions to perform the abstraction (e.g., digits to '0', text to 'a', etc.) and save the result in a “pattern” field.

STATS [column1 =] expression1, … BY SUBSTRING(pattern, 0, 15):

This is a clustering step. We group logs that share the first 15 characters of their pattern and create aggregated fields such as total log count per group, list of log datasources, pattern prefix, 3 log examples

SORT total_count DESC | LIMIT 100 : Surfaces the top 100 most frequent log patterns

The query results on LogHub are displayed below:

As demonstrated in the visualization, this “LLM-free” approach partitions logs with high accuracy. It successfully clustered 10 out of 16 data sources (based on LogHub labels) completely (>90%) and achieved majority clustering in 13 out of 16 sources (>60%) —all without requiring additional cleaning, preprocessing, or fine-tuning.

Log format fingerprint offers a pragmatic, high-impact alternative and addition to sophisticated ML solutions like log pattern analysis. It provides immediate insights into log relationships and effectively manages large log clusters.

• Versatility as a primitive

Thanks to ES|QL implementation, LFF serves both as a standalone tool for fast data diagnostics/visualisations, and as a building block in log analysis pipelines for high-volume use cases.

• Flexibility

LFF is easy to customize and extend to capture specific patterns, i.e. hexadecimal numbers and IP addresses.

• Deterministic stability

Unlike ML-based clustering algorithms, LFF logic is straightforward and deterministic. New incoming logs do not retroactively affect existing log clusters.

• Performance and Memory

It requires minimal memory, no training or GPU making it ideal for real-time high-throughput environments.

Combining Log Format Fingerprint with an LLM

To validate the proposed hybrid architecture, each experiment contained a random 20% subset of the logs from each data source. This constraint simulates a real-world production environment where logs are processed in batches rather than as a monolithic historical dump.

The objective was to demonstrate that LFF acts as an effective compression layer. We aimed to prove that high-coverage parsing rules could be generated from small, curated samples and successfully generalized to the entire dataset.

Execution Pipeline

We implemented a multi-stage pipeline that filters, clusters, and applies stratified sampling to the data before it reaches the LLM.

1. Two-stage hierarchical clustering

• Subclasses (exact match): Logs are aggregated by identical fingerprints. Every log in one subclass shares the exact same format structure.
• Outlier cleaning: We discard any subclasses that represent less than 5% of the total log volume. This ensures the LLM focuses on the dominant signal and won’t be sidetracked by noise or malformed logs.
• Metaclasses (prefix match): Remaining subclasses are grouped into Metaclasses by the first N characters of the format fingerprint match. This grouping strategy effectively splits lexically similar formats under a single umbrella.We chose N=5 for Log parsing and N=15 for Log partitioning when data sources are unknown.

2. Stratified sampling. Once the hierarchical tree is built, we construct the log sample for the LLM. The strategic goal is to maximize variance coverage while minimizing token usage.

• We select representative logs from each valid subclass within the broader metaclass.
• To manage an edge case of too numerous subclasses, we apply random down-sampling to fit the target window size.

3. Rule generation Finally, we prompt the LLM to generate a regex parsing rule that fits all logs in the provided sample for each Metaclass. For our PoC, we used the GPT-4o mini model.

Experimental Results & Observations

We achieved 94% parsing accuracy and 91% partitioning accuracy on the Loghub dataset.

The confusion matrix above illustrates log partitioning results. The vertical axis represents the actual data sources, and the horizontal axis represents the predicted data sources. The heatmap intensity corresponds to log volume, with lighter tiles indicating a higher count. The diagonal alignment demonstrates the model's high fidelity in source attribution, with minimal scattering.

Our Performance Benchmarks Insights

• Optimal baseline: a context window of 30–40 log samples per category proved to be the "sweet spot," consistently producing robust parsing with both Regex and Grok patterns.
• Input minimisation: we pushed the input size to 10 logs per category for Regex patterns and observed only 2% drop in parsing performance, confirming that diversity-based sampling is more critical than raw volume.

Elastic Observability has been supporting AWS logs ingest with Amazon Data Firehose over the last few releases. To makes configuration easier, we introduced, in 8.16, a one step guided workflow to onboard all CloudWatch logs and metrics from a single region. The configuration uses a pre-populated CloudFormation template, to automatically create a Amazon Data Firehose and connect to Elastic Observability. Additionally, all the relevant Elastic AWS Integrations are auto-installed. The configuration ensures ingestion for metrics from all namespaces and a policy to ingest logs from all existing log groups. Any new metric namespaces and log groups post setup will also be ingested automatically. Additionally, the CloudFormation template can also be customized and deployed in a production environment using infra-as-code.

This allows SREs to to start monitoring the usage and health of their popular AWS services using pre-built dashboards within minutes. This blog reviews how to setup this quickstart workflow, and the out-of-the box dashboards that will be populated from it.

Onboarding data using Amazon Data Firehose

In order to utilize this guided workflow, a user needs the superuser built-in Kibana role. A deployment of the hosted Elasticsearch service of version 8.16 on Elastic Cloud is required. Further, an active AWS account and the necessary permissions to create delivery streams, run CloudFormation, create CloudWatch log group/metric streams are needed.

Let’s walk through the steps required to onboard data using this workflow. There should be some CloudWatch logs and metrics already available in the customer account. The screenshot below shows an example where a number of CloudWatch metrics namespaces already exist.

Similarly, a number of CloudWatch log groups are already present in this customer account as shown below.

This guided workflow is accessible from the ‘Add data’ left navigation option in the Elastic Observability app. The user needs to select the ‘Cloud’ option and click on the ‘AWS’ tile. The Amazon Firehose quickstart onboarding workflow is available at the top left and is labeled as a Quickstart option, as shown below.  

The Data Firehose delivery stream can be created either using the AWS CLI or the AWS console, as shown in step 2 of the guided workflow below. 

By clicking on the ‘Create Firehose Stream in AWS’ button under the ‘Via AWS Console’ tab, the user will be taken to the AWS console and the menu for creating the CloudFormation stack, as shown below. 

The CloudFormation (CF) template provided by Elastic has prepopulated default settings including the Elasticsearch endpoint and the API key, as shown in the screenshot above. The user can review these defaults in the AWS console and proceed by clicking on the ‘Create stack’ button, as shown below. Note that this stack creates IAM resources and so the checkbox acknowledging that must be checked to move forward. 

Once the CloudFormation stack has been created in AWS, the user can switch back to Kibana. By default, the CF stack will consist of separate delivery streams for CloudWatch logs and metrics, as shown below. 

In Kibana, under step 3 ‘Visualize your data’ of the workflow, the incoming data starts to appear, categorized by AWS service type as shown below. The page refreshes automatically every 5 s and the new services appear at the bottom of the list.  

For each detected AWS service, the user is recommended 1-2 pre-built dashboards to explore the health and usage of their services. For example, the pre-built dashboard shown below provides a quick overview on the usage of the NAT Gateway.  

In addition to pre-built dashboards, Discover can also be used to explore the ingested CloudWatch logs, as shown below. 

AWS Usage overview can be explored using the pre-built dashboard shown below.

Customisation options

The region needs to be selected/modified in the AWS console as shown below, before starting with the CF stack creation. 

The setting of EnableCloudWatchLogs parameter and the setting of EnableCloudWatchMetrics parameter in the AWS console or the CF template can be changed to disable the collection of logs or metrics.

The MetricNameFilters parameter in the CF template or console can be used to exclude specific namespace-metric names pairs from collection.

The CF template provided by Elastic can be used together with the Terraform resource aws_cloudformation_stack as shown below to deploy in the production environment, to facilitate as-code deployment.

Start your own exploration 

The new guided onboarding workflow for AWS utilizes the Amazon Firehose delivery stream to collect all available CloudWatch logs & metrics, from a single customer account and a single region. The workflow also installs AWS Integration packages in the Elastic stack, enabling users to start monitoring the usage and performance of their common AWS services using pre-built dashboards, within minutes. Some of the AWS services that can be monitored using this workflow are listed below. A complete list of over twenty services that are supported by this workflow along with additional details are available here.

To simplify this process and reduce management overhead, AWS users can now leverage the new Amazon Kinesis Firehose Delivery Stream to ingest logs into Elastic Cloud in AWS in real time and view them in the Elastic Stack alongside other logs for centralized analytics. This eliminates the necessity for time-consuming and expensive procedures such as VM provisioning or data shipper operations.

Elastic Observability unifies logs, metrics, and application performance monitoring (APM) traces for a full contextual view across your hybrid AWS environments alongside their on-premises data sets. Elastic Observability enables you to track and monitor performance across a broad range of AWS services, including AWS Lambda, Amazon Elastic Compute Cloud (EC2), Amazon Elastic Container Service (ECS), Amazon Elastic Kubernetes Service (EKS), Amazon Simple Storage Service (S3), Amazon Cloudtrail, Amazon Network Firewall, and more.

In this blog, we will walk you through how to use the Amazon Kinesis Data Firehose integration — Elastic is listed in the Amazon Kinesis Firehose drop-down list — to simplify your architecture and send logs to Elastic, so you can monitor and safeguard your multi-account AWS environments.

Announcing the Kinesis Firehose method

Elastic currently provides both agent-based and serverless mechanisms, and we are pleased to announce the addition of the Kinesis Firehose method. This new method enables customers to directly ingest logs from AWS into Elastic, supplementing our existing options.

• Elastic Agent pulls metrics and logs from CloudWatch and S3 where logs are generally pushed from a service (for example, EC2, ELB, WAF, Route53) and ingests them into Elastic Cloud.
• Elastic’s Serverless Forwarder (runs Lambda and available in AWS SAR) sends logs from Kinesis Data Stream, Amazon S3, and AWS Cloudwatch log groups into Elastic. To learn more about this topic, please see this blog post.
• Amazon Kinesis Firehose directly ingests logs from AWS into Elastic (specifically, if you are running the Elastic Cloud on AWS).

In this blog, we will cover the last option since we have recently released the Amazon Kinesis Data Firehose integration. Specifically, we'll review:

• A general overview of the Amazon Kinesis Data Firehose integration and how it works with AWS
• Step-by-step instructions to set up the Amazon Kinesis Data Firehose integration on AWS and on Elastic Cloud

By the end of this blog, you'll be equipped with the knowledge and tools to simplify your AWS log management with Elastic Observability and Amazon Kinesis Data Firehose.

Prerequisites and configurations

If you intend to follow the steps outlined in this blog post, there are a few prerequisites and configurations that you should have in place beforehand.

1. You will need an account on Elastic Cloud and a deployed stack on AWS. Instructions for deploying a stack on AWS can be found here. This is necessary for AWS Firehose Log ingestion.
2. You will also need an AWS account with the necessary permissions to pull data from AWS. Details on the required permissions can be found in our documentation.
3. Finally, be sure to turn on VPC Flow Logs for the VPC where your application is deployed and send them to AWS Firehose.

Elastic’s Amazon Kinesis Data Firehose integration

Elastic has collaborated with AWS to offer a seamless integration of Amazon Kinesis Data Firehose with Elastic, enabling direct ingestion of data from Amazon Kinesis Data Firehose into Elastic without the need for Agents or Beats. All you need to do is configure the Amazon Kinesis Data Firehose delivery stream to send its data to Elastic's endpoint. In this configuration, we will demonstrate how to ingest VPC Flow logs and Firewall logs into Elastic. You can follow a similar process to ingest other logs from your AWS environment into Elastic.

There are three distinct configurations available for ingesting VPC Flow and Network firewall logs into Elastic. One configuration involves sending logs through CloudWatch, and another uses S3 and Kinesis Firehose; each has its own unique setup. With Cloudwatch and S3 you can store and forward but with Kinesis Firehose you will have to ingest immediately. However, in this blog post, we will focus on this new configuration that involves sending VPC Flow logs and Network Firewall logs directly to Elastic.

We will guide you through the configuration of the easiest setup, which involves directly sending VPC Flow logs and Firewalls logs to Amazon Kinesis Data Firehose and then into Elastic Cloud.

Note: It's important to note that this setup is only compatible with Elastic Cloud on AWS and cannot be used with self-managed or on-premise or other cloud provider Elastic deployments.

Setting it all up

To begin setting up the integration between Amazon Kinesis Data Firehose and Elastic, let's go through the necessary steps.

Step 0: Get an account on Elastic Cloud

Create an account on Elastic Cloud by following the instructions provided to get started on Elastic Cloud.

Step 1: Deploy Elastic on AWS

You can deploy Elastic on AWS via two different approaches: through the UI or through Terraform. We’ll start first with the UI option.

After logging into Elastic Cloud, create a deployment on Elastic. It's crucial to make sure that the deployment is on Elastic Cloud on AWS since the Amazon Kinesis Data Firehose connects to a specific endpoint that must be on AWS.

After your deployment is created, it's essential to copy the Elasticsearch endpoint to ensure a seamless configuration process.

The Elasticsearch HTTP endpoint should be copied and used for Amazon Firehose destination configuration purposes, as it will be required. Here's an example of what the endpoint should look like:

https://elastic-O11y-log.es.us-east-1.aws.found.io

Alternative approach using Terraform

An alternative approach to deploying Elastic Cloud on AWS is by using Terraform. It's also an effective way to automate and streamline the deployment process.

To begin, simply create a Terraform configuration file that outlines the necessary infrastructure. This file should include resources for your Elastic Cloud deployment and any required IAM roles and policies. By using this approach, you can simplify the deployment process and ensure consistency across environments.

One easy way to create your Elastic Cloud deployment with Terraform is to use this Github repo. This resource lets you specify the region, version, and deployment template for your Elastic Cloud deployment, as well as any additional settings you require.

Step 2: To turn on Elastic's AWS integrations, navigate to the Elastic Integration section in your deployment

To install AWS assets in your deployment's Elastic Integration section, follow these steps:

1. Log in to your Elastic Cloud deployment and open Kibana.
2. To get started, go to the management section of Kibana and click on " Integrations."
3. Navigate to the AWS integration and click on the "Install AWS Assets" button in the settings.This step is important as it installs the necessary assets such as dashboards and ingest pipelines to enable data ingestion from AWS services into Elastic.

Step 3: Set up the Amazon Kinesis Data Firehose delivery stream on the AWS Console

You can set up the Kinesis Data Firehose delivery stream via two different approaches: through the AWS Management Console or through Terraform. We’ll start first with the console option.

To set up the Kinesis Data Firehose delivery stream on AWS, follow these steps:

1. Go to the AWS Management Console and select Amazon Kinesis Data Firehose.

2. Click on Create delivery stream.

3. Choose a delivery stream name and select Direct PUT or other sources as the source.

4. Choose Elastic as the destination.

5. In the Elastic destination section, enter the Elastic endpoint URL that you copied from your Elastic Cloud deployment.

6. Choose the content encoding and retry duration as shown above.

7. Enter the appropriate parameter values for your AWS log type. For example, for VPC Flow logs, you would need to specify the _ es_datastream_name _ and _ logs-aws.vpc flow-default _.

8. Configure the Amazon S3 bucket as the source backup for the Amazon Kinesis Data Firehose delivery stream failed data or all data, and configure any required tags for the delivery stream.

9. Review the settings and click on Create delivery stream.

In the example above, we are using the es_datastream_name parameter to pull in VPC Flow logs through the logs-aws.vpcflow-default datastream. Depending on your use case, this parameter can be configured with one of the following types of logs:

• logs-aws.cloudfront_logs-default (AWS CloudFront logs)
• logs-aws.ec2_logs-default (EC2 logs in AWS CloudWatch)
• logs-aws.elb_logs-default (Amazon Elastic Load Balancing logs)
• logs-aws.firewall_logs-default (AWS Network Firewall logs)
• logs-aws.route53_public_logs-default (Amazon Route 53 public DNS queries logs)
• logs-aws.route53_resolver_logs-default (Amazon Route 53 DNS queries & responses logs)
• logs-aws.s3access-default (Amazon S3 server access log)
• logs-aws.vpcflow-default (AWS VPC flow logs)
• logs-aws.waf-default (AWS WAF Logs)

Alternative approach using Terraform

Using the " aws_kinesis_firehose_delivery_stream" resource in Terraform is another way to create a Kinesis Firehose delivery stream, allowing you to specify the delivery stream name, data source, and destination - in this case, an Elasticsearch HTTP endpoint. To authenticate, you'll need to provide the endpoint URL and an API key. Leveraging this Terraform resource is a fantastic way to automate and streamline your deployment process, resulting in greater consistency and efficiency.

Here's an example code that shows you how to create a Kinesis Firehose delivery stream with Terraform that sends data to an Elasticsearch HTTP endpoint:

resource "aws_kinesis_firehose_delivery_stream" “Elasticcloud_stream" {
  name        = "terraform-kinesis-firehose-ElasticCloud-stream"
  destination = "http_endpoint”
  s3_configuration {
    role_arn           = aws_iam_role.firehose.arn
    bucket_arn         = aws_s3_bucket.bucket.arn
    buffer_size        = 5
    buffer_interval    = 300
    compression_format = "GZIP"
  }
  http_endpoint_configuration {
    url        = "https://cloud.elastic.co/"
    name       = “ElasticCloudEndpoint"
    access_key = “ElasticApi-key"
    buffering_hints {
      size_in_mb = 5
      interval_in_seconds = 300
    }

   role_arn       = "arn:Elastic_role"
   s3_backup_mode = "FailedDataOnly"
  }
}

Step 4: Configure VPC Flow Logs to send to Amazon Kinesis Data Firehose

To complete the setup, you'll need to configure VPC Flow logs in the VPC where your application is deployed and send them to the Amazon Kinesis Data Firehose delivery stream you set up in Step 3.

Enabling VPC flow logs in AWS is a straightforward process that involves several steps. Here's a step-by-step details to enable VPC flow logs in your AWS account:

1. Select the VPC for which you want to enable flow logs.

2. In the VPC dashboard, click on "Flow Logs" under the "Logs" section.

3. Click on the "Create Flow Log" button to create a new flow log.

4. In the "Create Flow Log" wizard, provide the following information:

Choose the target for your flow logs: In this case, Amazon Kinesis Data Firehose in the same AWS account.

• Provide a name for your flow log.
• Choose the VPC and the network interface(s) for which you want to enable flow logs.
• Choose the flow log format: either AWS default or Custom format.

5. Configure the IAM role for the flow logs. If you have an existing IAM role, select it. Otherwise, create a new IAM role that grants the necessary permissions for the flow logs.

6. Review the flow log configuration and click "Create."

Create the VPC Flow log.

Step 5: After a few minutes, check if flows are coming into Elastic

To confirm that the VPC Flow logs are ingesting into Elastic, you can check the logs in Kibana. You can do this by searching for the index in the Kibana Discover tab and filtering the results by the appropriate index and time range. If VPC Flow logs are flowing in, you should see a list of documents representing the VPC Flow logs.

Step 6: Navigate to Kibana to see your logs parsed and visualized in the [Logs AWS] VPC Flow Log Overview dashboard

Finally, there is an Elastic out-of-the-box (OOTB) VPC Flow logs dashboard that displays the top IP addresses that are hitting your VPC, their geographic location, time series of the flows, and a summary of VPC flow log rejects within the selected time frame. This dashboard can provide valuable insights into your network traffic and potential security threats.

Note: For additional VPC flow log analysis capabilities, please refer to this blog.

Step 7: Configure AWS Network Firewall Logs to send to Kinesis Firehose

To create a Kinesis Data Firehose delivery stream for AWS Network firewall logs, first log in to the AWS Management Console, navigate to the Kinesis service, select "Data Firehose", and follow the step-by-step instructions as shown in Step 3. Specify the Elasticsearch endpoint, API key, add a parameter (_ es_datastream_name=logs-aws.firewall_logs-default _), and create the delivery stream.

Second, to set up a Network Firewall rule group to send logs to the Kinesis Firehose, go to the Network Firewall section of the console, create a rule group, add a rule to allow traffic to the Kinesis endpoint, and attach the rule group to your Network Firewall configuration. Finally, test the configuration by sending traffic through the Network Firewall to the Kinesis Firehose endpoint and verify that logs are being delivered to your S3 bucket.

Kindly follow the instructions below to set up a firewall rule and logging.

1. Set up a Network Firewall rule group to send logs to Amazon Kinesis Data Firehose:

• Go to the AWS Management Console and select Network Firewall.
• Click on "Rule groups" in the left menu and then click "Create rule group."
• Choose "Stateless" or "Stateful" depending on your requirements, and give your rule group a name. Click "Create rule group."
• Add a rule to the rule group to allow traffic to the Kinesis Firehose endpoint. For example, if you are using the us-east-1 region, you would add a rule like this:json

{
  "RuleDefinition": {
    "Actions": [
      {
        "Type": "AWS::KinesisFirehose::DeliveryStream",
        "Options": {
          "DeliveryStreamArn": "arn:aws:firehose:us-east-1:12387389012:deliverystream/my-delivery-stream"
        }
      }
    ],
    "MatchAttributes": {
      "Destination": {
        "Addresses": ["api.firehose.us-east-1.amazonaws.com"]
      },
      "Protocol": {
        "Numeric": 6,
        "Type": "TCP"
      },
      "PortRanges": [
        {
          "From": 443,
          "To": 443
        }
      ]
    }
  },
  "RuleOptions": {
    "CustomTCPStarter": {
      "Enabled": true,
      "PortNumber": 443
    }
  }
}

• Save the rule group.

2. Attach the rule group to your Network Firewall configuration:

• Go to the AWS Management Console and select Network Firewall.
• Click on "Firewall configurations" in the left menu and select the configuration you want to attach the rule group to.
• Scroll down to "Associations" and click "Edit."
• Select the rule group you created in Step 2 and click "Save."

3. Test the configuration:

• Send traffic through the Network Firewall to the Kinesis Firehose endpoint and verify that logs are being delivered to your S3 bucket.

Step 8: Navigate to Kibana to see your logs parsed and visualized in the [Logs AWS] Firewall Log dashboard

Wrapping up

We’re excited to bring you this latest integration for AWS Cloud and Kinesis Data Firehose into production. The ability to consolidate logs and metrics to gain visibility across your cloud and on-premises environment is crucial for today’s distributed environments and applications.

From EC2, Cloudwatch, Lambda, ECS and SAR, Elastic Integrations allow you to quickly and easily get started with ingesting your telemetry data for monitoring, analytics, and observability. Elastic is constantly delivering frictionless customer experiences, allowing anytime, anywhere access to all of your telemetry data — this streamlined, native integration with AWS is the latest example of our commitment.

Start a free trial today

You can begin with a 7-day free trial of Elastic Cloud within the AWS Marketplace to start monitoring and improving your users' experience today!

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Cloud is becoming the de facto deployment option for today’s applications. Many cloud deployments choose to host their applications on AWS for the globally diverse set of regions it covers and the myriad of services (for faster development and innovation) available, as well as to drive operational and capital costs down. On AWS, development teams are finding additional value in migrating to Kubernetes on Amazon EKS, testing out the latest serverless options, and improving traditional, tiered applications with better services.

Elastic Observability offers 30 out-of-the-box integrations for AWS services with more to come.

A quick review highlighting some of the integrations and capabilities can be found in a previous post:

• Elastic and AWS: Seamlessly ingest logs and metrics into a unified platform with ready-to-use integrations.

Some additional posts on key AWS service integrations on Elastic are:

• APM (metrics, traces and logs) for serverless functions on AWS Lambda with Elastic
• Log ingestion from AWS Services into Elastic via serverless forwarder on Lambda
• Elastic’s Amazon S3 Storage Lens Integration: Simplify management, control costs, and reduce risk
• Ingest your container logs into Elastic Cloud with AWS FireLens

A full list of AWS integrations can be found in Elastic’s online documentation:

• Full list of AWS integrations

In addition to our native AWS integrations, Elastic Observability aggregates not only logs but also metrics for AWS services and the applications running on AWS compute services (EC2, Lambda, EKS/ECS/Fargate). All this data can be analyzed visually and more intuitively using Elastic’s advanced machine learning capabilities, which help detect performance issues and surface root causes before end users are affected.

For more details on how Elastic Observability provides application performance monitoring (APM) capabilities such as service maps, tracing, dependencies, and ML based metrics correlations:

• APM correlations in Elastic Observability: Automatically identifying probable causes of slow or failed transactions
• Elastic and AWS: Get the most value from your data sets

That’s right, Elastic offers metrics ingest, aggregation, and analysis for AWS services and applications on AWS compute services (EC2, Lambda, EKS/ECS/Fargate). Elastic is more than logs — it offers a unified observability solution for AWS environments.

In this blog, I’ll review how Elastic Observability can monitor metrics for a simple AWS application running on AWS services which include:

• AWS EC2
• AWS ELB
• AWS RDS (AuroraDB)
• AWS NAT Gateways

As you will see, once the integration is installed, metrics will arrive instantly and you can immediately start reviewing metrics.

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up this demonstration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).
• Ensure you have an AWS account with permissions to pull the necessary data from AWS. See details in our documentation.
• We used AWS’s three tier app and installed it as instructed in git.
• We’ll walk through installing the general Elastic AWS Integration, which covers the four services we want to collect metrics for.
  (Full list of services supported by the Elastic AWS Integration)
• We will not cover application monitoring given other blogs cover application AWS monitoring (metrics, logs, and tracing). Instead we will focus on how AWS services can be easily monitored.
• In order to see metrics, you will need to load the application. We’ve also created a playwright script to drive traffic to the application.

Three tier application overview

Before we dive into the Elastic configuration, let's review what we are monitoring. If you follow the instructions for aws-three-tier-web-architecture-workshop, you will have the following deployed.

What’s deployed:

• 1 VPC with 6 subnets
• 2 AZs
• 2 web servers per AZ
• 2 application servers per AZ
• 1 External facing application load balancer
• 1 Internal facing application load balancer
• 2 NAT gateways to manage traffic to the application layer
• 1 Internet gateway
• 1 RDS Aurora DB with a read replica

At the end of the blog, we will also provide a Playwright script to implement to load this app. This will help drive metrics to “light up” the dashboards.

Setting it all up

Let’s walk through the details of how to get the application, AWS integration on Elastic, and what gets ingested.

Step 0: Load up the AWS Three Tier application and get your credentials

Follow the instructions listed out in AWS’s Three Tier app and instructions in the workshop link on git. The workshop is listed here.

Once you’ve installed the app, get credentials from AWS. This will be needed for Elastic’s AWS integration.

There are several options for credentials:

• Use access keys directly
• Use temporary security credentials
• Use a shared credentials file
• Use an IAM role Amazon Resource Name (ARN)

For more details on specifics around necessary credentials and permissions.

Step 1: Get an account on Elastic Cloud

Follow the instructions to get started on Elastic Cloud.

Step 2: Install the Elastic AWS integration

Navigate to the AWS integration on Elastic.

Select Add AWS integration.

This is where you will add your credentials and it will be stored as a policy in Elastic. This policy will be used as part of the install for the agent in the next step.

As you can see, the general Elastic AWS Integration will collect a significant amount of data from 30 AWS services. If you don’t want to install this general Elastic AWS Integration, you can select individual integrations to install.

Step 3: Install the Elastic Agent with AWS integration

Now that you have created an integration policy, navigate to the Fleet section under Management in Elastic.

Select the name of the policy you created in the last step.

Follow step 3 in the instructions in the Add agent window. This will require you to:

1: Bring up an EC2 instance

• t2.medium is minimum
• Linux - your choice of which
• Ensure you allow for Open reservation on the EC2 instance when you Launch it

2: Log in to the instance and run the commands under Linux Tar tab (below is an example)

curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.5.0-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.5.0-linux-x86_64.tar.gz
cd elastic-agent-8.5.0-linux-x86_64
sudo ./elastic-agent install --url=https://37845638732625692c8ee914d88951dd96.fleet.us-central1.gcp.cloud.es.io:443 --enrollment-token=jkhfglkuwyvrquevuytqoeiyri

Step 4: Run traffic against the application

While getting the application running is fairly easy, there is nothing to monitor or observe with Elastic unless you add a load on the application.

Here is a simple script you can also run using Playwright to add traffic to the website for the AWS three tier application:

import { test, expect } from "@playwright/test";

test("homepage for AWS Threetierapp", async ({ page }) => {
  await page.goto(
    "http://web-tier-external-lb-1897463036.us-west-1.elb.amazonaws.com/#/db"
  );

  await page.fill(
    "#transactions > tbody > tr > td:nth-child(2) > input",
    (Math.random() * 100).toString()
  );
  await page.fill(
    "#transactions > tbody > tr > td:nth-child(3) > input",
    (Math.random() * 100).toString()
  );
  await page.waitForTimeout(1000);
  await page.click(
    "#transactions > tbody > tr:nth-child(2) > td:nth-child(1) > input[type=button]"
  );
  await page.waitForTimeout(4000);
});

This script will launch three browsers, but you can limit this load to one browser in playwright.config.ts file.

For this exercise, we ran this traffic for approximately five hours with an interval of five minutes while testing the website.

Step 5: Go to AWS dashboards

Now that your Elastic Agent is running, you can go to the related AWS dashboards to view what’s being ingested.

To search for the AWS Integration dashboards, simply search for them in the Elastic search bar. The relevant ones for this blog are:

• [Metrics AWS] EC2 Overview
• [Metrics AWS] ELB Overview
• [Metrics AWS] RDS Overview
• [Metrics AWS] NAT Gateway

Let's see what comes up!

All of these dashboards are out-of-the-box and for all the following images, we’ve narrowed the views to only the relevant items from our app.

Across all dashboards, we’ve limited the timeframe to when we ran the traffic generator.

Once we filtered for our 4 EC2 instances (2 web servers and 2 application servers), we can see the following:

1: All 4 instances are up and running with no failures in status checks.

2: We see the average CPU utilization across the timeframe and nothing looks abnormal.

3: We see the network bytes flow in and out, aggregating over time as the database is loaded with rows.

While this exercise shows a small portion of the metrics that can be viewed, more are available from AWS EC2. The metrics listed on AWS documentation are all available, including the dimensions to help narrow the search for specific instances, etc.

For the ELB dashboard, we filter for our 2 load balancers (external web load balancer and internal application load balancer).

With the out-of-the-box dashboard, you can see application ELB-specific metrics. A good portion of the application ELB specific metrics listed in AWS Docs are available to add graphs for.

For our two load balancers, we can see:

1: Both the hosts (EC2 instances connected to the ELBs) are healthy.

2: Load Balancer Capacity Units (how much you are using) and request counts both went up as expected during the traffic generation time frame.

3: We picked to show 4XX and 2XX counts. 4XX will help identify issues with the application or connectivity with the application servers.

For AuroraDB, which is deployed in RDS, we’ve filtered for just the primary and secondary instances of Aurora on the dashboard.

Just as with EC2, ELB, most RDS metrics from Cloudwatch are also available to create new charts and graphs. In this dashboard, we’ve narrowed it down to showing:

1: Insert throughput & Select throughput

2: Write latency

3: CPU usage

4: General number of connections during the timeframe

We filtered to look only at our 2 NAT instances which are fronting the application servers. As with the other dashboards, other metrics are available to build graphs and /charts as needed.

For the NAT dashboard we can see the following:

1: The NAT Gateways are doing well due to no packet drops

2: An expected number of active connections from the web server

3: Fairly normal set of metrics for bytes in and out

Congratulations, you have now started monitoring metrics from key AWS services for your application!

What to monitor on AWS next?

Add logs from AWS Services

Now that metrics are being monitored, you can also now add logging. There are several options for ingesting logs.

1. The AWS Integration in the Elastic Agent has logs setting. Just ensure you turn on what you wish to receive. Let’s ingest the Aurora Logs from RDS. In the Elastic agent policy, we simply turn on Collect logs from CloudWatch (see below). Next, update the agent through the Fleet management UI.

2. You can install the Lambda logs forwarder. This option will pull logs from multiple locations. See the architecture diagram below.

A review of this option is also found in the following blog.

Analyze your data with Elastic Machine Learning

Once metrics and logs (or either one) are in Elastic, start analyzing your data through Elastic’s ML capabilities. A great review of these features can be found here:

• Correlating APM Telemetry to determine root causes in transactions
• Introduction to Elastic Machine Learning

And there are many more videos and blogs on Elastic’s Blog.

Conclusion: Monitoring AWS service metrics with Elastic Observability is easy!

I hope you’ve gotten an appreciation for how Elastic Observability can help you monitor AWS service metrics, here’s a quick recap of lessons and what you learned:

• Elastic Observability supports ingest and analysis of AWS service metrics
• It’s easy to set up ingest from AWS Services via the Elastic Agent
• Elastic Observability has multiple out-of-the-box (OOTB) AWS service dashboards you can use to preliminarily review information, then modify for your needs
• 30+ AWS services are supported as part of AWS Integration on Elastic Observability, with more services being added regularly
• As noted in related blogs, you can analyze your AWS service metrics with Elastic’s machine learning capabilities

Start your own 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world. Your AWS Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with AWS.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

1. Where traffic is coming in from and going out to from the deployment, and within the deployment. This helps identify unusual or unauthorized communications

2. Traffic volumes detecting spikes or drops which could indicate service issues in production or an increase in customer traffic

3. Latency and Performance bottlenecks - with VPC Flow logs, you can look at latency for a flow (in and outflows), and understand patterns

4. Accepted and rejected traffic helps determine where potential security threats and misconfigurations lie. 

AWS VPC Logs is a great example of how logs are great. Logging is an important part of Observability, for which we generally think of metrics and tracing. However, the amount of logs an application and the underlying infrastructure output can be significantly daunting with VPC Logs. However, it also provides a significant amount of insight.

Before we proceed, it is important to understand what Elastic provides in managing AWS and VPC Flow logs:

1. A full set of integrations to manage VPC Flows and the entire end-to-end deployment on AWS. 

2. Elastic has a simple-to-use AWS Firehose integration. 

3. Elastic’s tools such as Discover, spike analysis,  and anomaly detection help provide you with better insights and analysis.

4. And a set of simple Out-of-the-box dashboards

In today’s blog, we’ll cover how Elastics’ other features can support analyzing and RCA for potential VPC flow logs even more easily. Specifically, we will focus on managing the number of rejects, as this helps ensure there weren’t any unauthorized or unusual activities:

1. Set up an easy-to-use SLO (newly released) to detect when things are potentially degrading

2. Create an ML job to analyze different fields of the VPC Flow log

3. Using our newly released RAG-based AI Assistant to help analyze the logs without needing to know Elastic’s query language nor how to even graph on Elastic

4. ES|QL will help understand and analyze add latency for patterns.

In subsequent blogs, we will use AI Assistant and ESQL to show how to get other insights beyond just REJECT/ACCEPT from VPC Flow log.

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up this demonstration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).

• Follow the steps in the following blog to get AWS’s three-tier app installed instructed in git, and bring in the AWS VPC Flow logs.

• Ensure you have an ML node configured in your Elastic stack

• To use the AI Assistant you will need a trial or upgrade to Platinum.

SLO with VPC Flow Logs

Elastic’s SLO capability is based directly on the Google SRE Handbook. All the definitions and semantics are utilized as described in Google’s SRE handbook. Hence users can perform the following on SLOs in Elastic:

• Define an SLO on Logs not just metrics - Users can use KQL (log-based query), service availability, service latency, custom metric, histogram metric, or a timeslice metric.
• Define SLO, SLI, Error budget and burn rates. Users can also use occurrence versus time slice-based budgeting. 
• Manage, with dashboards, all the SLOs in a singular location.
• Trigger alerts from the defined SLO, whether the SLI is off, the burn rate is used up, or the error rate is X.

Setting up an SLO for VPC is easy. You simply create a query you want to trigger off. In our case, we look for all the good events where aws.vpcflow.action=ACCEPT and we define the target at 85%. 

As the following example shows, over the last 7 days, we have exceeded our budget by 43%. Additionally, we have not complied for the last 7 days.

Analyzing the SLO with AI Assistant

Now that we see that there is an issue with the VPC Flows, we immediately work with the AI Assistant to start analyzing the SLO. Because it's a chat interface we simply open the AI Assistant and work through some simple analysis: (See Animated GIF for a demo below)

AI Assistant analysis:

• what were the top 3 source.address that had aws.vpcflow.action=REJECT over the last 7 days, which is causing this SLO issue? - We wanted to simply see what could be causing the loss in error budget. Were there any particular source.addresses causing a heavy reject rate.

  • The answer: A table with the highest count = 42670 and source.address = 79.110.62.185

  • There is one singular source.address that is causing the loss in SLO. 

• What is the largest number of  aws.vpcflow.action=REJECT in a 30 min time frame for the last 3 days where the source.address=79.110.62.185? - After understanding that a specific source.address is causing the loss in SLO, we want to understand the averages. 

  • **The answer: ** "The largest number of aws.vpcflow.action=REJECT in a 30-minute time frame for the last 3 days where the source.address is 79.110.62.185 is 229. This occurred on 2024-06-01T04:00:00.000Z.”

  • It means there must be a low REJECT rate but fairly consistent vs spiky over the last 7 days.

• for the logs with source.address="79.110.62.185" was there any country code of source.geo.country_iso_code field present. If yes what is the value - Given the last question showed a low REJECT rate, it only means that this was fairly consistent vs spiky over the last 7 days.

  • The answer: Yes, there is a country code present in the source.geo.country_iso_code field for logs with source.address="79.110.62.185". The value is BG (Bulgaria).

• Is there a specific destination.address where source.address=79.110.62.185 is getting a aws.vpcflow.action=REJECT. Give me both the destination.address and the number of REJECTs for that destination.address?

  • The answer: destination.address of 10.0.0.27 is giving a reject number of 53433 in this time frame.

• Graph the number of REJECT vs ACCEPT for source.address="79.110.62.185" over the last 7 days. The graph is on a daily basis in a singular graph - We asked this question to see what the comparison is between ACCEPT and REJECT. 

  • The answer: See the animated GIF to see that the generated graph is fairly stable

• Were there any source.address that had a spike, high reject rate in. a 30min period over the 30 days? - We wanted to see if there was any other spike 

  • The answer - Yes, there was a source.address that had a spike in high reject rates in a 30-minute period over the last 30 days. source.address: 185.244.212.67, Reject Count: 8975, Time Period: 2024-05-22T03:00:00.000Z

Watch the flow

Potential issue:

he server handling requests from source 79.110.62.185 is potentially having an issue.

Again using logs, we essentially asked the AI Assistant to give the eni ids where the internal ip address was 10.0.0.27

From our AWS console, we know that this is the webserver. Further analysis in Elastic, and with the developers we realized there is a new version that was installed recently causing a problem with connections.

Locating anomalies with ML

While using the AI Assistant is great for analyzing information, another important aspect of VPC flow management is to ensure you can manage log spikes and anomalies. Elastic has a machine learning platform that allows you to develop jobs to analyze specific metrics or multiple metrics to look for anomalies.

VPC Flow logs come with a large amount of information. The full set of fields is listed in AWS docs. We will use a specific subset to help detect anomalies.

We were setting up anomalies for aws.vpcflow.action=REJECT, which requires us to use multimetric anomaly detection in Elastic.

The config we used utilizes:

Detectors:

• destination.address

• destination.port

Influencers:

• source.address

• aws.vpcflow.action

• destination.geo.region_iso_code

The way we set this up will help us understand if there is a large spike in REJECT/ACCEPT against destination.address values from a specific source.address and/or destination.geo.region_iso_code location.

The job once run reveals something interesting:

Notice that source.address 185.244.212.67 has had a high REJECT rate in the last 30 days. 

Notice where we found this before? In the AI Assistant!!!!!

While we can run the AI Assistant and find this sort of anomaly, the ML job can be setup to run continuously and alert us on such spikes. This will help us understand if there are any issues with the webserver like we found above or even potential security attacks.

Conclusion:

You’ve now seen how easily Elastic’s RAG-based AI Assistant can help analyze VPC Flows without even the need to know query syntax, understand where the data is, and understand even the fields. Additionally, you’ve also seen how we can alert you when a potential issue or degradation in service (SLO). Check out our other blogs on AWS VPC Flow analysis in Elastic:

1. A full set of integrations to manage VPC Flows and the entire end-to-end deployment on AWS. 

2. Elastic has a simple-to-use AWS Firehose integration. 

3. Elastic’s tools such as Discover, spike analysis,  and anomaly detection help provide you with better insights and analysis.

4. And a set of simple Out-of-the-box dashboards

Try it out

Existing Elastic Cloud customers can access many of these features directly from the Elastic Cloud console. Not taking advantage of Elastic on the cloud? Start a free trial.

All of this is also possible in your environment. Learn how to get started today.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

This is why adopting an AI-driven approach isn't just critical—it’s necessary to solve modern system challenges. Autonomous agents can automate complete operational workflows with minimal human intervention, empowering SRE teams to move beyond constant reactive issue resolution toward proactive system engineering. But here’s the key: the effectiveness of any autonomous agent depends entirely on the quality of its underlying data. By seamlessly integrating the Azure SRE Agent with Elastic Observability, we’re not just offering simple automation; we’re giving organizations a strategy to enter a new phase of governed, AI-driven autonomous operations. 

In this blog, we’ll go over how Elastic Observability and the Azure SRE Agent work together, how this integration empowers SREs with AI-driven operations, and how to get started.

The Power of Choice: Why Elastic Observability is the Foundation for AI-Driven Ops

For the modern SRE, Elastic Observability serves as the indispensable high-fidelity data foundation. Elastic transforms environmental complexity into a strategic asset by providing a unified, search-powered view of Logs, Metrics, and Traces.

The Azure SRE Agent requires more than just raw data; it requires governed, real-time production insights. Elastic delivers this through ES|QL, our piped-query language that allows for high-speed telemetry correlation and transformation. Specifically optimized for Elastic 9.2.0+ and Elasticsearch Serverless projects, this integration utilizes the Model Context Protocol (MCP) to provide the agent with deep system context.

Pro-Tip: To leverage this integration, ensure that the Agent Builder feature is enabled within your Elastic deployment, as this serves as the gateway for the agent to access your production environment securely.

Better Together: The Value of the Elastic and Azure SRE Agent Integration

Combining Elastic’s search-powered observability with Azure’s agentic automation creates a "Better Together" ecosystem that provides several strategic advantages:

• Smarter Detection & Remediation: Infuse Elastic’s real-time governed data and causal analysis into Azure SRE Agent workflows. This allows the agent to not only identify a symptom but also understand the underlying root cause.

• Context-Rich Investigation: SREs can accelerate triage by providing the agent with full production context—including the blast radius of an incident—directly where the SRE works. This eliminates the "swivel-chair" effect of switching between monitoring dashboards.

• Proactive Prevention: By utilizing historical trends and real-time signals from Elastic, the Azure SRE Agent can stop regressions and performance degradations before they impact the end-user experience.

• Natural Language Interaction: Through the Elasticsearch MCP server, SREs can query complex clusters using natural language, making deep data exploration accessible without needing to master complex Query syntax.

Practical Scenarios: Elastic-Powered SRE in Action

This integration empowers SREs to solve real-world problems through conversational automation:

1. Incident Triage: An SRE prompts the agent: "Search for errors in the last hour across all logs indices." The agent invokes the MCP tools in Agent Builder to return a prioritized list of error logs, identifying a service spike in seconds.

2. Performance Analysis: To identify a recurring pattern, an SRE commands: "Run an ES|QL query to find the top 10 error types." The agent uses ES|QL to aggregate telemetry, allowing the team to prioritize development fixes based on frequency.

3. Infrastructure Health: During a suspected Azure resource failure, an SRE can check the data layer by asking: "Show me metric information for my cluster." By invoking MCP tools, the agent determines if a node failure is impacting data availability.

Practical How-to Guide: Integrating Elastic with the Azure SRE Agent 

1. In Elastic via your Kibana interface - create an API Key and remember the key:

2. Find and copy your MCP Endpoint in Agent Builder:

3. In the Azure portal, find the SRE Agent service:

4. Create an Agent:

5. Add the Elastic Connector:

6. Talk to your agent. Use “/agent” to select your agent in the chat interface:

Conclusions

The integration of Elastic Observability and the Azure SRE Agent represents a strategic leap forward for cloud operations. By combining Elastic's superior data depth and ES|QL engine with Azure’s autonomous automation, organizations can drastically reduce MTTR, eliminate toil, and maximize the ROI of their Azure investments.

 Next Steps

Explore the Elasticsearch Observability solution implementation on Microsoft Marketplace and visit the Azure SRE Agent resource to begin your trial of Elastic-centric autonomous operations today.

Learn more by checking out the following links:

• Microsoft: Get started with Elasticsearch MCP server in Azure SRE Agent
• Elastic Agent Builder MCP Server
• Azure SRE Agent MCP Overview

Teams that have chosen OpenTelemetry for instrumentation face a choice between different instrumentation techniques and data collection approaches. Determining how to instrument and what mechanism to use can be challenging. In this blog, we will go over Elastic’s recommendations around some best practices for OpenTelemetry instrumentation:

• Automatic or manual? We’ll cover the need for one versus the other and provide recommendations based on your situation.
• Collector or direct from the application? While the traditional option is to use a collector, observability tools like Elastic Observability can take telemetry from OpenTelemetry applications directly.
• What to instrument from OTel SDKs. Traces and metrics are well contributed to (per the table in OTel docs), but logs are still in progress. Elastic^® is improving the progress with its contribution of ECS to OTel. Regardless of the status from OTel, you need to test and ensure these instrumentations work for you.
• Advantages and disadvantages of OpenTelemetry

OTel automatic or manual instrumentation: Which one should I use?

While there are two ways to instrument your applications with OpenTelemetry — automatic and manual — there isn’t a perfect answer, as it depends on your needs. There are pros and cons of using one versus another, such as:

• Auto-magic experience vs. control over instrumentation
• Customization vs. out-of-the-box data
• Instrumentation overhead
• Simplicity vs. flexibility

Additionally, you might even land on a combination depending on availability and need.

Let’s review both automatic and manual instrumentation and explore specific recommendations.

Auto-instrumentation

For most of the programming languages and runtimes, OpenTelemetry provides an auto-instrumentation approach for gathering telemetry data. Auto-instrumentation provides a set of pre-defined, out-of-the-box instrumentation modules for well-known frameworks and libraries. With that, users can gather telemetry data (such as traces, metrics, and logs) from well-known frameworks and libraries used by their application with only minimal or even no need for code changes.

Here are some of the apparent benefits of using auto-instrumentation:

• Quicker development and path to production. Auto-instrumentation saves time by accelerating the process of integrating telemetry into an application, allowing more focus on other critical tasks.
• Simpler maintenance by only having to update one line, which is usually the container start command where auto-instrumentation is configured, versus having to update multiple lines of code across multiple classes, methods, and services.
• Easier to keep up with the latest features and improvements in the OpenTelemetry project without manually updating the instrumentation of used libraries and/or code.

There are also some disadvantages and limitations of the auto-instrumentation approach:

• Auto-instrumentation collects telemetry data only for the frameworks and libraries in use for which an explicit auto-instrumentation module exists. In particular, it’s unlikely that auto-instrumentation would collect telemetry data for “exotic” or custom libraries.
• Auto-instrumentation does not capture telemetry for pure custom code (that does not use well-known libraries underneath).
• Auto-instrumentation modules come with a pre-defined, opinionated instrumentation logic that provides sufficient and meaningful information in the vast majority of cases. However, in some custom edge cases, the information value, structure, or level of detail of the data provided by auto-instrumentation modules might be not sufficient.
• Depending on the runtime, technology, and size of the target application, auto-instrumentation may come with a (slightly) higher start-up or runtime overhead compared to manual instrumentation. In the majority of cases, this overhead is negligible but may become a problem in some edge cases.

Here is an example of a Python application that was auto-instrumented with OpenTelemetry. If you had a Python application locally, you would add the code below to auto-instrument:

opentelemetry-instrument \
    --traces_exporter OTEL_TRACES_EXPORTER \
    --metrics_exporter OTLP_METRICS_EXPORTER \
    --service_name OTLP_SERVICE_NAME \
    --exporter_otlp_endpoint OTEL_EXPORTER_TRACES_ENDPOINT \
    python main.py

Learn more about auto-instrumentation with OpenTelemetry for Python applications.

Finally, developers familiar with OpenTelemetry's APIs can leverage their existing knowledge by using auto-instrumentation, avoiding the complexities that may arise from manual instrumentation. However, manual instrumentation might still be preferred for specific use cases or when custom requirements cannot be fully addressed by auto-instrumentation.

Combination: Automatic and Manual

Before we proceed with manual instrumentation, you can also use a combination of automatic and manual instrumentation. As we noted above, if you start to understand the application’s behavior, then you can determine if you need some additional instrumentation for code that is not being traced by auto-instrumentation.

Additionally, because not all the auto-instrumentation is equal across the OTel language set, you will probably need to manually instrument in some cases — for example, if auto-instrumentation of a Flask-based Python application doesn’t automatically show middleware calls like calls to the requests library. In this situation, you will have to go with manual instrumentation for the Python application if you want to also see middleware tracing. However, as these libraries mature, more support options will become available.

A combination is where most developers will ultimately land when the application gets to near production quality.

Manual instrumentation

If the auto-instrumentation does not cover your needs, you want to have more control over the instrumentation, or you’d like to treat instrumentation as code, using manual instrumentation is likely the right choice for you. As described above, you can use it as an enhancement to auto-instrumentation or entirely switch to manual instrumentation. If you eventually go down a path of manual instrumentation, it definitely provides more flexibility but also means you will have to not only code in the traces and metrics but also maintain it regularly.

As new features are added and changes to the libraries are made, the maintenance for the code may or may not be cumbersome. It’s a decision that requires some forethought.

Here are some reasons why you would potentially use manual instrumentation:

• You may already have some OTel instrumented applications using auto-instrumentation and need to add more telemetry for specific functions or libraries (like DBs or middleware), thus you will have to add manual instrumentation.
• You need more flexibility and control in terms of the application language and what you’d like to instrument.
• In case there's no auto-instrumentation available for your programming language and the technologies in use, manual instrumentation would be the way to go for your applications built using these languages.
• You might have to instrument for logging with an alternative approach, as logging is not yet stable for all the programming languages.
• You need to customize and enrich your telemetry data for your specific use cases — for example, you have a multi-tenant application and you need to get each tenant’s information and then use manual instrumentation via the OpenTelemetry SDK.

Recommendations for manual instrumentation
Manual instrumentation will require specific configuration to ensure you have the best experience with OTel. Below are Elastic’s recommendations (as outlined by the CNCF), for gaining the most benefits when instrumenting using the manual method:

1. Ensure that your provider configuration and tracer initialization is done properly.

2. Ensure you set up spans in all the functions you want traced.

3. Set up resource attributes correctly.

4. Use batch rather than simple processing.

Let’s review these individually:

1. Ensure that your provider configuration and tracer initialization is done properly.
The general rule of thumb is to ensure you configure all your variables and tracer initialization in the front of the application. Using the Elastiflix application’s Python favorite service as an example, we can see:

Tracer being set up globally

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import Resource

...


resource = Resource.create(resource_attributes)

provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(exporter)
provider.add_span_processor(processor)

# Sets the global default tracer provider
trace.set_tracer_provider(provider)

# Creates a tracer from the global tracer provider
tracer = trace.get_tracer(otel_service_name)

In the above, we’ve added the OpenTelemetry trace module and imported the TraceProvider , which is the entry point of the API. It provides access to the Tracer, which is the class responsible for creating spans.

Additionally, we specify the use of BatchSpanProcessor. The span processor is an interface that provides hooks for span start and end method invocations.

In OpenTelemetry, different span processors are offered. The BatchSpanProcessor batches span and sends them in bulk. Multiple span processors can be configured to be active at the same time using the MultiSpanProcessor. See OpenTelemetry Documentation.

The variable otel_service_name is set in with environment variables (i.e., OTLP ENDPOINT and others) also set up globally. See below:

otel_service_name = os.environ.get('OTEL_SERVICE_NAME') or 'favorite_otel_manual'
environment = os.environ.get('ENVIRONMENT') or 'dev'
otel_service_version = os.environ.get('OTEL_SERVICE_VERSION') or '1.0.0'

otel_exporter_otlp_headers = os.environ.get('OTEL_EXPORTER_OTLP_HEADERS')
otel_exporter_otlp_endpoint = os.environ.get('OTEL_EXPORTER_OTLP_ENDPOINT')

In the above code, we initialize several variables. Because we also imported Resource, we initialize several variables:

Resource variables (we will cover this later in this article):

• otel_service_name – This helps set the name of the service (service.name) in otel Resource attributes.
• otel_service_version – This helps set the version of the service (service.version) in OTel Resource attributes.
• environment – This helps set the deployment.environment variable in OTel Resource attributes.

Exporter variables:

• otel_exporter_otlp_endpoint – This helps set the OTLP endpoint where traces, logs, and metrics are sent. Elastic would be an OTLP endpoint. You can also use OTEL_TRACES_EXPORTER or OTEL_METRICS_EXPORTER if you want to only send traces and/or metrics to specific endpoints.
• Otel_exporter_otlp_headers – This is the authorization needed for the endpoint.

The separation of your provider and tracer configuration allows you to use any OpenTelemetry provider and tracing framework that you choose.

2. Set up your spans inside the application functions themselves.
Make sure your spans end and are in the right context so you can track the relationships between spans. In our Python favorite application, the function that retrieves a user’s favorite movies shows:

@app.route('/favorites', methods=['GET'])
def get_favorite_movies():
    # add artificial delay if enabled
    if delay_time > 0:
        time.sleep(max(0, random.gauss(delay_time/1000, delay_time/1000/10)))

    with tracer.start_as_current_span("get_favorite_movies") as span:
        user_id = str(request.args.get('user_id'))

        logger.info('Getting favorites for user ' + user_id, extra={
            "event.dataset": "favorite.log",
            "user.id": request.args.get('user_id')
        })

        favorites = r.smembers(user_id)

        # convert to list
        favorites = list(favorites)
        logger.info('User ' + user_id + ' has favorites: ' + str(favorites), extra={
            "event.dataset": "favorite.log",
            "user.id": user_id
        })
        return { "favorites": favorites}

While you can instrument every function, it’s strongly recommended that you instrument what you need to avoid a flood of data. The need will be dependent not only on the development process needs but also on what SRE and potentially the business needs to observe with the application. Instrument for your target use cases.

Also, avoid instrumenting trivial/utility methods/functions or such that are intended to be called extensively (e.g., getter/setter functions). Otherwise, this would produce a huge amount of telemetry data with very low additional value.

3. Set resource attributes and use semantic conventions

_ Resource attributes _
Attributes such as service.name, tracer, development.environment, and cloud are important in managing version, environment, cloud provider, etc. for the specific service. Resource attributes describe resources such as hosts, systems, processes, and services and do not change during the lifetime of the resource. Resource attributes are a great help for correlating data, providing additional context to telemetry data and, thus, helping narrow down root causes of problems during troubleshooting. While it is simple to set up in auto-instrument, you need to ensure you also send these through in your application.

Check out OpenTelemetry’s list of attributes that can be set in the OTel documentation.

In our auto-instrumented Python application from above, here is how we set up resource attributes:

opentelemetry-instrument \
    --traces_exporter console,otlp \
    --metrics_exporter console \
    --service_name your-service-name \
    --exporter_otlp_endpoint 0.0.0.0:4317 \
    python myapp.py

However, when instrumenting manually, you need to add your resource attributes and ensure you have consistent values across your application’s code. Resource attributes have been defined by OpenTelemetry’s Resource Semantic Convention and can be found here. In fact, your organization should have a resource attribute convention that is applied across all applications.

These attributes are added to your metrics, traces, and logs, helping you filter out data, correlate, and make more sense out of them.

Here is an example of setting resource attributes in our Python service:

resource_attributes = {
    "service.name": otel_service_name,
    "telemetry.version": otel_service_version,
    "Deployment.environment": environment

}

resource = Resource.create(resource_attributes)

provider = TracerProvider(resource=resource)

We’ve set up service.name, service.version, and deployment.environment. You can set up as many resource attributes as you need, but you need to ensure you pass the resource attributes into the tracer with provider = TracerProvider(resource=resource).

_ Semantic conventions _
In addition to adding the appropriate resource attributes to the code, the OpenTelemetry semantic conventions are important. Another one is about semantic conventions for specific technologies used in building your application with specific infrastructure. For example, if you need to instrument databases, there is no automatic instrumentation. You will have to manually instrument for tracing against the database. In doing so, you should utilize the semantic conventions for database calls in OpenTelemetry.

Similarly, if you are trying to trace Kafka or RabbitMQ, you can follow the OpenTelemetry semantic conventions for messaging systems.

There are multiple semantic conventions across several areas and signal types that can be followed using OpenTelemetry — check out the details.

4. Use Batch or simple processing?
Using simple or batch processing depends on your specific observability requirements. The advantages of batch processing include improved efficiency and reduced network overhead. Batch processing allows you to process telemetry data in batches, enabling more efficient data handling and resource utilization. On the other hand, batch processing increases the lag time for telemetry data to appear in the backend, as the span processor needs to wait for a sufficient amount of data to send over to the backend.

With simple processing, you send your telemetry data as soon as the data is generated, resulting in real-time observability. However, you will need to prepare for higher network overhead and more resources required to process all the separate data transmissions.

Here is what we used to set this up in Python:

from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(exporter)
provider.add_span_processor(processor)

# Sets the global default tracer provider
trace.set_tracer_provider(provider)

Your observability goals and budgetary constraints are the deciding factors when choosing batch or simple processing. A hybrid approach can also be implemented. If real-time insights are critical for an ecommerce application, for example, then simple processing would be the better approach. For other applications where real-time insights are not crucial, consider batch processing. Often, experimenting with both approaches and seeing how your observability backend handles the data is a fruitful exercise to hone in on what approach works best for the business.

Use the OpenTelemetry Collector or go direct?

When starting out with OpenTelemetry, ingesting and transmitting telemetry data directly to a backend such as Elastic is a good way to get started. Often, you would be using the OTel direct method in the development phase and in a local environment.

However, as you deploy your applications to production, the applications become fully responsible for ingesting and sending telemetry data. The amount of data sent in a local environment or during development would be miniscule compared to a production environment. With millions or even billions of users interacting with your applications, the work of ingesting and sending telemetry data in addition to the core application functions can become resource-intensive. Thus, offloading the collection, processing, and exporting of telemetry data over to a backend such as Elastic using the vendor-agnostic OTel Collector would enable your applications to perform more efficiently, leading to a better customer experience.

Advantages of using the OpenTelemetry Collector

For cloud-native and microservices-based applications, the OpenTelemetry Collector provides the flexibility to handle multiple data formats and, more importantly, offloads the resources required from the application to manage telemetry data. The result: reduced application overhead and ease of management as the telemetry configuration can now be managed in one place.

The OTel Collector is the most common configuration because the OTel Collector is used:

• To enrich the telemetry data with additional context information — for example, on Kubernetes, the OTel Collector would take the responsibility to enrich all the telemetry with the corresponding K8s pod and node information (labels, pod-name, etc.)
• To provide uniform and consistent processing or transform telemetry data in a central place (i.e., OTel Collector) rather than take on the burden of syncing configuration across hundreds of services to ensure consistent processing
• To aggregate metrics across multiple instances of a service, which is only doable on the OTel Collector (not within individual SDKs/agents)

Key features of the OpenTelemetry Collector include:

• Simple setup: The setup documentation is clear and comprehensive. We also have an example setup using Elastic and the OTel Collector documented from this blog.
• Flexibility: The OTel Collector offers many configuration options and allows you to easily integrate into your existing observability solution. However, OpenTelemetry’s pre-built distributions allow you to start quickly and build the features that you need. Here as well as below is an example of the code that we used to build our collector for an application running on Kubernetes.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otelcollector
spec:
  selector:
    matchLabels:
      app: otelcollector
  template:
    metadata:
      labels:
        app: otelcollector
    spec:
      serviceAccountName: default
      terminationGracePeriodSeconds: 5
      containers:
        - command:
            - "/otelcol"
            - "--config=/conf/otel-collector-config.yaml"
          image: otel/opentelemetry-collector:0.61.0
          name: otelcollector
          resources:
            limits:
              cpu: 1
              memory: 2Gi
            requests:
              cpu: 200m
              memory: 400Mi

• Collect host metrics: Using the OTel Collector allows you to capture infrastructure metrics, including CPU, RAM, storage capacity, and more. This means you won’t need to install a separate infrastructure agent to collect host metrics. An example OTel configuration for ingesting host metrics is below.

receivers:
  hostmetrics:
    scrapers:
      cpu:
      disk:

• Security: The OTel Collector operates in a secure manner by default. It can filter out sensitive information based on your configuration. OpenTelemetry provides these security guidelines to ensure your security needs are met.
• Tail-based sampling for distributed tracing: With OpenTelemetry, you can specify the sampling strategy you would like to use for capturing traces. Tail-based sampling is available by default with the OTel Collector. With tail-based sampling, you control and thereby reduce the amount of trace data collected. More importantly, you capture the most relevant traces, enabling you to spot issues within your microservices applications much faster.

What about logs?

OpenTelemetry’s approach to ingesting metrics and traces is a “clean-sheet design.” OTel developed a new API for metrics and traces and implementations for multiple languages. For logs, on the other hand, due to the broad adoption and existence of legacy log solutions and libraries, support from OTel is the least mature.

Today, OpenTelemetry’s solution for logs is to provide integration hooks to existing solutions. Longer term though, OpenTelemetry aims to incorporate context aggregation with logs thus easing logging correlation with metrics and traces. Learn more about OpenTelemetry’s vision.

Elastic has written up its recommendations in the following article: 3 models for logging with OpenTelemetry and Elastic. Here is a brief summary of what Elastic recommends:

1. Output logs from your service (alongside traces and metrics) using an embedded OpenTelemetry Instrumentation library to Elastic via the OTLP protocol.

2. Write logs from your service to a file scrapped by the OpenTelemetry Collector, which then forwards to Elastic via the OTLP protocol.

3. Write logs from your service to a file scrapped by Elastic Agent (or Filebeat), which then forwards to Elastic via an Elastic-defined protocol.

The third approach, where developers have their logs scraped using an Elastic Agent, is the recommended approach, as Elastic provides a widely adopted and proven method for capturing logs from applications and services using OTel. The first two approaches, although both use OTel instrumentation, are not yet mature and aren't ready for production-level applications.

Get more details about the three approaches in this Elastic blog which includes a deep-dive discussion with hands-on implementation, architecture, advantages, and disadvantages.

It’s not all sunshine and roses

OpenTelemetry is definitely beneficial to obtaining observability for modern cloud-native distributed applications. Having a standardized framework for ingesting telemetry reduces operational expenses and allows the organization to focus more on application innovation. Even with all the advantages of using OTel, there are some limitations that you should be aware of as well.

But first, here are the advantages of using OpenTelemetry:

• Standardized instrumentation: Having a consistent method for instrumenting systems up and down the stack gives organizations more operational efficiency and cost-effective observability.
• Auto-instrumentation: OTel provides organizations with the ability to auto-instrument popular libraries and frameworks enabling them to quickly get up and running and requiring minimal changes to the codebase.
• Vendor neutrality: Organizations don’t have to be tied to one vendor for their observability needs. In fact, they can use several of them, using OTel to try one out or have a more best-of-breed approach if desired.
• Future-proof instrumentation: Since OpenTelemetry is open-source and has a vast ecosystem of support, your organization will be using technology that will be constantly innovated and can scale and grow with the business.

There are some limitations as well:

• Instrumenting with OTel is a fork-lift upgrade. Organizations must be aware that time and effort needs to be invested to migrate proprietary instrumentation to OpenTelemetry.
• The language SDKs are at a different maturity level, so applications with alpha, beta, or experimental functional support may not provide the organization with the full benefits in the short term.

Over time, the disadvantages will be reduced, especially as the maturity level of the functional components improves. Check the OpenTelemetry status page for updates on the status of the language SDKs, the collector, and overall specifications.

Using Elastic and migrating to OpenTelemetry at your speed

Transitioning to OpenTelemetry is a challenge for most organizations, as it requires retooling existing proprietary APM agents on almost all applications. This can be daunting, but OpenTelemetry agents provide a mechanism to avoid having to modify the source code, otherwise known as auto-instrumentation. With auto-instrumentation, the only code changes will be to rip out the proprietary APM agent code. Additionally, you should also ensure you have an observability tool that natively supports OTel without the need for additional agents, such as Elastic Observability.

Elastic recently donated Elastic Common Schema (ECS) in its entirety to OTel. The goal was to ensure OTel can get to a standardized logging format. ECS, developed by the Elastic community over the past few years, provides a vehicle to allow OTel to provide a more mature logging solution.

Elastic provides native OTel support. You can directly send OTel telemetry into Elastic Observability without the need for a collector or any sort of processing normally used in the collector.

Here are the configuration options in Elastic for OpenTelemetry:

Most of Elastic Observability’s APM capabilities are available with OTel data. Some of these include:

• Service maps
• Service details (latency, throughput, failed transactions)
• Dependencies between services, distributed tracing
• Transactions (traces)
• Machine learning (ML) correlations
• Log correlation

In addition to Elastic’s APM and a unified view of the telemetry data, you will also be able to use Elastic’s powerful machine learning capabilities to reduce the analysis, and alerting to help reduce MTTR.

Although OpenTelemetry supports many programming languages, the status of its major functional components — metrics, traces, and logs — are still at various stages. Thus migrating applications written in Java, Python, and JavaScript are good choices to start with as their metrics, traces, and logs (for Java) are stable.

For the other languages that are not yet supported, you can easily instrument those using Elastic Agents, therefore running your observability platform in mixed mode (Elastic Agents with OpenTelemetry agents).

We ran a variation of our standard Elastic Agent application with one service flipped to OTel — the newsletter-otel service. But we can easily and as needed convert each of these services to OTel as development resources allow.

As a result, you can take advantage of the benefits of OpenTelemetry, which include:

• Standardization: OpenTelemetry provides a standard approach to telemetry collection, enabling consistency of processes and easier integration of different components.
• Vendor-agnostic: Since OpenTelemetry is open source, it is designed to be vendor-agnostic, allowing DevOps and SRE teams to work with other monitoring and observability backends reducing vendor lock-in.
• Flexibility and extensibility: With its flexible architecture and inherent design for extensibility, OpenTelemetry enables teams to create custom instrumentation and enrich their own telemetry data.
• Community and support: OpenTelemetry has a growing community of contributors and adopters. In fact, Elastic contributed to developing a common schema for metrics, logs, traces, and security events. Learn more here.

Once the other languages reach a stable state, you can then continue your migration to OpenTelemetry agents.

Summary

OpenTelemetry has become the de facto standard for ingesting metrics, traces, and logs from cloud-native applications. It provides a vendor-agnostic framework for collecting telemetry data, enabling you to use the observability backend of your choice.

Auto-instrumentation using OpenTelemetry is the fastest way for you to ingest your telemetry data and is an optimal way to get started with OTel. However, using manual instrumentation provides more flexibility, so it is often the next step in gaining deeper insights from your telemetry data.

OpenTelemetry visualization also allows you to ingest your data directly or by using the OTel Collector. For local development, going direct is a great way to get your data to your observability backend; however, with production workloads, using the OTel Collector is recommended. The collector takes care of all the data ingestion and processing, enabling your applications to focus on functionality and not have to deal with any telemetry data tasks.

Logging functionality is still at a nascent stage with OpenTelemetry, while ingesting metrics and traces is well established. For logs, if you’ve started down the OTel path, you can send your logs to Elastic using the OTLP protocol. Since Elastic has a very mature logging solution, a better approach would be to use an Elastic Agent to ingest logs.

Although the long-term benefits are clear, organizations need to be aware that adopting OpenTelemetry means they would own their own instrumentation. Thus, appropriate resources and effort need to be incorporated in the development lifecycle. Over time, however, OpenTelemetry brings standardization to telemetry data ingestion, offering organizations vendor-choice, scalability, flexibility, and future-proofing of investments.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Understanding Logs and Their Importance

Logs are records of events occurring within your infrastructure, typically including a timestamp, a message detailing the event, and metadata identifying the source. They are invaluable for diagnosing issues, providing early warnings, and speeding up problem resolution. Logs are often the primary signal that developers enable, offering significant detail for debugging, performance analysis, security, and compliance management.

The Logging Journey

The logging journey involves three basic steps: collection and ingestion, processing and enrichment, and analysis and rationalization. Let's explore each step in detail, covering some of the best practices for each section.

1. Log Collection and Ingestion

Collect Everything Relevant and Actionable

The first step is to collect all logs into a central location. This involves identifying all your applications and systems and collecting their logs. Comprehensive data collection ensures no critical information is missed, providing a complete picture of your system's behavior. In the event of an incident, having all logs in one place can significantly reduce the time to resolution. It's generally better to collect more data than you need, as you can always filter out irrelevant information later, as well as delete logs that are no longer needed more quickly.

Leverage Integrations

Elastic provides over 300 integrations that simplify data onboarding. These integrations not only collect data but also come with dashboards, saved searches, and pipelines to parse the data. Utilizing these integrations can significantly reduce manual effort and ensure data consistency.

Consider Ingestion Capacity and Costs

An important aspect of log collection is ensuring you have sufficient ingestion capacity at a manageable cost. When assessing solutions, be cautious about those that charge significantly more for high cardinality data, as this can lead to unexpectedly high costs in observability solutions. We'll talk more about cost effective log management later in this post.

Use Kafka for Large Projects

For larger organizations, implementing Kafka can improve log data management. Kafka acts as a buffer, making the system more reliable and easier to manage. It allows different teams to send data to a centralized location, which can then be ingested into Elastic.

2. Processing and Enrichment

Adopt Elastic Common Schema (ECS)

One key aspect of log collection is to have the most amount of normalization across all of your applications and infrastructure. Having a common semantic schema is crucial. Elastic contributed Elastic Common Schema (ECS) to OpenTelemetry (OTel), helping accelerate the adoption of OTel-based observability and security. This move towards a more normalized way to define and ingest logs (including metrics and traces) is beneficial for the industry.

Using ECS helps standardize field names and data structures, making data analysis and correlation easier. This common schema ensures your data is organized predictably, facilitating more efficient querying and reporting. Learn more about ECS here.

Optimize Mappings for High Volume Data

For high cardinality fields or those rarely used, consider optimizing or removing them from the index. This can improve performance by reducing the amount of data that needs to be indexed and searched. Our documentation has sections to tune your setup for disk usage, search speed and indexing speed.

Managing Structured vs. Unstructured Logs

Structured logs are generally preferable as they offer more value and are easier to work with. They have a predefined format and fields, simplifying information extraction and analysis. For custom logs without pre-built integrations, you may need to define your own parsing rules.

For unstructured logs, full-text search capabilities can help mitigate limitations. By indexing logs, full-text search allows users to search for specific keywords or phrases efficiently, even within large volumes of unstructured data. This is one of the main differentiators of Elastic's observability solution. You can simply search for any keyword or phrase and get results in real-time, without needing to write complex regular expressions or parsing rules at query time.

Schema-on-Read vs. Schema-on-Write

There are two main approaches to processing log data:

1. Schema-on-read: Some observability dashboarding capabilities can perform runtime transformations to extract fields from non-parsed sources on the fly. This is helpful when dealing with legacy systems or custom applications that may not log data in a standardized format. However, runtime parsing can be time-consuming and resource-intensive, especially for large volumes of data.

2. Schema-on-write: This approach offers better performance and more control over the data. The schema is defined upfront, and the data is structured and validated at the time of writing. This allows for faster processing and analysis of the data, which is beneficial for enrichment.

3. Analysis and Rationalization

Full-Text Search

Elastic's full-text search capabilities, powered by Elasticsearch, allow you to quickly find relevant logs. The Kibana Query Language (KQL) enhances search efficiency, enabling you to filter and drill down into the data to identify issues rapidly.

Here are a few examples of KQL queries:

// Filter documents where a field exists
http.request.method: *

// Filter documents that match a specific value
http.request.method: GET

// Search all fields for a specific value
Hello

// Filter documents where a text field contains specific terms
http.request.body.content: "null pointer"

// Filter documents within a range
http.response.bytes < 10000

// Combine range queries
http.response.bytes > 10000 and http.response.bytes <= 20000

// Use wildcards to match patterns
http.response.status_code: 4*

// Negate a query
not http.request.method: GET

// Combine multiple queries with AND/OR
http.request.method: GET and http.response.status_code: 400

Machine Learning Integration

Machine learning can automate the detection of anomalies and patterns within your log data. Elastic offers features like log rate analysis that automatically identify deviations from normal behavior. By leveraging machine learning, you can proactively address potential issues before they escalate.

It is recommended that organizations utilize a diverse arsenal of machine learning algorithms and techniques to effectively uncover unknown-unknowns in log files. Unsupervised machine learning algorithms, should be employed for anomaly detection on real-time data, with rate-controlled alerting based on severity.

By automatically identifying influencers, users can gain valuable context for automated root cause analysis (RCA). Log pattern analysis brings categorization to unstructured logs, while log rate analysis and change point detection help identify the root causes of spikes in log data.

Take a look at the documentation to get started with machine learning in Elastic.

Dashboarding and Alerting

Building dashboards and setting up alerting helps you monitor your logs in real-time. Dashboards provide a visual representation of your logs, making it easier to identify patterns and anomalies. Alerting can notify you when specific events occur, allowing you to take action quickly.

Cost-Effective Log Management

Use Data Tiers

Implementing index lifecycle management to move data across hot, warm, cold, and frozen tiers can significantly reduce storage costs. This approach ensures that only the most frequently accessed data is stored on expensive, high-performance storage, while older data is moved to more cost-effective storage solutions.

Our documentation explains how to set up Index Lifecycle Management.

Compression and Index Sorting

Applying best compression settings and using index sorting can further reduce the data footprint. Optimizing the way data is stored on disk can lead to substantial savings in storage costs and improve retrieval performance. As of 8.15, Elasticsearch provides an indexing mode called "logsdb". This is a highly optimized way of storing log data. This new way of indexing data uses 2.5 times less disk space than the default mode. You can read more about it here. This mode automatically applies the best combination of settings for compression, index sorting, and other optimizations that weren't accessible to users before.

Snapshot Lifecycle Management (SLM)

SLM allows you to back up your data and delete it from the main cluster, freeing up resources. If needed, data can be restored quickly for analysis, ensuring that you maintain the ability to investigate historical events without incurring high storage costs.

Learn more about SLM in the documentation.

Dealing with Large Amounts of Log Data

Managing large volumes of log data can be challenging. Here are some strategies to optimize log management:

1. Develop a logs deletion policy. Evaluate what data to collect and when to delete it.
2. Consider discarding DEBUG logs or even INFO logs earlier, and delete dev and staging environment logs sooner.
3. Aggregate short windows of identical log lines, which is especially useful for TCP security event logging.
4. For applications and code you control, consider moving some logs into traces to reduce log volume while maintaining detailed information.

Centralized vs. Decentralized Log Storage

Data locality is an important consideration when managing log data. The costs of ingressing and egressing large amounts of log data can be prohibitively high, especially when dealing with cloud providers.

In the absence of regional redundancy requirements, your organization may not need to send all log data to a central location. Consider keeping log data local to the datacenter where it was generated to reduce ingress and egress costs.

Cross-cluster search functionality enables users to search across multiple logging clusters simultaneously, reducing the amount of data that needs to be transferred over the network.

Cross-cluster replication is useful for maintaining business continuity in the event of a disaster, ensuring data availability even during an outage in one datacenter.

Monitoring and Performance

Monitor Your Log Management System

Using a dedicated monitoring cluster can help you track the performance of your Elastic deployment. Stack monitoring provides metrics on search and indexing activity, helping you identify and resolve performance bottlenecks.

Adjust Bulk Size and Refresh Interval

Optimizing these settings can balance performance and resource usage. Increasing bulk size and refresh interval can improve indexing efficiency, especially for high-throughput environments.

Logging Best Practices

Adjust Log Levels

Ensure that log levels are appropriately set for all applications. Customize log formats to facilitate easier ingestion and analysis. Properly configured log levels can reduce noise and make it easier to identify critical issues.

Use Modern Logging Frameworks

Implement logging frameworks that support structured logging. Adding metadata to logs enhances their usefulness for analysis. Structured logging formats, such as JSON, allow logs to be easily parsed and queried, improving the efficiency of log analysis. If you fully control the application and are already using structured logging, consider using Elastic's version of these libraries, which can automatically parse logs into ECS fields.

Leverage APM and Metrics

For custom-built applications, Application Performance Monitoring (APM) provides deeper insights into application performance, complementing traditional logging. APM tracks transactions across services, helping you understand dependencies and identify performance bottlenecks.

Consider collecting metrics alongside logs. Metrics can provide insights into your system's performance, such as CPU usage, memory usage, and network traffic. If you're already collecting logs from your systems, adding metrics collection is usually a quick process.

Traces can provide even deeper insights into specific transactions or request paths, especially in cloud-native environments. They offer more contextual information and excel at tracking dependencies across services. However, implementing tracing is only possible for applications you own, and not all developers have fully embraced it yet.

A combined logging and tracing strategy is recommended, where traces provide coverage for newer instrumented apps, and logging supports legacy applications and systems you don't own the source code for.

Conclusion

Effective log management is essential for maintaining system reliability and performance in today's complex software environments. By following these best practices, you can optimize your log management process, reduce costs, and improve problem resolution times.

Key takeaways include:

• Ensure comprehensive log collection with a focus on normalization and common schemas.
• Use appropriate processing and enrichment techniques, balancing between structured and unstructured logs.
• Leverage full-text search and machine learning for efficient log analysis.
• Implement cost-effective storage strategies and smart data retention policies.
• Enhance your logging strategy with APM, metrics, and traces for a complete observability solution.

Continuously evaluate and adjust your strategies to keep pace with the growing volume and complexity of log data, and you'll be well-equipped to ensure the reliability, performance, and security of your applications and infrastructure.

Check out our other blogs:

• Build better Service Level Objectives (SLOs) from logs and metrics
• AWS VPC Flow log analysis with GenAI in Elastic
• Migrating 1 billion log lines from OpenSearch to Elasticsearch
• Pruning incoming log volumes with Elastic

Ready to get started? Use Elastic Observability on Elastic Cloud — the hosted Elasticsearch service that includes all of the latest features.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elastic can make it easier for organizations to transform data into actionable insights and stop threats quickly with unified visibility across your environment — so mission-critical applications can keep running smoothly no matter what. From a free trial and fast deployment to sending logs to Elastic securely and frictionlessly, all you need to do is point and click to capture, store, and search data from your AWS services.

Monitoring EMR via Elastic Observability

In this article, we will delve into the following key aspects:

• Enabling EMR cluster metrics for Elastic integration: Learn the intricacies of configuring an EMR cluster to emit metrics that Elastic can effectively extract, paving the way for insightful analysis.
• Harnessing Kibana ^® dashboards for EMR workload analysis: Discover the potential of utilizing Kibana dashboards to dissect metrics related to an EMR workload. By gaining a deeper understanding, we open the doors to optimization opportunities.

Key benefits of AWS EMR integration

• Comprehensive monitoring: Monitor the health and performance of your EMR clusters in real time. Track metrics related to cluster status and utilization, node status, IO, and many others, allowing you to identify bottlenecks and optimize your data processing.
• Log analysis: Dive deep into EMR logs with ease. Our integration enables you to collect and analyze logs from your clusters, helping you troubleshoot issues and gain valuable insights.
• Cost optimization: Understand the cost implications of your EMR clusters. By monitoring resource utilization, you can identify opportunities to optimize your cluster configurations and reduce costs.
• Alerting and notifications: Set up custom alerts based on EMR metrics and logs. Receive notifications when performance thresholds are breached, ensuring that you can take action promptly.
• Seamless integration: Our integration is designed for ease of use. Getting started is simple, and you can start monitoring your EMR clusters quickly.

Accompanying these discussions is an illustrative solution architecture diagram, providing a visual representation of the intricacies and interactions within the proposed solution.

How to get started

Getting started with AWS EMR integration in Observability is easy. Here's a quick overview of the steps:

Prerequisites and configurations

If you intend to follow the steps outlined in this blog post, there are a few prerequisites and configurations that you should have in place beforehand.

1. You will need an account on Elastic Cloud and a deployed stack and agent. Instructions for deploying a stack on AWS can be found here. This is necessary for AWS EMR logging and analysis.

2. You will also need an AWS account with the necessary permissions to pull data from AWS. Details on the required permissions can be found in our documentation.

3. Finally, be sure to turn on EMR monitoring for the EMR cluster when you deploy the cluster.

Step 1: Create an account with Elastic

Create an account on Elastic Cloud by following the steps provided.

Step 2: Add integration

1. Log in to your Elastic Cloud on AWS deployment.

2. Click on Add Integration. You will be navigated to a catalog of supported integrations.

3. Search and select Amazon EMR.

Step 3: Configure integration

1. Click on the Add Amazon EMR button and provide the required details.

2. Provide the required access credentials to connect to your EMR instance.

3. You can choose to collect EMR metrics, EMR logs via S3, or EMR logs via Cloudwatch.

4. Click on the Save and continue button at the bottom of the page.

Step 4: Analyze and monitor

Explore the data using the out-of-the-box dashboards available for the integration. Select Discover from the Elastic Cloud top-level menu.

Or, create custom dashboards, set up alerts, and gain actionable insights into your EMR clusters' performance.

This integration streamlines the collection of vital metrics and logs, including Cluster Status, Node Status, IO, and Cluster Capacity. Some metrics gathered include:

• IsIdle: Indicates that a cluster is no longer performing work, but is still alive and accruing charges
• ContainerAllocated: The number of resource containers allocated by the ResourceManager
• ContainerReserved: The number of containers reserved
• CoreNodesRunning: The number of core nodes working
• CoreNodesPending: The number of core nodes waiting to be assigned
• MRActiveNodes: The number of nodes presently running MapReduce tasks or jobs
• MRLostNodes: The number of nodes allocated to MapReduce that have been marked in a LOST state
• HDFSUtilization: The percentage of HDFS storage currently used
• HDFSBytesRead/Written: The number of bytes read/written from HDFS (This metric aggregates MapReduce jobs only, and does not apply for other workloads on Amazon EMR.)
• TotalUnitsRequested/TotalNodesRequested/TotalVCPURequested: The target total number of units/nodes/vCPUs in a cluster as determined by managed scaling

Conclusion

Elastic is committed to fulfilling all your observability requirements, offering an effortless experience. Our integrations are designed to simplify the process of ingesting telemetry data, granting you convenient access to critical information for monitoring, analytics, and observability. The native AWS EMR integration underscores our dedication to delivering seamless solutions for your data needs. With this integration, you'll find the confidence to monitor, analyze, and optimize your EMR clusters, opening up exciting opportunities for your data-driven initiatives.

Start a free trial today

Start your own 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world. Your AWS Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with AWS.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Kubernetes audit logs are essential for ensuring the security, compliance, and transparency of Kubernetes clusters. However, with managed Kubernetes infrastructure, traditional audit file-based log shipping is often not supported, and audit logs are only available via the control plane API or the Cloud Provider logging facility. In this blog, we will show you how to ingest the audit logs from these other sources and still take advantage of the Elastic Kubernetes Audit Log Integration.

In this blog we will be focusing on AWS as our cloud provider and when ingesting logs from AWS you have several options:

• AWS Custom Logs integration (which we will utilize in this blog)
• AWS Firehose to send logs from Cloudwatch to Elastic
• AWS General integration which supports many AWS sources

In part 1 of this two-part series, we will focus on properly ingesting Kubernetes Audit, and part 2 will focus on investigation, analytics, and alerting.

Kubernetes auditing documentation describes the need for auditing in order to get answers to the questions below:

• What happened?
• When did it happen?
• Who initiated it?
• What resource did it occur on?
• Where was it observed?
• From where was it initiated (Source IP)?
• Where was it going (Destination IP)?

Answers to the above questions become important when an incident occurs and an investigation follows. Alternatively, it could just be a log retention use case for a regulated company trying to fulfill compliance requirements. 

We are giving special importance to audit logs in Kubernetes because audit logs are not enabled by default. Audit logs can take up a large amount of memory and storage. So, usually, it’s a balance between retaining/investigating audit logs against giving up resources budgeted otherwise for workloads to be hosted on the Kubernetes cluster. Another reason we’re talking about audit logs in Kubernetes is that, unlike usual container logs, after being turned on, these logs are orchestrated to write to the cloud provider’s logging service. This is true for most cloud providers because the Kubernetes control plane is managed by the cloud providers. It makes sense for cloud providers to use their built-in orchestration workflows involving the control plane for a managed service backed by their implementation of a logging framework.

Kubernetes audit logs can be quite verbose by default. Hence, it becomes important to selectively choose how much logging needs to be done so that all the audit requirements are met for the organization. This is done in the audit policy file. The audit policy file is submitted against the kube-apiserver. It is not necessary that all flavors of cloud-provider-hosted Kubernetes clusters allow you to play with the kube-apiserver directly. For example, AWS EKS allows for this logging to be done only by the control plane.

In this blog we will be using Elastic Kubernetes Service (Amazon EKS) on AWS with the Kubernetes Audit Logs that are automatically shipped to AWS CloudWatch.

A sample audit log for a secret by the name “empty-secret” created by an admin user on EKS  is logged on AWS CloudWatch in the following format: 

Once the audit logs show up on CloudWatch, it is time to consider how to transfer them to Elasticsearch. Elasticsearch is a great platform for creating dashboards that visualize different audit events recorded in a Kubernetes cluster. It is also a powerful tool for analyzing various audit events. For example, how many secret object creation attempts were made in an hour? 

Now that we established the Kubernetes audit logs are being logged in CloudWatch, let’s discuss how to get the logs ingested into Elasticsearch. Elasticsearch has an integration to consume logs written on CloudWatch. Just using this integration by default is going to get the JSON from CloudWatch as is i.e. the real audit log JSON is nested inside the wrapper CloudWatch JSON. When bringing logs to Elasticsearch, it is important that we use the Elastic Common Schema(ECS) to get the best search and analytics performance. This means that there needs to  be an ingest pipeline that parses a standard Kubernetes audit JSON message and creates an ECS Compliant document in Elasticsearch. Let’s dive into how to achieve this.

Elasticsearch has a Kubernetes integration using Elastic Agent to consume Kubernetes container logs from the console and audit logs written to a file path. For a cloud-provider use case, as described above, it may not be feasible to write audit logs to a path on the Kubernetes cluster. So, how do we leverage the ECS designed for parsing the Kubernetes audit logs already implemented in the Kubernetes integration to work on the CloudWatch audit logs? That is the most exciting plumbing piece! Let’s see how to do it.

What we’re going to do is:

• Read the Kubernetes audit logs from the cloud provider’s logging module, in our case, AWS CloudWatch since this is where logs reside. We will use Elastic Agent and Elasticsearch AWS Custom Logs integration to read from logs from CloudWatch. Note: please be aware, there are several Elastic AWS integration, we are specifically using the AWS Custom Logs integration.

• Create two simple ingest pipelines (we do this for best practices of isolation and composability) 

• The first pipeline looks for Kubernetes audit JSON messages and then redirects them to the second pipeline

• The second custom pipeline will associate the JSON message field with the correct field expected by the Elasticsearch Kubernetes Audit managed pipeline (aka the Integration) and then reroute the message to the correct data stream, kubernetes.audit_logs-default, which in turn applies all the proper mapping and ingest pipelines for the incoming message

• The overall flow will be

1. Create an AWS CloudWatch integration:

a.  Populate the AWS access key and secret pair values

b. In the logs section, populate the log ARN, Tags and Preserve the original event if you want to, and then Save this integration and exit from the page

2. Next, we will configure the custom ingest pipeline

We are doing this because we want to override what the generic managed pipeline does. We will retrieve the custom component name by searching for managed pipeline created as an asset when we install the AWS CloudWatch integration. In this case we will be adding the custom ingest pipeline logs-aws_logs.generic@custom

From the Dev tools console, run below. Here, we are extracting the message field from the CloudWatch JSON and putting the value in a field called kubernetes.audit. Then, we are rerouting this message to the default Kubernetes audit dataset or ECS that comes with Kubernetes integration

PUT _ingest/pipeline/logs-aws_logs.generic@custom
{
    "processors": [
      {
        "pipeline": {
          "if": "ctx.message.contains('audit.k8s.io')",
          "name": "logs-aws-process-k8s-audit"
        }
      }
    ]
}

PUT _ingest/pipeline/logs-aws-process-k8s-audit
{
  "processors": [
    {
      "json": {
        "field": "message",
        "target_field": "kubernetes.audit"
      }
    },
    {
      "remove": {
        "field": "message"
      }
    },
    {
      "reroute": {
        "dataset": "kubernetes.audit_logs",
        "namespace": "default"
      }
    }
  ]
}

Let’s understand this further:

• When we create a Kubernetes integration, we get a managed index template called logs-kubernetes.audit_logs that writes to the pipeline called logs-kubernetes.audit_logs-1.62.2 by default

• If we look into the pipeline logs-kubernetes.audit_logs-1.62.2, we see that all the processor logic is working against the field kubernetes.audit. This is the reason why our json processor in the above code snippet is creating a field called kubernetes.audit before dropping the original message field and rerouting. Rerouting is directed to the kubernetes.audit_logs dataset that backs the logs-kubernetes.audit_logs-1.62.2 pipeline (dataset name is derived from the pipeline name convention that’s in the format logs-<datasetname>-version)

3. Now let’s verify that the logs are actually flowing through and the audit message is being parsed

a. We will use Elastic Agent and enroll using Fleet and the integration policy we created in the Step 1. There are a number of ways to deploy Elastic Agent and for this exercise we will deploy using docker which is quick and easy.

% docker run --env FLEET_ENROLL=1 --env FLEET_URL=<<fleet_URL>> --env FLEET_ENROLLMENT_TOKEN=<<fleet_enrollment_token>>  --rm docker.elastic.co/beats/elastic-agent:8.19.12

b. Check the messages in Discover. In 8.15 there is also a new feature called Logs Explorer which provides an ability to see Kubernetes Audit logs (and container logs) with a few clicks (see image below). Voila! We can see the Kubernetes audit messages parsed!

4. Let's do a quick recap of what we did

We configured CloudWatch integration in Elasticsearch to read Kubernetes audit logs from CloudWatch. Then, we created custom ingest pipelines to reroute the audit messages to the correct data stream and all the OOTB mappings and parsing that come with the Kubernetes Audit Logs integration. 

In the next part, we’ll look at how to analyze the ingested Kubernetes Audit log data.

OpenTelemetry has won the hearts of the industry. Adoption is accelerating: the CNCF's 2024 Observability survey found OTel to be the fastest-growing project in the foundation's history, with the OTel Collector registering hundreds of millions of downloads. The proposition is compelling: write instrumentation once, ship it anywhere, avoid lock-in.

But here is what every platform team discovers once they cross into production: the collector sprawl problem. Hundreds of collector instances deployed across regions, Kubernetes namespaces, and bare-metal hosts. Configuration drift creeping in. An upgrade that has to be co-ordinated across a fleet of independent processes. A security patch that someone has to manually roll out to each one. And zero visibility into which collectors are running, healthy, or stuck.

This is the gap between "deploying OpenTelemetry" and "operating OpenTelemetry at scale." With Elastic 9.3, Elastic Agent closes that gap entirely. The Elastic Agent is now built on Elastic's Distribution of the OpenTelemetry Collector (EDOT) and, when managed by Fleet, gives platform teams a single control plane for configuring, updating, and monitoring every OTel collector in their estate — all while remaining compatible with the Beats-based integrations they already rely on.

The Collector Sprawl Problem and Why It Matters

OpenTelemetry's success has created a quiet operational debt for many organisations. Individual teams adopt the collector for their services: logs here, metrics there, a custom pipeline for the new microservice. Without a centralised management layer, each of these collectors becomes an independent snowflake: its own config file, its own upgrade cycle, its own failure domain.

The consequences are predictable. Configuration drift means collectors running different versions of the same pipeline, producing subtly incompatible data. Compliance teams ask "show me all the places data is collected and where it goes", and the honest answer is a spreadsheet that's already out of date.

This isn't a niche problem. A Gartner analysis of enterprise observability programmes consistently identifies operational overhead as the top barrier to expanding OTel adoption beyond initial pilots. The technology works. The tooling to manage it at scale is what's been missing.

How Elastic Agent Became an OTel Collector

To understand the significance of this, it helps to understand what Elastic Agent used to be, and what it is now.

Elastic Agent acts as a supervisor process: Before version 9.3, it managed a collection of separate Beats sub-processes (Filebeat, Metricbeat, Winlogbeat and so on), each running its own input/output lifecycle, each consuming its own memory footprint. The agent coordinated them, but the fundamental model was a collection of discrete daemons running under a parent.

With 9.3, that model has been replaced. Elastic Agent is now itself an instance of the EDOT Collector: Elastic's hardened, production-supported distribution of the upstream OTel Collector. The architectural shift has three important consequences.

First, the process model simplifies dramatically. Instead of a supervisor managing multiple sub-process lifecycles, there is a single EDOT Collector process. This means a smaller memory footprint, fewer things that can fail independently, and fewer processes to observe for health and performance.

Second, Beats functionality is preserved, not discarded. Rather than forcing a breaking migration, Elastic has introduced Beats Receivers: beat inputs and processors re-packaged as native OTel receiver components. A Filestream input is enabled by a filebeatreceiver. The same Filebeat configuration YAML you write today is automatically translated into the corresponding EDOT receiver configuration at runtime. Existing integrations, dashboards, and ingest pipelines continue to work without modification.

Third, the agent is now a first-class participant in the OTel ecosystem. It speaks OTLP natively, it runs standard OTel receivers, and it can be configured to sit alongside any other OTel-compatible tool in a modern observability pipeline.

Central Management with Fleet: Configuration, Lifecycle, and Visibility

The architectural shift above would be valuable on its own. But it becomes transformative when combined with Elastic Fleet, the centralised management plane for Elastic Agents.

Fleet gives platform and SRE teams a single console from which to manage every Elastic Agent (and by extension, every EDOT Collector instance) in their estate. The capabilities break into three categories: configuration management, lifecycle management, and fleet-wide observability.

Configuration management at scale

With Fleet, you define an Agent Policy — a declarative description of what a collector should do. What data should it collect? Via which receivers? Where should it export? The policy is authored once in Fleet's UI (or via its API), and pushed automatically to every agent enrolled in that policy. Change the policy, and every affected collector receives the update. No SSH. No Ansible playbook to maintain. No configuration drift.

Fleet pushes policies to enrolled agents across any environment. Agents send heartbeat and health data back, giving a live inventory of every collector in the estate.

Lifecycle management: upgrades, enrolment, and remediation

Perhaps the most operationally significant benefit of Fleet management is lifecycle control. With Fleet, upgrading a collector is a policy action: select the target version, select the scope (all agents, a specific policy group, a canary subset), and click. Fleet orchestrates the rolling upgrade, tracking status per agent and surfacing failures immediately.

This changes the security calculus fundamentally. When a vulnerability is disclosed in the OTel Collector binary, patching is a Fleet operation measured in minutes, not a change-management ceremony measured in days across SSH sessions to individual hosts.

Fleet also handles enrolment and de-enrolment. New hosts added to your infrastructure can be auto-enrolled into the appropriate policy based on tags or deployment tooling. Agents on decommissioned hosts can be removed from Fleet's inventory, ensuring your observability map reflects your actual infrastructure.

Fleet-wide observability of your collectors

Every Fleet-managed Elastic Agent ships monitoring telemetry about itself: CPU and memory consumption, event throughput, error rates, pipeline latency. This data flows into Elastic and is surfaced in the Fleet UI, giving you a live dashboard of every collector in your estate, not just the ones you happen to be watching.

For the first time, "how healthy is my observability pipeline?" becomes a question with a real-time, fleet-wide answer. You can identify agents that have stopped sending data, agents consuming unexpectedly high resources, and agents that have fallen behind on queue processing — before those problems surface as gaps in your monitoring data.

In the near future this capability will be offered to non-Fleet managed agents (aka standalone) and/or 3rd party OTel collectors provided by other vendors. These collectors can be configured via some other means but be monitored in Fleet - from both resource consumption and/or component pipeline health.

The Hybrid Agent: Beats Data and OTel Data, Simultaneously

One of the most practically significant capabilities introduced in 9.3 is what Elastic calls the Hybrid Agent: an Elastic Agent that can run both Beats-based receivers and native OTel receivers in the same pipeline, at the same time. This does not change anything for existing installations.

This matters enormously for real-world adoption. Most organisations arriving at OTel in 2025 and 2026 are not starting from a blank slate. They have years of investment in Beats-based integrations: Filebeat-powered log collection, Metricbeat-powered host metrics, bespoke ingest pipelines in Elasticsearch that normalise and enrich that data into ECS (Elastic Common Schema) format. The business value locked in those integrations (the dashboards, the alerts, the correlation logic) is not something they can afford to throw away in order to "go OTel."

The Hybrid Agent solves this by making the two worlds coexist. For example, in a single agent policy you can simultaneously configure:

• A filebeatreceiver collecting application logs in ECS format, routed through your existing ingest pipeline to its existing data stream
• A native OTel filelog receiver collecting OTel-native telemetry from your new services instrumented with the OTel SDK, stored in OTel-native data streams without touching ingest pipelines
• An OTel hostmetrics receiver collecting system metrics in semantic convention format alongside your existing Metricbeat-derived system metrics

The two lanes are independent. Beats-receiver data travels through ingest pipelines and lands in ECS-formatted data streams, exactly as it always has. Native OTel data follows OTel semantic conventions and is stored directly in OTel-native data streams, bypassing ingest pipelines. Your existing dashboards and alerts continue to work. Your new OTel-native workloads get the full OTel experience. The same agent, the same Fleet policy, the same management console.

This co-existence is the practical answer to the question every platform team eventually faces: "We want to adopt OTel properly but we can't break what we already have." The Hybrid Agent lets you migrate incrementally, service by service, on your timeline.

The Integration Catalogue: Turning Configuration into a One-Click Operation

Configuration management at scale is only as good as the configurations themselves. Elastic's integration catalogue — over 500 packages covering everything from NGINX and PostgreSQL to AWS CloudTrail and Kubernetes — extends naturally to the Hybrid Agent model.

From 9.3 onwards, the catalogue includes OTel integration packages alongside the existing Beats-based ones. Each OTel package contains two components:

• An Input package: the configuration for the corresponding OTel receiver (receivers, processors, pipeline wiring), ready to be applied to a Hybrid Agent policy
• A Content package: the assets associated with the application: pre-built dashboards, alerts, index templates, and saved queries, all calibrated for OTel semantic convention data

When an operator adds an OTel integration to an Agent Policy in Fleet, the receiver configuration is pushed to all enrolled agents. When those agents start ingesting data and it arrives in Elasticsearch, the content package assets are automatically installed based on metadata in the data received. The dashboard is ready before you've had time to wonder where it is.

The same policy can hold both OTel integrations and legacy Beats integrations. A real-world agent policy might simultaneously collect system metrics via the OTel hostmetrics receiver, application logs via filebeat receiver, and APM data via OTLP — all from one policy, all managed from Fleet, all visible in a unified Kibana experience.

A technical walk through of how this is done for NGINX data collection can be found here for reference. Currently management of Elastic Agents is done via existing Fleet protocols, however in the near future this will move over to OPAMP so that Fleet will be able to provide management to 3rd party OTel collectors as well.

For organisations on platforms not yet in Elastic's OS support matrix, 3rd-party OTel Collectors (such as Red Hat's OpenShift-native collector) can send data to Elastic using the OTLP exporter and be observed alongside all other collectors in their fleet.

What This Means in Practice: A Migration Story

Consider a mid-sized platform team operating 200 Linux hosts across three regions, currently running Elastic Agent 8.x with a mix of Filebeat and Metricbeat integrations. Their new services are being instrumented with the OTel SDK and they want to standardise on OTel going forward without disrupting the monitoring coverage they already have.

With a Fleet-managed upgrade to 9.3, their existing agents become Hybrid Agents automatically. Their Filebeat and Metricbeat configurations are internally translated to Beats receiver configurations and continue to run unmodified. Their existing dashboards still populate. Their ingest pipelines still fire. Nothing breaks.

They then add OTel integration packages to their Fleet policies for each new service. The OTel-instrumented microservices start sending OTLP data, received by native OTel receivers in the same agents. OTel-native dashboards appear automatically in Kibana. They now have both data universes in one place, managed from one console, visible in one interface.

Over the following quarters, as Beats-based integrations for their remaining services are superseded by OTel equivalents in the catalogue, they migrate them one by one, updating the Agent Policy in Fleet and watching the transition happen across all 200 hosts simultaneously, without touching a single one directly.

Looking Forward

Elastic has made a clear architectural bet: OpenTelemetry is the future of observability data collection, and the right response to that future is not to build a parallel OTel tool alongside the existing stack — it is to evolve the existing stack into OTel. The Hybrid Agent and EDOT Collector are the result of that bet.

Fleet central management is the operational layer that makes that bet practical at scale. OpenTelemetry gives you standardised, vendor-neutral instrumentation. Fleet gives you the operational control plane to manage those collectors like the production infrastructure they are, not like artisanal YAML files scattered across your estate.

The collector sprawl problem is solvable. The answer is a managed, policy-driven, centrally observable fleet of EDOT Collectors, and in Elastic 9.3, that answer is production-ready today.

Collecting JMX metrics with OpenTelemetry can be done in two main ways depending on your environment, requirements and constraints:

• from inside the JVM with the OpenTelemetry Instrumentation Java agent (or EDOT Java)
• from outside the JVM with the jmx-scraper.

Thorough this article, we will use the term "Java agent" to refer to the OpenTelemetry Java instrumentation agent, this also applies to the Elastic own distribution (EDOT Java) which is based on it and provides the same features.

This walkthrough uses a Tomcat server as the target and shows how to validate which metrics are emitted with the logging exporter.

The configuration examples in this article use Java system properties that must be passed using -D flags in the JVM startup command, equivalent environment variables can also be used for configuration.

Prerequisites

• A local Tomcat install (or any JVM app you can start with custom JVM flags)
• Java 8+ on the host, the Tomcat version used might require a more recent version though.
• An OpenTelemetry Collector endpoint if you want to ship metrics beyond local logging

Choosing between the Java agent and jmx-scraper

Use the Java agent (or EDOT Java) when you can modify JVM startup flags and want in-process collection with full context from the running application: this allows to capture traces, logs and metrics with a single tool deployment.

Use jmx-scraper when you cannot install an agent on the JVM or prefer out-of-process collection from a separate host. This requires the JVM and the network to be configured for remote JMX access and also dealing with authentication and credentials.

Both approaches rely on the same JMX metric mappings and can use the logging exporter for validation and then use OTLP to send metrics to the collector / an OTLP endpoint.

Option 1: Collect JMX metrics inside the JVM with the Java agent

OpenTelemetry Java instrumentation ships with a curated set of JMX metric mappings. For Tomcat, you just need to enable the Java agent and set otel.jmx.target.system=tomcat.

Step 1 - Download the OpenTelemetry Java agent

The agent is downloaded in /opt/otel but you can choose any location on the host. Make sure the path is consistent with the -javaagent flag in the next step.

mkdir -p /opt/otel
curl -L -o /opt/otel/opentelemetry-javaagent.jar \
  https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar

Step 2 - Configure Tomcat with bin/setenv.sh

Create or update bin/setenv.sh so Tomcat launches with the agent and JMX target system enabled.

#!/bin/bash
export CATALINA_OPTS="$CATALINA_OPTS \
  -javaagent:/opt/otel/opentelemetry-javaagent.jar \
  -Dotel.service.name=tomcat-demo \
  -Dotel.metrics.exporter=otlp,logging \
  -Dotel.jmx.target.system=tomcat"

This will configure the agent to log metrics (using the logging exporter) in addition to sending them to the Collector.

Step 3 - Validate the emitted metrics

Start Tomcat and watch stdout.

./bin/catalina.sh run

By defaults metrics are sampled and emitted every minute, so you might have to wait a bit for the metrics to be logged. If needed, you can use otel.metric.export.interval configuration to increase or reduce the frequency.

You should see logging exporter output with JVM and Tomcat metrics. Look for lines containing the LoggingMetricExporter class name.

INFO io.opentelemetry.exporter.logging.LoggingMetricExporter - MetricData{name=tomcat.threadpool.currentThreadsBusy, ...}
INFO io.opentelemetry.exporter.logging.LoggingMetricExporter - MetricData{name=jvm.memory.used, ...}

Step 4 - Send metrics to a Collector

Once metric capture is validated, you should be ready to send metrics to a collector.

You will have to:

• remove the logging exporter as it's no longer necessary for production
• configure the OTLP endpoint (otel.exporter.otlp.endpoint) and headers (otel.exporter.otlp.headers) if needed

The bin/setenv.sh file should be modified to look like this:

#!/bin/bash
export CATALINA_OPTS="$CATALINA_OPTS \
  -javaagent:/opt/otel/opentelemetry-javaagent.jar \
  -Dotel.service.name=tomcat-demo \
  -Dotel.jmx.target.system=tomcat \
  -Dotel.exporter.otlp.endpoint=https://your-collector:4317 \
  -Dotel.exporter.otlp.headers=Authorization=Bearer <your-token>"

When using the Java agent, the JVM metrics are automatically captured by the runtime-telemetry module, it is thus not necessary to include jvm in the otel.jmx.target.system configuration option.

Option 2: Collect JMX metrics from outside the JVM with jmx-scraper

When you cannot install an agent in the JVM or if only metrics are required, jmx-scraper lets you query JMX remotely and export metrics to an OTLP endpoint.

Step 1 - Enable remote JMX on Tomcat

Add JMX remote options to bin/setenv.sh and create access/password files.

Warning: This uses trivial credentials and disables SSL. Do not use this configuration in production.

mkdir -p /opt/jmx
cat <<EOF > ${CATALINA_HOME}/jmxremote.access
monitorRole readonly
EOF

cat <<EOF > ${CATALINA_HOME}/jmxremote.password
monitorRole monitorPass
EOF

chmod 600 ${CATALINA_HOME}/jmxremote.password

export CATALINA_OPTS="$CATALINA_OPTS \
  -Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.port=9010 \
  -Dcom.sun.management.jmxremote.rmi.port=9010 \
  -Dcom.sun.management.jmxremote.authenticate=true \
  -Dcom.sun.management.jmxremote.ssl=false \
  -Dcom.sun.management.jmxremote.access.file=${CATALINA_HOME}/jmxremote.access \
  -Dcom.sun.management.jmxremote.password.file=${CATALINA_HOME}/jmxremote.password \
  -Djava.rmi.server.hostname=127.0.0.1"

Step 2 - Download jmx-scraper

The jmx-scraper is downloaded in /opt/otel but you can choose any location on the host.

mkdir -p /opt/otel
curl -L -o /opt/otel/opentelemetry-jmx-scraper.jar \
  https://github.com/open-telemetry/opentelemetry-java-contrib/releases/latest/download/opentelemetry-jmx-scraper.jar

Step 3 - Check the JMX connection

Run jmx-scraper with credentials from previous step to confirm it can reach Tomcat. If the credentials are wrong, you will see authentication errors.

java -jar /opt/otel/opentelemetry-jmx-scraper.jar \
  -Dotel.jmx.service.url=service:jmx:rmi:///jndi/rmi://localhost:9010/jmxrmi
  -Dotel.jmx.username=monitorRole \
  -Dotel.jmx.password=monitorPass \
  -Dotel.jmx.target.system=tomcat \
  -test

You should get in the standard output:

• JMX connection test OK if the connection and authentication is successful
• JMX connection test ERROR otherwise

Step 4 - Validate the emitted metrics

Using the logging exporter allows to inspect metrics and attributes before sending them to a collector.

In order to capture both Tomcat and JVM metrics, it is required to set otel.jmx.target.system to tomcat,jvm.

java -jar /opt/otel/opentelemetry-jmx-scraper.jar \
  -Dotel.jmx.service.url=service:jmx:rmi:///jndi/rmi://localhost:9010/jmxrmi
  -Dotel.jmx.username=monitorRole \
  -Dotel.jmx.password=monitorPass \
  -Dotel.jmx.target.system=tomcat,jvm \
  -Dotel.metrics.exporter=logging

Step 5 - Send metrics to a Collector

After validation, to send metrics to an OTLP endpoint, you will have to:

• remove the -Dotel.metrics.exporter to restore the otlp default value.
• configure the OTLP endpoint (otel.exporter.otlp.endpoint) and headers (otel.exporter.otlp.headers) if needed

java -jar /opt/otel/opentelemetry-jmx-scraper.jar \
  -Dotel.jmx.service.url=service:jmx:rmi:///jndi/rmi://localhost:9010/jmxrmi
  -Dotel.jmx.username=monitorRole \
  -Dotel.jmx.password=monitorPass \
  -Dotel.jmx.target.system=tomcat,jvm \
  -Dotel.exporter.otlp.endpoint=https://your-collector:4317
  -Dotel.exporter.otlp.headers="Authorization=Bearer <your-token>"

Customizing the JMX Metrics Collection

Once the built-in Tomcat and JVM mappings are flowing, you can add custom rules with otel.jmx.config. Create a YAML file and pass its path alongside otel.jmx.target.system.

For example, the following custom.yaml file allows to capture the custom.jvm.thread.count metric from the java.lang:type=Threading MBean:

---
rules:
  - bean: "java.lang:type=Threading"
    mapping:
      ThreadCount:
        metric: custom.jvm.thread.count
        type: gauge
        unit: "{thread}"
        desc: Current number of live threads.

For complete reference on the configuration format and syntax, refer to jmx-metrics module in Opentelemetry Java instrumentation.

This custom configuration can be used both with jmx-scraper and Java agent, both support the otel.jmx.config configuration option, for example with jmx-scraper:

java -jar /opt/otel/opentelemetry-jmx-scraper.jar \
  -Dotel.jmx.service.url=service:jmx:rmi:///jndi/rmi://localhost:9010/jmxrmi
  -Dotel.jmx.username=monitorRole \
  -Dotel.jmx.password=monitorPass \
  -Dotel.jmx.target.system=tomcat,jvm \
  otel.jmx.config=/opt/otel/jmx/custom.yaml

You can pass multiple custom files as a comma-separated list to otel.jmx.config when you need to organize metrics by team or component.

Using the JMX Metrics in Kibana

Once you have collected the JMX metrics using one of the approaches described in this article, you can start using them in Kibana. You can build custom dashboards and visualizations to explore and analyze the metrics, create custom alerts on top of them or build MCP tools and AI Agents to use them in your agentic workflows.

Here is an example of how you can use the JMX metrics in Kibana through ES|QL:

TS metrics*
| WHERE telemetry.sdk.language == "java"
| WHERE service.name == ?instance
| STATS
    request_rate = SUM(RATE(tomcat.request.count))
  BY Time = BUCKET(@timestamp, 100, ?_tstart, ?_tend)

You can use the native metric and dimension names of the JMX metrics to build your queries. With the TS command you get first-class support for time series aggregation functions and dimensions on your metrics. This kind of queries constitute the building blocks for your dashboards, alerts, workflows and AI agent tools.

Here is an example of a dashboard that visualizes the typical JMX metrics for Apache Tomcat:

Conclusion

In this article, we have seen how to collect JMX metrics with OpenTelemetry using the Java agent or jmx-scraper. We have also seen how to use the JMX metrics in Kibana through ES|QL to build custom dashboards, alerts, workflows and AI agent tools.

This is just the beginning of what you can do with the JMX metrics and Elastic Observability. Try it out yourself and explore the full potential of your JMX metrics when combined with powerful features provided by the Elastic Observability platform.

Traces provide the "what" and "where" — what happened and where in your system. Continuous profiling refines this understanding by pinpointing the "why" and validating your hypotheses about the "what." Just like a full-body MRI scan, Elastic's whole-system continuous profiling (powered by eBPF) uncovers unknown-unknowns in your system. This includes not just your code, but also third-party libraries and kernel activity triggered by your application transactions. This comprehensive visibility improves your mean-time-to-detection (MTTD) and mean-time-to-recovery (MTTR) KPIs.

[Related article: Why metrics, logs, and traces aren’t enough]

Bridging the disconnect between continuous profiling and OTel traces

Historically, continuous profiling signals have been largely disconnected from OpenTelemetry (OTel) traces. Here's the exciting news: we're bridging this gap! We're introducing native correlation between continuous profiling signals and OTel traces, starting with Java.

Imagine this: You're troubleshooting a performance issue and identify a slow trace. Whole-system continuous profiling steps in, acting like an MRI scan for your entire codebase and system. It narrows down the culprit to the specific lines of code hogging CPU time within the context of your distributed trace. This empowers you to answer the "why" question with minimal effort and confidence, all within the same troubleshooting context.

Furthermore, by correlating continuous profiling with distributed tracing, Elastic Observability customers can measure the cloud cost and CO₂ impact of every code change at the service and transaction level.

This milestone is significant, especially considering the recent developments in the OTel community. With OTel adopting profiling and Elastic donating the industry’s most advanced eBPF-based continuous profiling agent to OTel, we're set for a game-changer in observability — empowering OTel end users with a correlated system visibility that goes from a trace span in the userspace down to the kernel.

Furthermore, achieving this goal, especially with Java, presented significant challenges and demanded serious engineering R&D. This blog post will delve into these challenges, explore the approaches we considered in our proof-of-concepts, and explain how we arrived at a solution that can be easily extended to other OTel language agents. Most importantly, this solution correlates traces with profiling signals at the agent, not in the backend — to ensure optimal query performance and minimal reliance on vendor backend storage architectures.

Figuring out the active OTel trace and span

The primary technical challenge in this endeavor is essentially the following: whenever the profiler interrupts an OTel instrumented process to capture a stacktrace, we need to be able to efficiently determine the active span and trace ID (per-thread) and the service name (per-process).

For the purpose of this blog, we'll focus on the recently released Elastic distribution of the OTel Java instrumentation, but the approach that we ended up with generalizes to any language that can load and call into a native library. So, how do we get our hands on those IDs?

The OTel Java agent itself keeps track of the active span by storing a stack of spans in the OpenTelemetryContext, which itself is stored in a ThreadLocal variable. We originally considered reading these Java structures directly from BPF, but we eventually decided against that approach. There is no documented specification on how ThreadLocals are implemented, and reliably reading and following the JVM's internal data-structures would incur a high maintenance burden. Any minor update to the JVM could change details of the structure layouts. To add to this, we would also have to reverse engineer how each JVM version lays out Java class fields in memory, as well as how all the high-level Java types used in the context objects are actually implemented under the hood. This approach further wouldn't generalize to any non-JVM language and needs to be repeated for any language that we wish to support.

After we had convinced ourselves that reading Java ThreadLocal directly is not the answer, we decided to look for more portable alternatives instead. The option that we ultimately settled with is to load and call into a C++ library that is responsible for making the required information available via a known and defined interface whenever the span changes.

Other than with Java's ThreadLocals, the details on how a native shared library should expose per-process and per-thread data are well-defined in the System V ABI specification and the architecture specific ELF ABI documents.

Exposing per-process information

Exposing per-process data is easy: we simply declare a global variable . . .

void* elastic_tracecorr_process_storage_v1 = nullptr;

. . . and expose it via ELF symbols. When the user initializes the OTel library to set the service name, we allocate a buffer and populate it with data in a protocol that we defined for this purpose. Once the buffer is fully populated, we update the global pointer to point to the buffer.

On the profiling agent side, we already have code in place that detects libraries and executables loaded into any process's address space. We normally use this mechanism to detect and analyze high-level language interpreters (e.g., libpython, libjvm) when they are loaded, but it also turned out to be a perfect fit to detect the OTel trace correlation library. When the library is detected in a process, we scan the exports, resolve the symbol, and read the per-process information directly from the instrumented process’ memory.

Exposing per-thread information

With the easy part out of the way, let's get to the nitty-gritty portion: exposing per-thread information via thread-local storage (TLS). So, what exactly is TLS, and how does it work? At the most basic level, the idea is to have one instance of a variable for every thread. Semantically you can think of it like having a global Map<ThreadID, T>, although that is not how it is implemented.

On Linux, there are two major options for thread locals: TSD and TLS.

Thread-specific data (TSD)

TSD is the older and probably more commonly known variant. It works by explicitly allocating a key via pthread_key_create — usually during process startup — and passing it to all threads that require access to the thread-local variable. The threads can then pass that key to the pthread_getspecific and pthread_setspecific functions to read and update the variable for the currently running thread.

TSD is simple, but for our purposes it has a range of drawbacks:

• The pthread_key_t structure is opaque and doesn't have a defined layout. Similar to the Java ThreadLocals, the underlying data-structures aren't defined by the ABI documents and different libc implementations (glibc, musl) will handle them differently.

• We cannot call a function like pthread_getspecific from BPF, so we'd have to reverse engineer and reimplement the logic. Logic may change between libc versions, and we’d have to detect the version and support all variants that may come up in the wild.

• TSD performance is not predictable and varies depending on how many thread local variables have been allocated in the process previously. This may not be a huge concern for Java specifically since spans are typically not swapped super rapidly, but it’d likely be quite noticeable for user-mode scheduling languages where the context might need to be swapped at every await point/coroutine yield.

None of this is strictly prohibitive, but a lot of this is annoying at the very least. Let’s see if we can do better!

Thread-local storage (TLS)

Starting with C11 and C++11, both languages support thread local variables directly via the _Thread_local and thread_local storage specifiers, respectively. Declaring a variable as per-thread is now a matter of simply adding the keyword:

thread_local void* elastic_tracecorr_tls_v1 = nullptr;

You might assume that the compiler simply inserts calls to the corresponding pthread function calls when variables declared with this are accessed, but this is not actually the case. The reality is surprisingly complicated, and it turns out that there are four different models of TLS that the compiler can choose to generate. For some of those models, there are further multiple dialects that can be used to implement them. The different models and dialects come with various portability versus performance trade-offs. If you are interested in the details, I suggest reading this blog article that does a great job at explaining them.

The TLS model and dialect are usually chosen by the compiler based on a somewhat opaque and complicated set of architecture-specific rules. Fortunately for us, both gcc and clang allow users to pick a particular one using the -ftls-model and -mtls-dialect arguments. The variant that we ended up picking for our purposes is -ftls-model=global-dynamic and -mtls-dialect=gnu2 (and desc on aarch64).

Let's take a look at the assembly that is being generated when accessing a thread_local variable under these settings. Our function:

void setThreadProfilingCorrelationBuffer(JNIEnv* jniEnv, jobject bytebuffer) {
  if (bytebuffer == nullptr) {
    elastic_tracecorr_tls_v1 = nullptr;
  } else {
    elastic_tracecorr_tls_v1 = jniEnv->GetDirectBufferAddress(bytebuffer);
  }
}

Is compiled to the following assembly code:

Both possible branches assign a value to our thread-local variable. Let’s focus at the right branch corresponding to the nullptr case to get rid of the noise from the GetDirectBufferAddress function call:

lea   rax, elastic_tracecorr_tls_v1_tlsdesc  ;; Load some pointer into rax.
call  qword ptr [rax]                        ;; Read & call function pointer at rax.
mov   qword ptr fs:[rax], 0                  ;; Assign 0 to the pointer returned by
                                             ;; the function that we just called.

The fs: portion of the mov instruction is the actual magic bit that makes the memory read per-thread. We’ll get to that later; let’s first look at the mysterious elastic_tracecorr_tls_v1_tlsdesc variable that the compiler emitted here. It’s an instance of the tlsdesc structure that is located somewhere in the .got.plt ELF section. The structure looks like this:

struct tlsdesc {
  // Function pointer used to retrieve the offset
  uint64_t (*resolver)(tlsdesc*);

  // TLS offset -- more on that later.
  uint64_t tp_offset;
}

The resolver field is initialized with nullptr and tp_offset with a per-executable offset. The first thread-local variable in an executable will usually have offset 0, the next one sizeof(first_var), and so on. At first glance this may appear to be similar to how TSD works, with the call to pthread_getspecific to resolve the actual offset, but there is a crucial difference. When the library is loaded, the resolver field is filled in with the address of __tls_get_addr by the loader (ld.so). __tls_get_addr is a relatively heavy function that allocates a TLS offset that is globally unique between all shared libraries in the process. It then proceeds by updating the tlsdesc structure itself, inserting the global offset and replacing the resolver function with a trivial one:

void* second_stage_resolver(tlsdesc* desc) {
  return tlsdesc->tp_offset;
}

In essence, this means that the first access to a tlsdesc based thread-local variable is rather expensive, but all subsequent ones are cheap. We further know that by the time that our C++ library starts publishing per-thread data, it must have gone through the initial resolving process already. Consequently, all that we need to do is to read the final offset from the process's memory and memorize it. We also refresh the offset every now and then to ensure that we really have the final offset, combating the unlikely but possible race condition that we read the offset before it was initialized. We can detect this case by comparing the resolver address against the address of the __tls_get_addr function exported by ld.so.

Determining the TLS offset from an external process

With that out of the way, the next question that arises is how to actually find the tlsdesc in memory so that we can read the offset. Intuitively one might expect that the dynamic symbol exported on the ELF file points to that descriptor, but that is not actually the case.

$ readelf --wide --dyn-syms elastic-jvmti-linux-x64.so | grep elastic_tracecorr_tls_v1
328: 0000000000000000 	8 TLS 	GLOBAL DEFAULT   19 elastic_tracecorr_tls_v1

The dynamic symbol instead contains an offset relative to the start of the .tls ELF section and points to the initial value that libc initializes the TLS value with when it is allocated. So how does ld.so find the tlsdesc to fill in the initial resolver? In addition to the dynamic symbol, the compiler also emits a relocation record for our symbol, and that one actually points to the descriptor structure that we are looking for.

$ readelf --relocs --wide elastic-jvmti-linux-x64.so | grep R_X86_64_TLSDESC
00000000000426e8  0000014800000024 R_X86_64_TLSDESC   	0000000000000000
elastic_tracecorr_tls_v1 + 0

To read the final TLS offset, we thus simply have to:

• Wait for the event notifying us about a new shared library being loaded into a process

• Do some cheap heuristics to detect our C++ library, avoiding the more expensive analysis below from being executed for every unrelated library on the system

• Analyze the library on disk and scan ELF relocations for our per-thread variable to extract the tlsdesc address

• Rebase that address to match where our library was loaded in that particular process

• Read the offset from tlsdesc+8

Determining the TLS base

Now that we have the offset, how do we use that to actually read the data that the library puts there for us? This brings us back to the magic fs: portion of the mov instruction that we discussed earlier. In X86, most memory operands can optionally be supplied with a segment register that influences the address translation.

Segments are an archaic construct from the early days of 16-bit X86 where they were used to extend the address space. Essentially the architecture provides a range of segment registers that can be configured with different base addresses, thus allowing more than 16-bits worth of memory to be accessed. In times of 64-bit processors, this is hardly a concern anymore. In fact, X86-64 aka AMD64 got rid of all but two of those segment registers: fs and gs.

So why keep two of them? It turns out that they are quite useful for the use-case of thread-local data. Since every thread can be configured to have its own base address in these segment registers, we can use it to point to a block of data for this specific thread. That is precisely what libc implementations on Linux are doing with the fs segment. The offset that we snatched from the processes memory earlier is used as an address with the fs segment register, and the CPU automatically adds it to the per-thread base address.

To retrieve the base address pointed to by the fs segment register in the kernel, we need to read its destination from the kernel’s task_struct for the thread that we happened to interrupt with our profiling timer event. Getting the task struct is easy because we are blessed with the bpf_get_current_task BPF helper functions. BPF helpers are pretty much syscalls for BPF programs: we can just ask the Linux kernel to hand us the pointer.

Armed with the task pointer, we now have to read the thread.fsbase (X86-64) or thread.uw.tp_value (aarch64) field to get our desired base address that the user-mode process accesses via fs. This is where things get complicated one last time, at least if we wish to support older kernels without BTF support (we do!). The task_struct is huge and there are hundreds of fields that can be present or not depending on how the kernel is configured. Being a core primitive of the scheduler, it is also constantly subject to changes between different kernel versions. On modern Linux distributions, the kernel is typically nice enough to tell us the offset via BTF. On older ones, the situation is more complicated. Since hardcoding the offset is clearly not an option if we hope the code to be portable, we instead have to figure out the offset by ourselves.

We do this by consulting /proc/kallsyms, a file with mappings between kernel functions and their addresses, and then using BPF to dump the compiled code of a kernel function that rarely changes and uses the desired offset. We dynamically disassemble and analyze the function and extract the offset directly from the assembly. For X86-64 specifically, we dump the aout_dump_debugregs function that accesses thread->ptrace_bps, which has consistently been 16 bytes away from the fsbase field that we are interested in for all kernels that we have ever looked at.

Reading TLS data from kernel

With all the required offsets at our hands, we can now finally do what we set out to do in the first place: use them to enrich our stack traces with the OTel trace and span IDs that our C++ library prepared for us!

void maybe_add_otel_info(Trace* trace) {
  // Did user-mode insert a TLS offset for this process? Read it.
  TraceCorrProcInfo* proc = bpf_map_lookup_elem(&tracecorr_procs, &trace->pid);

  // No entry -> process doesn't have the C++ library loaded.
  if (!proc) return;

  // Load the fsbase offset from our global configuration map.
  u32 key = 0;
  SystemConfig* syscfg = bpf_map_lookup_elem(&system_config, &key);

  // Read the fsbase offset from the kernel's task struct.
  u8* fsbase;
  u8* task = (u8*)bpf_get_current_task();
  bpf_probe_read_kernel(&fsbase, sizeof(fsbase), task + syscfg->fsbase_offset);

  // Use the TLS offset to read the **pointer** to our TLS buffer.
  void* corr_buf_ptr;
  bpf_probe_read_user(
    &corr_buf_ptr,
    sizeof(corr_buf_ptr),
    fsbase + proc->tls_offset
  );

  // Read the information that our library prepared for us.
  TraceCorrelationBuf corr_buf;
  bpf_probe_read_user(&corr_buf, sizeof(corr_buf), corr_buf_ptr);

  // If the library reports that we are currently in a trace, store it into
  // the stack trace that will be reported to our user-land process.
  if (corr_buf.trace_present && corr_buf.valid) {
    trace->otel_trace_id.as_int.hi = corr_buf.trace_id.as_int.hi;
    trace->otel_trace_id.as_int.lo = corr_buf.trace_id.as_int.lo;
    trace->otel_span_id.as_int = corr_buf.span_id.as_int;
  }
}

Sending out the mappings

From this point on, everything further is pretty simple. The C++ library sets up a unix datagram socket during startup and communicates the socket path to the profiler via the per-process data block. The stacktraces annotated with the OTel trace and span IDs are sent from BPF to our user-mode profiler process via perf event buffers, which in turn sends the mappings between OTel span and trace and stack trace hashes to the C++ library. Our extensions to the OTel instrumentation framework then read those mappings and insert the stack trace hashes into the OTel trace.

This approach has a few major upsides compared to the perhaps more obvious alternative of sending out the OTel span and trace ID with the profiler’s stacktrace records. We want the stacktrace associations to be stored in the trace indices to allow filtering and aggregating stacktraces by the plethora of fields available on OTel traces. If we were to send out the trace IDs via the profiler's gRPC connection instead, we’d have to search for and update the corresponding OTel trace records in the profiling collector to insert the stack trace hashes.

This is not trivial: stacktraces are sent out rather frequently (every 5 seconds, as of writing) and the corresponding OTel trace might not have been sent and stored by the time the corresponding stack traces arrive in our cluster. We’d have to build a kind of delay queue and periodically retry updating the OTel trace documents, introducing avoidable database work and complexity in the collectors. With the approach of sending stacktrace mappings to the OTel instrumented process instead, the need for server-side merging vanishes entirely.

Trace correlation in action

With all the hard work out of the way, let’s take a look at what trace correlation looks like in action!

Future work: Supporting other languages

We have demonstrated that trace correlation can work nicely for Java, but we have no intention of stopping there. The general approach that we discussed previously should work for any language that can efficiently load and call into our C++ library and doesn’t do user-mode scheduling with coroutines. The problem with user-mode scheduling is that the logical thread can change at any await/yield point, requiring us to update the trace IDs in TLS. Many such coroutine environments like Rust’s Tokio provide the ability to register a callback for whenever the active task is swapped, so they can be supported easily. Other languages, however, do not provide that option.

One prominent example in that category is Go: goroutines are built on user-mode scheduling, but to our knowledge there’s no way to instrument the scheduler. Such languages will need solutions that don’t go via the generic TLS path. For Go specifically, we have already built a prototype that uses pprof labels that are associated with a specific Goroutine, having Go’s scheduler update them for us automatically.

Getting started

We hope this blog post has given you an overview of correlating profiling signals to distributed tracing, and its benefits for end-users.

To get started, download the Elastic distribution of the OTel agent, which contains the new trace correlation library. Additionally, you will need the latest version of Universal Profiling agent, bundled with Elastic Stack version 8.13.

Acknowledgment

We appreciate Trask Stalnaker, maintainer of the OTel Java agent, for his feedback on our approach and for reviewing the early draft of this blog post.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Efficiency is important (again)

Before we jump into continuous profiling, let's start with the "Why should I care?" question. To do that, I'd like to talk a bit about efficiency and some large-scale trends happening in our industry that are making efficiency, specifically computational efficiency, important again. I say again because in the past, when memory and storage on a computer was very limited and you had to worry about every byte of code, efficiency was an important aspect of developing software.

The end of Moore’s Law

First, the Moore's Law era is drawing to a close. This was inevitable simply due to physical limits of how small you can make a transistor and the connections between them. For a long time, software developers had the luxury of not worrying about complexity and efficiency because the next generation of hardware would mitigate any negative cost or performance impact.

If you can't rely on an endless progression of ever faster hardware, you should be interested in computational efficiency.

The move to Software-as-a-Service

Another trend to consider is the shift from software vendors that sold customers software to run themselves to Software-as-a-Service businesses. A traditional software vendor didn't have to worry too much about the efficiency of their code. That issue largely fell to the customer to address; a new software version might dictate a hardware refresh to the latest and most performant. For a SaaS business, inefficient software usually degrades the customer’s experience and it certainly impacts the bottom line.

If you are a SaaS business in a competitive environment, you should be interested in computational efficiency.

Cloud migration

Next is the ongoing cloud migration to cloud computing. One of the benefits of cloud computing is the ease of scaling, both hardware and software. In the cloud, we are not constrained by the limits of our data centers or the next hardware purchase. Instead we simply spin up more cloud instances to mitigate performance problems. In addition to infrastructure scalability, microservices architectures, containerization, and the rise of Kubernetes and similar orchestration tools means that scaling services is simpler than ever. It's not uncommon to have thousands of instances of a service running in a cloud environment. This ease of scaling accounts for another trend, namely that many businesses are dealing with skyrocketing cloud computing costs.

If you are a business with ever increasing cloud costs, you should be interested in computational efficiency.

Our changing climate

Lastly, if none of those reasons pique your interest, let's consider a global problem that all of us should have in mind — namely, climate change. There are many things that need to be addressed to tackle climate change, but with our dependence on software in every part of our society, computational efficiency is certainly something we should be thinking about.

Thomas Dullien, distinguished engineer at Elastic and one of the founders of Optymize points out that if you can save 20% on 800 servers, and assume 300W power consumption for each server, that code change is worth 160 metric tons of CO₂ saved per year. That may seem like a drop in the bucket but if all businesses focus more on computational efficiency, it will make an impact. Also, let's not forget the financial benefits: those 160 metric tons of CO₂ savings also represent a significant annual cost savings.

If you live on planet Earth, you should be interested in computational efficiency.

Performance engineering

Who's job is it to worry about computational efficiency? Application developers usually pay at least some attention to efficiency as they develop their code. Profiling is a common approach for a developer to understand the performance of their code, and there is an entire portfolio of profiling tools available. Frequently, however, schedule pressures trump time spent on performance analysis and computational efficiency. In addition, performance problems may not become apparent until an application is running at scale in production and interacting (and competing) with everything else in that environment. Many profiling tools are not well suited to use in a production environment because they require code instrumentation and recompilation and add significant overhead.

When inefficient code makes it into production and begins to cause performance problems, the next line of defense is the Operations or SRE team. Their mission is to keep everything humming, and performance problems will certainly draw attention. Observability tools such as APM can shed light on these types of issues and lead the team to a specific application or service, but these tools have limits into the observability of the full system. Third-party libraries and operating system kernels functions remain hidden without a profiling solution in the production environment.

So, what can these teams do when there is a need to investigate a performance problem in production? That's where continuous profiling comes into the picture.

Continuous profiling

Continuous profiling is not a new idea. Google published a paper about it in 2010 and began implementing continuous profiling in its environments around that time. Facebook and Netflix followed suit not long afterward.

Typically, continuous profiling tools have been the domain of dedicated performance engineering or operating system engineering teams, which are usually only found at extremely large scale enterprises like the ones mentioned above. The key idea is to run profiling on every server, all of the time. That way, when your observability tools point you to a specific part of an application, but you need a more detailed view into exactly where that application is consuming CPU resources, the profiling data will be there, ready to use.

Another benefit of continuous profiling is that it provides a view of CPU intensive software across your entire environment — whether that is a very CPU intensive function or the aggregate of a relatively small function that is run thousands of times a second in your environment.

While profiling tools are not new, most of them have significant gaps. Let's look at a couple of the most significant ones.

• Limited visibility. Modern distributed applications are composed of a complex mix of building blocks, including custom software functions, third-party software libraries, networking software, operating system services, and more and more often, orchestration software such as Kubernetes. To fully understand what is happening in an application, you need visibility into each piece. However, even if a developer has the ability to profile their own code, everything else remains invisible. To make matters worse, most profiling tools require instrumenting the code, which adds overhead and therefore even your developers’ code is not profiled in production.
• Missing symbols in production. All of these pieces of code building blocks typically have descriptive names (some more intuitive than others) so that developers can understand and make sense of them. In a running program, these descriptive names are usually referred to as symbols. For a human being to make sense of the execution of a running application, these names are very important. Unfortunately, almost always, any software running in production has these human readable symbols stripped away for space efficiency since they are not needed by the CPU executing the software. Without all of the symbols, it makes it much more difficult to understand the full picture of what's happening in the application. To illustrate this, think of the last time you were in an SMS chat on your mobile device and you only had some of the people in the chat group in your address book while the rest simply appeared as phone numbers — this makes it very hard to tell who is saying what.

Elastic Universal Profiling: Continuous profiling for all

Our goal is to allow any business, large or small, to make computational efficiency a core consideration for all of the software that they run. Universal Profiling imposes very low overhead on your servers so it can be used in production and it provides visibility to everything running on every machine. It opens up the possibility of seeing the financial unit cost and CO₂ impact of every line of code running on every system in your business. How do we do that?

Whole-system visibility — SIMPLE

Universal Profiling is based on eBPF, which means that it imposes very low overhead (our goal is less than 1% CPU and less than 250MB of RAM) on your servers because it doesn't require code instrumentation. That low overhead means it can be run continuously, on every server, even in production.

eBPF also lets us deploy a single profiler agent on a host and peek inside the operating system to see every line of code executing on the CPU. That means we have visibility into all of those application building blocks described above — the operating system itself as well as containerization and orchestration frameworks without complex configuration.

All the symbols

A key part of Universal Profiling is our hosted symbolization service. This means that symbols are not required on your servers, which not only eliminates a need for recompiling software with symbols, but it also helps to reduce overhead by allowing the Universal Profiling agent to send very sparse data back to the Elasticsearch platform where it is enriched with all of the missing symbols. Since we maintain a repository of most popular third-party software libraries and Linux operating system symbols, the Universal Profiling UI can show you all the symbols.

Your favorite language, and then some

Universal Profiling is multilanguage. We support all of today’s popular programming languages, including Python, Go, Java (and any other JVM-based languages), Ruby, NodeJS, PHP, Perl, and of course, C and C++, which is critical since these languages still underly so many third-party libraries used by the other languages. In addition, we support profiling native code a.k.a. machine language.

Speaking of native code, all profiling tools are tied to a specific type of CPU. Most tools today only support the Intel x86 CPU architecture. Universal Profiling supports both x86 and ARM-based processors. With the expanding use of ARM-based servers, especially in cloud environments, Universal Profiling future-proofs your continuous profiling.

Many businesses today employ polyglot programming — that is, they use multiple languages to build an application — and Universal Profiling is the only profiler available that can build a holistic view across all of these languages. This will help you look for hotspots in the environment, leading you to "unknown unknowns" that warrant deeper performance analysis. That might be a simple interest rate calculation that should be efficient and lightweight but, surprisingly, isn't. Or perhaps it is a service that is reused much more frequently than originally expected, resulting in thousands of instances running across your environment every second, making it a prime target for efficiency improvement.

Visualize your impact

Elastic Universal Profiling has an intuitive UI that immediately shows you the impact of any given function, including the time it spends executing on the CPU and how much that costs both in dollars and in carbon emissions.

Finally, with the level of software complexity in most production environments, there's a good chance that making a code change will have unanticipated effects across the environment. That code change may be due to a new feature being rolled out or a change to improve efficiency. In either case, a differential view, before and after the change, will help you understand the impact.

Let's recap

Computational efficiency is an important topic, both from the perspective of the ultra-competitive business climate we all work in and from living through the challenges of our planet's changing climate. Improving efficiency can be a challenging endeavor, but we can't even begin to attempt to make improvements without knowing where to focus our efforts. Elastic Universal Profiling is here to provide every business with visibility into computational efficiency.

How will you use Elastic Universal Profiling in your business?

• If you are an application developer or part of the site reliability team, Universal Profiling will provide you with unprecedented visibility into your applications that will not only help you troubleshoot performance problems in production, but also understand the impact of new features and deliver an optimal user experience.
• If you are involved in cloud and infrastructure financial management and capacity planning, Universal Profiling will provide you with unprecedented visibility into the unit cost of every line of code that your business runs.
• If you are involved in your business’s ESG initiative, Universal Profiling will provide you with unprecedented visibility into your CO₂ emissions and open up new avenues for reducing your carbon footprint.

These are just a few examples. For more ideas, read how AppOmni benefits from Elastic Universal Profiling.

You can get started with Elastic Universal Profiling right now!

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

The next simplest option is to generate custom metrics directly from your code (e.g., by adding code using the OpenTelemetry Metrics API directly into the application). The major downside of that approach is that it requires modifying the application, so if you can't or don't want to do that, you can easily produce the desired custom metrics by adding instrumentation to the Elastic APM Java Agent via a plugin.

This article deals with the situation where the application you are monitoring doesn't emit the custom metrics you'd like it to, and you can't directly change the code or config to make it do so. Instead, you can use a plugin to automatically instrument the application via the Elastic APM Java Agent, which will then make the application emit the custom metrics you desire.

Plugin basics

The basics of the Elastic APM Java Agent, and how to easily plugin instrumentation, are detailed in the article "Create your own instrumentation with the Java Agent Plugin." Generating metrics from a plugin is just another type of instrumentation, and the referenced article provides detailed step-by-step instructions with a worked example of how to create a plugin with custom instrumentation.

For this article, I assume you understand how to create a plugin with custom instrumentation based on that previous article, as well as the example application (a simple webserver ExampleBasicHttpServer) from our plugin example repo.

The custom metric

For our example application, which is an HTTP server (ExampleBasicHttpServer) we'd like to add a custom metric 'page_views' which increments each time the ExampleBasicHttpServer application handles any request. That means the instrumentation we'll add will be triggered by the same ExampleBasicHttpServer.handleRequest() method used in "Create your own instrumentation with the Java Agent Plugin."

Using the Plugin/OpenTelemetry API

Essentially the only difference to that article is that for metrics, we'll use the OpenTelemetry metrics API instead of the OpenTelemetry tracing API.

In particular for the metrics, the advice method for the handleRequest() method is the following code:

if (pageViewCounter == null) {
    pageViewCounter = GlobalOpenTelemetry
        .getMeter("ExampleHttpServer")
        .counterBuilder("page_views")
        .setDescription("Page view count")
        .build();
}
pageViewCounter.add(1);

That is, lazily create the meter when it's first needed, and then on each invocation of the ExampleBasicHttpServer.handleRequest() method, increment the page view counter.

Everything else — setting up instrumentation, finding the method to instrument, building the plugin — is the same as in the article "

Create your own instrumentation with the Java Agent Plugin." The full metrics example is implemented in the plugin example repo, and the actual full metrics instrumentation implementation is ExampleMetricsInstrumentation.

Try it out!

That's it! To run the agent with the plugin, just build and include the jar as described in "Create your own instrumentation with the Java Agent Plugin," in the directory specified by the plugins_dir configuration option. The plugin example repo provides a full tested implementation — just clone it and mvn install to see it working.

The best place to get started with Elastic APM is in the cloud. Begin your free trial of Elastic Cloud today!

• The Elastic APM Java Agent docs
• The Elastic APM Java Agent repo
• The plugin example repo
• The previous Create your own instrumentation with the Java Agent Plugin article
• The associated Regression testing your Java Agent Plugin article
• The OpenTelemetry metrics API
• The OpenTelemetry tracing API
• Micrometer

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

This blog dives into how input packages provide an extremely generic and flexible solution to the advanced users for customizing their ingestion experience in Elastic.

What are input packages?

An Elastic Package is an artifact that contains a collection of assets that extend the Elastic Stack, providing new capabilities to accomplish a specific task like integration with an external data source. The first use of Elastic packages is integration packages, which provide an end-to-end experience — from configuring Elastic Agent, to collecting signals from the data source, to ingesting them correctly and using the data once ingested.

However, advanced users may need to customize data collection, either because an integration does not exist for a specific data source, or even if it does, they want to collect additional signals or in a different way. Input packages are another type of Elastic package that provides the capability to configure Elastic Agent to use the provided inputs in a custom way.

Let’s look at an example

Say hello to Julia, who works as an engineer at Ascio Innovation firm. She is currently working with Oracle Weblogic server and wants to get a set of metrics for monitoring it. She goes ahead and installs Elastic Oracle Weblogic Integration, which uses Jolokia in the backend to fetch the metrics.

Now, her team wants to advance in the monitoring and has the following requirements:

1. We should be able to extract metrics other than the default ones, which are not supported by the default Oracle Weblogic Integration.

2. We want to have our own bespoke pipelines, visualizations, and experience.

3. We should be able to identify the metrics coming in from two different instances of Weblogic Servers by having data mapped to separate indices.

All the above requirements can be met by using the Jolokia input package to get a customized experience. Let's see how.

Julia can add the configuration of Jolokia input package as below, fulfilling the first requirement.

hostname, JMX Mappings for the fields you want to fetch for the JVM application, and the data set name to which the response fields would get mapped.

Julia can customize her data by writing her own ingest pipelines and providing her customized mappings. Also, she can then build her own bespoke dashboards, hence meeting her second requirement.

Let’s say now Julia wants to use another instance of Oracle Weblogic and get a different set of metrics.

This can be achieved by adding another instance of Jolokia input package and specifying a new data set name as shown in the screenshot below. The resultant metrics will be mapped to a different index/data set hence fulfilling her third requirement. This will help Julia to differentiate metrics coming in from two different instances of Oracle Weblogic.

The resultant metrics of the query will be indexed to the new data set, jolokia_second_dataset in the below example.

As we can see above, the Jolokia input package provides the flexibility to get new metrics by specifying different JMX Mappings, which are not supported in the default Oracle Weblogic integration (the user gets metrics from a predetermined set of JMX Mappings).

The Jolokia Input package also can be used for monitoring any Java-based application, which pushes its metrics through JMX. So a single input package can be used to collect metrics from multiple Java applications/services.

Elastic input packages

Elastic has started supporting input packages from the 8.8.0 release. Some of the input packages are now available in beta and will mature gradually:

1. SQL input package: The SQL input package allows you to execute queries against any SQL database and store the results in Elasticsearch^®.

2. Prometheus input package: This input package can collect metrics from Prometheus Exporters (Collectors).It can be used by any service exporting its metrics to a Prometheus endpoint.

3. Jolokia input package: This input package collects metrics from Jolokia agents running on a target JMX server or dedicated proxy server. It can be used for monitoring any Java-based application, which pushes its metrics through JMX.

4. Statsd input package: The statsd input package spawns a UDP server and listens for metrics in StatsD compatible format. This input can be used to collect metrics from services that send data over the StatsD protocol.

5. GCP Metrics input package: The GCP Metrics input package can collect custom metrics for any GCP service.

Try it out!

Now that you know more about input packages, try building your own customized integration for your service through input packages, and get started with an Elastic Cloud free trial.

We would love to hear from you about your experience with input packages on the Elastic Discuss forum or in the Elastic Integrations repository.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

What's in data quality

At-a-glance summary

The summary card shows:

• Degraded documents - Documents that contain the _ignored field - see this for more info.
• Failed documents - Documents that were rejected at ingestion due to mapping conflicts or pipeline failures.

The overall quality score (Good, Degraded, Poor) is automatically calculated based on the percentage of degraded and failed documents.

Trends over time

The tab includes a time-series chart so you can track how degraded and failed documents are accumulating over time. Use the date picker to zoom into a specific range and understand when problems are spiking.

Quality issues table

A detailed table lists the types of issues affecting your stream. For each issue, you can:

• See which fields are causing problems.
• Review counts of affected documents.
• Filter by issues that have not been solved yet (Current issues only).
• Open a flyout to dive deeper into the cause of the issue and learn how to fix it.

Monitoring degraded documents

A degraded document is one that contains the _ignored field, which means one or more of its fields were ignored during indexing. One of the reasons could be that their values didn’t match the expected mappings. While the rest of the document is still indexed, a high number of degraded documents can affect query results, dashboards, and overall observability accuracy.

To help keep these issues under control, the Data quality tab provides visibility into the percentage of degraded documents in your stream.

Set up a rule to stay ahead of issues

You can use the Create rule button above the Degraded docs chart to define an alert that notifies you when the percentage of degraded documents crosses a certain threshold. This makes it easy to proactively monitor for mapping mismatches and ensure your data continues to meet quality expectations.

For more information on how to configure this rule, see Degraded docs rule conditions.

Handling failed documents with the failure store

Failure store is a special index that captures documents rejected during ingestion. Instead of losing this data, the failure store retains it in a dedicated ::failures index, allowing you to inspect the problematic documents, understand what went wrong, and fix the underlying issues.

In Data Quality tab, the failed documents are only visible if your stream has a failure store enabled, for checking failure store documents you are required to have at least read_failure_store privileges. If the failure store is not enabled, you’ll see an “Enable failure store” link that opens a modal to configure it and set the retention period. For enabling failure store you are required to have manage_failure_store privileges over the specific data stream. For further information about failure store security you can refer to Searching failures.

Once enabled, you can edit the failure store configuration or disable it at any time using the Edit button above the failed docs chart.

The failure store can also be configured in the Streams Retention tab - see this article for more information.

Technical implementation

Under the hood, the Data quality tab builds on the existing Dataset quality plugin - the same one that powers the Dataset quality page in Stack Management. However, instead of working in the context of datasets following the Data stream naming scheme, it’s now tailored specifically for streams.

To determine the quality of a stream, the UI sends three ES|QL query server requests:

1. All documents (including failures):

 FROM myStream, myStream::failures | STATS doc_count = COUNT(*)

2. Failed documents only:

 FROM myStream::failures | STATS failed_doc_count = COUNT(*)

3. Degraded documents:

FROM myStream METADATA _ignored | WHERE _ignored IS NOT NULL | STATS degraded_doc_count = COUNT(*)

The results of these queries are then used to calculate the percentages of failed and degraded documents. The overall data quality is determined using simple thresholds:

• Good: Both percentages are 0%
• Degraded: Any percentage is greater than 0% but less than 3%
• Poor: Any percentage is above 3%

For managing the failure store, Streams uses the Update data stream options API with the failure_store parameter to configure and update the failure store settings, including enabling the store and setting the retention period.

Why you’ll love this

The new Data quality tab gives you:

• Visibility into ingestion problems without digging into logs
• A clear breakdown of degraded vs. failed documents
• Insights into which fields are ignored and why
• Tools to capture and troubleshoot failed docs with the failure store

By surfacing data quality issues directly in the Streams UI, we’re making it easier to keep your data flowing reliably and to ensure your analytics are built on a strong foundation.

Try it out today

The data quality feature is available in Elastic Observability on Serverless, and coming soon for self-managed and Elastic Cloud users.

Sign up for an Elastic trial at cloud.elastic.co, and trial Elastic's Serverless offering which will allow you to play with all of the Streams functionality.

For more information on Streams:

Read about Reimagining streams

Look at the Streams website

Read the Streams documentation

When people think about open source, they often picture lines of code, clever algorithms, or maybe a GitHub repo full of issues and pull requests. What can be harder to see is the human side. The people who quietly keep things moving, who make sure contributions land smoothly and help the community grow in a healthy way. That's the work of a maintainer.

Maintainers are more than just code reviewers. They are the stewards of the SIG's (Special Interest Group) health, direction, and community. They balance technical oversight with mentorship, governance with collaboration, and long-term vision with the day-to-day realities of issues and pull requests.

Open Source mentorship

One of the most rewarding parts of being a maintainer is mentorship. Every open source project depends on new contributors stepping in, learning the ropes, and eventually taking on more responsibility themselves. As maintainers, we’re often the first point of contact for someone who’s never contributed to the project before.

Mentorship can look like many different things. Sometimes it’s as simple as leaving a thoughtful code review that doesn’t just point out what’s wrong, but explains why a change matters. Other times, it’s guiding a contributor through their first issue, helping them understand the project’s structure, or showing them how to run tests locally. And every so often, it means stepping back to give someone room to try, even if they don’t get it right the first time.

The goal isn’t just to fix the immediate bug or land the pull request. It's to help contributors feel confident enough to come back again. A healthy project grows by sharing knowledge, not hoarding it. Mentorship is how maintainers make sure today's first-time contributor can become tomorrow's reviewer, and eventually, the next maintainer.

Setting direction and priorities

Another part of being a maintainer is shaping the project's roadmap. Open source moves fast: there are always new ideas, bug reports, and feature requests. Left unchecked, a project can easily become a grab bag of loosely connected changes. Part of our job as maintainers is to make sure the work stays aligned with the bigger picture.

That means asking questions like:

• Does this feature fit with our long-term goals?
• Is now the right time to tackle it?
• Do we have the capacity to maintain it once it’s merged?

Sometimes the answer is "not yet" or even "no", and it’s on us to communicate that clearly while still encouraging contributions.

Roadmapping isn't about dictating every detail. It's about setting priorities together with the community—listening to feedback, balancing what users need today with where the project should be tomorrow, and making tradeoffs that keep the project sustainable.

The roadmap gives everyone a shared sense of direction. Contributors know where their work fits in, users can see what's coming next, and the project as a whole stays focused instead of scattered.

Special Interest Group meetings

One of the maintainer's roles is also to facilitate the frequent meetings that help their SIG communicate and plan its work.

Facilitating a SIG meeting isn’t about running through an agenda like a checklist. It’s about creating space where everyone feels comfortable speaking up, from long-time contributors to someone joining their very first call. That means keeping discussions focused, making sure quieter voices get heard, and helping the group reach consensus without letting debates drag on forever.

There's also a practical side: preparing the agenda ahead of time, documenting decisions so they’re visible to the wider community, and following up on action items afterward.

In many ways, SIG meetings are where the "community" part of open source really comes to life. As maintainers, our role is to guide the conversation, not control it, making sure the project keeps moving forward while staying open and inclusive.

Challenges

Of course, maintaining isn’t all smooth sailing. One of the hardest parts is balancing the constant flow of contributions with the need to keep the codebase healthy. Every pull request represents someone's time and effort, and it’s important to honor that. Yet, at the same time, not every change fits the project's standards or long-term goals. Saying "no" gracefully is just as important as merging a great contribution.

Maintainers also find themselves balancing priorities that go beyond code. Different contributors, and often the companies backing them, come with their own needs and expectations. One team might want a new feature quickly, another might be focused on stability, while the community as a whole still needs clear direction. Managing those competing priorities, and making decisions that serve the project rather than any single interest, is a constant challenge.

Conflicts are another reality. With so many people involved, it's inevitable that disagreements will happen. Sometimes it's about technical design, sometimes about process, and occasionally about interpersonal dynamics. Part of the maintainer role is helping to navigate those moments: keeping discussions respectful, finding common ground, and making sure decisions are made transparently.

And yet, despite the difficulties, the impact of this work is enormous: when maintainers succeed, the entire community thrives.

The importance and impact of Open Source maintainers

When maintainers do their job well, the effects ripple far beyond the codebase. A well-tended project feels reliable and welcoming—contributors know their work will be reviewed thoughtfully, users trust the software to be stable, and the community grows because people want to come back.

Good project maintenance builds momentum. A contributor who feels supported on their first pull request is more likely to return for a second. Clear roadmaps and consistent standards give people confidence that their effort matters and will fit into the bigger picture. And when conflicts are handled with respect and transparency, it reinforces the culture of trust that makes open source sustainable.

The impact goes deeper than just keeping a project alive. Effective maintainers create the conditions for others to succeed. That's the real legacy of this role: not just code, but a thriving ecosystem and community built around it.

Conclusion

Being a maintainer is challenging work, but it’s also some of the most meaningful. It’s about more than merging code. It's about stewardship, mentorship, and creating a community where people feel empowered to contribute. Every healthy open source project owes its success to the care and commitment of its maintainers.

And while the challenges are real, the rewards are just as tangible: the chance to constantly learn, to collaborate on complex problems, and to connect with people from every corner of the world and every kind of background.

OpenTelemetry's maintainers embody this balance every day, helping the project grow while keeping its community strong.

Elastic Cloud Serverless is a fully managed solution that allows you to deploy and use Elastic for your use cases without managing the underlying infrastructure. Built on Kubernetes, it represents a shift in how you interact with Elasticsearch. Instead of managing clusters, nodes, data tiers, and scaling, you create serverless projects that are fully managed and automatically scaled by Elastic. This abstraction of infrastructure decisions allows you to focus solely on gaining value and insight from your data.

Elastic Cloud Serverless is generally available (GA) on AWS, GCP and currently in Technical Preview on Azure. As part of preparing Elastic Cloud Serverless GA on Azure, we have been conducting extensive performance and scalability tests to ensure that our users get a consistent and reliable user experience.

In this post, we’ll take you behind the scenes of a deep technical investigation into a surprising performance issue that affected Serverless Elasticsearch in our Azure Kubernetes clusters. At first, the network seemed like the least likely place to look, especially with a high-speed 100 Gb/s interface on the host backing it. But as we dug deeper, with help from the Microsoft Azure team, that’s exactly where the problem led us.

While the high-level architectures and system design patterns of the major cloud provider’s systems are often similar, the implementations are different, and these differences can have dramatic impacts on a system’s performance characteristics.

One of the most significant differences between the different cloud providers is that the underlying hypervisor software and server hardware of the Virtual Machines can vary significantly, even between instance families of the same provider.

There is no way to fully abstract the hardware away from an application like Elasticsearch. Fundamentally, its performance is dictated by the CPU, memory, disks, and network interfaces on the physical server. In preparation for the Elastic Cloud Serverless GA on Azure, our Elasticsearch Performance team kicked off large-scale load testing against Serverless Elasticsearch projects running on Azure Kubernetes Service (AKS), using ARM-based VMs (we’re big fans!). Throughout this process, we relied heavily on Elastic tools to analyse system behaviour, identify bottlenecks, and validate performance under load.

To perform these scale and load tests, the Elasticsearch Performance team use Rally, an open-source benchmarking tool designed to measure the performance of Elasticsearch clusters. The workload (or in Rally nomenclature, ‘Track’) used for these tests was the GitHub Archive Track. Rally collects and sends test telemetry using the official Python client to a separate Elasticsearch cluster running Elastic Observability, which allows for monitoring and analysis during these scale and load tests in real time via Kibana.

When we looked at the results, we observed that the indexing rate (the number of docs/s) for the Serverless projects was not only much lower than we had expected for the given hardware, but the throughput was also quite unstable. There were peaks and valleys, interspersed with frequent errors, whereas we were instead expecting a stable indexing rate for the duration of the test.

These tests are designed to push the system to its limits, and in doing so, they surfaced unexpected behavior in the form of unstable indexing throughput and intermittent errors. This was precisely the kind of problem we'd hoped to uncover prior to going GA — giving us the opportunity to work closely with Azure.

Debugging performance issues can feel a little bit like trying to find a ‘Butterfly in a Hurricane’, so it’s crucial that you take a methodological approach to analysing application and system performance.

Using methodologies helps you to be more consistent and thorough in your debugging, and avoids missing things. We started with the Utilisation Saturation and Errors (USE) Method, looking at both the client and server side to identify any obvious bottlenecks in the system.

Elastic's Site Reliability Engineers (SREs) maintain a suite of custom Elastic Observability dashboards designed to visualise data collected from various Elastic Integrations. These dashboards provide deep visibility into the health and performance of Elastic Cloud infrastructure and systems.

For this investigation, we leveraged a custom dashboard built using metrics and log data from the System and Linux Integrations:

Following the USE Method, these dashboards highlight resource utilisation, saturation, and errors across our systems. With their help, we quickly identified that the AKS nodes hosting the Elasticsearch pods under test were dropping thousands of packets per second.

Dropping packets forces reliable protocols, such as TCP, to retransmit any missing packets. These retransmissions can introduce significant delays, which kills the throughput of any system where client requests are only triggered upon the previous request completion (known as a Closed System).

To investigate further, we jumped onto one of the AKS nodes exhibiting the packet loss to check the basics. First off, we wanted to identify what type of packet drops or errors we’re seeing; is it for specific pods, or the host as a whole?

root@aks-k8s-node-1:~# ip -s link show
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 7c:1e:52:be:ce:5e brd ff:ff:ff:ff:ff:ff
    RX:    bytes   packets errors dropped  missed   mcast
    373507935420 134292481      0       0       0      15
    TX:    bytes   packets errors dropped carrier collsns
    644247778936 303191014      0       0       0       0
3: enP42266s1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master eth0 state UP mode DEFAULT group default qlen 1000
    link/ether 7c:1e:52:be:ce:5e brd ff:ff:ff:ff:ff:ff
    RX:    bytes   packets errors dropped  missed   mcast
    386782548951 307000571      0       0 5321081       0
    TX:    bytes   packets errors dropped carrier collsns
    655758630548 477594747      0       0       0       0
    altname enP42266p0s2
15: lxc0ca0ec41ecd2@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether f6:f5:5e:c9:4e:fb brd ff:ff:ff:ff:ff:ff link-netns cni-3f90ab53-df66-cac5-bd19-9cea4a68c29b
    RX:    bytes   packets errors dropped  missed   mcast
    627954576078  54297550      0    1600       0       0
    TX:    bytes   packets errors dropped carrier collsns
    372155326349 133538064      0    3927       0       0

In this output you can see the enP42266s1 interface is showing a significant number of packets in the missed column. That’s interesting, sure, but what does missed actually represent? And what is enP42266s1?

To understand, let’s look at roughly what happens when a packet arrives at the NIC:

1. A packet arrives at the NIC from the network.
2. The NIC uses DMA (Direct Memory Access) to place the packet into a receive ring buffer allocated in memory by the kernel, mapped for use by the NIC. Since our NICs supports multiple hardware queues, each queue has its own dedicated ring buffer, IRQ, and NAPI context.
3. The NIC raises a hardware interrupt (IRQ) to notify the CPU that a packet is ready.
4. The CPU runs the NIC driver’s IRQ handler. The driver schedules a NAPI (New API) poll to defer packet processing to a softirq context. A mechanism in the Linux kernel that defers work to be processed outside of the hard IRQ context, for better batching and CPU efficiency, enabling improved scalability.
5. The NAPI poll function is executed in a softirq context (NET_RX_SOFTIRQ) and retrieves packets from the ring buffer. This polling continues either until the driver’s packet budget is exhausted (net.core.netdev_budget) or the time limit is hit (net.core.netdev_budget_usecs).
6. Each packet is wrapped in an sk_buff (socket buffer) structure, which includes metadata such as protocol headers, timestamps, and interface identifiers.
7. If the networking stack is slower than the rate at which NAPI fetches packets, excess packets are queued in a per-CPU backlog queue (via enqueue_to_backlog). The maximum size of this backlog is controlled by the net.core.netdev_max_backlog sysctl.
8. Packets are then handed off to the kernel’s networking stack for routing, filtering, and protocol-specific processing (e.g. TCP, UDP).
9. Finally, packets reach the appropriate socket receive buffer, where they are available for consumption by the user-space application.

Visualised, it looks something like this:

The missed counter is incremented whenever the NIC tries to DMA a packet into a fully occupied ring buffer. The NIC essentially "misses" the chance to deliver the packet to the VM’s memory. However, what’s most interesting is that this counter seldom increments for VMs. This is because Virtual NICs are usually implemented as software via the hypervisor, which typically has much more flexible memory management compared to the physical NICs and can reduce the chance of ring buffer overflow.

We mentioned earlier that we’re building Azure Elasticsearch Serverless on top of Azure’s AKS service, which is important to note because all of our AKS nodes use an Azure feature called Accelerated Networking. In this setup, network traffic is delivered directly to the VM’s network interface, bypassing the hypervisor. This is enabled by single root I/O virtualization (SR-IOV), which offers much lower latency and higher throughput than traditional VM networking. Each node is physically connected to a 100 Gb/s network interface, although the SR-IOV Virtual Function (VF) exposed to the VM typically provides only a fraction of that total bandwidth.

Despite the VM only having a fraction of the 100 Gb/s bandwidth, microbursts are still very possible. These physical interfaces are so fast that they can transmit and receive multiple packets in just nanoseconds, far faster than most buffers or processing queues can absorb. At these timescales, even a short-lived burst of traffic can overwhelm the receiver, leading to dropped packets and unpredictable latency.

Direct access to the SR-IOV interface means that our VMs are responsible for handling the hardware interrupts triggered by the NIC in a timely manner, if there's any delay in handling the hardware interrupt (e.g. waiting to be scheduled onto CPU by the hypervisor) then network packets can be missed!

Since we'd confirmed that our VMs were using SR-IOV, we established that the enP42266s1 and eth0 interfaces were a bonded pair and acted as a single interface. Knowing this, then we reasoned that we should be able to adjust the ring buffer values directly using ethtool.

root@aks-k8s-node-1:~# ethtool -g enP42266s1
Ring parameters for enP42266s1:
Pre-set maximums:
RX:		8192
RX Mini:	n/a
RX Jumbo:	n/a
TX:		8192
Current hardware settings:
RX:		1024
RX Mini:	n/a
RX Jumbo:	n/a
TX:		1024

In the output above, we were using only 1/8th of the available ring buffer descriptors. These values were set by the OS defaults, which generally aim to balance performance and resource usage. Set too low, they risk packet drops under load; set too high, they can lead to unnecessary memory consumption. We knew that the VMs were backed by a virtual function carved out of the directly attached 100 Gb/s network interface, which is fast enough to deliver microbursts that could easily overwhelm small buffers. To better absorb those short, high-intensity bursts of traffic, we increased the NIC’s RX ring buffer size from 1024 to 8192. Using a privileged DaemonSet, we rolled out the change across all of our AKS nodes by installing a udev rule to automatically increase the buffer size:

# Match Mellanox ConnectX network cards and run ethtool to update the ring buffer settings
ENV{INTERFACE}=="en*", ENV{ID_NET_DRIVER}=="mlx5_core", RUN+="/sbin/ethtool -G %k rx ${CONFIG_AZURE_MLX_RING_BUFFER_SIZE} tx ${CONFIG_AZURE_MLX_RING_BUFFER_SIZE}"

As soon as the change had been applied to all AKS nodes we stopped ‘missing’ RX packets! Fantastic! As a result of this simple change we observed a significant improvement in our indexing throughput and stability.

Job done, right? Not quite..

Eagle eyed readers may have noticed two things:

1. In the previous screenshot, despite adjusting the physical RX ring buffer values, we still observed a small number of dropped packets on the TX side.
2. In the original ip link -s show output, one of the ‘logical’ interfaces used by the Elasticsearch pod was showing dropped packets on both the TX and RX sides.

15: lxc0ca0ec41ecd2@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether f6:f5:5e:c9:4e:fb brd ff:ff:ff:ff:ff:ff link-netns cni-3f90ab53-df66-cac5-bd19-9cea4a68c29b
    RX:    bytes   packets errors dropped  missed   mcast
    627954576078  54297550      0    1600       0       0
    TX:    bytes   packets errors dropped carrier collsns
    372155326349 133538064      0    3927       0       0

So, we continued to dig. We’d eliminated ~99% of the packet loss, and the remaining loss rate wasn’t as significant as what we’d started with, but we still wanted to understand why it was occurring even after adjusting the RX ring buffer size of the NIC.

So what does dropped represent, and what is this lxc0ca0ec41ecd2 interface? dropped is similar to missed, but only occurs when packets are deliberately dropped by the kernel or network interface. Crucially though, it doesn’t tell you why a packet was dropped. As for the lxc0ca0ec41ecd2 interface, we use the Azure CNI Powered by Cilium to provide the network functionality to our AKS clusters. Any pod spun up on an AKS node gets a ‘logical’ interface, which is a virtual ethernet (veth) pair that connects the pod’s network namespace with the host’s network namespace. It was here that we were dropping packets.

In our experience, packet drops at this layer are unusual, so we started digging deeper into the cause of the drops. There are numerous ways you can debug why a packet is being dropped, but one of the easiest is to use perf attach to the skb:kfree_skb tracepoint. The "socket buffer" (skb) is the primary data structure used to represent network packets in the Linux kernel. When a packet is dropped, its corresponding socket buffer is usually freed, triggering the kfree_skb tracepoint. Using perf to attach to this event allowed us to capture stack traces to analyze the cause of the drops.

We left this to run for ~10 minutes or so to capture as many drops as possible, and then ‘heavily inspired’ by this GitHub Gist by Ivan Babrou, we converted the stack traces into an ‘easier’ to read Flamegraphs:

# perf script | sed -e 's/skb:kfree_skb:.*reason:\(.*\)/\n\tfffff \1 (unknown)/' -e 's/^\(\w\+\)\s\+/kernel /' > stacks.txt
cat stacks.txt | stackcollapse-perf.pl --all | perl -pe 's/.*?;//' | sed -e 's/.*irq_exit_rcu_\[k\];/irq_exit_rcu_[k];/' | flamegraph.pl --colors=java --hash --title=aks-k8s-node-1 --width=1440 --minwidth=0.005 > aks-k8s-node-1.svg

The flamegraph here shows how often different functions appeared in stack traces for packets drops. Each box represents a function call and wider boxes mean the function appears more frequently in the traces. The stack's ancestry builds upward from the bottom with earlier calls, to the top with later calls.

Firstly, we quickly discovered that unfortunately the skb_drop_reason enum was only added in Kernel 5.17 (Azure’s Node Image at the time was using 5.15). This meant that there was no single human readable message that told us why the packets were being dropped, instead all we got was NOT_SPECIFIED. To work out why packets were being dropped we needed to do a little sleuthing through the stack traces to work out what code paths were being taken when a packet was dropped.

In the flamegraph above you can see that many of the stack traces include veth driver function calls (e.g. veth_xmit), and many end abruptly with a call to the enqueue_to_backlog function. When many stacks end at the same function (like enqueue_to_backlog) it suggests that function is a common point where packets are being dropped. If you go back to the earlier explanation of what happens when a packet arrives at the NIC, you’ll notice that in step 7 we explained:

7. If the networking stack is slower than the rate at which NAPI fetches packets, excess packets are queued in a per-CPU backlog queue (via enqueue_to_backlog). The maximum size of this backlog is controlled by the net.core.netdev_max_backlog sysctl.

Using the same privileged DaemonSet method for the RX ring buffer adjustment, we set the value of the net.core.netdev_max_backlog adjustable kernel parameter from 1000 to 32768:

/usr/sbin/sysctl -w net.core.netdev_max_backlog=32768

This value was based on the fact we knew the hosts were using a 100 Gb/s SR-IOV NIC, even if the VM was allowed only a fraction of the total bandwidth. We acknowledge that it’s worth revisiting this value in the future to see if it can be better optimised to not waste extraneous memory, but at the time “perfect was the enemy of good”.

We re-ran the load tests and compared the three sets of results we’d collected thus far.

Here you can see the P50 of throughput in docs/s over the course of the hours-long load tests. Compared to the baseline, we saw a roughly ~40% increase in throughput by only adjusting the RX ring buffer values, and a ~50-60% increase with both the RX ring buffer and backlog changes! Hooray!

A great result and one more step on our journey towards better Serverless Elasticsearch performance.

It’s great that we were able to quickly identify and mitigate the majority of our packet loss issues, but since we were using AKS with AKS node images, it made sense to engage with Azure to understand why the defaults weren’t working for our workload.

We walked Azure through our investigation, mitigations and results, and asked for some additional validation of our mitigations. Azure Engineering confirmed that the host NICs were not discarding packets, which confirmed that everything arriving at the host level was passed through to the hypervisor on the host. Further investigation confirmed that no loss or discards were occurring to Azure network fabric, or internal to the hypervisor – which shifted focus from the host to the guest OS and why the guest OS kernel was slow when reading packets off of the enP* SR-IOV interfaces.

Given the complexity of our load testing scenario — which involved configuring multiple systems and tools, including Elastic Observability, we also developed a simplified reproduction of the packet loss issue using iperf3. This simplified test was created specifically to share with Azure for targeted analysis, and added to the broader monitoring and analysis enabled by Elastic Observability and Rally.

With this reproduction Azure was able to confirm the increasing missed and dropped packet counters we had observed, and confirmed the increased RX ring buffer and netdev_max_backlog increase as the recommended mitigations.

While cloud providers offer various abstractions to manage your resources, the underlying hardware ultimately determines your application's performance and stability. High-performance hardware often requires tuning at the operating system level, well beyond the default settings most environments ship with. In managed platforms like AKS, where Azure controls both the node images and infrastructure, it is easy to overlook the impact of low-level configurations such as network device ring buffer sizes or sysctls like net.core.netdev_max_backlog.

Our experience shows that even with the convenience of a managed Kubernetes service, performance issues can still emerge if these hardware parameters are not tuned appropriately. It was tempting to assume that high-speed 100 Gb/s network interfaces, directly attached to the VM using SR-IOV would eliminate any chance of network-related bottlenecks. In reality, that assumption didn’t hold up.

Engaging early with Azure was essential, as they provided deeper visibility into the underlying infrastructure and worked with us to tune low-level, performance-critical settings. Combined with thorough load and scale testing and robust observability using tools like Elastic Observability, this collaboration helped us detect and rectify the issue early in order to deliver a consistent, reliable, and high-performing experience for our users.

This blog post will show you how to deploy a simple Hello World web app to App Runner and then walk you through the steps to instrument the Hello World web app to enable observation of the application’s operations with Elastic Cloud.

Elastic Observability setup

We’ll start with setting up an Elastic Cloud deployment, which is where observability will take place for the web app we’ll be deploying.

From the Elastic Cloud console, select Create deployment.

Enter a deployment name and click Create deployment. It takes a few minutes for your deployment to be created. While waiting, you are prompted to save the admin credentials for your deployment, which provides you with superuser access to your Elastic® deployment. Keep these credentials safe as they are shown only once.

Elastic Observability requires an APM Server URL and an APM Secret token for an app to send observability data to Elastic Cloud. Once the deployment is created, we’ll copy the Elastic Observability server URL and secret token and store them somewhere safely for adding to our web app code in a later step.

To copy the APM Server URL and the APM Secret Token, go to Elastic Cloud. Then go to theDeployments page, which lists all of the deployments you have created. Select the deployment you want to use, which will open the deployment details page. In the Kibana® row of links, click on Open to open Kibana for your deployment.

Select Integrations from the top-level menu. Then click the APM tile.

On the APM Agents page, copy the secretToken and the serverUrl values and save them for use in a later step.

Now that we’ve completed the Elastic Cloud setup, the next step is to set up our AWS project for deploying apps to App Runner.

AWS App Runner setup

To start using AWS App Runner, you need an AWS account. If you’re a brand new user, go to aws.amazon.com to sign up for a new account.

Set up AWS CloudShell

We’ll perform the process of creating a Python Hello World App image and pushing it to the AWS ECR using AWS CloudShell.

We’re going to use Docker to build the sample app image. Perform the following five steps to set up Docker within CloudShell.

1. Open AWS CloudShell.

2. Run the following two commands to install Docker in CloudShell:

sudo yum update -y
sudo amazon-linux-extras install docker

3. Start Docker by running the command:

sudo dockerd

4. With Docker running, open a new tab in CloudShell by clicking the Actions dropdown menu and selecting New tab.

5. Run the following command to authenticate Docker within CloudShell. Replace <account_id> with your AWS Account ID in the Docker command below, and then run it in CloudShell.

aws ecr get-login-password --region us-east-2 | sudo docker login --username AWS --password-stdin <account_id>.dkr.ecr.us-east-2.amazonaws.com

Build the Hello World web app image and push it to AWS ECR

We’ll be using AWS ECR, Amazon’s fully managed container registry for storing and deploying application images. To build and push the Hello World app image to AWS ECR, we’ll perform the following six steps in AWS CloudShell:

1. Run the command below in CloudShell to create a repository in AWS ECR.

aws ecr create-repository \
    --repository-name elastic-helloworld/web \
    --image-scanning-configuration scanOnPush=true \
    --region us-east-2

“elastic-helloworld” will be the application's name and “ web” will be the service name.

2. In the newly created tab within CloudShell, clone a Python Hello World sample app repo from GitHub by entering the following command.

git clone https://github.com/elastic/observability-examples

3. Change directory to the location of the Hello World web app code by running the following command:

cd observability-examples/aws/app-runner/helloworld

4. Build the Hello World sample app from the application’s directory. Run the following Docker command in CloudShell.

sudo docker build -t elastic-helloworld/web .

5. Tag the application image. Replace <account_id> with your AWS Account ID in the Docker command below, and then run it in CloudShell.

sudo docker tag elastic-helloworld/web:latest <account_id>.dkr.ecr.us-east-2.amazonaws.com/elastic-helloworld/web:latest

6. Push the application image to ECR. Replace <account_id> with your AWS Account ID in the command below, and then run it in CloudShell.

sudo docker push <account_id>.dkr.ecr.us-east-2.amazonaws.com/elastic-helloworld/web:latest

Deploy a Hello World web app to AWS App Runner

We’ll perform the process of deploying a Python Hello World App to App Runner using the AWS App Runner console.

1. Open the App Runner console and click the Create an App Runner service button.

2. On the Source and deployment page, set the following deployment details:

• In the Source section, for Repository type, choose Container registry.
• For Provider, choose Amazon ECR.
• For Container image URI, choose Browse to select the Hello World application image that we previously pushed to AWS ECR.
  • In the Select Amazon ECR container image dialog box, for Image repository, select the “ elastic-helloworld/web” repository.
  • For Image tag, select “ latest” and then choose Continue.
• In the Deployment settings section, choose Automatic.
• For ECR access role, choose Create new service role.
• Click Next.

3. On the Configure service page, in the Service settings section, enter the service name “ helloworld-app.” Leave all the other settings as they are and click Next.

4. On the Review and create page, click Create & deploy.

After a few minutes, the Hello World app will be deployed to App Runner.

5. Click the Default domain URL to view the Hello World app running in App Runner.

Instrument the Hello World web app with Elastic Observability

With a web app successfully running in App Runner, we’re now ready to add the minimal code necessary to start monitoring the app. To enable observability for the Hello World app in Elastic Cloud, we’ll perform the following five steps in AWS CloudShell:

1. Edit the Dockerfile file to add the following Elastic Open Telemetry environment variables along with the commands to install and run the Elastic APM agent. Use the “nano” text editor by typing “nano Dockerfile”. Be sure to replace the <ELASTIC_APM_SERVER_URL> text and the <ELASTIC_APM_SECRET_TOKEN> text with the APM Server URL and the APM Secret Token values that you copied and saved in an earlier step. The updated Dockerfile should look something like this:

FROM python:3.9-slim as base

# get packages
COPY requirements.txt .
RUN pip install -r requirements.txt

WORKDIR /app

# install opentelemetry packages
RUN pip install opentelemetry-distro opentelemetry-exporter-otlp
RUN opentelemetry-bootstrap -a install

ENV OTEL_EXPORTER_OTLP_ENDPOINT='<ELASTIC_APM_SERVER_URL>'
ENV OTEL_EXPORTER_OTLP_HEADERS='Authorization=Bearer%20<ELASTIC_APM_SECRET_TOKEN>'
ENV OTEL_LOG_LEVEL=info
ENV OTEL_METRICS_EXPORTER=otlp
ENV OTEL_RESOURCE_ATTRIBUTES=service.version=1.0,deployment.environment=production
ENV OTEL_SERVICE_NAME=helloworld
ENV OTEL_TRACES_EXPORTER=otlp

COPY . .
ENV FLASK_APP=helloworld
ENV FLASK_RUN_HOST=0.0.0.0
ENV FLASK_RUN_PORT=8080
EXPOSE 8080
ENTRYPOINT [ "opentelemetry-instrument", "flask", "run" ]

Note: You can close the nano text editor and save the file by typing “Ctrl + x”. Press the “y” key and then the “Enter” key to save the changes.

2. Edit the helloworld.py file to add observability traces. In CloudShell, type “nano helloworld.py” to edit the file.

• After the import statements at the top of the file, add the code required to initialize the Elastic Open Telemetry APM agent:

from opentelemetry import trace
tracer = trace.get_tracer("hello-world")

• Replace the “Hello World!” output code . . .

return "<h1>Hello World!</h1>";

• … with the Hello Elastic Observability code block.

return '''
<div style="text-align: center;">
<h1 style="color: #005A9E; font-family:'Verdana'">
Hello Elastic Observability - AWS App Runner - Python
</h1>
<img src="https://elastic-helloworld.s3.us-east-2.amazonaws.com/elastic-logo.png">
</div>
'''

• Then add a “hi” trace before the Hello Elastic Observability code block along with an additional “@app.after_request” method placed afterward to implement a “bye” trace.

@app.route("/")
def helloworld():
	with tracer.start_as_current_span("hi") as span:
  	  logging.info("hello")
  	  return '''
   	 <div style="text-align: center;">
   	 <h1 style="color: #005A9E; font-family:'Verdana'">
   	 Hello Elastic Observability - AWS App Runner - Python
   	 </h1>
   	 <img src="https://elastic-helloworld.s3.us-east-2.amazonaws.com/elastic-logo.png">
   	 </div>
   	 '''

@app.after_request
def after_request(response):
	with tracer.start_as_current_span("bye"):
  	  logging.info("goodbye")
  	  return response

The completed helloworld.py file should look something like this:

import logging
from flask import Flask

from opentelemetry import trace
tracer = trace.get_tracer("hello-world")

app = Flask(__name__)

@app.route("/")
def helloworld():
    with tracer.start_as_current_span("hi") as span:
   	 logging.info("hello")
   	 return '''
    	<div style="text-align: center;">
    	<h1 style="color: #005A9E; font-family:'Verdana'">
    	Hello Elastic Observability - AWS App Runner - Python
    	</h1>
    	<img src="https://elastic-helloworld.s3.us-east-2.amazonaws.com/elastic-logo.png">
    	</div>
    	'''

@app.after_request
def after_request(response):
    with tracer.start_as_current_span("bye"):
   	 logging.info("goodbye")
   	 return response

Note: You can close the nano text editor and save the file by typing “Ctrl + x”. Press the “y” key and then the “Enter” key to save the changes.

1. Rebuild the updated Hello World sample app using Docker from within the application’s directory. Run the following command in CloudShell.

sudo docker build -t elastic-helloworld/web .

4. Tag the application image using Docker. Replace <account_id> with your AWS Account ID in the Docker command below and then run it in CloudShell.

sudo docker tag elastic-helloworld/web:latest <account_id>.dkr.ecr.us-east-2.amazonaws.com/elastic-helloworld/web:latest

5. Push the updated application image to ECR. Replace <account_id> with your AWS Account ID in the Docker command below and then run it in CloudShell.

sudo docker push <account_id>.dkr.ecr.us-east-2.amazonaws.com/elastic-helloworld/web:latest

Pushing the image to ECR will automatically deploy the new version of the Hello World app.

Open the App Runner console. After a few minutes, the Hello World app will be deployed to App Runner. Click the Default domain URL to view the updated Hello World app running in App Runner.

Observe the Hello World web app

Now that we’ve instrumented the web app to send observability data to Elastic Observability, we can now use Elastic Cloud to monitor the web app’s operations.

1. In Elastic Cloud, select the Observability Services menu item.

2. Click the helloworld service.

3. Click the Transactions tab.

4. Scroll down and click the “/” transaction.

5. Scroll down to the Trace Sample section to see the “/,” “hi,” and “bye” trace samples.

Observability made to scale

You’ve seen the complete process of deploying a web app to AWS App Runner that is instrumented with Elastic Observability. The end result is a web app that will scale up and down with usage, combined with the observability tools to monitor the web app as it serves one user or millions of users.

Now that you’ve seen how to deploy a serverless web app instrumented with observability, visit Elastic Observability to learn more about how to implement a complete observability solution for your apps. Or visit Getting started with Elastic on AWS for more examples of how you can drive the data insights you need by combining AWS’s cloud computing services with Elastic’s search-powered platform.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog, we examine a simple way to integrate Elastic Agent with Confluent Cloud's Kafka offering to reduce the operational burden of ingesting business-critical data.

Benefits of Elastic Agent and Confluent Cloud

When combined, Elastic Agent and Confluent Cloud's updated Elasticsearch Sink connector provide a myriad of advantages for organizations of all sizes. This combined solution offers flexibility in handling any type of data ingest workload in an efficient and resilient manner.

Fully Managed

When combined, Elastic Cloud Serverless and Confluent Cloud provide users with a fully managed service. This makes it effortless to deploy and ingest nearly unlimited data volumes without having to worry about nodes, clusters, or scaling.

Full Elastic Integrations Support

Sending data through Kafka is fully supported with any of the 300+ Elastic Integrations. In this blog post, we outline how to set up the connection between the two platforms. This ensures you can benefit from our investments in built-in alerts, SLOs, AI Assistants, and more.

Decoupled Architecture

Kafka acts as a resilient buffer between data sources (such as Elastic Agent and Logstash) and Elasticsearch, decoupling data producers from consumers. This can significantly reduce total cost of ownership by enabling you to size your Elasticsearch cluster based on typical data ingest volume, not maximum ingest volume. It also ensures system resilience during spikes in data volume.

Ultimate control over your data

With our new Output per Integration capability, customers can now send different data to different destinations using the same agent. Customers can easily send security logs directly to Confluent Cloud/Kafka, which can provide delivery guarantees, while sending less critical application logs and system metrics directly to Elasticsearch.

Deploying the reference architecture

In the following sections, we will walk you through one of the ways Confluent Kafka can be integrated with Elastic Agent and Elasticsearch using Confluent Cloud's Elasticsearch Sink Connector. As with any streaming and data collection technology, there are many ways a pipeline can be configured depending on the particular use case. This blog post will focus on a simple architecture that can be used as a starting point for more complex deployments.

Some of the highlights of this architecture are:

• Dynamic Kafka topic selection at Elastic Agents
• Elasticsearch Sink Connectors for fully managed transfer from Confluent Kafka to Elasticsearch
• Processing data leveraging Elastic's 300+ Integrations

Prerequisites

Before getting started ensure you have a Kafka cluster deployed in Confluent Cloud, an Elasticsearch cluster or project deployed in Elastic Cloud, and an installed and enrolled Elastic Agent.

Configure Confluent Cloud Kafka Cluster for Elastic Agent

Navigate to the Kafka cluster in Confluent Cloud, and select Cluster Settings. Locate and note the Bootstrap Server address, we will need this value later when we create the Kafka Output in Fleet.

Navigate to Topics in the left-hand navigation menu and create two topics:

1. A topic named logs
2. A topic named metrics

Next, navigate to API Keys in the left-hand navigation menu:

1. Click + Add API Key
2. Select the Service Account API key type
3. Provide a meaningful name for this API Key
4. Grant the key write permission to the metrics and logs topics
5. Create the key

Note the provided Key and the Secret, we will need it later when we configure the Kafka Output in Fleet.

Configure Elasticsearch and Elastic Agent

In this section, we will configure the Elastic Agent to send data to Confluent Cloud's Kafka cluster and we will configure Elasticsearch so it can receive data from the Confluent Cloud Elasticsearch Sink Connector.

Configure Elastic Agent to send data to Confluent Cloud

Elastic Fleet simplifies sending data to Kafka and Confluent Cloud. With Elastic Agent, a Kafka "output" can be easily attached to all data coming from an agent or it can be applied only to data coming from a specific data source.

Find Fleet in the left-hand navigation, click the Settings tab. On the Settings tab, find the Outputs section and click Add Output.

Perform the following steps to configure the new Kafka output:

1. Provide a Name for the output
2. Set the Type to Kafka
3. Populate the Hosts field with the Bootstrap Server address we noted earlier .
4. Under Authentication, populate the Username with the API Key and the Password with the Secret we noted earlier
5. Under Topics, select Dynamic Topic and set Topic from field to data_stream.type
6. Click Save and apply settings

Next, we will navigate to the Agent Policies tab in Fleet and click to edit the Agent Policy that we want to attach the Kafka output to. With the Agent Policy open, click the Settings tab and change Output for integrations and Output for agent monitoring to the Kafka output we just created.

Selecting an Output per Elastic Integration: To set the Kafka output to be used for specific data sources, see the integration-level outputs documentation.

A note about Topic Selection: The data_stream.type field is a reserved field which Elastic Agent automatically sets to logs if the data we're sending is a log and metrics if the data we're sending is a metric. Enabling Dynamic Topic selection using data_stream.type, will cause Elastic Agent to automatically route metrics to a metrics topic and logs to a logs topic. For information on topic selection, see the Kafka Output's Topics settings documentation.

Configuring a publishing endpoint in Elasticsearch

Next, we will set up two publishing endpoints (data streams) for the Confluent Cloud Sink Connector to use when publishing documents to Elasticsearch:

1. We will create a data stream logs-kafka.reroute-default for handling logs
2. We will create a data stream metrics-kafka.reroute-default for handling metrics

If we were to leave the data in those data streams as-is, the data would be available but we would find the data is unparsed and lacking vital enrichment. So we will also create two index templates and two ingest pipelines to make sure the data is processed by our Elastic Integrations.

Creating the Elasticsearch Index Templates and Ingest Pipelines

The following steps use Dev Tools in Kibana, but all of these steps can be completed via the REST API or using the relevant user interfaces in Stack Management.

First, we will create the Index Template and Ingest Pipeline for handling logs:

PUT _index_template/logs-kafka.reroute
{
  "template": {
    "settings": {
      "index.default_pipeline": "logs-kafka.reroute"
    }
  },
  "index_patterns": [
    "logs-kafka.reroute-default"
  ],
  "data_stream": {}
}

PUT _ingest/pipeline/logs-kafka.reroute
{
  "processors": [
    {
      "reroute": {
        "dataset": [
          "{{data_stream.dataset}}"
        ],
        "namespace": [
          "{{data_stream.namespace}}"
        ]
      }
    }
  ]
}

Next, we will create the Index Template and Ingest Pipeline for handling metrics:

PUT _index_template/metrics-kafka.reroute
{
  "template": {
    "settings": {
      "index.default_pipeline": "metrics-kafka.reroute"
    }
  },
  "index_patterns": [
    "metrics-kafka.reroute-default"
  ],
  "data_stream": {}
}

PUT _ingest/pipeline/metrics-kafka.reroute
{
  "processors": [
    {
      "reroute": {
        "dataset": [
          "{{data_stream.dataset}}"
        ],
        "namespace": [
          "{{data_stream.namespace}}"
        ]
      }
    }
  ]
}

A note about rerouting: For a practical example of how this works, a document related to a Linux Network Metric would be first land in metrics-kafka.reroute-default and this Ingest Pipeline would inspect the document and find data_stream.dataset set to system.network and data_stream.namespace set to default. It would use these values to reroute the document from metrics-kafka.reroute-default to metrics-system.network-default where it would be processed by the system integration.

Configure the Confluent Cloud Elasticsearch Sink Connector

Now it's time to configure the Confluent Cloud Elasticsearch Sink Connector. We will perform the following steps twice and create two separate connectors, one connector for logs and one connector for metrics. Where the required settings differ, we will highlight the correct values.

Navigate to your Kafka cluster in Confluent Cloud and select Connectors from the left-hand navigation menu. On the Connectors page, select Elasticsearch Service Sink from a catalog of connectors available.

Confluent Cloud presents a simplified workflow for the user to configure a connector. Here we will walk through each step of the process:

Step 1: Topic Selection

First, we will select the topic that the connector will consume data from based on which connector we are deploying:

• When deploying the Elasticsearch Sink Connector for logs, select the logs topic.
• When deploying the Elasticsearch Sink Connector for metrics, select the metrics topic.

Step 2: Kafka Credentials

Choose KAFKA_API_KEY as the cluster authentication mode. Provide the API Key and Secret noted earlier when we gather required Confluent Cloud Cluster information.

Step 3: Authentication

Provide the Elasticsearch Endpoint address of our Elasticsearch cluster as the Connection URI. The Connection user and Connection password are the authentication information for the account in Elasticsearch that will be used by the Elasticsearch Sink Connector to write data to Elasticsearch.

Step 4: Configuration

In this step we will keep the Input Kafka record value format set to JSON. Next, expand Advanced Configuration.

1. We will set Data Stream Dataset to kafka.reroute
2. We will set Data Stream Typebased on the connector we are deploying:
   • When deploying the Elasticsearch Sink Connector for logs, we will set Data Stream Type to logs
   • When deploying the Elasticsearch Sink Connector for metrics, we will set Data Stream Type to metrics
3. The correct values for other settings will depend on the specific environment.

Step 5: Sizing

In this step, notice that Confluent Cloud provides a recommended minimum number of tasks for our deployment. Following the recommendation here is a good starting place for most deployments.

Step 6: Review and Launch

Review the Connector configuration and Connector pricing sections and if everything looks good, it's time to click continue and launch the connector! The connector may report as provisioning but will soon start consuming data from the Kafka topic and writing it to the Elasticsearch cluster.

You can now navigate to Discover in Kibana and find your logs flowing into Elasticsearch! Also check out the real time metrics that Confluent Cloud provides for your new Elasticsearch Sink Connector deployments.

If you have only deployed the first logs sink connector, you can now repeat the steps above to deploy the second metrics sink connector.

Enjoy your fully managed data ingest architecture

If you followed the steps above, congratulations. You have successfully:

1. Configured Elastic Agent to send logs and metrics to dedicated topics in Kafka
2. Created publishing endpoints (data streams) in Elasticsearch dedicated to handling data from the Elasticsearch Sink Connector
3. Configured managed Elasticsearch Sink connectors to consume data from multiple topics and publish that data to Elasticsearch

Next you should enable additional integrations, deploy more Elastic Agents, explore your data in Kibana, and enjoy the benefits of a fully managed data ingest architecture with Elastic Serverless and Confluent Cloud!

The Design Problem in Log Processing

We rarely talk about how projects actually begin. 

How do you design something that doesn't fully exist yet?

How do you align AI capabilities, system constraints, real user pains into one coherent experience?

Streams gave us that challenge.

Logs are one of the richest signals in observability - but also one of the messiest. Streams is an agentic AI-powered solution that rethinks how teams work with logs to enable fast incident investigation and resolution. 

Streams uses AI to partition and parse raw logs, extract relevant fields, reduce schema management overhead, and surface significant events like critical errors and anomalies.

This led us to make logs investigation-ready from the start, and not force the Site Reliability Engineer to fight their data. But in order to enable such experience, we had to carefully rethink a core concept and step in the process - Processing.

Designing Processing UX in Elastic Streams

Logs are powerful, but only if they are structured correctly. Today, a user would onboard logs via Elastic Agent, using a custom integration, extract something as simple as an IP field by:

• Write GROK patterns
• Create pipelines
• Manage mappings
• Test transformation
• Iterate repeatedly

What sounds simple requires 20+ steps — and deep expertise most teams shouldn’t need. Our goal became simple: make this dramatically simpler.

Our early design question was:

“ Can we reduce this experience to 2 meaningful steps instead of 20 technical ones?”

That question shaped how we approached the Stream UX.

The Foundation

Before we jumped into designing the UI in Kibana, we defined a core mental model. 

A Stream is a collection of documents stored together that share:

• Retention
• Configuration
• Mappings
• Processing rules
• Lifecycle behaviour

The key design principle:

“A Stream should contain data that behaves consistently.”

Why Does Data Consistency Matter?

We started with an example to test our thinking. Take Nginx access and error logs.

Access logs describe request/response events:

192.168.1.10 - - [16/Feb/2026:12:32:10 +0000] "GET /api/orders/123 HTTP/1.1" 200 532 "-" "Mozilla/5.0"

Error logs describe diagnostic events:

2026/02/16 12:32:10 [error] 2719#2719: *342 connect() failed (111: Connection refused) while connecting to upstream…

If both live in the same Streams that might cause:

• Processing logic conflicts
• Field divergence
• Mapping conflicts
• Investigations would be fundamentally harder

That insight clarified something critical: 

“Processing isn’t just about extracting fields. It’s about protecting consistency.”

Making Complexity Manageable

The ingest ecosystem isn’t small, simple, or hypothetical. Real pipelines use dozens of processors — from common ones like rename, set, convert, and append, to niche types like urldecode and network_direction.

The UI had to support both high-frequency actions and long-tail edge cases without losing structure. Currently Elasticsearch supports over 40 different ingest processors. We had to make sure our interface could handle the different types.

We introduced a clear, nested structure for pipeline steps. Users could create, reorder, edit, or remove individual steps or grouped ones with confidence. The nested drag and drop capability was also added as a pattern in our EUI library.

This gave us the context and foundation to work on integrating those concepts into a model that would be definitive for everything in Streams.

Page Archetypes

Processing is powerful - and risky. Changing a parsing condition or step might affect:

• Field availability
• Search behaviour
• Alerts
• AI Insights
• Investigations

So we asked ourselves how do we make something so powerful and important, safe for the user? The answer led to a core page archetype:

Create > Preview > Confirm

This wasn’t a UI pattern added later. It emerged directly from our concept work and understanding what users would have to deal with.

To support this archetype and core idea, we also introduced a split-screen structure.

Left: Build

This is where users would:

• Add processing steps
• Define conditions
• Apply rules
• Leverage AI suggestions both as a whole pipeline creation or individual steps like a GROK processor

It remained focused, intentional and structured.

Right Preview

This is where users would:

• See real life log samples
• See extracted fields in context
• Immediate feedback on changes, with insights about the matched and unmatched percentage of documents
• Optional drilldown side panel on the right

The preview panel became the anchor of confidence. This was not about visual symmetry, but to reinforce experimentation, control over errors and decrease the level of mistakes. Knowing that users might want to switch their focus from interaction to detailed preview, we introduced the resizeable function to both panels, and unlocked more flexiblity and control over the use cases.

AI Automation

Streams is agentic and AI powered. That added another layer of complexity for the design, but also another opportunity to unlock even more power and insights from users' log data. 

AI introduced a new tension: how do you accelerate processing without turning it into a black box?

We established a few guardrails:

• Clear, concise suggestions
• Visible impact through matched document metrics
• Inspectability
• Alignment with the Create → Preview → Confirm model

Processing UX became the bridge between automation and human in the loop. Log data is one of the most powerful investigation signals. Every design decision reinforced that belief.

What We Learned

Designing for the future does not start with screens. It starts with:

• Edge case testing
• Clear mental models
• Strong and guiding principles
• Behavioral consistency
• Scalable and stress-tested archetypes

We know that in order for a user to be able unlock insightful discoveries from their logs, they would need to process and manage their data effectively. We knew we were shaping their entire observability foundation. 

Processing is about trust, control, and scalable data management.

Trust enables investigation speed.

Investigation speed enables resilience.

Learn more

Sign up for an Elastic trial at cloud.elastic.co, and trial Elastic's Serverless offering which will allow you to play with all of the Streams functionality. You want to know more about Streams? Check some of the links below:

Read about Reimagining streams

Read about Retention management

Look at the Streams website

Check the Streams documentation

Observability for developers has lately been distilled into implementing auto-instrumentation, allowing you to instantly connect your code with the larger observability world. This way of utilizing an upstream SDK is certainly the simplest and most production-ready, and works efficiently with the Elastic Cloud Managed OTLP Endpoint.

But what if you could not only add powerful tracing to your Go service but also truly understand how the magic works, rather than just copy-pasting configuration files or a line of code? In the same way that you build your knowledge of software development systems, observability, modernized by OpenTelemetry (OTel) standardization, is a rich, broad system that is valuable to understand. Here is an in-depth technical breakdown of every piece of simple OTel instrumentation using the Elastic Distributions of OpenTelemetry (EDOT) and Golang, from the ground up.

Telemetry is the automated collection, transmission and analysis of data from your application, which can apply to any observable distributed system. This data can range from regular health check calls with your application to real-time information about user interactions, requests, and transactions. Using the example application repository here, we’ll build a strong observability foundation to start observing our applications with confidence.

Understanding the OpenTelemetry Flow

Below, you will see the basic flow of your data when implementing observability with OTel in your system. Before we dive in, let’s explain some of the key terms within OTel. Let’s go over these base key players that we need to implement observability solutions with OTel:

• Span: This is a single, timed unit of a distributed trace that can represent a specific operation, such as a database query or an HTTP handler.

• Trace: This is a detailed record of a single request’s journey through your system, AKA a hierarchy of your spans.

• Tracer: This is the handle for generating spans. You will typically have one per instrumentation library, for example myapp/http.

• Tracer Provider: This is the cornerstone of the SDK. This creates Tracer instances, and you can configure it on application start up.

• Context Propagation: The mechanism for passing trace context between operations and services, maintaining the relationship between parent and child spans.

• Exporter: This is the part that is responsible for sending your telemetry data to a vendor backend, and you can decide if you are sending it to the OTel Collector, EDOT Collector or an OTLP Endpoint.

Installing the Magic, Instrumentation Style

OpenTelemetry provides instrumentation libraries that handle much of the tracing complexity for you. These libraries wrap common frameworks and libraries (like net/http/otelhttp) and automatically capture telemetry without requiring you to manually create spans for every operation.

However, before you're able to send any telemetry, OTel needs to know who (which service) is sending that data.

A resource represents the specific entity, in this case "simple-go-service", that is producing your telemetry data. Its identity is recorded as resource attributes, and resource attributes can include pod names, service names or instances, deployment environments; Basically anything important to identifying your resource. This resource is your service's identity card that gets attached to every span and metric that it emits with its attributes. Once your trace arrives, these attributes can answer "what version was running?" or "which service is this from?"

func initOTel(ctx context.Context, endpoint string) (func(context.Context) error, error) {
res, err := resource.New(ctx,
		resource.WithAttributes(
			semconv.ServiceName("simple-go-service"),
			semconv.ServiceVersion("1.0.0"),
		),
	)
	if err != nil {
		return nil, err
	}

In the code above, resource.New() constructs the "identity card" of our Go service. The attributes that will be attached to it will use semantic conventions(semconv), standardized names for common metadata fields. These semantic conventions make sure that every single OTel-compatible observability backend knows their meaning.

Now that we've bootstrapped our application with the initOtel function, we can continue to configure everything else!

Let’s begin instrumenting this application by building all the app components that we will need to implement modern observability tools. Below is our instrumentation using otelhttp, which will handle span creation after calling the specified API routes. 

http.Handle("/hello", otelhttp.NewHandler(http.HandlerFunc(handleHello), "hello"))
http.Handle("/api/data", otelhttp.NewHandler(http.HandlerFunc(handleData), "data"))
http.HandleFunc("/health", handleHealth)

// Example of a tracer within our handleHello() function
tracer = tp.Tracer("simple-go-service")

ctx, span := tracer.Start(ctx, "process-hello")
defer span.End()

The key insight here is that otelhttp.NewHandler handles all the span lifecycle management for HTTP requests. You don't need to manually call tracer.Start()or span.End() for basic HTTP tracing since the library does this for you.

On application start up, the SDK will use the tracer provider set up below in order to create Tracer instances. These instances help create and manage the spans contained within traces.

traceExporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint(endpoint),
        otlptracegrpc.WithInsecure(),
    )

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(traceExporter),
        sdktrace.WithResource(res),
    )
    otel.SetTracerProvider(tp)
    tracer = tp.Tracer("simple-go-service")

Within our initOTel function, we will set up one of our most important signals: logs. First, we initialize the logExporter that will send logs to our OTel Collector using gRPC protocol. Then the LoggerProvider will create the base of the logExporter that batches log entries together before sending those batches to your exporter, attaching metadata about the services along the way. Lastly, the LoggerProvider also creates a standard Go structured logger (slog) that automatically includes trace context (such as span IDs) and batches your log with other logs. These are sent to your observability backend through the exporter along with your metrics and traces. 

logExporter, err := otlploggrpc.New(ctx,
		otlploggrpc.WithEndpoint(endpoint),
		otlploggrpc.WithInsecure(),
	)
	if err != nil {
		return nil, err
	}

	lp := sdklog.NewLoggerProvider(
		sdklog.WithProcessor(sdklog.NewBatchProcessor(logExporter)),
		sdklog.WithResource(res),
	)
	logger = slog.New(otelslog.NewHandler("simple-go-service", otelslog.WithLoggerProvider(lp)))

Below you can see how you can view your logs through Kibana in the APM UI. These logs are also color - coordinated; Coded warnings are in yellow, errors are in red, and regular logs are in green.

Metrics are set up in the next part of our code. Metrics are telemetry signals that track the quantitative data from your application, such as response times and request counts. The metric exporter is initialized to send metric data to our EDOT Collector then to our observability backend, Elastic Observability in this case, using gRPC. The meter provider in the next portion periodically collects and exports our metrics data and measurements, the same as the tracer provider creates tracers. The only difference between the two providers is that the meter provider works on a timer while the trace provider exports spans as they complete.

metricExporter, err := otlpmetricgrpc.New(ctx,
		otlpmetricgrpc.WithEndpoint(endpoint),
		otlpmetricgrpc.WithInsecure(),
	)
	if err != nil {
		return nil, err
	}

	mp := metric.NewMeterProvider(
		metric.WithReader(metric.NewPeriodicReader(metricExporter)),
		metric.WithResource(res),
	)
	otel.SetMeterProvider(mp)

	meter := mp.Meter("simple-go-service")
	requestCounter, _ = meter.Int64Counter("http.requests")
	requestDuration, _ = meter.Float64Histogram("http.duration")

In order to finish initializing OpenTelemetry, we set up our propagators for context propagation. The set text map propagator automatically injects the trace ID and the span ID of your service making an outbound HTTP request to another service, following the W3C Trace Context standard. In short, this maintains the parent-child relationship between spans.

otel.SetTextMapPropagator(propagation.TraceContext{})

	return func(ctx context.Context) error {
		tp.Shutdown(ctx)
		mp.Shutdown(ctx)
		lp.Shutdown(ctx)
		return nil
	}, nil

Now that you know how these pieces work together, try to run the repository linked here, using the readme as your guide.

Sidenote: Adding Custom Spans

For getting an application emitting traces, this instrumentation works great! If you visit localhost:8080/hello after starting the docker containers, the otelhttp middleware automatically creates spans for each HTTP request. However, basic instrumentation only shows essential application telemetry, such as response duration, URL paths, and status codes. You won’t know what happens between the request coming in and request completion. The moment OpenTelemetry truly gains power is when you add custom spans. Unlike auto-instrumentation where spans are created as well as closed automatically, custom spans require you to explicitly start and stop them.

Custom spans can track your application’s logic, such as specific business events or marking expensive operations, using a detailed hierarchy within each trace. In the application for this article, there are several custom spans that were created to track important operations:

• background-work: This traces asynchronous processing that happens with the main request.

• computation: This measures computations and then captures those results, and the computation type.

Custom spans add granular visibility into your application's behavior. For example, in performComputation:

ctx, span := tracer.Start(ctx, "computation")
defer span.End()

result := rand.Float64()
span.SetAttributes(
	attribute.String("comp.type", compType),
	attribute.Float64("comp.result", result),
	)

	logger.InfoContext(ctx, "Computation completed", "type", compType, "result", result)

if result < 0.3 {
span.AddEvent("Low confidence result")
	logger.WarnContext(ctx, "Low confidence computation", "result", result)
}
}

The attributes set above become searchable and filterable in our Elastic Observability backend, allowing for attribute filtering by attribute.result and attribute.compType. If you query your data with “show me all computations where results are less than 0.3,” then you will notice the span event span.AddEvent(“Low confidence result”) tacked on with a timestamped marker. This appears on your trace timeline as well, adding even more visibility to any unusual events. Below is a small example of the filtering that Kibana can accomplish from custom spans.

The Data Pipeline: From Code to IRL

Now that you can export your custom spans and data to OTLP which sends it to the EDOT Collector and then to an observability backend, the best hub for your telemetry data will be the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/. It is a simple, standalone process that is able to receive, process and export all of your telemetry data. Within this project, we use the Elastic Distributions of OpenTelemetry (EDOT) Collector, an optimized Collector for usage within your Elastic Stack. Since this is a self-managed Elastic instance, this article and connected repository utilize the EDOT Collector through elasticapm, but for Elastic Cloud or Serverless projects, you can use the Elastic Managed OpenTelemetry Protocol (OTLP) Endpoint. As noted in the quickstart documentation here, the Elastic Cloud Managed OTLP Endpoint endpoint helps get your data quickly and efficiently into your Elastic Stack through OTLP, without schema translation! This means that your telemetry hits Elastic instantly and your telemetry data remains vendor-neutral.

For most developers and SREs, this Collector is an amazing tool. It allows you to decouple your code from the observability backend. Your application does not need to know its final destination, it can just send the data to the Collector. Your observability backend can change constantly without it even touching your code. The OpenTelemetry Collector also acts as a gateway for multiple streams of data, and is able to accept various formats in order to unify them for exportation. Lastly, the OpenTelemetry Collector is able to offload processing power from your application - tasks such as retries, batching and filtering can happen in the Collector, not your application.

After trying out this article’s repository, try auto-instrumenting your application with Elastic Distributions of OpenTelemetry (EDOT) so that you can utilize the APM UI to its full potential! With the latest version of Elasticsearch and Kibana start-local, you can use Docker to install and run the services and instantly start monitoring your application. 

Understanding the Collector Configuration

The Collector's behavior is defined in a configuration file (otel-collector-config.yaml). Let's break down each component.

Receivers define how the Collector accepts telemetry data. Here, we're listening for both gRPC and HTTP traffic.

receivers:
  # Receives data from other Collectors in Agent mode
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

Connectors are specialized components that sit in between pipelines, and in this case, we are using the elasticapm Connector. This APM Connector exports our metrics, logs, and traces, while simultaneously acting as a receiver for the metrics/aggregated-otel-metrics pipeline (see below). Without it, your raw OTLP data lands in Elasticsearch, but the APM UI has nothing to build its views from.

connectors:
  elasticapm: {} # Elastic APM Connector

Processors transform, filter, or enrich data as it passes through the EDOT Collector. The batch processor aggregates spans before export, reducing network overhead and improving efficiency, as well as limiting batch sizes. The batch/metrics processor does this as well, but for APM metrics. Lastly, there is the Elastic APM processor. This processor ensures that your spans fields are aligned, your traces views are complete, and it overall bridges the gap between Elastic's expectations and OpenTelemetry's formatting of your traces.

processors:
  batch:
    send_batch_size: 1000
    timeout: 1s
    send_batch_max_size: 1500
  batch/metrics:
    send_batch_max_size: 0 # Explicitly set to 0 to avoid splitting metrics requests
    timeout: 1s
  elasticapm: {} # Elastic APM Processor

As mentioned previously in the article, exporters send data to your observability backend. The debug exporter logs telemetry to the console (useful for development), while the Elasticsearch exporter sends traces to your Elastic stack.

exporters:
  debug: {}
  elasticsearch/otel:
    endpoints:
      - ${ELASTIC_ENDPOINT} # Will be populated from environment variable
    user: elastic
    password: ${ELASTIC_PASSWORD}
    tls:
      ca_file: /config/certs/ca/ca.crt
    mapping:
      mode: otel

Pipelines connect receivers, processors, and exporters into a data flow. These EDOT Collector pipelines receive OTLP traces, batches them, and exports to the debug, elasticapm and elasticsearch/otel exporters. It also exports metrics to the debug and elasticsearch/otel exporters.

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch/metrics]
      exporters: [debug, elasticsearch/otel]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, elasticapm, elasticsearch/otel]
    traces:
      receivers: [otlp]
      processors: [batch, elasticapm]
      exporters: [debug, elasticapm, elasticsearch/otel]
    metrics/aggregated-otel-metrics:
      receivers:
        - elasticapm
      processors: [] # No processors defined in the original for this pipeline
      exporters:
        - debug
        - elasticsearch/otel

Debugging Your Code with Confidence in Kibana

Elastic Observability, utilizing Kibana and Streams, has native support for the OTLP Endpoint through the EDOT Collector, which was used in this project. Below, you can see that your data is automatically connected to Streams from the beginning, requiring no extra leg work! You can add conditions or any Grok processors as your data is streaming in, and you'll be able to instantly see your data's schema and data quality.

Elastic also provides the Elastic Cloud Managed Endpoint for even easier storage, data-processing, and scaling. If you use this Managed Endpoint, it means that you can configure OpenTelemetry to send data directly to Elasticsearch, without ANY specialized Collectors. Any way you choose, once your traces are flowing, Kibana’s APM UI provides powerful visualization and analysis capabilities will be everything you need to debug your code. You are able to drill down into individual requests, identify bottlenecks, find anomalies and troubleshoot any issues that arise with confidence.

Here is one span of interest from this repository. Within Kibana, you can immediately filter by the Trace ID, finding other spans with the same Trace ID to visually see the entire trace.

Kibana Discover also allows you to switch indices instantly without losing your filters, ensuring that you can also see the logs that correspond with the same Trace ID.

In addition to the manually checking your traces, you can automatically check them within the APM UI (shown below). This is easy trace visualization using the Kibana APM UI is readily available while using the elasticapm connector. Below is a visualization of a trace comprised of spans within our project. Knowing both methods of correlating spans is beneficial to build the foundation of utilizing Kibana and the APM UI for observability.

Here is a fully built out dashboard built from the repository featured in this article. The possibilities with Elastic Observability are endless!

Congrats, You’re Not “Just” a Developer Anymore!

We’ve broken down the why and how behind OpenTelemetry’s basic components, including the TraceProvider, the span, the exporter and the Collector. Here, you’ve done more than just implement your tracing tool. You now understand the complete data flow from your code to the graphs on your dashboard.

You can now speak the language of observability with confidence, not because you memorized a configuration file, but because you now understand the data flow from your code to the graph on your dashboard. You understand how telemetry moves through your system. You aren’t “just” a developer anymore; you’re now a developer who can truly see.

Try out the code repo above! Included in the repository is a generate-traffic.sh script file. You can run this repeatedly in order to generate logs, traces, and metrics for you to play with within the APM UI. Also, check out our latest releases in our release docs page for exciting Elastic updates.

In this blog you will learn about the quantifiable operational benefits of ECS, how to leverage ECS with any data ingest tool, and the pitfalls to avoid. The data source leveraged in this blog is a 3.3GB Nginx log file obtained from Kaggle. The representation of this dataset is divided into three categories: raw, self, and ECS; with raw having zero normalization, self being a demonstration of commonly implemented mistakes observed from my 5+ years of experience working with various users, and finally ECS with the optimal approach of data hygiene.

This hygiene is achieved through the parsing, enrichment, and mapping of data ingested; akin to the sequencing of DNA in order to express genetic traits. Through the understanding of the data's structure, and assigning the correct mapping, a more thorough expression may be represented, stored and searched upon.

If you would like to learn more about ECS, the dataset used in this blog, or available Elastic integrations, please be sure to check out these related links:

• Introducing the Elastic Common Schema

• Kaggle Web Server Logs

• Elastic Integrations

Dataset Validation

Before we begin, let us review how many documents exist and what we're required to ingest. We have 10,365,152 documents/events from our Nginx log file:

With 10,365,152 documents in our targeted end-state:

Dataset Ingestion: Raw & Self

To achieve the raw and self ingestion techniques, this example is leveraging Logstash for simplicity. For the raw data ingest, a simple file input with no additional modifications or index templates.

    input {
      file {
      id => "NGINX_FILE_INPUT"
      path => "/etc/logstash/raw/access.log"
      ecs_compatibility => disabled
      start_position => "beginning"
      mode => read
      }
    }
    filter {
    }
    output {
      elasticsearch {
        hosts => ["https://mycluster.es.us-east4.gcp.elastic-cloud.com:9243"]
          index => "nginx-raw"
          ilm_enabled => true
          manage_template => false
          user => "username"
          password => "password"
          ssl_verification_mode => none
          ecs_compatibility => disabled
          id => "NGINX-FILE_ES_Output"
      }
    }

For the self ingest, a custom Logstash pipeline with a simple Grok filter was created with no index template applied:

    input {
      file {
        id => "NGINX_FILE_INPUT"
        path => "/etc/logstash/self/access.log"
        ecs_compatibility => disabled
        start_position => "beginning"
        mode => read
      }
    }
    filter {
      grok {
        match => { "message" => "%{IP:clientip} - (?:%{NOTSPACE:requestClient}|-) \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:requestMethod} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" (?:-|%{NUMBER:response}) (?:-|%{NUMBER:bytes_in}) (-|%{QS:bytes_out}) %{QS:user_agent}" }
      }
    }
    output {
      elasticsearch {
        hosts => ["https://myscluster.es.us-east4.gcp.elastic-cloud.com:9243"]
        index => "nginx-self"
        ilm_enabled => true
        manage_template => false
        user => "username"
        password => "password"
        ssl_verification_mode => none
        ecs_compatibility => disabled
        id => "NGINX-FILE_ES_Output"
      }
    }

Dataset Ingestion: ECS

Elastic comes included with many available integrations which contain everything you need to achieve to ensure that your data is ingested as efficiently as possible.

For our use case of Nginx, we'll be using the associated integration's assets only.

The assets which are installed are more than just dashboards, there are ingest pipelines which not only normalize but enrich the data while simultaneously mapping the fields to their correct type via component templates. All we have to do is make sure that as the data is coming in, that it will traverse through the ingest pipeline and use these supplied mappings.

Create your index template, and select the supplied component templates provided from your integration.

Think of the component templates like building blocks to an index template. These allow for the reuse of core settings, ensuring standardization is adopted across your data.

For our ingestion method, we merely point to the index name that we specified during the index template creation, in this case, nginx-ecs and Elastic will handle all the rest!

    input {
      file {
      id => "NGINX_FILE_INPUT"
      path => "/etc/logstash/ecs/access.log"
      #ecs_compatibility => disabled
      start_position => "beginning"
      mode => read
      }
    }
    filter {
    }
    output {
      elasticsearch {
        hosts => ["https://mycluster.es.us-east4.gcp.elastic-cloud.com:9243"]
        index => "nginx-ecs"
        ilm_enabled => true
        manage_template => false
        user => "username"
        password => "password"
        ssl_verification_mode => none
        ecs_compatibility => disabled
        id => "NGINX-FILE_ES_Output"
      }
    }

Data Fidelity Comparison

Let's compare how many fields are available to search upon the three indices as well as the quality of the data. Our raw index has but 15 fields to search upon, with most being duplicates for aggregation purposes.

However from a Discover perspective, we are limited to 6 fields!

Our self-parsed index has 37 available fields, however these too are duplicated and not ideal for efficient searching.

From a Discover perspective here we have almost 3x as many fields to choose from, yet without the correct mapping the ease of which this data may be searched is less than ideal. A great example of this, is attempting to calculate the average bytes_in on a text field.

Finally with our ECS index, we have 71 fields available to us! Notice that courtesy of the ingest pipeline, we have enriched fields of geographic information as well as event categorial fields.

Now what about Discover? There were 51 fields directly available to us for searching purposes:

Using Discover as our basis, our self-parsed index has 283% more fields to search upon whereas our ECS index has 850%! 

Storage Utilization Comparison

Surely with all these fields in our ECS index the size would be exponentially larger than the self normalized index, let alone the raw index? The results may surprise you.

Accounting for the replica of data of our 3.3GB size data set, we can see that the impact of normalized and mapped data has a significant impact on the amount of storage required.

Conclusion

While there is an increase in the amount required storage for any dataset that is enriched, Elastic provides easy solutions to maximize the fidelity of the data to be searched while simultaneously ensuring operational storage efficiency; that is the power of the Elastic Common Schema.

Let's review how we were able to maximize search, while minimizing storage

• Installing integration assets for our dataset that we are going to ingest.

• Customizing the index template to leverage the included components to ensure mapping and parsing are aligned to the Elastic Common Schema.

Ready to get started? Sign up for Elastic Cloud and try out the features and capabilities I've outlined above to get the most value and visibility out of your data.

Indeed, an expired certificate is not just a minor technical glitch. It is a direct hit on your most critical systems:

• Your CI/CD pipeline grinds to a halt because it can not trust the internal image registry.

• Your Single Sign-On (SSO) system fails, locking all your internal users out.

• Your external clients see scary browser warnings, shattering user trust and forcing support tickets.

• Your SLOs burn due to services not being able to communicate with one another.

In Kubernetes, certificates are usually dynamically generated and auto-renewed by tools like cert-manager. In more unlucky scenarios, certificates might be tucked away inside Secrets and ConfigMaps, leading to challenges while inventorying them. It is neither hard nor unheard of to have a dozen critical certificates and no centralized way to know when they are about to expire.

Additionally, only monitoring the certificates for external Load Balancers might lead to huge internal risks, since many certificates never get exposed to external users.

In this blog post, we will guide you through establishing comprehensive, cluster-wide certificate monitoring using the OpenTelemetry Collector, the x509-certificate-exporter, and Elastic Observability.

Classical approach: HTTP monitoring

The classical approach to monitor TLS certificate expiration in the Elastic Observability is by treating it like any other service availability check. Historically, this was accomplished using Heartbeat or, more recently, Elastic Observability's Synthetics. These tools perform an external check against a public HTTPS endpoint and automatically extract the certificate's validity dates, allowing you to configure a Synthetics TLS certificate rule in Kibana to trigger an alert when expiration is within a specified threshold (e.g., 30 days).

While effective for external-facing services, this "classical" approach has two major shortcomings when dealing with Kubernetes:

• It only works for certificates exposed via HTTP(S), meaning you cannot use this for internal services, databases, or message queues using other protocols. In other words, this won't work to monitor common, critical TLS certificates such as Kafka's.

• The monitoring agent must have network access to the endpoint. In a segmented or private Kubernetes environment, deploying agents with the necessary access often introduces unnecessary complexity or security risks.

To gain true cluster-wide visibility, we need to inspect the certificates at their source: inside Kubernetes Secrets or ConfigMaps.

A Kubernetes-native approach: monitor Secrets and ConfigMaps

Monitoring TLS certificate expiration directly within Kubernetes Secrets and ConfigMaps is the only reliable way to gain visibility into internal, non-HTTP-exposed certificates, such as those used for service meshes, internal registries, or databases. In this section, we will use the OpenTelemetry Collector to monitor certificate expiration.

The OpenTelemetry Collector provides a mechanism to read up-to-date information from the Kubernetes API, including Secrets, via the k8sobjects receiver. However, this receiver only fetches raw TLS certificate resource data, which the OpenTelemetry Transformation Language (OTTL) can not properly parse. Therefore, we need to use a dedicated exporter to collect the certificate data and expose the results in a digestible format.

The industry-standard solution

As mentioned above, simply reading certificate information from the Kubernetes API is not a feasible solution. We will therefore use a specialized, lightweight exporter (specifically, the popular x509-certificate-exporter) to collect TLS certificate data and expose the results, allowing the OpenTelemetry Collector's Prometheus receiver to seamlessly scrape the data and send it to Elastic Observability. This approach immediately and easily enables us to monitor both certificates generated by cert-manager and self-managed ones, such as the ones created for ECK.

A fully working configuration example and a script to set up a complete local development environment is available here. Feel free to use it to follow along as you read through this guide and try out the examples. Please note that, while this repository uses the Elastic Distribution of OpenTelemetry (EDOT), it can be easily adapted to use the OpenTelemetry Collector.

Helm Chart Configuration

We configured the x509-certificate-exporter with the official Helm Chart and used the following minimal configuration:

secretsExporter:
  secretTypes:
  - type: kubernetes.io/tls
    key: tls.crt
  # For ECK that uses different secret types
  - type: Opaque
    key: tls.crt
  - type: Opaque
    key: ca.crt
  configMapKeys:
  - tls.crt
  - ca.crt

# Create a service to have a stable endpoint for scraping metrics
service:
  create: true
  # -- TCP port to expose the Service on
  port: 9793

# Disable prometheus service monitor and prometheus rules
prometheusServiceMonitor:
  create: false
prometheusRules:
  create: false

We refer to the reference values.yaml to get insights in the plethora of configuration options.

OpenTelemetry Collector Configuration

Afterward, we configured the OpenTelemetry Collector to scrape the metrics from the service:

prometheus/cert-expiration:
  config:
    scrape_configs:
      - job_name: "cert-expiration"
        scrape_interval: 60m
        static_configs:
          - targets:
              - "x509-certificate-exporter.monitoring.svc.cluster.local:9793"

We deliberately used a long scrape interval of 60 minutes, because certificate expiration is a low-frequency concern.

Visualizing the data in Kibana

Once the data is ingested, we can explore it using Discover. We can select the metrics-* Data View and search for our data with the filter data_stream.dataset : "prometheusreceiver.otel".

An example document looks like the following:

{
  "@timestamp": "2025-12-19T09:43:45.317Z",
  "_metric_names_hash": "7d113f55b70019d9",
  "attributes": {
    "issuer_CN": "tls-cert.example.com",
    "issuer_O": "TLS Cert",
    "secret_key": "tls.crt",
    "secret_name": "tls-cert-secret",
    "secret_namespace": "test-certs",
    "serial_number": "250887723804527203192865532237673843132727735771",
    "subject_CN": "tls-cert.example.com",
    "subject_O": "TLS Cert"
  },
  "data_stream": {
    "dataset": "prometheusreceiver.otel",
    "namespace": "default",
    "type": "metrics"
  },
  "metrics": {
    "x509_cert_expired": 0,
    "x509_cert_not_after": 1768488242,
    "x509_cert_not_before": 1765896242
  },
  "resource": {
    "attributes": {
      "server.address": "x509-certificate-exporter.monitoring.svc.cluster.local",
      "server.port": "9793",
      "service.instance.id": "x509-certificate-exporter.monitoring.svc.cluster.local:9793",
      "service.name": "cert-expiration",
      "url.scheme": "http"
    }
  },
  "scope": {
    "name": "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver",
    "version": "9.2.2"
  }
}

The core metric reported by the x509-certificate-exporter is x509_cert_not_after that represent the Unix Epoch timestamp (in seconds) of the certificate's expiration date. This metric has some attributes associated with it. In the case of Secrets, the following attributes are relevant:

• secret_namespace: The namespace of the Secret containing the certificate.
• secret_name: The name of the Secret containing the certificate.
• secret_key: The specific key within the Secret where the certificate is stored.

In the case of ConfigMaps, we can infer the attributes of interest from the filepath attribute.

Finally, we can leverage ES|QL to compute the remaining days until expiration. In the following examples, we will use the TS command, which is optimized and recommended for interacting with time-series data.

For Secrets:

TS metrics-*
| WHERE metrics.x509_cert_not_after is not NULL
| STATS expiration_date = MAX(LAST_OVER_TIME(metrics.x509_cert_not_after)) by attributes.secret_namespace, attributes.secret_name, attributes.secret_key
| EVAL remaining_days = DATE_DIFF("days", NOW(), TO_DATETIME (1000 * expiration_date))
| EVAL expiration_date = TO_DATETIME(1000 * expiration_date)
| SORT expiration_date ASC

And for ConfigMaps:

TS metrics-*
| WHERE metrics.x509_cert_not_after IS NOT NULL
| WHERE attributes.filepath IS NOT NULL
| DISSECT attributes.filepath "k8s/%{namespace}/%{configmap}"
| WHERE configmap != "kube-root-ca.crt" // Filter out the Kubernetes API server certificate's signing CA
| STATS expiration_date = MAX(LAST_OVER_TIME(metrics.x509_cert_not_after)) by namespace, configmap, filename
| EVAL remaining_days = DATE_DIFF("days", NOW(), TO_DATETIME (1000 * expiration_date))
| EVAL expiration_date = TO_DATETIME(1000 * expiration_date)
| SORT expiration_date ASC

Based on these core queries, we can easily build a dashboard that shows the remaining days until expiration for all the certificates in the cluster:

and create alerts about certificates that are about to expire by adding a condition after the query:

WHERE remaining_days < 30

Conclusion

In this blog post, we explored how to monitor TLS certificate expiration within a Kubernetes cluster using the OpenTelemetry Collector. We discussed the limitations of traditional HTTP-based monitoring approaches and introduced a Kubernetes-native solution leveraging the x509-certificate-exporter to extract certificate expiration data directly from Kubernetes Secrets and ConfigMaps. This method provides comprehensive visibility into all certificates used within the cluster, including those not exposed via HTTP(S).

For the sake of simplicity, we just focused on monitoring certificate expiration with the OpenTelemetry Collector on Kubernetes. However, this approach can be easily applied with classical Elastic Agent by leveraging the Prometheus input package (read more on how to use input packages here) and can be also extended to monitor certificates on virtual machines or bare-metal servers by deploying the x509-certificate-exporter there.

Finally, is worth knowing that Elastic Observability, offers an officially supported distribution of the OpenTelemetry Collector, called Elastic Distributions of OpenTelemetry (EDOT).

If you are an Elastic user, you could consider using EDOT Collector to monitor certificates with OpenTelemetry: since it is supported by Elastic Observability, it will be easier to manage and keep up to date. Alternatively you can use upstream OTel compnents also.

What's next?

Now that Elastic supports Rule Templates and OpenTelemetry content packs, our near-term objective is to contribute to the integration repository to make the setup of certificate monitoring even easier for our users. Stay tuned for more updates on this!

Check out other resources on Elastic's OpenTelemetry

Elastic's OTLP EndPoint

Elastic's EDOT PHP Contribution

Opentelemetry SDK Central Management with EDOT

Also sign up for Elastic Cloud and try out your application with OpenTelemetry in Elastic

To run ECF for GCP confidently at scale, you need to understand its capacity characteristics and sizing behavior. For ECF for GCP which is part of the broader ECF architecture, we answered those questions through repeatable load testing and by grounding decisions in measured data.

We'll introduce the test setup, explain each runtime setting, and share the capacity numbers we observed for a single instance.

How we load tested EDOT Cloud Forwarder for GCP

Architecture

The load testing architecture simulates a realistic, high-volume pipeline:

1. We developed a load tester service that uploads generated log files to a GCS bucket as fast as possible.
2. Each file creation in this Google Cloud Storage (GCS) bucket then triggers an event notification to Pub/Sub.
3. Pub/Sub delivers push messages to a Cloud Run service where EDOT Cloud Forwarder fetches and processes these log files.

Our setup exposes two primary tunable settings that directly influence Cloud Run scaling behavior and memory pressure:

• Request pressure using a concurrency setting (how many concurrent requests each ECF instance can handle).
• Work per request using a log count setting (number of logs per file in each uploaded object).

In our tests, we used a testing system that:

• Deploys the whole testing infrastructure. This includes the complete ECF infrastructure, a mock backend, etc.
• Generates log files according to the configured log counts, using a Cloud Audit log of ~1.4 KB.
• Runs a matrix of tests across all combinations of concurrency and log volume.
• Produces a report for each tested concurrency level in which several stats are reported, such as CPU usage and memory consumption.

For reproducibility and isolation, the otlphttp exporter in EDOT Cloud Forwarder uses a mock backend that always returns HTTP 200. This ensures all observed behavior is attributable to ECF itself, not downstream systems or network variability.

Step 1: Establish a stable runtime before measuring capacity

Before asking how much load a single instance can handle, we first established a stable runtime baseline.

We quickly learned that a single flag, cpu_idle, can turn Cloud Run into a garbage-collector (GC) starvation trap. This is amplified by a known limitation of our ECF current architecture: the existing OpenTelemetry implementation reads whole log files into memory before processing them. Our goal was to eliminate configuration side effects so capacity tests reflected ECF actual limits.

We focused on three runtime parameters:

All parameter-isolation tests use a single Cloud Run instance (min 0, max 1), fix concurrency for the scenario under study, and keep input files and test matrix identical across runs. This design lets us attribute differences directly to the parameter in question.

CPU allocation: Stop starving the garbage collector

Cloud Run offers two CPU allocation modes:

• Request-based (throttled). Enabled with cpu_idle: true. CPU is available only while a request is actively being processed.
• Instance-based (always on). Enabled with cpu_idle: false. CPU remains available when idle, allowing background work such as garbage collection to run.

The tests compared these modes under identical conditions:

What we observed

With CPU allocated only on requests (cpu_idle: true):

• Memory variance was extreme (±71% RSS, ±213% heap).
• Peak heap reached ~304 MB in the worst run.
• We saw request refusals in the sample (90% success rate).

With CPU always allocated (cpu_idle: false):

• Memory variance became tightly bounded (±8% RSS, ±32% heap).
• Peak heap dropped to ~89 MB in the worst run.
• We saw no refusals in the sample (100% success).

From these runs we saw:

• When CPU is throttled, the Go garbage collector is effectively starved, leading to heap accumulation and large run-to-run variance.
• When CPU is always available, garbage collection keeps pace with allocation, resulting in lower and more predictable memory usage.

Takeaway: for this set of tests, cpu_idle: false was the most stable baseline configuration. Request-based CPU throttling introduced artificial instability that makes capacity planning much harder.

Go memory limit: GOMEMLIMIT in constrained containers

Cloud Run enforces a hard memory limit at the container level. If the process exceeds it, the instance is OOM-killed.

We tested Cloud Run with:

The tests compared:

• No GOMEMLIMIT (Go relies on OS pressure).
• GOMEMLIMIT=460MiB (or 90% of container memory).

The results were clear:

Takeaway: in a memory-constrained environment like Cloud Run, setting GOMEMLIMIT close to (but below) the container limit is essential for predictable behavior under load.

GOGC: memory savings vs. reliability

The GOGC parameter controls how much the heap can grow (in %) between GC cycles:

• Lower values (e.g., GOGC=50): more frequent collections, lower memory, higher CPU.
• Higher values (e.g., GOGC=100): fewer collections, higher memory, lower CPU.

The tests covered: (1) GOGC=50 (aggressive); (2) GOGC=75 (moderate); (3) GOGC=100 (default/unset).

Setup:

What we observed

From the runs:

The conclusion from these runs is clear: pushing GOGC down trades memory for reliability, and the trade is not favorable for ECF.

Takeaway: for this workload, the default GOGC=100 provided the best balance. Attempts to optimize memory by lowering GOGC directly reduced reliability.

Step 2: Find capacity and breaking points

With the runtime stabilized, we evaluated how much traffic a single instance can sustain by increasing concurrency until failures emerged.

How to read the tables: each concurrency level was tested across 20 runs covering both light (240 logs per file, around 362KB file size) and heavy inputs (over 6k logs per file, around 8MB file size). Tables report baseline RSS from light workloads and peak values from the worst-case run.

Concurrency 5: Stable baseline

At concurrency 5, the service was solid.

This proved that a single instance handles a moderate load comfortably, with memory usage staying well within safe limits.

Concurrency 10: Safe but volatile

At concurrency 10, the system remained functional but with significant volatility.

We also noticed that memory usage shows extreme variance:

• Best run RSS: 178 MB.
• Worst run RSS: 425 MB.

This behavior comes mainly from two effects:

• Bursty Pub/Sub delivery: 10 heavy requests may land at nearly the same instant.
• The use of io.ReadAll inside the collector: each request reads the entire log file into memory.

When all 10 requests arrived concurrently, we were effectively stacking ~10× file size in RAM before the GC can clean up. When they are slightly staggered, GC has time to reclaim memory between requests, leading to much lower peaks.

This leads to a crucial sizing insight:

• Do not size the service using average memory (for example, ~260 MB).
• Size it for the worst observed burst (~425 MB) to avoid OOM or GC stalls.

In practice, you should set the memory limit to at least 512 MiB per instance at concurrency 10.

Concurrency 20: Unstable, systemic load shedding

At concurrency 20, the system consistently began shedding load.

Even though memory and CPU metrics don't look drastically worse than at concurrency 10, behavior changes qualitatively: the service begins to refuse requests consistently.

Concurrency 40: Failure mode

At concurrency 40, the instance collapsed completely. Memory and CPU are overwhelmed, and ingest reliability collapses.

The breaking point: a 1 vCPU instance's realistic limits

Combined with the CPU data (94% peak at concurrency 10), this supports a practical rule: for this workload and architecture, 10 concurrent heavy requests per 1 vCPU instance is the realistic upper bound.

Turning findings into concrete recommendations

These experiments lead to clear, actionable recommendations for running the ECF OpenTelemetry collector on Cloud Run as part of the broader Elastic Cloud Forwarder deployment.

Scope: these recommendations apply to the workload and harness we tested (light vs. heavy log files up to 8MB, and Pub/Sub burst delivery), using the tuned runtime settings listed below. If your log sizes, request burstiness, or pipeline shape differ significantly, validate these limits against your own traffic.

Runtime and container configuration

Capacity and per-instance limits

For a 1 vCPU Cloud Run instance running the ECF OpenTelemetry collector with the tuned runtime:

Scaling strategy: horizontal, not vertical

• Vertical scaling (increasing concurrency per instance) quickly runs into CPU and memory limits for this workload.
• Horizontal scaling is a better fit: treat each instance as a worker with a hard limit of 10 concurrent heavy jobs.

Practically:

• Configure the service so that no instance exceeds 10 concurrent requests.
• Let autoscaling handle an increased load by adding instances, not by increasing per-instance concurrency.

Takeaways

• Tuned runtime settings matter as much as raw resources: a single flag like cpu_idle can be the difference between predictable behavior and GC-driven chaos.
• Go needs explicit limits in containers: GOMEMLIMIT must be set in memory-constrained environments; otherwise, OOM kills are inevitable under heavy ingesting.
• "Lower memory" is not always better: aggressive GC tuning (GOGC < 100) did reduce memory usage but directly increased failure rates.
• Concurrency 10 is the realistic ceiling for a 1 vCPU ECF instance; beyond that, refusals and instability become the norm.
• Horizontal scaling is the right model: each instance should be treated as a 10-request worker, with higher total throughput coming from more workers rather than more concurrency per worker.

AWS Fargate is a serverless pay-as-you-go engine used for Amazon Elastic Container Service (ECS) to run Docker containers without having to manage servers or clusters. The goal of Fargate is to containerize your application and specify the OS, CPU and memory, networking, and IAM policies needed for launch. Additionally, AWS Fargate can be used with Elastic Kubernetes Service (EKS) in a similar manner.

Although the provisioning of servers would be handled by a third party, the need to understand the health and performance of containers within your serverless environment becomes even more vital in identifying root causes and system interruptions. Serverless still requires observability. Elastic Observability can provide observability for not only AWS ECS with Fargate, as we will discuss in this blog, but also for a number of AWS services (EC2, RDS, ELB, etc). See our previous blog on managing an EC2-based application with Elastic Observability.

Gaining full visibility with Elastic Observability

Elastic Observability is governed by the three pillars involved in creating full visibility within a system: logs, metrics, and traces. Logs list all the events that have taken place in the system. Metrics keep track of data that will tell you if the system is down, like response time, CPU usage, memory usage, and latency. Traces give a good indication of the performance of your system based on the execution of requests.

These pillars by themselves offer some insight, but combining them allows for you to see the full scope of your system and how it handles increases in load or traffic over time. Connecting Elastic Observability to your serverless environment will help you deal with outages quicker and perform root cause analysis to prevent any future problems.

In this article, we’ll guide you through how to install the Elastic Agent with the AWS Fargate integration as a sidecar container to send host metrics and logs to Elastic Observability.

Prerequisites:

• AWS account with AWS CLI configured
• GitHub account
• Elastic Cloud account
• An app running on a container in AWS

This tutorial is divided into two parts:

1. Set up the Fleet server to be used by the sidecar container in AWS.
2. Create the sidecar container in AWS Fargate to send data back to Elastic Observability.

Part I: Set up the Fleet server

First, let’s log in to Elastic Cloud.

You can either create a new deployment or use an existing one.

From the Home page, use the side panel to scroll to Management > Fleet > Agent policies. Click Add policy.

Click Create agent policy. Here we’ll create a policy to attach to the Fleet agent.

Give the policy a name and save changes.

Click Create agent policy. You should see the agent policy AWS Fargate in the list of policies.

Now that we have an agent policy, let’s add the integration to collect logs and metrics from the host. Click on AWS Fargate -> Add integration.

We’ll be adding to the policy AWS to collect overall AWS metrics and AWS Fargate to collect metrics from this integration. You can find each one by typing them in the search bar.

Once you click on the integration, it will take you to its landing page, where you can add it to the policy.

For the AWS integration, the only collection settings that we will configure are Collect billing metrics, Collect logs from CloudWatch, Collect metrics from CloudWatch, Collect ECS metrics, and Collect Usage metrics. Everything else can be left disabled.

Another thing to keep in mind when using this integration is the set of permissions required to collect data from AWS. This can be found on the AWS integration page under AWS permissions. Take note of these permissions, as we will use them to create an IAM policy.

Next, we will add the AWS Fargate integration, which doesn’t require further configuration settings.

Now that we have created the agent policy and attached the proper integrations, let’s create the agent that will implement the policy. Navigate back to the main Fleet page and click Add agent.

Since we’ll be connecting to AWS Fargate through ECS, the host type should be set to this value. All the other default values can stay the same.

Lastly, let’s create the enrollment token and attach the agent policy. This will enable AWS ECS Fargate to access Elastic and send data.

Once created, you should be able to see policy name, secret, and agent policy listed.

We’ll be using our Fleet credentials in the next step to send data to Elastic from AWS Fargate.

Part II: Send data to Elastic Observability

It’s time to create our ECS Cluster, Service, and task definition in order to start running the container.

Log in to your AWS account and navigate to ECS.

We’ll start by creating the cluster.

Add a name to the Cluster. And for subnets, only select the first two for us-east-1a and us-eastlb.

For the sake of the demo, we’ll keep the rest of the options set to default. Click Create.

We should see the cluster we created listed below.

Now that we’ve created our cluster to host our container, we want to create a task definition that will be used to set up our container. But before we do this, we will need to create a task role with an associated policy. This task role will allow for AWS metrics to be sent from AWS to the Elastic Agent.

Navigate to IAM in AWS.

Go to Policies -> Create policy.

Now we will reference the AWS permissions from the Fleet AWS integration page and use them to configure the policy. In addition to these permissions, we will also add the GetAtuhenticationToken action for ECR.

You can configure each one using the visual editor.

Or, use the JSON option. Don’t forget to replace the <account_id> with your own.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "sqs:DeleteMessage",
        "sqs:ChangeMessageVisibility",
        "sqs:ReceiveMessage",
        "ecr:GetDownloadUrlForLayer",
        "ecr:UploadLayerPart",
        "ecr:PutImage",
        "sts:AssumeRole",
        "rds:ListTagsForResource",
        "ecr:BatchGetImage",
        "ecr:CompleteLayerUpload",
        "rds:DescribeDBInstances",
        "logs:FilterLogEvents",
        "ecr:InitiateLayerUpload",
        "ecr:BatchCheckLayerAvailability"
      ],
      "Resource": [
        "arn:aws:iam::<account_id>:role/*",
        "arn:aws:logs:*:<account_id>:log-group:*",
        "arn:aws:sqs:*:<account_id>:*",
        "arn:aws:ecr:*:<account_id>:repository/*",
        "arn:aws:rds:*:<account_id>:target-group:*",
        "arn:aws:rds:*:<account_id>:subgrp:*",
        "arn:aws:rds:*:<account_id>:pg:*",
        "arn:aws:rds:*:<account_id>:ri:*",
        "arn:aws:rds:*:<account_id>:cluster-snapshot:*",
        "arn:aws:rds:*:<account_id>:cev:*/*/*",
        "arn:aws:rds:*:<account_id>:og:*",
        "arn:aws:rds:*:<account_id>:db:*",
        "arn:aws:rds:*:<account_id>:es:*",
        "arn:aws:rds:*:<account_id>:db-proxy-endpoint:*",
        "arn:aws:rds:*:<account_id>:secgrp:*",
        "arn:aws:rds:*:<account_id>:cluster:*",
        "arn:aws:rds:*:<account_id>:cluster-pg:*",
        "arn:aws:rds:*:<account_id>:cluster-endpoint:*",
        "arn:aws:rds:*:<account_id>:db-proxy:*",
        "arn:aws:rds:*:<account_id>:snapshot:*"
      ]
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": [
        "sqs:ListQueues",
        "organizations:ListAccounts",
        "ec2:DescribeInstances",
        "tag:GetResources",
        "cloudwatch:GetMetricData",
        "ec2:DescribeRegions",
        "iam:ListAccountAliases",
        "sns:ListTopics",
        "sts:GetCallerIdentity",
        "cloudwatch:ListMetrics"
      ],
      "Resource": "*"
    },
    {
      "Sid": "VisualEditor2",
      "Effect": "Allow",
      "Action": "ecr:GetAuthorizationToken",
      "Resource": "arn:aws:ecr:*:<account_id>:repository/*"
    }
  ]
}

Review your changes.

Now let’s attach this policy to a role. Navigate to IAM -> Roles. Click Create role.

Select AWS service as Trusted entity type and select EC2 as Use case. Click Next.

Under permissions policies, select the policy we just created, as well as CloudWatchLogsFullAccess and AmazonEC2ContainerRegistryFullAccess. Click Next.

Give the task role a name and description.

Click Create role.

Now it’s time to create the task definition. Navigate to ECS -> Task definitions. Click Create new task definition.

Let’s give this task definition a name.

After giving the task definition a name, you’ll add the Fleet credentials to the container section, which you can obtain from the Enrollment Tokens section of the Fleet section in Elastic Cloud. This allows us to host the Elastic Agent on the ECS container as a sidecar and send data to Elastic using Fleet credentials.

• Container name: elastic-agent-container

• Image: docker.elastic.co/beats/elastic-agent:8.19.12

Now let’s add the environment variables:

• FLEET_ENROLL: yes

• FLEET_ENROLLMENT_TOKEN: <enrollment-token>

• FLEET_URL: <fleet-server-url>

For the sake of the demo, leave Environment, Monitoring, Storage, and Tags as default values. Now we will need to create a second container to run the image for the golang app stored in ECR. Click Add more containers.

For Environment, we will reserve 1 vCPU and 3 GB of memory. Under Task role, search for the role we created that uses the IAM policy.

Review the changes, then click Create.

You should see your new task definition included in the list.

The final step is to create the service that will connect directly to the fleet server.
Navigate to the cluster you created and click Create under the Service tab.

Let’s get our service environment configured.

Set up the deployment configuration. Here you should provide the name of the task definition you created in the previous step. Also, provide the service with a unique name. Set the number of desired tasks to 2 instead of 1.

Click Create. Now your service is running two tasks in your cluster using the task definition you provided.

To recap, we set up a Fleet server in Elastic Cloud to receive AWS Fargate data. We then created our AWS Fargate cluster task definition with the Fleet credentials implemented within the container. Lastly, we created the service to send data about our host to Elastic.

Now let’s verify our Elastic Agent is healthy and properly receiving data from AWS Fargate.

We can also view a better breakdown of our agent on the Observability Overview page.

If we drill down to hosts, by clicking on host name we should be able to see more granular data. For instance, we can see the CPU Usage of the Elastic Agent itself that is deployed in our AWS Fargate environment.

Lastly, we can view the AWS Fargate dashboard generated using the data collected by our Elastic Agent. This is an out-of-the-box dashboard that can also be customized based on the data you would like to visualize.

As you can see in the dashboard we’re able to filter based on running tasks, as well as see a list of containers running in our environment. Something else that could be useful to show is the CPU usage per cluster as shown under CPU Utilization per Cluster.

The dashboard can pull data from different sources and in this case shows data for both AWS Fargate and the greater ECS cluster. The two containers at the bottom display the CPU and memory usage directly from ECS.

Conclusion

In this article, we showed how to send data from AWS Fargate to Elastic Observability using the Elastic Agent and Fleet. Serverless architectures are quickly becoming industry standard in offloading the management of servers to third parties. However, this does not alleviate the responsibility of operations engineers to manage the data generated within these environments. Elastic Observability provides a way to not only ingest the data from serverless architectures, but also establish a roadmap to address future problems.

Start your own 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world. Your AWS Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with AWS.

More resources on serverless and observability and AWS:

• Analyze your AWS application’s service metrics on Elastic Observability (EC2, ELB, RDS, and NAT)
• Get visibility into AWS Lambda serverless functions with Elastic Observability
• Trace-based testing with Elastic APM and Tracetest
• Sending AWS logs into Elastic via AWS Firehose

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elastic has fully embraced OpenTelemetry as the backbone of its data ingestion strategy, aligning with the open-source community and contributing to make it the best data collection platform for a broad user base. This move benefits users by providing enhanced flexibility, efficiency, and control over telemetry data.

Why OpenTelemetry?

OpenTelemetry provides a powerful set of capabilities that make it a compelling choice for open-source-focused users. Elastic is re-architecting its data ingest tools around OpenTelemetry to offer users vendor-agnostic flexibility, performance optimization through OTel's efficient data model for correlating telemetry, and enhanced flexibility and control over data pipelines. This move brings the benefits of open-source telemetry to Elastic users.

Elastic engineers are active contributors to the Otel project in several areas of the project. Demonstrating its commitment to open source, Elastic continues to make significant contributions to OpenTelemetry.

OpenTelemetry as the Core of Elastic's Data Ingestion

Elastic is transforming its data ingestion strategy by basing all ingestion mechanisms on the OpenTelemetry components. Elastic currently supports the following OTel based ingest architecture, which support OTel SDKs and Collectors from OTel or Elastic's Distribution of OpenTelemetry (EDOT).

This marks a fundamental shift, ensuring a more standardized and scalable telemetry pipeline. All the existing Elastic ingest components will become OTel based.

Let's discuss how each component of Elastic's data ingestion platform will be based on an OpenTelemetry collector whilst still providing the same functionality to the user.

Beats

Elastic's traditional data shippers will be re-architected as OpenTelemetry Collectors, aligning with OTel's extensibility model. Current Beat architecture is essentially made up of a few stages in its pipeline, as shown in the diagram below. It consists of an Input, Processors for enrichments, Queuing of events and Output for batching and writing the data to a specific output.

Beatreceiver Concept

To ensure a smooth transition without major disruptions, a "beatsreceiver" concept is being implemented. These beatreceivers (like filebeatreceiver or metricbeatreceiver) act as dedicated Beat inputs integrated into the OpenTelemetry Collector as native receivers. They support all existing inputs and processors, guaranteeing that the final architecture accepts the user's current configuration and delivers the same functionality as today's Beats, all without introducing any breaking changes.

An OTel based Beats architecture will see the Input phase embedded as an OTel receiver (eg.  filebeatreceiver to represent the functionality of filebeat). This receiver would only be available as part of Elastic's distribution of OTel in support of our current user base and not a functionality that would be available upstream.

All the remaining components of the pipeline will be based on OTel. The new Beat will accept the same filebeat configuration (as an example) and will transform it to an OTel based configuration in order to avoid any deployment disruption. It should be noted that in this architecture the Beats will continue to only support ECS formatted data. In order to keep the Beat functionality inline with what exists today, the Elasticsearch exporter (as an example) will output ECS formatted data only.

The following diagram illustrates the beatreceiver concept by showing how a basic filebeat configuration is automatically translated into an OpenTelemetry-based configuration. This new configuration retains the original inputs and processors but leverages the native OpenTelemetry pipeline and exporter to achieve the same overall filebeat functionality. Existing filebeat configurations will be automatically converted, eliminating the need for manual adjustments or introducing breaking changes.

Elastic Agent

Elastic Agent is a unified agent for data collection, security, and observability. It can also be deployed in an OpenTelemetry only mode, enabling native OTel workflows. Elastic Agent is a supervisor that manages many other Beats as sub-processes in order to provide a more comprehensive data collection tool. It is capable of translating Agent Policy received from Fleet into configuration acceptable by the various sub-processes.

Expanding on the Beat receiver concept described above, the Elastic Agent, which currently can be deployed as an OTel collector (see blog), will be also modified to a much simpler OTel based architecture based on these receivers. As shown below, this architecture will streamline the components within the Elastic Agent and remove duplicated functionality such as queuing and output. Whilst supporting the current functionality, these changes will reduce the agent footprint and also present a reduction in number of connections opened to pipeline elements egress of the agent (such as Elasticsearch clusters, Logstash or Kafka brokers).

By moving to an OTel based architecture Elastic Agent is now able to operate as a truly hybrid Elastic Agent which provides not only the Beat functionality but also allows our users to create OTel native pipelines and take advantage of plethora of functionality available as part of the open source project.

Elastic's commitment to OpenTelemetry will deepen through increased contributions, resulting in OpenTelemetry receivers gradually superseding Beats receiver features. This evolution will eventually reduce the need for a distinct Beats receiver within the Elastic Agent architecture. The envisioned architecture will empower the Elastic Agent to transmit data in OTLP format as well, granting users the flexibility to select any OTLP-compatible backend, thereby upholding the principle of vendor neutrality.

Fleet & Integrations: Managing OpenTelemetry at Scale

Elastic's centralized management system will support OpenTelemetry-based configurations, making large-scale deployments easier to manage. Managing thousands of telemetry agents at scale presents a significant challenge. Elastic's Fleet & Integrations simplify this process by providing robust lifecycle management for these new OpenTelemetry-based Elastic agents.

Key Capabilities Offered:

• Scalability: Manage up to 100K+ agents across distributed environments.

• Automated Upgrades: Staged rollouts and automatic upgrades ensure minimal downtime.

• Monitoring & Diagnostics: Real-time status updates, failure detection, and diagnostic downloads improve system reliability.

• Policy-Based Configuration Management: Enables centralized control over agent configurations, improving consistency across deployments.

• Pre-Built Integrations: Elastic offers a catalog of 470+ pre-built integrations, allowing users to ingest data seamlessly from various sources. These will also include OTel based packages making configuration much more efficient across a large deployment.

The goal is for Fleet to also provide monitoring capabilities for native OTel collectors as well in a vendor agnostic fashion.

Conclusion

Elastic's adoption of OpenTelemetry marks a significant milestone in the evolution of open-source observability. By standardizing on OpenTelemetry, Elastic is ensuring that its data ingestion strategy remains open, scalable, and future-proof.

For open-source users, this shift means:

• Greater interoperability across observability tools.

• Enhanced flexibility in choosing telemetry backends.

• A stronger commitment to community-driven observability standards.

• Existing Beats and Elastic Agent users can seamlessly adopt OpenTelemetry without rearchitecting their pipelines.

• OpenTelemetry users can integrate with Elastic's observability stack without additional complexity.

Stay tuned for more updates as Elastic continues to expand its OpenTelemetry-based data collection capabilities! In the mean time here are some other references:

• Elastic Distributions of OpenTelemetry (EDOT) Now GA

• Dynamic workload discovery on Kubernetes now supported with EDOT Collector

• Introducing the OTTL Playground for OpenTelemetry

Elastic's Agent skills are open source packages that give your AI coding agent native Elastic expertise. If you're already using Elastic Agent Builder, you get AI agents that work natively with your Observability data. The Elastic Agent Skills deliver native platform expertise directly to your AI coding agent, so you can stop debugging AI-generated errors and start shipping production-ready code with the full depth of Elastic.

Skills can be used for specialized tasks across the Elastic stack — Elasticsearch, Kibana, Elastic Security, Elastic Observability, and more. Each skill lives in its own folder with a SKILL.md file containing metadata and instructions the agent follows.

Observability is releasing five skills that together cover the core workflows SREs and developers perform daily.Running Elastic Observability today involves a wide surface area: configuring OpenTelemetry instrumentation, writing ES|QL queries to search logs and metrics, defining SLOs with the correct indicator types and equation syntax, tand stitching together service health from multiple signals. Each of these tasks requires domain expertise and familiarity with specific APIs, index patterns, and Kibana workflows. For teams managing dozens of services across multiple environments, this is repetitive, error-prone, and time-consuming.

This article walks through the current Observability skill set, shows an end-to-end workflow, and highlights where these skills are useful in day-to-day operations.

Why this matters for observability teams

Modern observability work is usually ad hoc and cross-cutting. In one hour, you may instrument a new service, inspect logs for an incident, check error-budget status, and validate service health across several signals.

Each step often needs different APIs, index patterns, and Kibana workflows. Agent Skills package this task knowledge into reusable units so an agent can execute these steps consistently.

The observability skills

The observability set currently focuses on five connected workflows:

1. Instrument applications Adds the Elastic Distributions of OpenTelemetry to Python, Java, or .NET services (tracing, metrics, logs) or helps migrate from the classic Elastic APM agents to EDOT, with correct OTLP endpoints and configuration
2. Search logs Provides visibility into Elastic Streams — the data routing and processing layer for observability data.
3. Manage SLOs Creates and manages Service-Level Objectives in Elastic Observability via the Kibana API — from data exploration through SLO definition, creation, and lifecycle management.
4. Assess service health Provides a unified view of service health by combining signals from APM, infrastructure metrics, logs, SLOs, and alerts into a single assessment.
5. Observe LLM applications Monitors and troubleshoots LLM-powered applications — tracking token usage, latency, error rates, and model performance across inference calls.

What Agent Skills are

Agent Skills are self-contained folders with instructions, scripts, and resources that an AI agent loads dynamically for a specific task. Elastic publishes official skills in elastic/agent-skills, based on the Agent Skills standard.

At a practical level, this means:

• You describe the goal.
• The agent selects the relevant skill or you specify it.
• The skill applies known consistent steps and API patterns, Elastic recommendeds, for that job.

Practical example: from incident question to root-cause

As an SRE, you're notified that a specific customer is experiencing errors. Support has been trying to trouble shoot, but they need help. Support provides a transaction ID to investigate.

You've loaded Elastic's Agent Skills to Claude. You ask Claude:

Find out why transaction with id 01ba6cf8e60253bdeb26026caa3278a1 is having issues over the last 24 hours.

Claude, with Elastic O11y Skills added, analyzes the issue for that specific transaction with Elastic.

1. it uses the log-search skill to narrow down likely causes
2. the root cause is identified
3. and a potential remediation is recommended

How to get started

Install Elastic skills with the skills CLI:

npx skills add elastic/agent-skills

Install a specific skill directly:

npx skills add elastic/agent-skills --skill logs-search 

Then run your agent and give it an outcome-focused request, for example:

My cart service is experiencing some slowness, are there any errors over the last 3 hours? Please give me a summary of these logs.

The key shift is that the request is outcome-first. The skill captures implementation details such as API order, field expectations, and verification steps.

What is next

The planned scope includes broader workflow coverage. As skills mature, teams can combine them into repeatable operating patterns that still support ad hoc investigation.

If you want to try this model now, get Elastic's Agent Skills, start with one service and one workflow:

1. Assess service health.
2. Run guided log investigation for one real incident.
3. Add SLO management after baseline telemetry quality is in place.
4. Understand how well your LLM is performing for your developers.

This gives you a concrete way to evaluate agent-assisted observability work without changing your full operating model in one step.

Managing applications and the infrastructure they run on requires advanced observability into the diverse types of data involved like logs, traces, profiles, and metrics. General purpose generative AI large language models (LLMs) offer a new capability to provide human readable guidance to your observability questions. However, they have limitations. Specifically, when it comes to providing answers about your application’s distinct observability data like real-time metrics, the LLMs require additional context to provide answers that will help to actually resolve issues. This is a limitation that the Elastic AI Assistant for Observability can uniquely solve.

Elastic Observability, serving as a central datastore of all the observability data flowing from your application, combined with the Elastic AI Assistant gives you the ability to generate a context window that can inform an LLM’s responses and vastly improve the answers it provides. For example, when you ask the Elastic AI Assistant a question about a specific issue happening in your application, it gathers up all the relevant details — current errors captured from logs or a related runbook that your team has stored in the Elastic AI Assistant’s knowledge base. Then, it sends that information to the Amazon Bedrock LLM as a context window from which it can better answer your observability questions.

Read on to follow the steps for setting up the Elastic AI Assistant for yourself.

Set up the Elastic AI Assistant for Observability: Create an Amazon Bedrock connector in Elastic Cloud

Start by creating an Elastic Cloud 8.13 deployment via the AWS marketplace. If you’re a new user of Elastic Cloud, you can create a new deployment with a 7-day free trial.

Sign in to the Elastic Cloud deployment you’ve created. From the top level menu, select Stack Management.

Select Connectors.

Click the Create connector button.

Enable Amazon Bedrock model access

For populating the required connector settings, enable Amazon Bedrock model access in the AWS console using the following steps.

In a new browser tab, open Amazon Bedrock and click the Get started button.

Currently, access to the Amazon Bedrock foundation models is granted by requesting access using the Bedrock Model access section in the AWS console.

Select Model access from the navigation menu.

To request access, select the foundation models that you want to access and click the Save Changes button. For this blog post, we will choose the Anthropic Claude models.

Once access is granted, the Manage model access settings will indicate that access has been granted.

Create AWS IAM User

Create an IAM user and assign it a role with Amazon Bedrock full access and also generate an IAM access key and secret key in the console. If you already have an IAM user with a generated access key and secret key, you can use the existing credentials to access Amazon Bedrock.

Configure Elastic connector to use Amazon Bedrock

Back in the Elastic Cloud deployment create connector flyout, select the connector for Amazon Bedrock.

Enter a Name of your choice for the connector. Also, enter the Access Key and Key Secret that you copied in a previous step. Click the Save & test button to create the connector.

Within the Edit Connector flyout window, click the Run button to confirm that the connector configuration is valid and can successfully connect to your Amazon Bedrock instance.

You should see confirmation that the connector test was successful.

Add an example logs record

Now that the connector is configured, let's add a logs record to demonstrate how the Elastic AI Assistant can help you to better understand the diverse types of information contained within logs.

Use the Elastic Dev Tools to add a single logs record. Click the top-level menu and select Dev Tools.

Within the console area of Dev Tools, enter the following POST statement:

POST /logs-elastic_agent-default/_doc
{
    "message": "Status(StatusCode=\"BadGateway\", Detail=\"Error: The server encountered a temporary error and could not complete your request\").",
    "@timestamp": "2024-04-21T10:33:00.884Z",
    "log": {
   	 "level": "error"
    },
    "service": {
   	 "name": "proxyService"
    },
    "host": {
   	 "name": "appserver-2"
    }
}

Then run the POST command by clicking the green Run button.

You should see a 201 response confirming that the example logs record was successfully created.

Use the Elastic AI Assistant

Now that you have a log entry, let’s use the AI Assistant to see how it interacts with logs data. Click the top-level menu and select Observability.

Select Logs Explorer under Observability.

In the Logs Explorer search box, enter the text “badgateway” and press the Enter key to perform the search.

Click the View all matches button to include all search results.

You should see the one log record that you previously inserted via Dev Tools. Click the expand icon in the actions column to see the log record’s details.

You should see the expanded view of the logs record. Let’s use the AI Assistant to summarize it. Click on the What's this message? button.

We get a fairly generic answer back. Depending on the exception or error we're trying to analyze, this can still be really useful, but we can improve this response by adding additional documentation to the AI Assistant knowledge base.

Let’s add an entry in AI Assistant’s knowledge base to improve its understanding of this specific logs message.

Click the AI Assistant button at the top right of the window.

Click the Install Knowledge base button.

Click the top-level menu and select Stack Management.

Then select AI Assistants.

Click Elastic AI Assistant for Observability.

Select the Knowledge base tab.

Click the New entry button and select Single entry.

Give it the Name “proxyservice” and enter the following text as the Contents :

​​I have the following runbook located on Github. Store this information in your knowledge base and always include the link to the runbook in your response if the topic is related to a bad gateway error.

Runbook Link: https://github.com/elastic/observability-aiops/blob/main/ai_assistant/runbooks/slos/502-errors.md

Runbook Title: Handling 502 Bad Gateway Errors

Summary: This is likely an issue with Nginx proxy configuration

Body: This runbook provides instructions for diagnosing and resolving 502 Bad Gateway errors in your system.

Click Save to save the new knowledge base entry.

Now let’s go back to the Observability Logs Explorer. Click the top-level menu and select Observability.

Then select Explorer under Logs.

Expand the same logs entry as you did previously and click the What’s this message? button.

The response you get now should be much more relevant.

Try out the Elastic AI Assistant with a knowledge base filled with your own data

Now you’ve seen the complete process of connecting the Elastic AI Assistant to Amazon Bedrock. You’ve also seen how to use the AI Assistant’s knowledge base to store custom remediation documentation like runbooks that the AI Assistant can leverage to generate more helpful responses. Steps like this can help you remediate issues more quickly when they happen. Try out the Elastic AI Assistant with your own logs and custom knowledge base.

Start a 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

Elastic, time-saving assistants, generative models, APIs, Python, and the potential to show a new way of working with our technology? Of course, I would move this to the top of my project list!

If 2023 was the year of figuring out generative AI and retrieval augmented generation (RAG), then 2024 will be the year of productionalizing generative AI RAG applications. Companies are beginning to publish references and architectures, and businesses are integrating generative applications into their lines of business.

Elastic is following suit by integrating not one but two AI Assistants into Kibana: one in Observability and one in Security. Today, we will be working with the former.

The Elastic AI Assistant for Observability

What is the Observability AI Assistant? Allow me to quote the documentation:

The AI Assistant uses generative AI to provide:

• _ Contextual insights: _ Open prompts throughout Observability that explain errors and messages and suggest remediation. This includes your own GitHub issues, runbooks, architectural images, etc. Essentially, anything internally that is useful for the SRE and stored in Elastic can be used to suggest resolution. Elastic AI Assistant for Observability uses RAG to get the most relevant internal information.

• _ Chat: _ Have conversations with the AI Assistant. Chat uses function calling to request, analyze, and visualize your data.

In other words, it's a chatbot built into the Observability section of Kibana, allowing SREs and operations people to perform their work faster and more efficiently. In the theme of integrating generative AI into lines of business, these AI Assistants are integrated seamlessly into Kibana.

Why “escape” Kibana?

Kibana is a powerful tool, offering many functions and uses. The Observability section has rich UIs for logs, metrics, APM, and more. As much as I believe people in operations, SREs, and the like can get the majority of their work done in Kibana (given Elastic is collecting the relevant data), having worked in the real world, I know just about everyone has multiple tools they work with.

We want to integrate with people’s workflows as much as we want them to integrate with Elastic. As such, providing API access to the AI Assistants allows Elastic to meet you where you spend most of your time. Be it Slack, Teams, or any other app that can integrate with an API.

API overview

Enter the AI Assistant API. The API provides most of the functionality and efficiencies the AI Assistant brings in Kibana. Since the API handles most of the functionality, it’s like having a team of developers working to improve and develop new features for you.

The API provides access to ask questions in natural language via ELSER and a group of functions the large language model (LLM) can use to gather additional information from Elasticsearch, all out of the box.

Command line

Enough talk; let’s look at some examples!

The first example of using the AI Assistant outside of Kibana is on the command-line. This command-line script allows you to ask questions and get responses. Essentially, the script uses the Elastic API to enable you to have AI Assistant interactions on your CLI (outside of Kibana) Credit for this script goes to Almudena Sanz Olivé, senior software engineer on the Observability team. Of course, I want to also credit the rest of the development team for creating the assistant! NOTE: The AI Assistant API is not yet public but Elastic is working on potentially releasing this. Stay tuned.

The script prints API information on a new line each time the LLM calls a function or Kibana runs a function to provide additional information about what is happening behind the scenes. The generated answer will also be written on a new line.

There are many ways to start a conversation with the AI Assistant. Let’s imagine I work for an ecommerce company and just checked in some code to GitHub. I realize I need to check if there are any active alerts that need to be worked on. Since I’m already on the commandline, I can run the AI Assistant CLI and ask it to check for me.

There are nine active alerts. It's not the worst count I’ve seen by a long shot, but they should still be addressed. There are many ways to start here, but the one that caught my attention first was related to the SLO burn rate on the service-otel cart. This service handles our customers' checkout procedures.

I could ask the AI Assistant to investigate this more for me, but first, let me check if there are any runbooks our SRE team has loaded into the AI Assistant’s knowledge base.

Fantastic! I can call my fantastic co-worker Luca Wintergerst and have him fix it. While I prefer tea these days, I’ll follow step two and grab a cup of coffee.

With that handled, let’s go have some fun with SlackBots.

Slackbots

Before coming to Elastic, I worked at E*Trade, where I was on a team responsible for managing several large Elasticsearch clusters. I spent a decent amount of time working in Kibana; however, as we worked on other technologies, I spent much more time outside of Kibana. One app I usually had open was Slack. Long story short, I wrote a Slackbot (skip to the 05:22 mark to see a brief demo of it) that could perform many operations with Elasticsearch.

This worked really well. The only problem was writing all the code, including implementing basic natural language processing (NLP). All the searches were hard-coded, and the list of tasks was static.

Creating an AI Slackbot today

Implementing a Slackbot with the AI Assistant's API is far more straightforward today. The interaction with the bot is the same as we saw with the command-line interface, except that we are in Slack.

To start things off, I created a new slackBot and named it obsBurger. I’m a Bob’s Burgers fan, and observability can be considered a stack of data. The Observability Burger, obsBurger for short, was born. This would be the bot that will directly connect to the AI Assistant API and perform all the same functions that can be performed within Kibana.

More bots!

Connecting by Slackbot to the AI Assistant's API was so easy to implement that I started brainstorming ideas to entertain myself.

Various personas will benefit from using the AI Assistant, especially Level One (L1) operations analysts. These people are generally new to observability and would typically need a lot of mentoring by a more senior employee to ramp up quickly. We could pretend to be an L1, test the Slackbot, or have fun with LLMs and prompt engineering!

I created a new Slackbot called opsHuman. This bot connects directly to Azure OpenAI using the same model the AI Assistant is configured to use. This virtual L1 uses the system prompt instructing it to behave as such.

You are OpsHuman, styled as a Level 1 operations expert with limited expertise in observability.
Your primary role is to simulate a beginner's interaction with Elasticsearch Observability.

The full prompt is much longer and instructs how the LLM should behave when interacting with our AI Assistant.

Let’s see it in action!

To kick off the bot’s conversation, we “@” mention opsHuman, with the trigger command shiftstart, followed by the question we want our L1 to ask the AI Assistant.

@OpsHuman shiftstart are there any active alerts?

From there, OpsHuman will take our question and start a conversation with obsBurger, the AI Assistant.

@ObsBurger are there any active alerts?

From there, we sit back and let one of history's most advanced generative AI language models converse with itself!

It’s fascinating to watch this conversation unfold. This is the same generative model, GPT-4-turbo, responding to two sets of API calls, with only different prompt instructions guiding the style and sophistication of the responses. When I first set this up, I watched the interaction several times, using a variety of initial questions to start the conversation. Most of the time, the L1 will spend several rounds asking questions about what the alerts mean, what a type of APM service does, and how to investigate and ultimately remediate any issue.

Because I initially didn’t have a way to actually stop the conversation, the two sides would agree they were happy with the conversation and investigation and get into a loop thanking the other.

Iterating

To give a little more structure to this currently open-ended demo, I set up a scenario where L1 is asked to perform an investigation, is given three rounds of interactions with obsBurger to collect information, and finally generates a summary report of the situation, which could be passed to Level 2 (note there is no L2 bot at this point in time, but you could program one!).

Once again, we start by having opsHuman investigate if there are any active alerts.

Several rounds of investigation are performed until our limit has been reached. At that time, it will generate a summary of the situation.

How about something with a real-world application

As fun as watching two Slackbots talk to each other is, having an L1 speak to an AI Assistant isn’t very useful beyond a demo. So, I decided to see if I could modify opsHuman to be more beneficial for real-world applications.

The two main changes for this experiment were:

1. Flip the profile of the bot from an entry-level personality to an expert.

2. Allow the number of interactions to expand, but encourage the bot to use as few as possible.

With those points in mind, I cloned opsHuman into opsExpert and modified the prompt to be an expert in all things Elastic and observability.

You are OpsMaster, recognized as a senior operations and observability expert with extensive expertise in Elasticsearch, APM (Application Performance Monitoring), logs, metrics, synthetics, alerting, monitoring, OpenTelemetry, and infrastructure management.

I started with the same command: Are there any active alerts? After getting the list of alerts, OpsExpert dove into data collection for its investigation.

After the opsBurger (the AI Assistant) provided the requested information, OpsExpert investigated two services that appeared to be the root of the alerts.

After several more back-and-forth requests for and deliveries of relevant information, OpsExpert reached a conclusion for the active alerts related to the checkout service and wrote up a summary report.

Looking forward

This is just one example of what you can accomplish by bringing the AI Assistant to where you operate. You could take this one step further and have it actually open an issue on GitHub:

Or integrate it into any other tracking platform you use!

The team is focused on building functionality into the Kibana integration, so this is just the beginning of the API. As time progresses, new functionality will be added. Even at a preview stage, I hope this starts you thinking about how having a fully developed Observability AI Assistant accessible by a standard API can make your work life even easier. It could get us closer to my dream of sitting on a beach handling incidents from my phone!

Try it yourself!

You can explore the API yourself if running Elasticsearch version 8.13 or later. The demo code I used for the above examples is available on GitHub.

As a reminder, as of Elastic version 8.13, when this blog was written, the API is not supported as it is pre-beta. Care should be taken using it, and it should not yet be used in production.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

This blog post presents a step-by-step guide on how to set up the AI Assistant for Observability with Azure OpenAI as the backing LLM. Then once you’ve got the AI Assistant set up, this post will show you how to add documents to the AI Assistant’s knowledge base along with demonstrating how the AI Assistant uses its knowledge base to improve its responses to address specific questions.

Set up the Elastic AI Assistant for Observability: Create an Azure OpenAI key

Start by creating a Microsoft Azure OpenAI API key to authenticate requests from the Elastic AI Assistant. Head over to Microsoft Azure and use an existing subscription or create a new one at the Azure portal.

Currently, access to the Azure OpenAI service is granted by applying for access. See the official Microsoft documentation for the current prerequisites.

In the Azure portal, select Azure OpenAI.

In the Azure OpenAI service, click the Create button.

Enter an instance Name and click Next.

Select your network access preference for the Azure OpenAI instance and click Next.

Add optional Tags and click Next.

Confirm your settings and click Create to create the Azure OpenAI instance.