[azure logs] enable event hub processor v2#12802

zmoog · 2025-02-17T11:43:15Z

Proposed commit message

The PR adds the new processor_version configuration option (an related ones) to enable the azure-eventhub processor v2 in the Azure Logs integration v2 preview.

We increase the min stack version to 8.15.1.
processor v2 is only available to for integration v2 preview (no integration v1 support)
Add storage_account_connection_string config option (required for processor v2); the policy template builds a default value using the storage_account_key but also offers an override option.

The processor v2 uses the modern Event Hub SDK.

Notes for reviewers

In this PR, I am adding the processor v2 settings to the advanced section of the integration v2 only (events data stream).

The goal is to enable only processor v2 for the data stream that can avoid contention among event hub partition consumers.

However, nothing can stop users from enabling v2 AND one or more v1 integrations. So, adding the processor v2 settings to the global scope is better for simplicity.

I would love to hear your thoughts about placing the processor v2 settings: data stream vs. global level.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

Update the integration docs

How to test this PR locally

Bootstrap a local stack using elastic-package (min stack version 8.15.1)
Build the Azure Logs package
Install the Azure Logs integration v2 preview
In the integration settings > advanced > set processor_version to v2

Related issues

Closes [Azure Logs] Enable azure-eventhub processor v2 #12800

Screenshots

elastic-vault-github-plugin-prod · 2025-02-17T11:59:59Z

🚀 Benchmarks report

Package `azure` 👍(8) 💚(2) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`activitylogs`	1626.02	1254.71	-371.31 (-22.84%)	💔

To see the full report comment with /test benchmark fullreport

elasticmachine · 2025-02-20T21:41:36Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

packages/azure/_dev/build/docs/events.md

packages/azure/changelog.yml

muthu-mps · 2025-02-21T06:15:57Z

packages/azure/data_stream/events/agent/stream/stream.yml.hbs

 storage_account_key: {{storage_account_key}}
 {{/if}}
+{{#if storage_account_connection_string}}
+ : {{storage_account_connection_string}}


Are we not processing the storage_account_connection_string through user input?
If this values doesn't come as input from the user, Then we could remove the IF condition. WDYT?

Make sense.

We can change the UI elements once when we'll add the Entra ID authentication options for the processor v2.

packages/azure/docs/events.md

muthu-mps · 2025-02-21T06:23:19Z

packages/azure/docs/events.md

+
+The input v2 is in preview. Input v1 is still the default and is recommended for typical use cases.
+
+See the "Settings" section for more details about enabling the input v2.


There are two different settings section, one which covers the details Event hub config and the other has Input v2 settings (advanced). Shouldn't this be input V2 setting?

I should remove the "(advanced)" from the title, and only leave "input/processor v2" only settings.

muthu-mps · 2025-02-21T06:26:08Z

packages/azure/docs/events.md

+_boolean_
+(v2 only) Flag to control if the input should perform the checkpoint information migration from v1 to v2 at startup. The checkpoint migration converts the checkpoint information from the v1 format to the v2 format.
+
+Default is `false`, which means the input will not perform the checkpoint migration.


The Default can also go as bullet point. In the document it looks as a separate section.

Please see below.

Good point!

zmoog · 2025-02-24T12:15:31Z

I'm also replacing "input" with "processor" in the PR description to simplify.

MichaelKatsoulis

LGTM

zmoog · 2025-02-27T10:50:22Z

@muthu-mps @efd6, I pushed a few changes to address the review comments. Could you take a new look at this PR, when you have time? 🙇

muthu-mps

LGTM!

zmoog · 2025-03-04T09:59:49Z

@efd6, could you take another look at the updates in this PR, when you have time? 🙇

efd6

LGTM

Made suggestions that I think improve clarity (except the <blink>), but the decision is obviously up to you.

packages/azure/_dev/build/docs/events.md

efd6 · 2025-03-04T21:53:35Z

packages/azure/_dev/build/docs/events.md

 The integration will automatically detect the log category and forward the logs to the appropriate data stream. When the integration v2 preview cannot find a matching data stream for a log category, it forwards the logs to the platform logs data stream.

-IMPORTANT: To use the v2 preview, you must turn off all the existing v1 integrations and turn on only the v2 preview integration.
+IMPORTANT: **To use the integration v2 preview, you must turn off all the existing v1 integrations and turn on only the v2 preview integration.**


Suggested change

IMPORTANT: **To use the integration v2 preview, you must turn off all the existing v1 integrations and turn on only the v2 preview integration.**

IMPORTANT: **<blink>To use the integration v2 preview, you must turn off all the existing v1 integrations and turn on only the v2 preview integration.</blink>**

🤣

Maybe it's a little too whimsical, but I appreciate it! 😄

packages/azure/_dev/build/docs/events.md

packages/azure/data_stream/events/manifest.yml

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

We offer no storage_account_connection_string setting option in the UI yet. We'll revise the UI once for the upcoming Entra ID auth settings.

Introduces the concept of processor as a configurable option available starting from stack 8.15.1 and integration 1.23.0. I removed 'input' because technically, we have one input, with a a configurable internal component called 'processor.' Both legacy and modern SDKs identify the 'processor' as the internal component responsible for processing messages fetched by event hub partition consumers.

I want to simplify the mental model for the user. From their perspective, there's the integration. Then the integration now has an optional 'processor' option with v1 and v2 variants.

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

zmoog · 2025-03-06T19:05:08Z

@efd6, it's ready for another lap!

elasticmachine · 2025-03-06T19:15:13Z

💚 Build Succeeded

Buildkite Build
Commit: b5426ee

History

💔 Build #23088 failed e99283f
💔 Build #23087 failed 99d821c
💚 Build #22812 succeeded 641cfdc
💚 Build #22629 succeeded 0cbf97f
💚 Build #22620 succeeded c965aa5
💔 Build #22613 failed 24cff86

cc @zmoog

elastic-sonarqube · 2025-03-06T19:15:13Z

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

elastic-vault-github-plugin-prod · 2025-03-10T21:56:17Z

Package azure - 1.23.0 containing this change is available at https://epr.elastic.co/package/azure/1.23.0/

* Add input v2 configuration options * Improve config option descriptions * Update docs and UI elements description * Clarify + simplify docs & UI descriptions * Apply suggestions from code review Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co> * Apply suggestions from code review Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co> * Remove storage_account_connection_string We offer no storage_account_connection_string setting option in the UI yet. We'll revise the UI once for the upcoming Entra ID auth settings. * Clarify input v2 only settings * Update rendered docs * Replace input with processor in the docs Introduces the concept of processor as a configurable option available starting from stack 8.15.1 and integration 1.23.0. I removed 'input' because technically, we have one input, with a a configurable internal component called 'processor.' Both legacy and modern SDKs identify the 'processor' as the internal component responsible for processing messages fetched by event hub partition consumers. * Minor alignment to UI descriptions * Replace all references to 'input' with 'processor' I want to simplify the mental model for the user. From their perspective, there's the integration. Then the integration now has an optional 'processor' option with v1 and v2 variants. * Apply suggestions from code review Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co> * Apply suggestions from code review Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co> * Updated rendered docs --------- Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

zmoog added enhancement New feature or request Integration:azure Azure Logs Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] labels Feb 17, 2025

zmoog self-assigned this Feb 17, 2025

zmoog linked an issue Feb 17, 2025 that may be closed by this pull request

[Azure Logs] Enable azure-eventhub processor v2 #12800

Closed

zmoog changed the title ~~[azure logs] Enable azure-eventhub input v2 configuration options~~ [azure logs] enable azure-eventhub input v2 Feb 17, 2025

zmoog marked this pull request as ready for review February 18, 2025 22:24

zmoog requested review from a team as code owners February 18, 2025 22:25

zmoog requested review from alaudazzi and muthu-mps February 18, 2025 22:25

MichaelKatsoulis self-requested a review February 19, 2025 12:45

zmoog added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Feb 20, 2025

efd6 reviewed Feb 20, 2025

View reviewed changes

muthu-mps reviewed Feb 21, 2025

View reviewed changes

zmoog changed the title ~~[azure logs] enable azure-eventhub input v2~~ [azure logs] enable azure-eventhub processor v2 Feb 24, 2025

MichaelKatsoulis approved these changes Feb 24, 2025

View reviewed changes

muthu-mps approved these changes Feb 27, 2025

View reviewed changes

efd6 reviewed Mar 4, 2025

View reviewed changes

zmoog and others added 5 commits March 6, 2025 19:44

Add input v2 configuration options

9a9b815

Improve config option descriptions

5fb7716

Update docs and UI elements description

60b8dab

Clarify + simplify docs & UI descriptions

0103142

Apply suggestions from code review

aaab69b

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

zmoog and others added 9 commits March 6, 2025 19:45

Apply suggestions from code review

7edc17a

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Remove storage_account_connection_string

2dee01d

We offer no storage_account_connection_string setting option in the UI yet. We'll revise the UI once for the upcoming Entra ID auth settings.

Clarify input v2 only settings

18e0cab

Update rendered docs

d88f2fb

Minor alignment to UI descriptions

c33e158

Replace all references to 'input' with 'processor'

145f915

I want to simplify the mental model for the user. From their perspective, there's the integration. Then the integration now has an optional 'processor' option with v1 and v2 variants.

Apply suggestions from code review

bf1b4b6

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Apply suggestions from code review

e99283f

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

zmoog force-pushed the zmoog/azure-logs-inputv2 branch from 99d821c to e99283f Compare March 6, 2025 18:45

Updated rendered docs

b5426ee

zmoog changed the title ~~[azure logs] enable azure-eventhub processor v2~~ [azure logs] enable event hub processor v2 Mar 6, 2025

zmoog merged commit bbdc042 into elastic:main Mar 10, 2025
6 of 7 checks passed

zmoog deleted the zmoog/azure-logs-inputv2 branch March 10, 2025 21:41


		The input v2 is in preview. Input v1 is still the default and is recommended for typical use cases.

		See the "Settings" section for more details about enabling the input v2.

	IMPORTANT: To use the integration v2 preview, you must turn off all the existing v1 integrations and turn on only the v2 preview integration.
	IMPORTANT: <blink>To use the integration v2 preview, you must turn off all the existing v1 integrations and turn on only the v2 preview integration.</blink>

Conversation

zmoog commented Feb 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Notes for reviewers

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elastic-vault-github-plugin-prod bot commented Feb 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Package azure 👍(8) 💚(2) 💔(1)

Uh oh!

elasticmachine commented Feb 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zmoog commented Feb 24, 2025

Uh oh!

MichaelKatsoulis left a comment

Choose a reason for hiding this comment

Uh oh!

zmoog commented Feb 27, 2025

Uh oh!

muthu-mps left a comment

Choose a reason for hiding this comment

Uh oh!

zmoog commented Mar 4, 2025

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zmoog commented Mar 6, 2025

Uh oh!

elasticmachine commented Mar 6, 2025

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Mar 6, 2025

Quality Gate failed

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Mar 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

zmoog commented Feb 17, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented Feb 17, 2025 •

edited

Loading

Package `azure` 👍(8) 💚(2) 💔(1)