[AWS VPC Flow] Add support for v6, v7 and v8 vpcflow logs#15077

moxarth-rathod · 2025-08-28T05:58:03Z

Proposed commit message

aws: add support for v6, v7 and v8 formats of vpc logs.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/aws directory.
Run the following command to run tests.

elastic-package test -v

Related issues

Closes [AWS VPC Flow] Support newer VPC log formats #14835

elasticmachine · 2025-08-28T05:58:07Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

andrewkroh · 2025-08-28T13:32:26Z

packages/aws/data_stream/vpcflow/elasticsearch/ingest_pipeline/default.yml

+           field: aws.vpcflow.packets_lost_ttl_expired
+  - set:
+      field: orchestrator.cluster.id
+      copy_from: aws.vpcflow.cs_cluster_arn


s/\.cs_cluster_arn/.ecs_cluster_arn/g

andrewkroh · 2025-08-28T13:36:31Z

packages/aws/_dev/build/docs/vpcflow.md


 * The default pattern of 14 version 2 fields
-* A custom pattern including all 29 fields, version 2 though 5:
+* A custom pattern including all 39 fields, version 2 though 7:


There is one v8 field, reject-reason. Should we add support for that now too?

https://docs.aws.amazon.com/vpc/latest/userguide/flow-log-records.html#flow-logs-fields

We also need to document support for the default transit gateway vpc flow format covering v2-6.

There is one v8 field, reject-reason. Should we add support for that now too?

Yes, i've added a support for the v8 field.

packages/aws/data_stream/vpcflow/elasticsearch/ingest_pipeline/default.yml

packages/aws/changelog.yml

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

elastic-vault-github-plugin-prod · 2025-08-29T09:09:37Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

andrewkroh · 2025-08-29T16:47:23Z

packages/aws/changelog.yml

 - version: "3.15.0"
  changes:
-    - description: Add support for v6 and v7 vpcflow logs.
+    - description: Add support for v6, v7 and v8 vpcflow logs.


Suggested change

- description: Add support for v6, v7 and v8 vpcflow logs.

- description: Add support for v6, v7, and v8 vpcflow logs.

I think the Oxford actually breaks the semantics here; it goes from ((v6 v7 v8) × vpcflow-logs) to ((v6 v7) (v8 vpcflow-logs)).

I don't think the semantics are changed by the addition of the comma. Both sentences are ((v6, v7, v8) × vpcflow-logs). The comma is only clarifying where one list item ends and the next begins, not which items are affected by the "vpcflow logs" modifier.

I think this one sounds more natural, and has the same intention ((VPC-Flow-logs × (versions-6, versions-7, versions-8))). Is there any issue with this wording?

Add support for VPC Flow logs versions 6, 7, and 8.

Add support for VPC Flow logs versions 6, 7, and 8.

@andrewkroh @efd6 should I update the description to this?

The latter is unambiguous. I'm happy with that.

elasticmachine · 2025-09-08T07:36:30Z

💚 Build Succeeded

Buildkite Build
Commit: bdff3c2

History

💚 Build #30612 succeeded 682cd0b
💚 Build #30598 succeeded 718977b
💔 Build #30594 failed 29e5638
💔 Build #30549 failed 250871a

cc @moxarth-rathod

elastic-sonarqube · 2025-09-08T07:36:36Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elastic-vault-github-plugin-prod · 2025-09-08T08:48:28Z

Package aws - 3.15.0 containing this change is available at https://epr.elastic.co/package/aws/3.15.0/

…R) workflow (#15230) aws: Add transforms to Config and Inspector data streams for extended protections (CDR) workflow. - Add latest transform to Config and Inspector data streams to help with Cloud Native Vulnerability Management (CNVM)[1] and Cloud Security Posture Management (CSPM)[2] workflows. - Add ILM policy to AWS Config as it does full sync every interval. - Update minimum kibana version to "^8.19.0 || ^9.1.0" to ensure necessary permissions for transform[3]. - Re-add 3.14.2 changelog entry as it is overwritten in VPC Flow PR[4]. - Skip system tests for securityhub* data streams to avoid fleet health degradation due to empty template values by httpjson. This is fixed in 8.19.4 and 9.1.4 by beats#45810[5] and beats#46332[6]. This skip can be removed when the stack version is upgraded to ones containing the fix. [1] https://www.elastic.co/guide/en/security/current/vuln-management-overview.html [2] https://www.elastic.co/docs/solutions/security/cloud/cloud-security-posture-management [3] elastic/elasticsearch#128350 [4] #15077 [5] elastic/beats#45810 [6] elastic/beats#46332

strawgate · 2025-10-16T19:10:55Z

/test benchmark fullreport

) aws: add support for v6, v7 and v8 formats of vpc logs.

…R) workflow (elastic#15230) aws: Add transforms to Config and Inspector data streams for extended protections (CDR) workflow. - Add latest transform to Config and Inspector data streams to help with Cloud Native Vulnerability Management (CNVM)[1] and Cloud Security Posture Management (CSPM)[2] workflows. - Add ILM policy to AWS Config as it does full sync every interval. - Update minimum kibana version to "^8.19.0 || ^9.1.0" to ensure necessary permissions for transform[3]. - Re-add 3.14.2 changelog entry as it is overwritten in VPC Flow PR[4]. - Skip system tests for securityhub* data streams to avoid fleet health degradation due to empty template values by httpjson. This is fixed in 8.19.4 and 9.1.4 by beats#45810[5] and beats#46332[6]. This skip can be removed when the stack version is upgraded to ones containing the fix. [1] https://www.elastic.co/guide/en/security/current/vuln-management-overview.html [2] https://www.elastic.co/docs/solutions/security/cloud/cloud-security-posture-management [3] elastic/elasticsearch#128350 [4] elastic#15077 [5] elastic/beats#45810 [6] elastic/beats#46332

[AWS VPC Flow] Add support for v6 and v7 vpcflow logs

250871a

moxarth-rathod self-assigned this Aug 28, 2025

moxarth-rathod added the enhancement New feature or request label Aug 28, 2025

moxarth-rathod requested review from a team as code owners August 28, 2025 05:58

moxarth-rathod added the Integration:aws AWS label Aug 28, 2025

moxarth-rathod requested a review from a team as a code owner August 28, 2025 05:58

moxarth-rathod added Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] labels Aug 28, 2025

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Aug 28, 2025

andrewkroh reviewed Aug 28, 2025

View reviewed changes

moxarth-rathod and others added 2 commits August 29, 2025 11:30

Update PR link in the changelog

29e5638

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

fix lint

718977b

Address PR comments

682cd0b

moxarth-rathod changed the title ~~[AWS VPC Flow] Add support for v6 and v7 vpcflow logs~~ [AWS VPC Flow] Add support for v6, v7 and v8 vpcflow logs Aug 29, 2025

moxarth-rathod requested a review from andrewkroh August 29, 2025 11:13

andrewkroh approved these changes Aug 29, 2025

View reviewed changes

Update the changelog description

bdff3c2

moxarth-rathod requested a review from efd6 September 8, 2025 05:46

efd6 approved these changes Sep 8, 2025

View reviewed changes

moxarth-rathod merged commit 1b506d1 into elastic:main Sep 8, 2025
9 checks passed

tehbooom pushed a commit to tehbooom/integrations that referenced this pull request Nov 19, 2025

[AWS VPC Flow] Add support for v6, v7 and v8 vpcflow logs (elastic#15077

3980ba1

) aws: add support for v6, v7 and v8 formats of vpc logs.

	- description: Add support for v6, v7 and v8 vpcflow logs.
	- description: Add support for v6, v7, and v8 vpcflow logs.

Conversation

moxarth-rathod commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Uh oh!

elasticmachine commented Aug 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Aug 29, 2025

🚀 Benchmarks report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Sep 8, 2025

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Sep 8, 2025

Quality Gate passed

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Sep 8, 2025

Uh oh!

strawgate commented Oct 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

moxarth-rathod commented Aug 28, 2025 •

edited

Loading