Skip to content

aws: improve error.message and add event.kind:pipeline_error for pipeline errors#11112

Merged
efd6 merged 3 commits intoelastic:mainfrom
efd6:11090-aws
Oct 15, 2024
Merged

aws: improve error.message and add event.kind:pipeline_error for pipeline errors#11112
efd6 merged 3 commits intoelastic:mainfrom
efd6:11090-aws

Conversation

@efd6
Copy link
Copy Markdown
Contributor

@efd6 efd6 commented Sep 12, 2024 •
edited
Loading

Proposed commit message

Also fix instances of incorrect yaml for script processors.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@efd6 efd6 added enhancement New feature or request Integration:aws AWS Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Sep 12, 2024
@efd6 efd6 self-assigned this Sep 12, 2024
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Sep 12, 2024 •
edited by elastic-vault-github-plugin-prod bot
Loading

🚀 Benchmarks report

Package aws 👍(8) 💚(5) 💔(6)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
guardduty 1152.07 801.28 -350.79 (-30.45%) 💔
waf 7407.41 5917.16 -1490.25 (-20.12%) 💔
cloudfront_logs 2518.89 1923.08 -595.81 (-23.65%) 💔
cloudwatch_logs 500000 250000 -250000 (-50%) 💔
elb_logs 4739.34 3937.01 -802.33 (-16.93%) 💔
firewall_logs 3184.71 2538.07 -646.64 (-20.3%) 💔

To see the full report comment with /test benchmark fullreport

@efd6 efd6 marked this pull request as ready for review September 12, 2024 08:12
@efd6 efd6 requested review from a team as code owners September 12, 2024 08:12
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Sep 12, 2024

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@andrewkroh andrewkroh added Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Sep 12, 2024
@efd6 efd6 mentioned this pull request Oct 7, 2024
Merged
4 tasks
andrewkroh
andrewkroh approved these changes Oct 8, 2024
View reviewed changes
packages/aws/data_stream/apigateway_logs/elasticsearch/ingest_pipeline/default.yml Outdated
Copy link
Copy Markdown
Member

@andrewkroh andrewkroh Oct 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you have an easy way to update these, consider making the "with tag" part conditional on whether the processor has a tag.

For example:

https://github.com/andrewkroh/go-examples/blob/ee88091db24ef39d92d4431fa4a112e6fcb00d02/ecs-update/ecs-update.go#L1697-L1700

@efd6
Copy link
Copy Markdown
Contributor Author

efd6 commented Oct 9, 2024

@romulets I would like to understand how to resolve the conflicts in this PR. There are three possible approaches (four):

  1. Merge this as 2.27.0-preview01 and leave the benefits unavailable to users until the the features in 2.26.0-preview01 are accepted as GA.
  2. Merge this as 2.26.0 after reverting the changes in 2.26.0-preview01.
  3. Merge this as 2.27.0 and release the features in 2.26.0-preview01.
  4. (abandon this change)

agithomas
agithomas reviewed Oct 9, 2024
View reviewed changes
packages/aws/data_stream/billing/elasticsearch/ingest_pipeline/default.yml Outdated
Copy link
Copy Markdown
Contributor

@agithomas agithomas Oct 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

For a few values, with tag (whitespace in front of with) is used, for a few this whitespace is missing.

Copy link
Copy Markdown
Contributor Author

@efd6 efd6 Oct 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I thought I got all of them with a s///g.

@chemamartinez chemamartinez mentioned this pull request Oct 10, 2024
Merged
5 tasks
@efd6
Copy link
Copy Markdown
Contributor Author

efd6 commented Oct 14, 2024

@ishleenk17 and @constanca-m it would be really helpful if you could also check out the blocking PR #11397 that this depends on.

efd6 added 2 commits October 15, 2024 17:50
@efd6
aws: improve error.message and add event.kind:pipeline_error for pipe…
bd91dc7 
…line errors

Also fix instances of incorrect yaml for script processors.
@efd6
address pr comment
7751e31
@efd6 efd6 enabled auto-merge (squash) October 15, 2024 07:21
@elastic-sonarqube
Copy link
Copy Markdown

elastic-sonarqube bot commented Oct 15, 2024

Quality Gate failed Quality Gate failed

Failed conditions
Image 2.3% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

@efd6 efd6 merged commit 6dd07a5 into elastic:main Oct 15, 2024
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Oct 15, 2024

💚 Build Succeeded

History

  • 💚 Build #17071 succeeded 306831c435e436e814322c4da64b01130dcb4426
  • 💔 Build #16956 failed 7a43dd42b75de8e907c2cdb238b6b3537b91e671
  • 💔 Build #16944 failed 7c1bb793fbe0a3c4943341012f941631b0be5ec8
  • 💔 Build #16943 failed 8c687ec561594866737c5a32e79edca7d0294b4b
  • 💚 Build #16392 succeeded 2aefed683b8815994036a67447801b458a29c4cb
  • 💚 Build #15908 succeeded b012926a001d13a8ebe96375d83953ed9d8265f0

cc @efd6

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Oct 15, 2024

Package aws - 2.27.0 containing this change is available at https://epr.elastic.co/search?package=aws

harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
@efd6
aws: improve error.message and add event.kind:pipeline_error for pipe…
3ee396f 
…line errors (elastic#11112)

Also fix instances of incorrect yaml for script processors.
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
@efd6
aws: improve error.message and add event.kind:pipeline_error for pipe…
c77fd8f 
…line errors (elastic#11112)

Also fix instances of incorrect yaml for script processors.
@efd6 efd6 deleted the 11090-aws branch February 5, 2025 22:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@agithomas agithomas agithomas left review comments

@andrewkroh andrewkroh andrewkroh approved these changes

@kcreddy kcreddy kcreddy approved these changes

@kruskall kruskall kruskall approved these changes

@ishleenk17 ishleenk17 ishleenk17 approved these changes

@constanca-m constanca-m constanca-m approved these changes

+1 more reviewer

@kaiyan-sheng kaiyan-sheng kaiyan-sheng approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@efd6 efd6

Labels

enhancement New feature or request Integration:aws AWS Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[aws]: guardduty error message is unhelpful

9 participants

@efd6 @elasticmachine @andrewkroh @kcreddy @kaiyan-sheng @kruskall @agithomas @ishleenk17 @constanca-m