aws: Add Config and Inspector transforms for extended protections (CDR) workflow

[kcreddy]

Proposed commit message

aws: Add transforms to Config and Inspector data streams for extended protections (CDR) workflow.

- Add latest transform to Config and Inspector data streams
to help with Cloud Native Vulnerability Management (CNVM)[1] 
and Cloud Security Posture Management (CSPM)[2] workflows.
- Add ILM policy to AWS Config as it does full sync every interval.
- Update minimum kibana version to "^8.19.0 || ^9.1.0"  to ensure 
necessary permissions for transform[3].
- Re-add 3.14.2 changelog entry as it is overwritten in VPC Flow PR[4].
- Skip system tests for securityhub* data streams to avoid fleet health 
degradation due to empty template values by httpjson. This is fixed in 
8.19.4 and 9.1.4 by beats#45810[5] and beats#46332[6]. This skip can 
be removed when the stack version is upgraded to ones containing the fix. 

[1] https://www.elastic.co/guide/en/security/current/vuln-management-overview.html
[2] https://www.elastic.co/docs/solutions/security/cloud/cloud-security-posture-management
[3] https://github.com/elastic/elasticsearch/pull/128350
[4] https://github.com/elastic/integrations/pull/15077
[5] https://github.com/elastic/beats/pull/45810
[6] https://github.com/elastic/beats/pull/46332

Note

• Necessary ECS mappings are already added by PRs: Add new AWS Config datastream. #13830 and [aws] Update inspector data stream for Cloud Detection and Response (CDR) workflow #14306
• This PR follows transform definition from guide: https://docs.elastic.dev/security-solution/cloud-security/cdr/3p-dev-guide#integration-assets

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

• Closes AWS Inspector: Implement transform for Cloud Security Workflows #13903
• Closes AWS Config: Implement transform for Cloud Security Workflows #15233

[kcreddy]

/test

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[kcreddy]

@maxcold, the destination pattern was the only change since #14306 (comment).
Earlier it was security_solution-aws.vulnerability_latest-v1, and now security_solution-awsinspector.vulnerability_latest-v1. Notice change from aws to awsinspector.

This is to make it consistent with awsconfig. We will reserve aws when adding vulnerability data via SecurityHub which has aws pattern already for misconfigurations.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 8e4a86a

History

• 💔 Build #31687 failed 74558a1
• 💚 Build #31412 succeeded a4bb051
• 💔 Build #31401 failed df5c462
• 💔 Build #31335 failed 6563059
• 💔 Build #31333 failed d5d3703
• 💔 Build #31247 failed 69f9a98

cc @kcreddy

[elastic-sonarqube]

Quality Gate failed

Failed conditions
1.8% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[elastic-vault-github-plugin-prod]

Package aws - 4.0.0 containing this change is available at https://epr.elastic.co/package/aws/4.0.0/