Skip to content

[azure logs] Update time formats parsing and fix durationMs parsing in signinlogs data stream#15976

Merged
zmoog merged 6 commits intomainfrom
zmoog/azure/signin/update-date-formats-and-duration
Nov 17, 2025
Merged

[azure logs] Update time formats parsing and fix durationMs parsing in signinlogs data stream#15976
zmoog merged 6 commits intomainfrom
zmoog/azure/signin/update-date-formats-and-duration

Conversation

@zmoog
Copy link
Copy Markdown
Contributor

@zmoog zmoog commented Nov 14, 2025 •
edited
Loading

Proposed commit message

  • Add MM/dd/yyyy h:mm:ss a to the list of supported formats to parse time into @timestamp alongside ISO8601.
  • Convert azure.signinlogs.duration_ms to the long type if the field value comes as a string.

These changes handle events from Azure that come with these off-spec values.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

POST _ingest/pipeline/logs-azure.signinlogs-1.29.1/_simulate 
POST _ingest/pipeline/logs-azure.signinlogs-1.29.1/_simulate
{
 "docs": [
 {
 "_source": {
 "message": """
 {
  "callerIpAddress": "0.0.0.0",
  "category": "ServicePrincipalSignInLogs",
  "correlationId": "11111111-1111-1111-1111-111111111111",
  "durationMs": "0",
  "level": "Informational",
  "location": "AU",
  "operationName": "Sign-in activity",
  "operationVersion": "1.0",
  "properties": {
    "alternateSignInName": "",
    "appDisplayName": "Placeholder Application Name",
    "appId": "22222222-2222-2222-2222-222222222222",
    "appServicePrincipalId": null,
    "appliedConditionalAccessPolicies": [],
    "authenticationContextClassReferences": [],
    "authenticationDetails": [],
    "authenticationProcessingDetails": [
      {
        "key": "Azure AD App Authentication Library",
        "value": "Family: MSAL Library: MSAL.NET 4.54.1.0 Platform: .NET FW"
      },
      {
        "key": "Legacy TLS (TLS 1.0, 1.1, 3DES)",
        "value": "False"
      },
      {
        "key": "Is Legacy Store Used",
        "value": "False"
      },
      {
        "key": "Is CAE Token",
        "value": "True"
      }
    ],
    "authenticationProtocol": "none",
    "authenticationRequirement": "",
    "authenticationRequirementPolicies": [],
    "autonomousSystemNumber": 7545,
    "clientAppUsed": "Unknown",
    "conditionalAccessStatus": "notApplied",
    "correlationId": "11111111-1111-1111-1111-111111111111",
    "createdDateTime": "2025-11-14T01:46:16.4282975+00:00",
    "crossTenantAccessType": "none",
    "deviceDetail": {
      "browser": "Whatever Client 4.54.1.0",
      "deviceId": "55555555-5555-5555-5555-555555555555",
      "displayName": "DevicePlaceholder",
      "isManaged": false,
      "operatingSystem": "Windows10",
      "trustType": "Azure AD registered"
    },
    "flaggedForReview": false,
    "globalSecureAccessIpAddress": "",
    "homeTenantId": "",
    "homeTenantName": "",
    "id": "66666666-6666-6666-6666-666666666666",
    "incomingTokenType": "none",
    "ipAddress": "0.0.0.0",
    "ipAddressFromResourceProvider": "",
    "isInteractive": false,
    "isTenantRestricted": false,
    "location": {
      "city": "Placeholder City",
      "countryOrRegion": "ZZ",
      "geoCoordinates": {
        "latitude": -0.000000,
        "longitude": 0.000000
      },
      "state": "Placeholder State"
    },
    "networkLocationDetails": [
      {
        "networkNames": [
          "PlaceholderNetwork1"
        ],
        "networkType": "namedNetwork"
      },
      {
        "networkNames": [
          "PlaceholderNetwork2"
        ],
        "networkType": "trustedNamedLocation"
      }
    ],
    "originalRequestId": "66666666-6666-6666-6666-666666666666",
    "privateLinkDetails": {},
    "processingTimeInMilliseconds": 49,
    "redirectUrl": "",
    "resourceDisplayName": "MDATPNetworkScanAgent",
    "resourceId": "44444444-4444-4444-4444-444444444444",
    "resourceTenantId": "33333333-3333-3333-3333-333333333333",
    "riskDetail": "none",
    "riskEventTypes": [],
    "riskEventTypes_v2": [],
    "riskLevelAggregated": "none",
    "riskLevelDuringSignIn": "none",
    "riskState": "none",
    "servicePrincipalCredentialKeyId": "",
    "servicePrincipalCredentialThumbprint": "",
    "servicePrincipalId": "99999999-9999-9999-9999-999999999999",
    "sessionLifetimePolicies": [],
    "signInIdentifier": "",
    "sourceAppClientId": "",
    "status": {
      "errorCode": 0
    },
    "tokenIssuerName": "",
    "tokenIssuerType": "AzureAD",
    "uniqueTokenIdentifier": "mmmmmmmmmmmmmmmmm",
    "userAgent": "",
    "userDisplayName": "",
    "userId": "",
    "userPrincipalName": "",
    "userType": "Member"
  },
  "resourceId": "/tenants/33333333-3333-3333-3333-333333333333/providers/Microsoft.aadiam",
  "resultSignature": "SUCCESS",
  "resultType": "0",
  "tenantId": "33333333-3333-3333-3333-333333333333",
  "time": "11/14/2025 1:48:53 AM"
}
"""
 }
 }
 ]
}
Response 
{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_version": "-3",
        "_id": "_id",
        "_source": {
          "geo": {
            "city_name": "Placeholder City",
            "region_name": "Placeholder State",
            "location": {
              "lon": 0,
              "lat": 0
            },
            "country_iso_code": "AU"
          },
          "cloud": {
            "provider": "azure"
          },
          "@timestamp": "2025-11-14T01:48:53.000Z",
          "ecs": {
            "version": "8.11.0"
          },
          "related": {
            "entity": [
              "22222222-2222-2222-2222-222222222222",
              "44444444-4444-4444-4444-444444444444",
              "99999999-9999-9999-9999-999999999999",
              "55555555-5555-5555-5555-555555555555"
            ],
            "ip": [
              "0.0.0.0"
            ]
          },
          "log": {
            "level": "Informational"
          },
          "client": {
            "ip": "0.0.0.0"
          },
          "source": {
            "address": "0.0.0.0",
            "ip": "0.0.0.0"
          },
          "event": {
            "duration": 0,
            "original": """
 {
  "callerIpAddress": "0.0.0.0",
  "category": "ServicePrincipalSignInLogs",
  "correlationId": "11111111-1111-1111-1111-111111111111",
  "durationMs": "0",
  "level": "Informational",
  "location": "AU",
  "operationName": "Sign-in activity",
  "operationVersion": "1.0",
  "properties": {
    "alternateSignInName": "",
    "appDisplayName": "Placeholder Application Name",
    "appId": "22222222-2222-2222-2222-222222222222",
    "appServicePrincipalId": null,
    "appliedConditionalAccessPolicies": [],
    "authenticationContextClassReferences": [],
    "authenticationDetails": [],
    "authenticationProcessingDetails": [
      {
        "key": "Azure AD App Authentication Library",
        "value": "Family: MSAL Library: MSAL.NET 4.54.1.0 Platform: .NET FW"
      },
      {
        "key": "Legacy TLS (TLS 1.0, 1.1, 3DES)",
        "value": "False"
      },
      {
        "key": "Is Legacy Store Used",
        "value": "False"
      },
      {
        "key": "Is CAE Token",
        "value": "True"
      }
    ],
    "authenticationProtocol": "none",
    "authenticationRequirement": "",
    "authenticationRequirementPolicies": [],
    "autonomousSystemNumber": 7545,
    "clientAppUsed": "Unknown",
    "conditionalAccessStatus": "notApplied",
    "correlationId": "11111111-1111-1111-1111-111111111111",
    "createdDateTime": "2025-11-14T01:46:16.4282975+00:00",
    "crossTenantAccessType": "none",
    "deviceDetail": {
      "browser": "Whatever Client 4.54.1.0",
      "deviceId": "55555555-5555-5555-5555-555555555555",
      "displayName": "DevicePlaceholder",
      "isManaged": false,
      "operatingSystem": "Windows10",
      "trustType": "Azure AD registered"
    },
    "flaggedForReview": false,
    "globalSecureAccessIpAddress": "",
    "homeTenantId": "",
    "homeTenantName": "",
    "id": "66666666-6666-6666-6666-666666666666",
    "incomingTokenType": "none",
    "ipAddress": "0.0.0.0",
    "ipAddressFromResourceProvider": "",
    "isInteractive": false,
    "isTenantRestricted": false,
    "location": {
      "city": "Placeholder City",
      "countryOrRegion": "ZZ",
      "geoCoordinates": {
        "latitude": -0.000000,
        "longitude": 0.000000
      },
      "state": "Placeholder State"
    },
    "networkLocationDetails": [
      {
        "networkNames": [
          "PlaceholderNetwork1"
        ],
        "networkType": "namedNetwork"
      },
      {
        "networkNames": [
          "PlaceholderNetwork2"
        ],
        "networkType": "trustedNamedLocation"
      }
    ],
    "originalRequestId": "66666666-6666-6666-6666-666666666666",
    "privateLinkDetails": {},
    "processingTimeInMilliseconds": 49,
    "redirectUrl": "",
    "resourceDisplayName": "MDATPNetworkScanAgent",
    "resourceId": "44444444-4444-4444-4444-444444444444",
    "resourceTenantId": "33333333-3333-3333-3333-333333333333",
    "riskDetail": "none",
    "riskEventTypes": [],
    "riskEventTypes_v2": [],
    "riskLevelAggregated": "none",
    "riskLevelDuringSignIn": "none",
    "riskState": "none",
    "servicePrincipalCredentialKeyId": "",
    "servicePrincipalCredentialThumbprint": "",
    "servicePrincipalId": "99999999-9999-9999-9999-999999999999",
    "sessionLifetimePolicies": [],
    "signInIdentifier": "",
    "sourceAppClientId": "",
    "status": {
      "errorCode": 0
    },
    "tokenIssuerName": "",
    "tokenIssuerType": "AzureAD",
    "uniqueTokenIdentifier": "mmmmmmmmmmmmmmmmm",
    "userAgent": "",
    "userDisplayName": "",
    "userId": "",
    "userPrincipalName": "",
    "userType": "Member"
  },
  "resourceId": "/tenants/33333333-3333-3333-3333-333333333333/providers/Microsoft.aadiam",
  "resultSignature": "SUCCESS",
  "resultType": "0",
  "tenantId": "33333333-3333-3333-3333-333333333333",
  "time": "11/14/2025 1:48:53 AM"
}
""",
            "kind": "event",
            "action": "Sign-in activity",
            "id": "66666666-6666-6666-6666-666666666666",
            "category": [
              "authentication"
            ],
            "type": [
              "info"
            ],
            "outcome": "success"
          },
          "user_agent": {
            "name": "Other",
            "device": {
              "name": "Other"
            }
          },
          "azure": {
            "tenant_id": "33333333-3333-3333-3333-333333333333",
            "correlation_id": "11111111-1111-1111-1111-111111111111",
            "signinlogs": {
              "operation_name": "Sign-in activity",
              "result_type": "0",
              "operation_version": "1.0",
              "caller_ip_address": "0.0.0.0",
              "result_signature": "SUCCESS",
              "category": "ServicePrincipalSignInLogs",
              "properties": {
                "is_tenant_restricted": false,
                "risk_level_aggregated": "none",
                "client_app_used": "Unknown",
                "resource_tenant_id": "33333333-3333-3333-3333-333333333333",
                "created_at": "2025-11-14T01:46:16.4282975+00:00",
                "risk_level_during_signin": "none",
                "authentication_protocol": "none",
                "token_issuer_type": "AzureAD",
                "user_type": "Member",
                "original_request_id": "66666666-6666-6666-6666-666666666666",
                "conditional_access_status": "notApplied",
                "id": "66666666-6666-6666-6666-666666666666",
                "app_id": "22222222-2222-2222-2222-222222222222",
                "network_location_details": [
                  {
                    "network_type": "namedNetwork",
                    "network_names": [
                      "PlaceholderNetwork1"
                    ]
                  },
                  {
                    "network_type": "trustedNamedLocation",
                    "network_names": [
                      "PlaceholderNetwork2"
                    ]
                  }
                ],
                "is_interactive": false,
                "service_principal_id": "99999999-9999-9999-9999-999999999999",
                "flagged_for_review": false,
                "app_display_name": "Placeholder Application Name",
                "authentication_processing_details": {
                  "Is Legacy Store Used": "False",
                  "Legacy TLS (TLS 1.0, 1.1, 3DES)": "False",
                  "Is CAE Token": "True",
                  "Azure AD App Authentication Library": "Family: MSAL Library: MSAL.NET 4.54.1.0 Platform: .NET FW"
                },
                "device_detail": {
                  "device_id": "55555555-5555-5555-5555-555555555555",
                  "browser": "Whatever Client 4.54.1.0",
                  "operating_system": "Windows10",
                  "trust_type": "Azure AD registered",
                  "display_name": "DevicePlaceholder",
                  "is_managed": false
                },
                "risk_detail": "none",
                "autonomous_system_number": 7545,
                "resource_display_name": "MDATPNetworkScanAgent",
                "risk_state": "none",
                "incoming_token_type": "none",
                "cross_tenant_access_type": "none",
                "processing_time_ms": 49,
                "resource_id": "44444444-4444-4444-4444-444444444444",
                "correlation_id": "11111111-1111-1111-1111-111111111111",
                "unique_token_identifier": "mmmmmmmmmmmmmmmmm",
                "status": {
                  "error_code": 0
                }
              }
            },
            "resource": {
              "provider": "Microsoft.aadiam",
              "id": "/tenants/33333333-3333-3333-3333-333333333333/providers/Microsoft.aadiam"
            }
          }
        },
        "_ingest": {
          "timestamp": "2025-11-14T08:45:42.714994545Z"
        }
      }
    }
  ]
}

@zmoog zmoog self-assigned this Nov 14, 2025
@zmoog zmoog added Integration:azure Azure Logs Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] bugfix Pull request that fixes a bug issue Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] labels Nov 14, 2025
@zmoog zmoog changed the title [azure logs] update time date formats and convert duration_ms when string [azure logs] update time date formats and convert duration_ms when string Nov 14, 2025
@zmoog zmoog changed the title [azure logs] update time date formats and convert duration_ms when string [azure logs] Update time formats parsing and fix durationMs parsing in signinlogs data stream Nov 14, 2025
@zmoog zmoog marked this pull request as ready for review November 14, 2025 08:50
@zmoog zmoog requested review from a team as code owners November 14, 2025 08:50
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Nov 14, 2025

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@zmoog zmoog mentioned this pull request Nov 14, 2025
Closed
@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Nov 14, 2025

🚀 Benchmarks report

Package azure 👍(5) 💚(3) 💔(4)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
springcloudlogs 3610.11 2638.52 -971.59 (-26.91%) 💔
activitylogs 1597.44 1222.49 -374.95 (-23.47%) 💔
events 31250 18518.52 -12731.48 (-40.74%) 💔
firewall_logs 2132.2 1351.35 -780.85 (-36.62%) 💔

To see the full report comment with /test benchmark fullreport

kcreddy
kcreddy approved these changes Nov 14, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@kcreddy kcreddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nits. LGTM otherwise.

zmoog and others added 2 commits November 17, 2025 15:17
@zmoog @kcreddy
Apply suggestions from code review
d0a1653 
Co-authored-by: Krishna Chaitanya Reddy Burri <krishnachaitanyareddy.burri@elastic.co>
@zmoog
Remove redundant checks
4f2d665
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Nov 17, 2025

💚 Build Succeeded

History

cc @zmoog

@zmoog zmoog merged commit e78bc4b into main Nov 17, 2025
7 checks passed
@zmoog zmoog deleted the zmoog/azure/signin/update-date-formats-and-duration branch November 17, 2025 16:23
@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Nov 17, 2025

Package azure - 1.29.1 containing this change is available at https://epr.elastic.co/package/azure/1.29.1/

tehbooom pushed a commit to tehbooom/integrations that referenced this pull request Nov 19, 2025
@zmoog
[azure logs] Update time formats parsing and fix durationMs parsi…
fd591d0 
…ng in signinlogs data stream (elastic#15976)

- Add `MM/dd/yyyy h:mm:ss a` to the list of supported formats to parse `time` into `@timestamp` alongside `ISO8601`.
- Convert `azure.signinlogs.duration_ms` to the `long` type if the field value comes as a string.

These changes handle events from Azure that come with these off-spec values.
@gregorywychowaniec-zt gregorywychowaniec-zt mentioned this pull request Dec 23, 2025
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gizas gizas gizas approved these changes

@kcreddy kcreddy kcreddy approved these changes

@muthu-mps muthu-mps muthu-mps approved these changes

Assignees

@zmoog zmoog

Labels

bugfix Pull request that fixes a bug issue Integration:azure Azure Logs Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@zmoog @elasticmachine @gizas @kcreddy @muthu-mps