Overview

Use this guide to get started monitoring your Google Cloud environment. This approach simplifies the setup for Google Cloud environments with multiple projects, allowing you to maximize your monitoring coverage.

Setup

Set up Datadog’s Google Cloud integration to collect metrics and logs from your Google Cloud services.

Prerequisites

2. Enable the following APIs for every project you want to monitor, including the project where the service account has been created.

Cloud Monitoring API

Allows Datadog to query your Google Cloud metric data.

Compute Engine API

Allows Datadog to discover compute instance data.

Cloud Asset API

Allows Datadog to request Google Cloud resources and link relevant labels to metrics as tags.

Cloud Resource Manager API

Allows Datadog to append metrics with the correct resources and tags.

IAM API

Allows Datadog to authenticate with Google Cloud.

Cloud Billing API

Allows developers to manage billing for their Google Cloud Platform projects programmatically. See the Cloud Cost Management (CCM) documentation for more information.

3. Ensure that any projects being monitored are not configured as scoping projects that pull in metrics from multiple other projects.

Metric collection

Installation

Organization-level (or folder-level) monitoring is recommended for comprehensive coverage of all projects, including any future projects that may be created in an org or folder.

Note: Your Google Cloud Identity user account must have the Admin role assigned to it at the desired scope to complete the setup in Google Cloud (for example, Organization Admin).

1. Create a Google Cloud service account in the default project

1. Open your Google Cloud console.
2. Navigate to IAM & Admin > Service Accounts.
3. Click Create service account at the top.
4. Give the service account a unique name.
5. Click Done to complete creating the service account.

2. Add the service account at the organization or folder level

1. In the Google Cloud console, go to the IAM page.
2. Select a folder or organization.
3. To grant a role to a principal that does not already have other roles on the resource, click Grant Access, then enter the email of the service account you created earlier.
4. Enter the service account’s email address.
5. Assign the following roles:

• Compute Viewer provides read-only access to get and list Compute Engine resources
• Monitoring Viewer provides read-only access to the monitoring data availabile in your Google Cloud environment
• Cloud Asset Viewer provides read-only access to cloud assets metadata
• Browser provides read-only access to browse the hierarchy of a project
• Service Usage Consumer (optional, for multi-project environments) provides per-project cost and API quota attribution after this feature has been enabled by Datadog support

6. Click Save.

Note: The Browser role is only required in the default project of the service account. Other projects require only the other listed roles.

3. Add the Datadog principal to your service account

Note: If you previously configured access using a shared Datadog principal, you can revoke the permission for that principal after you complete these steps.

1. In Datadog, navigate to Integrations > Google Cloud Platform.
2. Click Add Google Cloud Account. If you have no configured projects, you are automatically redirected to this page.
3. Copy your Datadog principal and keep it for the next section.

Note: Keep this window open for Section 4.

4. In the Google Cloud console, under the Service Accounts menu, find the service account you created in Section 1.
5. Go to the Permissions tab and click Grant Access.

6. Paste your Datadog principal into the New principals text box.
7. Assign the role of Service Account Token Creator.
8. Click Save.

4. Complete the integration setup in Datadog

1. In your Google Cloud console, navigate to the Service Account > Details tab. On this page, find the email associated with this Google service account. It has the format <SA_NAME>@<PROJECT_ID>.iam.gserviceaccount.com.
2. Copy this email.
3. Return to the integration configuration tile in Datadog (where you copied your Datadog principal in the previous section).
4. Paste the email you copied in Add Service Account Email.
5. Click Verify and Save Account.

Metrics appear in Datadog approximately 15 minutes after setup.

Best practices for monitoring multiple projects

Enable per-project cost and API quota attribution

By default, Google Cloud attributes the cost of monitoring API calls, as well as API quota usage, to the project containing the service account for this integration. As a best practice for Google Cloud environments with multiple projects, enable per-project cost attribution of monitoring API calls and API quota usage. With this enabled, costs and quota usage are attributed to the project being queried, rather than the project containing the service account. This provides visibility into the monitoring costs incurred by each project, and also helps to prevent reaching API rate limits.

To enable this feature:

1. Ensure that the Datadog service account has the Service Usage Consumer role at the desired scope (folder or organization).
2. Click the Enable Per Project Quota toggle in the Projects tab of the Google Cloud integration page.

You can use service account impersonation and automatic project discovery to integrate Datadog with Google Cloud. Note: Projects with a prefix of sys- will not be picked up as part of project discovery.

This method enables you to monitor all projects visible to a service account by assigning IAM roles in the relevant projects. You can assign these roles to projects individually, or you can configure Datadog to monitor groups of projects by assigning these roles at the organization or folder level. Assigning roles in this way allows Datadog to automatically discover and monitor all projects in the given scope, including any new projects that may be added to the group in the future.

1. Create a Google Cloud service account

1. Open your Google Cloud console.
2. Navigate to IAM & Admin > Service Accounts.
3. Click on Create service account at the top.
4. Give the service account a unique name, then click Create and continue.
5. Add the following roles to the service account:
   • Monitoring Viewer provides read-only access to the monitoring data availabile in your Google Cloud environment
   • Compute Viewer provides read-only access to get and list Compute Engine resources
   • Cloud Asset Viewer provides read-only access to cloud assets metadata
   • Browser provides read-only access to discover accessible projects
6. Click Continue, then Done to complete creating the service account.

2. Add the Datadog principal to your service account

1. In Datadog, navigate to the Integrations > Google Cloud Platform.

2. Click on Add GCP Account. If you have no configured projects, you are automatically redirected to this page.

3. If you have not generated a Datadog principal for your org, click the Generate Principal button.

4. Copy your Datadog principal and keep it for the next section.

   Note: Keep this window open for the next section.

5. In Google Cloud console, under the Service Accounts menu, find the service account you created in the first section.

6. Go to the Permissions tab and click on Grant Access.

7. Paste your Datadog principal into the New principals text box.

8. Assign the role of Service Account Token Creator and click SAVE.

Note: If you previously configured access using a shared Datadog principal, you can revoke the permission for that principal after you complete these steps.

3. Complete the integration setup in Datadog

1. In your Google Cloud console, navigate to the Service Account > Details tab. There, you can find the email associated with this Google service account. It resembles <sa-name>@<project-id>.iam.gserviceaccount.com.
2. Copy this email.
3. Return to the integration configuration tile in Datadog (where you copied your Datadog principal in the previous section).
4. In the box under Add Service Account Email, paste the email you previously copied.
5. Click on Verify and Save Account.

In approximately fifteen minutes, metrics appear in Datadog.

Validation

To view your metrics, use the left menu to navigate to Metrics > Summary and search for gcp:

Configuration

Limit metric collection by metric namespace, and by granular metric filters

Optionally, you can choose which Google Cloud services you monitor with Datadog, and further define which specific metrics you want to collect from each individual service. Configuring metric collection lets you optimize your Google Cloud Monitoring API costs, while retaining visibility into your critical services.

For granular control of metric collection from a service, define inclusion and exclusion filters for that service. A metric is collected only if it matches at least one inclusion filter, and does not match any exclusion filters.

Under the Metric Collection tab in Datadog’s Google Cloud integration page, unselect the metric namespaces to exclude. To apply granular filtering for enabled services, click on the service in question and apply your filters in the Add filters for gcp.<service> field.

Example filters:

subscription.* topic.*

Limit collection to metrics matching either gcp.<service>.subscription.* or gcp.<service>.topic.*

!*_cost !*_count

Limit collection to metrics matching neither gcp.<service>.*_cost nor gcp.<service>.*_count

snapshot.* !*_by_region

Limit collection to metrics matching gcp.<service>.snapshot.* but not matching gcp.<service>.*_by_region

Limit metric collection by Google Cloud region and by global resources

Under the Metric Collection tab in Datadog’s Google Cloud integration page, deselect the region you want to exclude from metrics collection.

You can also specify additional locations not listed and disable any global metrics not associated with a region.

datadog:monitored,env:production,!env:staging,instance-type:c1.*

Leveraging the Datadog Agent

Use the Datadog Agent to collect the most granular, low-latency metrics from your infrastructure. Install the Agent on any host, including GKE, to get deeper insights from the traces and logs it can collect. For more information, see Why should I install the Datadog Agent on my cloud instances?

Log collection

See the Google Cloud Log Forwarding Setup page for log forwarding setup options and instructions.

Expanded BigQuery monitoring

Expanded BigQuery monitoring provides granular visibility into your BigQuery environments. See the BigQuery Data Observability documentation for more information.

Resource changes collection

Select Enable Resource Collection in the Resource Collection tab of the Google Cloud integration page. This allows you to receive resource events in Datadog when Google’s Cloud Asset Inventory detects changes in your cloud resources.

Then, follow the steps below to forward change events from a Pub/Sub topic to the Datadog Event Explorer.

gcloud asset feeds create <FEED_NAME>
--project=<PROJECT_ID>
--pubsub-topic=projects/<PROJECT_ID>/topics/<TOPIC_NAME>
--asset-names=<ASSET_NAMES>
--asset-types=<ASSET_TYPES>
--content-type=<CONTENT_TYPE>

gcloud asset feeds create <FEED_NAME>
--folder=<FOLDER_ID>
--pubsub-topic=projects/<PROJECT_ID>/topics/<TOPIC_NAME>
--asset-names=<ASSET_NAMES>
--asset-types=<ASSET_TYPES>
--content-type=<CONTENT_TYPE>

gcloud asset feeds create <FEED_NAME>
--organization=<ORGANIZATION_ID>
--pubsub-topic=projects/<PROJECT_ID>/topics/<TOPIC_NAME>
--asset-names=<ASSET_NAMES>
--asset-types=<ASSET_TYPES>
--content-type=<CONTENT_TYPE>

locals {
  project_id = "<PROJECT_ID>"
}

resource "google_pubsub_topic" "pubsub_topic" {
  project = local.project_id
  name    = "<TOPIC_NAME>"
}

resource "google_pubsub_subscription" "pubsub_subscription" {
  project = local.project_id
  name    = "export-asset-changes-to-datadog"
  topic   = google_pubsub_topic.pubsub_topic.id
}

resource "google_pubsub_subscription_iam_member" "subscriber" {
  project      = local.project_id
  subscription = google_pubsub_subscription.pubsub_subscription.id
  role         = "roles/pubsub.subscriber"
  member       = "serviceAccount:<SERVICE_ACCOUNT_EMAIL>"
}

resource "google_cloud_asset_project_feed" "project_feed" {
  project      = local.project_id
  feed_id      = "<FEED_NAME>"
  content_type = "<CONTENT_TYPE>" # Optional. Remove if unused.

  asset_names = ["<ASSET_NAMES>"] # Optional if specifying asset_types. Remove if unused.
  asset_types = ["<ASSET_TYPES>"] # Optional if specifying asset_names. Remove if unused.

  feed_output_config {
    pubsub_destination {
      topic = google_pubsub_topic.pubsub_topic.id
    }
  }
}

locals {
  project_id = "<PROJECT_ID>"
}

resource "google_pubsub_topic" "pubsub_topic" {
  project = local.project_id
  name    = "<TOPIC_NAME>"
}

resource "google_pubsub_subscription" "pubsub_subscription" {
  project = local.project_id
  name    = "export-asset-changes-to-datadog"
  topic   = google_pubsub_topic.pubsub_topic.id
}

resource "google_pubsub_subscription_iam_member" "subscriber" {
  project      = local.project_id
  subscription = google_pubsub_subscription.pubsub_subscription.id
  role         = "roles/pubsub.subscriber"
  member       = "serviceAccount:<SERVICE_ACCOUNT_EMAIL>"
}

resource "google_cloud_asset_folder_feed" "folder_feed" {
  billing_project = local.project_id
  folder          = "<FOLDER_ID>"
  feed_id         = "<FEED_NAME>"
  content_type    = "<CONTENT_TYPE>" # Optional. Remove if unused.

  asset_names = ["<ASSET_NAMES>"] # Optional if specifying asset_types. Remove if unused.
  asset_types = ["<ASSET_TYPES>"] # Optional if specifying asset_names. Remove if unused.

  feed_output_config {
    pubsub_destination {
      topic = google_pubsub_topic.pubsub_topic.id
    }
  }
}

locals {
  project_id = "<PROJECT_ID>"
}

resource "google_pubsub_topic" "pubsub_topic" {
  project = local.project_id
  name    = "<TOPIC_NAME>"
}

resource "google_pubsub_subscription" "pubsub_subscription" {
  project = local.project_id
  name    = "export-asset-changes-to-datadog"
  topic   = google_pubsub_topic.pubsub_topic.id
}

resource "google_pubsub_subscription_iam_member" "subscriber" {
  project      = local.project_id
  subscription = google_pubsub_subscription.pubsub_subscription.id
  role         = "roles/pubsub.subscriber"
  member       = "serviceAccount:<SERVICE_ACCOUNT_EMAIL>"
}

resource "google_cloud_asset_organization_feed" "organization_feed" {
  billing_project = local.project_id
  org_id          = "<ORGANIZATION_ID>"
  feed_id         = "<FEED_NAME>"
  content_type    = "<CONTENT_TYPE>" # Optional. Remove if unused.

  asset_names = ["<ASSET_NAMES>"] # Optional if specifying asset_types. Remove if unused.
  asset_types = ["<ASSET_TYPES>"] # Optional if specifying asset_names. Remove if unused.

  feed_output_config {
    pubsub_destination {
      topic = google_pubsub_topic.pubsub_topic.id
    }
  }
}

Datadog recommends setting the asset-types parameter to the regular expression .* to collect changes for all resources.

Note: You must specify at least one value for either the asset-names or asset-types parameter.

See the gcloud asset feeds create reference for the full list of configurable parameters.

Enable resource changes collection

Click to Enable Resource Changes Collection in the Resource Collection tab of the Google Cloud integration page.

Validation

Find your asset change events in the Datadog Event Explorer.

Private Service Connect

Use the Google Cloud Private Service Connect integration to visualize connections, data transferred, and dropped packets through Private Service Connect. This gives you visibility into important metrics from your Private Service Connect connections, both for producers as well as consumers. Private Service Connect (PSC) is a Google Cloud networking product that enables you to access Google Cloud services, third-party partner services, and company-owned applications directly from your Virtual Private Cloud (VPC).

See Access Datadog privately and monitor your Google Cloud Private Service Connect usage in the Datadog blog for more information.

Data Collected

Metrics

Cumulative metrics

Cumulative metrics are imported into Datadog with a .delta metric for each metric name. A cumulative metric is a metric where the value constantly increases over time. For example, a metric for sent bytes might be cumulative. Each value records the total number of bytes sent by a service at that time. The delta value represents the change since the previous measurement.

For example:

gcp.gke.container.restart_count is a CUMULATIVE metric. While importing this metric as a cumulative metric, Datadog adds the gcp.gke.container.restart_count.delta metric which includes the delta values (as opposed to the aggregate value emitted as part of the CUMULATIVE metric). See Google Cloud metric kinds for more information.

Events

All service events generated by your Google Cloud Platform are forwarded to your Datadog Events Explorer.

Service Checks

The Google Cloud Platform integration does not include any service checks.

Tags

Tags are automatically assigned based on a variety of Google Cloud Platform and Google Compute Engine configuration options. The project_id tag is added to all metrics. Additional tags are collected from the Google Cloud Platform when available, and varies based on metric type.

Additionally, Datadog collects the following as tags:

• Any hosts with <key>:<value> labels.
• Custom labels from Google Pub/Sub, GCE, Cloud SQL, and Cloud Storage.

Troubleshooting

Incorrect metadata for user defined gcp.logging metrics?

For non-standard gcp.logging metrics, such as metrics beyond Datadog’s out of the box logging metrics, the metadata applied may not be consistent with Google Cloud Logging.

In these cases, the metadata should be manually set by navigating to the metric summary page, searching and selecting the metric in question, and clicking the pencil icon next to the metadata.

Need help? Contact Datadog support.

Further Reading

お役に立つドキュメント、リンクや記事:

Improve the compliance and security posture of your Google Cloud environment with DatadogBLOG Monitor Google Cloud Vertex AI with DatadogBLOG Monitor your Dataflow pipelines with DatadogBLOG Google Cloud PlatformOTHER Monitor BigQuery with DatadogBLOG Google Cloud Platform blogBLOG