GCP workload identity for ClickPipes

Configure keyless access from ClickPipes to supported Google Cloud data sources using GCP workload identity.

GCP workload identity authentication is in Private Preview. Contact ClickHouse Support to enable it for your organization.

GCP workload identity lets ClickPipes access supported Google Cloud sources without storing a customer service account key. This guide shows you how to retrieve the ClickPipes-managed Google service account, grant it access to your source, and create a ClickPipe that uses short-lived credentials.

Before you begin

You need:

• Private Preview access enabled for your ClickHouse Cloud organization.
• A ClickHouse Cloud service running on GCP. Its ClickPipes deployment must also run on GCP.
• A ClickHouse Cloud API key with permission to manage ClickPipes on the service.
• Permission to grant IAM roles on the Google Cloud source resources.
• For Terraform, a ClickHouse provider release that includes the clickhouse_clickpipes_service_context data source. Check the provider changelog before selecting a version.

How it works

1. ClickPipes creates and manages a Google service account for your ClickHouse Cloud service.
2. You retrieve that service account from the ClickPipes service context.
3. You grant the service account access to your Google Cloud source resources.
4. ClickPipes uses workload identity to obtain short-lived credentials for the service account and access the source.

The service account is unique to the ClickHouse Cloud service. You control its access to your resources through Google Cloud IAM and can revoke that access at any time. Network connectivity, including private connectivity, is configured separately.

Retrieve the ClickPipes service account

Use OpenAPI or Terraform to retrieve the service context. Do not create the ClickPipe until the context reports that workload identity is supported and ready.

OpenAPI

Set your API credentials and resource IDs:

export KEY_ID=<your_key_id>
export KEY_SECRET=<your_key_secret>
export ORG_ID=<your_organization_id>
export SERVICE_ID=<your_service_id>

Get the service context:

curl -u "$KEY_ID:$KEY_SECRET" \
  "https://api.clickhouse.cloud/v1/organizations/$ORG_ID/services/$SERVICE_ID/clickpipes/context"

Once the identity is ready, the response includes the managed service account in principal:

{
  "result": {
    "gcpWorkloadIdentity": {
      "supported": true,
      "ready": true,
      "principal": "[e-mail]"
    }
  }
}

If the ClickHouse service has not used ClickPipes before, the response can initially report ready: false without a principal. Retry until ready is true and principal is populated.

See the service context API reference for the complete response schema.

Terraform

Use the clickhouse_clickpipes_service_context data source:

data "clickhouse_clickpipes_service_context" "service" {
  service_id    = var.service_id
  ready_timeout = "2m"
}

output "clickpipes_gcp_service_account" {
  value = data.clickhouse_clickpipes_service_context.service.gcp_workload_identity.principal
}

By default, the data source fails when workload identity is unsupported. It waits up to ready_timeout for the identity to become ready and for principal to be populated. Set wait_for_identity = false only when you need a snapshot of the current capability state without waiting.

The service context fields have these meanings:

Field	Meaning
supported	The ClickPipes deployment supports GCP workload identity.
ready	The managed identity and its impersonation configuration are ready. This field can be omitted when workload identity is unsupported.
principal	The Google service account email to authorize on your source resources. This field can be omitted until the identity is ready.

Grant access to the source

Use serviceAccount:<principal> as the IAM member, where <principal> is the email returned in the service context. Grant it only the roles required by your source, as described in the source’s permissions section:

• BigQuery
• Google Managed Service for Apache Kafka
• Google Cloud Storage
• Google Cloud Pub/Sub

ClickPipes cannot create IAM bindings in your Google Cloud project. Create the grants before creating the ClickPipe. When managing the grants and ClickPipe in the same Terraform configuration, use depends_on to enforce this ordering.

Create the ClickPipe

Set the source’s authentication method to SERVICE_ACCOUNT_WORKLOAD_IDENTITY. Do not configure a service account key, access key, connection string, or IAM role together with workload identity.

See the programmatic examples for each source:

• Google Managed Kafka with Terraform
• Google Cloud Storage with Terraform
• Google Cloud Pub/Sub with Terraform
• BigQuery with Terraform
• OpenAPI examples

Troubleshooting

Workload identity is not supported

Confirm that Private Preview access is enabled and that both the ClickHouse Cloud service and its ClickPipes deployment run on GCP. Contact ClickHouse Support if the service meets these requirements but supported is false.

The identity is not ready

Identity provisioning may still be in progress. Retry the service context request, or increase ready_timeout when using Terraform. Do not create the ClickPipe until ready is true and principal is populated.

The source rejects access

Confirm that the IAM member is serviceAccount:<principal> using the exact principal from the service context. Verify that roles are granted at the correct project, dataset, bucket, subscription, or cluster scope. For Google Managed Kafka, also check the broker-side Kafka ACLs and TLS configuration.