> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-vortex-format.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# BYOC observability

> Monitor and observe your BYOC ClickHouse deployment using built-in dashboards and Prometheus metrics

export const Image = ({img, alt, size = "lg"}) => {
  const normalizedSize = ["sm", "md", "lg"].includes(size) ? size : "lg";
  return <div className={`ch-image-${normalizedSize}`}>
      <Frame>
        <img src={img} alt={alt} />
      </Frame>
    </div>;
};

BYOC deployments include comprehensive observability capabilities, allowing you to monitor your ClickHouse services through a dedicated Prometheus monitoring stack, and direct metric endpoints from ClickHouse Servers. All observability data remains within your cloud account, giving you complete control over your monitoring infrastructure.

<h2 id="prometheus-monitoring">
  Prometheus Monitoring Approaches
</h2>

BYOC offers two main ways to collect and visualize metrics using Prometheus:

1. **Connect to the Built-In Prometheus Stack**: Access the centralized, pre-installed Prometheus instance running inside your BYOC Kubernetes cluster.
2. **Scrape ClickHouse Metrics Directly**: Point your own Prometheus deployment to the `/metrics_all` endpoint exposed by each ClickHouse service.

<h3 id="monitoring-approaches-comparison">
  Comparing Monitoring Methods
</h3>

| Capability                | Built-In Prometheus Stack                                                                           | Direct Scraping from ClickHouse Services                                        |
| ------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| **Metrics Scope**         | Consolidates metrics from ClickHouse, Kubernetes, and supporting services (full cluster visibility) | Metrics from individual ClickHouse servers only                                 |
| **Setup Process**         | Requires setting up private network access (e.g., via private load balancer)                        | Simply configure Prometheus to scrape the public or private ClickHouse endpoint |
| **How You Connect**       | Through the private load balancer within your VPC/network                                           | The same endpoint you use for database access                                   |
| **Authentication**        | None needed (private-network-restricted)                                                            | Uses ClickHouse service credentials                                             |
| **Network Prerequisites** | Private load balancer and appropriate network connectivity                                          | Available to any network with access to your ClickHouse endpoint                |
| **Best Suited For**       | Holistic infrastructure & service monitoring                                                        | Service-specific monitoring and integration                                     |
| **How to Integrate**      | Configure federation in external Prometheus to ingest cluster metrics                               | Add ClickHouse metric endpoints directly to your Prometheus config              |

**Recommendation**: For most use cases, we recommend integrating with the built-in Prometheus stack, as it provides comprehensive metrics from all components in your BYOC deployment (ClickHouse services, Kubernetes cluster, and supporting services) rather than just ClickHouse server metrics alone.

<h2 id="builtin-prometheus-stack">
  The Built-in BYOC Prometheus Stack
</h2>

ClickHouse BYOC deploys a complete Prometheus monitoring stack within your Kubernetes cluster, including Prometheus, Grafana, AlertManager, and optionally Thanos for long-term metric storage. This stack collects metrics from:

* ClickHouse servers and ClickHouse Keeper
* Kubernetes cluster and system components
* Underlying infrastructure nodes

<h3 id="accessing-prometheus-stack">
  Accessing the Prometheus Stack
</h3>

To connect to the built-in Prometheus stack:

1. **Contact ClickHouse Support** to enable the private load balancer for your BYOC environment.
2. **Request the Prometheus endpoint URL** from ClickHouse Support.
3. **Verify private network connectivity** to the Prometheus endpoint—typically via VPC/VNet peering or other private network setup.

Endpoint formats vary by connectivity type:

| Connectivity     | Endpoint format                                                                |
| ---------------- | ------------------------------------------------------------------------------ |
| VPC/VNet peering | `https://prometheus-internal.<subdomain>.<region>.<cloud>.clickhouse-byoc.com` |
| Private endpoint | `https://prometheus.<label>.<subdomain>.<region>.<cloud>.clickhouse-byoc.com`  |

`<cloud>` is `aws`, `gcp`, or `azure`. For private endpoints, `<label>` is cloud-specific: `vpce` on AWS, `p` on GCP, and `privatelink` on Azure.

<Note>
  The Prometheus stack URL is only accessible via private network connections and doesn't require authentication. Access is restricted to networks that can reach your BYOC VPC/VNet through peering or other private connectivity options.
</Note>

<h3 id="prometheus-stack-integration">
  Integrating with Your Monitoring Tools
</h3>

You can utilize the BYOC Prometheus stack in your monitoring ecosystem in several ways:

**Option 1: Query the Prometheus API**

* Access the Prometheus API endpoint directly from your preferred monitoring platform or custom dashboards.
* Use PromQL queries to extract, aggregate, and visualize the metrics you need.
* Ideal for building bespoke dashboards or alerting pipelines.

Prometheus query endpoint `/query`:

```text theme={null}
https://prometheus-internal.<subdomain>.<region>.<cloud>.clickhouse-byoc.com/query
```

**Option 2: Federate Metrics to Your Own Prometheus**

* Configure your external Prometheus instance to federate (pull) metrics from the ClickHouse BYOC Prometheus stack.
* This enables you to unify and centralize metrics collection from multiple environments or clusters.
* Example Prometheus federation configuration:

```yaml theme={null}
scrape_configs:
  - job_name: 'federate-clickhouse-byoc'
    scrape_interval: 15s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job="clickhouse"}'
        - '{job="kubernetes"}'
    static_configs:
      - targets:
        - 'prometheus-internal.<subdomain>.<region>.<cloud>.clickhouse-byoc.com'
```

<h2 id="direct-prometheus-integration">
  ClickHouse service Prometheus Integration
</h2>

ClickHouse services expose a Prometheus-compatible metrics endpoint that you can scrape directly using your own Prometheus instance. This approach provides ClickHouse-specific metrics but doesn't include Kubernetes or supporting service metrics.

<h3 id="metrics-endpoint">
  Accessing the Metrics Endpoint
</h3>

The metrics endpoint is available at `/metrics_all` on your ClickHouse service endpoint:

```bash theme={null}
curl --user <username>:<password> https://<service-subdomain>.<byoc-subdomain>.<region>.<provider>.clickhouse-byoc.com:8443/metrics_all
```

**Sample Response:**

```bash theme={null}
# HELP ClickHouse_CustomMetric_StorageSystemTablesS3DiskBytes The amount of bytes stored on disk `s3disk` in system database
# TYPE ClickHouse_CustomMetric_StorageSystemTablesS3DiskBytes gauge
ClickHouse_CustomMetric_StorageSystemTablesS3DiskBytes{hostname="c-jet-ax-16-server-43d5baj-0"} 62660929
# HELP ClickHouse_CustomMetric_NumberOfBrokenDetachedParts The number of broken detached parts
# TYPE ClickHouse_CustomMetric_NumberOfBrokenDetachedParts gauge
ClickHouse_CustomMetric_NumberOfBrokenDetachedParts{hostname="c-jet-ax-16-server-43d5baj-0"} 0
# HELP ClickHouse_CustomMetric_TotalNumberOfErrors The total number of errors on server since the last restart
# TYPE ClickHouse_CustomMetric_TotalNumberOfErrors gauge
ClickHouse_CustomMetric_TotalNumberOfErrors{hostname="c-jet-ax-16-server-43d5baj-0"} 9
```

<h3 id="authentication">
  Authentication
</h3>

The metrics endpoint requires authentication using ClickHouse credentials. We recommend use `default` user or creating a dedicated user with minimal permissions specifically for metric scraping.

**Required Permissions:**

* `REMOTE` permission to connect to the service
* `SELECT` permissions on relevant system tables

**Example User Setup:**

```sql theme={null}
CREATE USER scrapping_user IDENTIFIED BY 'secure_password';
GRANT REMOTE ON *.* TO scrapping_user;
GRANT SELECT ON system._custom_metrics_dictionary_custom_metrics_tables TO scrapping_user;
GRANT SELECT ON system._custom_metrics_dictionary_database_replicated_recovery_time TO scrapping_user;
GRANT SELECT ON system._custom_metrics_dictionary_failed_mutations TO scrapping_user;
GRANT SELECT ON system._custom_metrics_dictionary_group TO scrapping_user;
GRANT SELECT ON system._custom_metrics_dictionary_shared_catalog_recovery_time TO scrapping_user;
GRANT SELECT ON system._custom_metrics_dictionary_table_read_only_duration_seconds TO scrapping_user;
GRANT SELECT ON system._custom_metrics_view_error_metrics TO scrapping_user;
GRANT SELECT ON system._custom_metrics_view_histograms TO scrapping_user;
GRANT SELECT ON system._custom_metrics_view_metrics_and_events TO scrapping_user;
GRANT SELECT(description, metric, value) ON system.asynchronous_metrics TO scrapping_user;
GRANT SELECT ON system.custom_metrics TO scrapping_user;
GRANT SELECT(name, value) ON system.errors TO scrapping_user;
GRANT SELECT(description, event, value) ON system.events TO scrapping_user;
GRANT SELECT(description, labels, metric, value) ON system.histogram_metrics TO scrapping_user;
GRANT SELECT(description, metric, value) ON system.metrics TO scrapping_user;
```

<h3 id="configuring-prometheus">
  Configuring Prometheus
</h3>

Configure your Prometheus instance to scrape the ClickHouse metrics endpoint:

```yaml theme={null}
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "clickhouse"
    static_configs:
      - targets: ["<service-subdomain>.<byoc-subdomain>.<region>.<provider>.clickhouse-byoc.com:8443"]
    scheme: https
    metrics_path: "/metrics_all"
    basic_auth:
      username: <username>
      password: <password>
    honor_labels: true
```

Replace:

* `<service-subdomain>.<byoc-subdomain>.<region>.<provider>.clickhouse-byoc.com:8443` with your actual service endpoint
* `<username>` and `<password>` with your scraping user credentials

<h2 id="clickhouse-mixin">
  ClickHouse Mixin
</h2>

For teams that want a ready-made set of dashboards, ClickHouse provides a Prometheus **ClickHouse Mixin**. This is a pre-built  Grafana dashboard designed specifically for monitoring ClickHouse clusters.

<h3 id="setup-grafana-mixin">
  Setting up Grafana & Importing the ClickHouse Mix-in
</h3>

Once your Prometheus instance is integrated with your ClickHouse monitoring stack, you can visualize metrics in Grafana by following these steps:

1. **Add Prometheus as a Data Source in Grafana**\
   Go to "Data sources" in the Grafana sidebar, click "Add data source," and select "Prometheus." Enter your Prometheus instance URL and any required credentials to connect.

<Image img="https://mintcdn.com/private-7c7dfe99-vortex-format/N5J92FpLvYKK3vEv/images/cloud/reference/byoc-mixin-1.webp?fit=max&auto=format&n=N5J92FpLvYKK3vEv&q=85&s=2153aff596ba7d3f8b9e466e32766e4e" size="lg" alt="BYOC Mixin 1" background="black" width="3928" height="1310" data-path="images/cloud/reference/byoc-mixin-1.webp" />

<Image img="https://mintcdn.com/private-7c7dfe99-vortex-format/N5J92FpLvYKK3vEv/images/cloud/reference/byoc-mixin-2.webp?fit=max&auto=format&n=N5J92FpLvYKK3vEv&q=85&s=013d7d12025c9f2aa911b3390800dc03" size="lg" alt="BYOC Mixin 2" background="black" width="5342" height="832" data-path="images/cloud/reference/byoc-mixin-2.webp" />

<Image img="https://mintcdn.com/private-7c7dfe99-vortex-format/N5J92FpLvYKK3vEv/images/cloud/reference/byoc-mixin-3.webp?fit=max&auto=format&n=N5J92FpLvYKK3vEv&q=85&s=52cc246e174e8785612c4455791540ae" size="lg" alt="BYOC Mixin 3" background="black" width="1620" height="350" data-path="images/cloud/reference/byoc-mixin-3.webp" />

2. **Import the ClickHouse Dashboard**\
   In Grafana, navigate to the dashboard area and choose "Import." You can either upload the dashboard JSON file or paste its contents directly. Obtain the JSON file from the ClickHouse mixin repository:\
   [ClickHouse Mix-in Dashboard JSON](https://github.com/ClickHouse/clickhouse-mixin/blob/main/dashboard_byoc.json)

<Image img="https://mintcdn.com/private-7c7dfe99-vortex-format/N5J92FpLvYKK3vEv/images/cloud/reference/byoc-mixin-4.webp?fit=max&auto=format&n=N5J92FpLvYKK3vEv&q=85&s=3449bf50c6bf0636d5636a3499024836" size="lg" alt="BYOC Mixin 4" background="black" width="2490" height="1574" data-path="images/cloud/reference/byoc-mixin-4.webp" />

3. **Explore Your Metrics**\
   Once the dashboard is imported and configured with your Prometheus data source, you should see real-time metrics from your ClickHouse Cloud services.

<Image img="https://mintcdn.com/private-7c7dfe99-vortex-format/N5J92FpLvYKK3vEv/images/cloud/reference/byoc-mixin-5.webp?fit=max&auto=format&n=N5J92FpLvYKK3vEv&q=85&s=371a2e72c726ad4ba69a68e3cb535288" size="lg" alt="BYOC Mixin 5" background="black" width="5336" height="2932" data-path="images/cloud/reference/byoc-mixin-5.webp" />