Managed services

Create, watch, and retire ClickHouse services that ClickHouse Cloud operates in your cluster through the connector executor

In managed mode the connector runs a third component, the executor. ClickHouse Cloud operates ClickHouse services in your Kubernetes cluster through it. This page covers creating a service, reading its status, what ClickHouse Cloud does with it, and how it is retired.

What managed mode does

The executor is a daemon in the same clicklink binary as the scraper and the troubleshooter. It holds an outbound WebSocket channel to your connector endpoint and receives lifecycle commands from ClickHouse Cloud. It applies each command to the one Kubernetes cluster it is configured for and reports the outcome over that channel. Like the other components it makes only outbound connections: ClickHouse Cloud never connects into your cluster, and the executor’s local API binds to loopback.

Managed mode is available on Amazon EKS with S3 storage during the private preview. Your account team enables it when registering your environment. init refuses a kube context that does not point at an EKS cluster.

Each service needs three cloud resources you own: a data bucket, a backup bucket, and an IAM role the ClickHouse pods assume to reach them. You create them with your own credentials before the service exists and remove them after it is gone. The executor holds no credentials for your buckets or IAM and never deletes data. Its only cloud calls go to Amazon ECR and STS. It logs in to the registry when a platform sync or create pulls a chart, and checks the images under the read-only pull role during a platform update. It makes no S3 or IAM calls. For each service it creates a Kubernetes Service of type LoadBalancer, which your cluster’s load-balancer controller realizes as an internal NLB in your account.

On a VM the executor runs as the clicklink-executor systemd unit next to the other two. On Kubernetes it is a single-replica Deployment in the connector namespace. It serves health and metrics on port 8086 and its local API on 127.0.0.1:9999.

Enable managed mode

You choose managed mode when you enroll: pass --managed to clicklink clctl init, or answer the prompt on a terminal. On a VM, init takes the cluster from this host’s kubeconfig (--cluster-name when it holds more than one EKS cluster) and grants the executor’s cluster-wide access inline. On Kubernetes, pass --egress-cidrs with the CIDRs of your connector endpoint so the chart stages its default-deny NetworkPolicy enabled.

The install itself is unchanged. See onboarding for the full flow and the CLI reference for the flags.

Create a service

Service creation through the connector endpoint is enabled per environment; check with your account team before you start.

Creating a service takes two commands. prepare creates everything on your side; instances create submits the create to ClickHouse Cloud through your connector endpoint. ClickHouse Cloud renders the service definition and sends the create to the executor over its outbound channel. The executor applies it to your cluster and reports back.

Only prepare needs AWS credentials: it creates the S3 buckets and the IAM role. instances create needs the connector’s configuration and credentials, and reaches the executor’s local API only for --wait. On a VM, run both as root on the connector host. On Kubernetes, run both from a workstation with a kube context for the managed cluster:

• prepare needs a copy of the connector configuration (--config), a kubectl port-forward to the executor’s local API, and a writable --output-dir. It checks the service name against the executor and refuses to run when it cannot reach it.
• When this host lacks the credential files the configuration names, instances create reads them from the clicklink-hmac and clicklink-mtls Secrets and announces each read. Add --connector-namespace when the connector namespace is not clicklink. Only instances create has this fallback.

1. Prepare the service

   Kubernetes

   CONNECTOR_NAMESPACE='clicklink'   # the connector namespace you chose at init
   kubectl -n "${CONNECTOR_NAMESPACE}" get configmap clicklink-connector-executor \
     -o jsonpath='{.data.config\.yaml}' > clicklink-config.yaml
   kubectl -n "${CONNECTOR_NAMESPACE}" port-forward deployment/clicklink-connector-executor 9999:9999 &
   clicklink clctl executor prepare --config clicklink-config.yaml --output-dir ./clicklink-access
   

   Keep the output directory: it holds the create body and the name record that instances create and any retry read.

   Linux VM

   sudo clicklink clctl executor prepare --config /etc/clicklink/config.yaml
   

   The command runs four steps in order and stops at the first failure:

   1. Name. Picks a service name, or validates the one you pass with --instance <name>. A generated name is recorded in <output-dir>/_prepare/<eks-cluster-name>.name and resumed by the next run, so a retry reuses the first run’s buckets and role. --new-name picks another. The command refuses a name the executor still holds, or whose namespace already contains a ClickHouse cluster.
   2. Storage. Creates the data and backup buckets and the IAM role CH-S3-<name>-<region>-00-Role. The default bucket names are <cluster>-clickhouse-data-<rand> and <cluster>-clickhouse-backup-<rand>; --data-bucket and --backup-bucket override them. With --role-arn it verifies a role you bring and writes no IAM.
   3. Grant and apply. On a VM, renders the executor’s access bundle for the service namespace (ns-<name>), applies its RBAC, and registers the service in the executor’s registry. On Kubernetes this step is skipped: the executor runs as its pod ServiceAccount and registers the service itself when the create arrives.
   4. Summary. Mints the default user’s password and writes the create body to <output-dir>/_prepare/<name>.create.json (mode 0600; it carries only the password’s hashes). Prints the next command on a Next: line.

   prepare prints the default user’s password once, on stderr. Neither the create body nor --output json contains it, and ClickHouse Cloud receives only its hashes. Store it before moving on. A re-run that finds the create body keeps the hashes already written, and says so.

   Pass --context <name> when your kubeconfig holds several contexts; the context must point at the EKS cluster named in executor.cluster in the config. --dry-run runs every step without creating or writing anything. The hashing step uses SHA-1, which Go refuses under GODEBUG=fips140=only; run prepare on a host without that setting.

2. Submit the create

   Kubernetes

   clicklink clctl instances create \
     --from-prepare ./clicklink-access/_prepare/<name>.create.json \
     --config clicklink-config.yaml \
     --wait
   

   Leave the port-forward from prepare open: --wait polls the executor through it.

   Linux VM

   sudo clicklink clctl instances create \
     --from-prepare /etc/clicklink/access/executor/_prepare/<name>.create.json \
     --config /etc/clicklink/config.yaml \
     --wait
   

   The command posts the prepared body to your connector endpoint with the connector’s own credentials. It prints created <spoken-name> (state provisioning) and a watch: hint. <spoken-name> is the name ClickHouse Cloud assigned (for example amberaws-kq-42), not the service name you prepared. The watch: hint and every clctl command use your service name.

   A retry with the same inputs is safe. The idempotency key is derived by default from your environment and the service name, so resubmitting returns the first create.

   --wait polls the executor’s local API every 10 seconds until the service is running, for up to --wait-timeout (default 30m). It fails fast, with the recorded error, when the executor records a failed create for the name or the service turns terminating, terminated, or stale.

3. Verify

   clicklink clctl instances get --name <name> --cluster <eks-cluster-name>
   

   The service is ready when its status is running. Its default user takes the password prepare printed. --cluster can also come from the CLCTL_CLUSTER environment variable.

Status

The executor answers status questions from its local API, which binds to 127.0.0.1:9999 and carries no authentication of its own. On a VM, run the commands on the host. On Kubernetes, open a port-forward first and point the commands at it:

kubectl -n <connector-namespace> port-forward deployment/clicklink-connector-executor 9999:9999
clicklink clctl instances list --endpoint http://127.0.0.1:9999

clicklink clctl instances list prints every service the executor knows as JSON. clicklink clctl instances get --name <name> --cluster <eks-cluster-name> prints one, with the storage it was created with. The executor derives status from the service’s ClickHouseCluster resource (ready versus expected server replicas) and refreshes it every sync_interval (30 seconds by default):

• provisioning: no server replica is ready yet, or the ClickHouseCluster does not exist yet
• running: every expected server replica is ready
• degraded: some, but not all, server replicas are ready
• terminating: the executor is uninstalling the service
• terminated: the service’s namespace is gone
• stale: the service was dropped from the executor’s registry without a delete; --wait and teardown treat it like terminated

A terminated service stays listed, with its storage, until teardown removes its cloud resources.

The executor records every command ClickHouse Cloud sends it. clicklink clctl commands list prints them, filtered with --status (pending, running, completed, failed), --action (for example create_instance), or --cluster. clicklink clctl commands get <id> prints one command with its stage and result. A failed command’s result holds the error: why a create did not converge, or which platform update is waiting for your approval.

Service lifecycle

Once a service exists, ClickHouse Cloud operates it through the executor. The commands it sends are:

• Scale. ClickHouse Cloud sets a fixed replica count, capped by a ClickHouse Cloud setting (20 in the default configuration). There is no autoscaling.
• Stop and start. Stopping scales the servers to zero and keeps Keeper; starting restores the replica counts. Data stays in your buckets throughout.
• Restart. The whole service, its Keeper, or a single pod.
• Backups. ClickHouse Cloud triggers them; they land in your backup bucket, and backup deletion removes them.
• Version upgrades and configuration changes. ClickHouse Cloud re-renders the service definition with the new version or setting. It arrives as a create_instance command, so commands list --action create_instance also shows upgrades. The executor applies it and waits for the replicas to become ready again.
• Delete. Described in delete a service.

Version upgrades and configuration changes have no customer approval step. ClickHouse Cloud applies them to a managed service the same way it applies a scale or a restart: it re-sends the definition and the executor converges the cluster on it.

Creates, scales, and starts complete asynchronously. The executor reports the command as running as soon as it applies the definition and sends a progress report every 2 minutes. It reports the final outcome once the server replicas are ready. If they are not ready within 2 hours it reports the command as failed; if they come up later, it moves the service back to running.

The instances scale, instances patch, and instances delete subcommands post commands straight to the executor’s local API, bypassing ClickHouse Cloud. Run them only when your account team asks; the CLI reference describes each.

Support sessions for a managed service

The executor wires the scraper into each service it creates. It does not provision the troubleshooter, so a support session cannot run diagnostics on a managed service until you do. Provision it once per service, with the service namespace ns-<name>, the same way as for an instance you registered yourself:

Kubernetes

printf '%s\n' "$CH_DEFAULT_PASSWORD" | clicklink clctl troubleshoot access provision --target helm \
  --target-namespace "${CONNECTOR_NAMESPACE}" \
  --instance <name> --instance-namespace ns-<name> \
  --server <kubernetes-api-server-url> --ch-admin-password-stdin \
  --apply-ch-grants --ch-pod <clickhouse-pod-or-label-selector> --ch-pod-namespace ns-<name>

$CH_DEFAULT_PASSWORD is the default user’s password that prepare printed. The command authenticates with it to apply the SQL grants and does not prompt for it. Then add the Secret and ServiceAccount pair to troubleshooter.accessBundles and run helm upgrade, as shown under adding ClickHouse instances.

Linux VM

printf '%s\n' "$CH_DEFAULT_PASSWORD" | sudo clicklink clctl troubleshoot access provision --provider local \
  --instance <name> --instance-namespace ns-<name> \
  --server <kubernetes-api-server-url> --ch-admin-password-stdin

$CH_DEFAULT_PASSWORD is the default user’s password that prepare printed.

Grant the ClickHouse user with SQL, as above. A user added with --ch-user-via cr would not survive: ClickHouse Cloud owns the service definition and re-applies it. When the service is deleted, the executor removes the troubleshooter’s bundle on a VM. On Kubernetes the bundle Secret and ServiceAccount stay until you delete them, as listed under uninstall.

Delete a service

There is no customer delete command through your connector endpoint: ask your account team to delete the service. ClickHouse Cloud terminates it, and the executor deletes the workload and its namespace (terminating, then terminated). Nothing in AWS is touched: the buckets, their data, and the IAM role remain until you remove them.

Once the service reports terminated, tear down what prepare created. Run teardown where you ran prepare, with the same AWS credentials. On Kubernetes that means the same workstation, the same configuration copy and output directory, and an open port-forward to the executor:

Kubernetes

kubectl -n "${CONNECTOR_NAMESPACE}" port-forward deployment/clicklink-connector-executor 9999:9999 &
clicklink clctl executor teardown --instance <name> --config clicklink-config.yaml --output-dir ./clicklink-access

Linux VM

sudo clicklink clctl executor teardown --instance <name>

The command reads the executor’s record of the service: where its data lives and which role it had. It deletes the IAM role that prepare created and makes the executor forget the service, which frees the name. On a VM it also removes the service’s cluster-wide RBAC objects, the local access bundle, and the registry entry.

By default it keeps the service’s data and backups. It re-tags a kept bucket from clicklink:deployed-name to clicklink:retained-from=<name>, so a re-created service with the same name never inherits it, and the summary says where the data is. To delete them, add --delete-data --delete-backups --yes to the same command.

--delete-data and --delete-backups empty and delete the buckets they name; there is no recovery afterwards. Confirm the bucket names in the record step before adding --yes.

Without --yes the run stops after the record step and names the buckets it would empty. --dry-run reads everything and writes nothing. A role you brought with --role-arn, or a bucket prepare did not create, is reported as kept and never touched. The run refuses while the service’s namespace still holds a ClickHouse cluster. It reads everything before it deletes anything, so a refused run changes nothing.

When the connector is offline

Running services do not depend on the executor. The ClickHouse operator in your cluster keeps them running, and a disconnected connector interrupts nothing that is already serving queries.

While no executor is connected, ClickHouse Cloud cannot hand it new work. A create, delete, or platform sync it has already accepted is kept and retried. A create is retried for 30 minutes from its last progress report, a delete for 2 hours, a platform sync for up to 10 attempts. Past that budget your connector endpoint marks the command failed. Every other lifecycle command (scale, stop, start, restart, backup, backup deletion) is refused rather than queued. A create or delete you submit while no executor is connected is refused the same way; only commands already accepted are retried.

A command the executor has already accepted runs to completion; a result it could not report is sent on its next connection.

instances create needs your connector endpoint reachable from the host it runs on. --wait, instances list, instances get, and commands list need the executor’s local API, so the executor must be running. A platform approval’s token expires on its own clock, regardless of connectivity; see the approval window.