> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-vortex-format.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
> Integrate ClickHouse and Amazon Glue
# Integrating Amazon Glue with ClickHouse and Spark
export const ClickHouseSupportedBadge = () => {
return ;
};
export const Image = ({img, alt, size = "lg"}) => {
const normalizedSize = ["sm", "md", "lg"].includes(size) ? size : "lg";
return ;
};
[Amazon Glue](https://aws.amazon.com/glue/) is a fully managed, serverless data integration service provided by Amazon Web Services (AWS). It simplifies the process of discovering, preparing, and transforming data for analytics, machine learning, and application development.
Installation
To integrate your Glue code with ClickHouse, you can use our official Spark connector in Glue via one of the following:
* Installing the ClickHouse Glue connector from the AWS Marketplace (recommended).
* Manually adding the Spark Connector's jars to your Glue job.
To access the connector in your account, subscribe to the ClickHouse AWS Glue Connector from AWS Marketplace.
Ensure your Glue job’s IAM role has the necessary permissions, as described in the minimum privileges [guide](https://docs.aws.amazon.com/glue/latest/dg/getting-started-min-privs-job.html#getting-started-min-privs-connectors).
After subscribing, select the Glue version that matches your job requirements. In the **Additional details** section, under **Usage instructions**, click the link to **Open Glue Studio - Add ClickHouse connector**. This opens the Glue connection creation page with key fields pre-filled. Give the connection a name and press create (no need to provide the ClickHouse connection details at this stage).
In your Glue job, select the `Job details` tab, and expend the `Advanced properties` window. Under the `Connections` section, select the connection you just created. The connector automatically injects the required JARs into the job runtime.
Make sure to select the connector version that matches your Glue job configuration:
* **Glue 4**: Spark 3.3, Scala 2, Python 3
* **Glue 5**: Spark 3.5, Scala 2, Python 3
To add the required jars manually, please follow the following:
Upload the latest Spark connector JAR (`clickhouse-spark-runtime-3.X_2.X-0.10.X.jar`) to an S3 bucket.
Make sure the Glue job has access to this bucket.
Under the `Job details` tab, scroll down and expend the `Advanced properties` drop down, and fill the jars path in `Dependent JARs path`:
Using AWS Secrets Manager for credentials
Rather than hardcoding your ClickHouse user and password in the job, store them in [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/) and reference the secret from your Glue connection or job script. At runtime, Glue fetches the secret and merges its key-value pairs into the connector's connection options.
Create the secret
In AWS Secrets Manager, create a secret of type **Other type of secret** with key-value pairs whose keys match the connector's option names:
| Key | Value |
| ---------- | ------------------------ |
| `user` | your ClickHouse username |
| `password` | your ClickHouse password |
Any key you put in the secret is forwarded to the connector, so you can also store `host`, `database`, or any other option there if you'd like to keep them out of code.
Reference the secret
There are two ways to wire the secret into a job.
**Option 1: attach it to the Glue connection.** When creating or editing the ClickHouse connection in Glue Studio, set the **AWS secret** field to the secret's name. Any job that uses this connection resolves the secret automatically — no code changes needed.
**Option 2: pass `secretId` in connection options.** Use this when the secret isn't attached to the connection. Add `secretId` alongside `connectionName`:
```python theme={null}
source = glueContext.create_dynamic_frame.from_options(
connection_type="marketplace.spark",
connection_options={
"connectionName": "",
"secretId": "clickhouse/glue/credentials",
"database": "default",
"table": "example_table"
},
transformation_ctx="clickhouse_source"
)
```
```scala theme={null}
val source = glueContext.getSource(
connectionType = "marketplace.spark",
connectionOptions = JsonOptions(Map(
"connectionName" -> "",
"secretId" -> "clickhouse/glue/credentials",
"database" -> "default",
"table" -> "example_table"
)),
transformationContext = "clickhouseSource"
)
```
The secret's `user` and `password` keys are merged into the connector options at runtime, so you never need to read them in your script.
Examples
The examples below use `marketplace.spark` and reference the connector by `connectionName`. If you installed the connector manually (Manual Installation tab), use `connection_type="custom.spark"` and pass `className`, `host`, `http_port`, `user`, and `password` directly in the options instead.
If you attached an AWS secret to the connection itself (Option 1 in [Using AWS Secrets Manager for credentials](#secrets-manager)), drop `secretId` from the options — Glue resolves credentials from the connection automatically.
You can use the ClickHouse connector as either a source or a target in the Glue Studio visual editor. Simply drag the ClickHouse Spark Connector component onto the canvas and connect it to your data pipeline.
```java theme={null}
import com.amazonaws.services.glue.GlueContext
import com.amazonaws.services.glue.util.{GlueArgParser, Job, JsonOptions}
import org.apache.spark.SparkContext
import scala.collection.JavaConverters._
object ClickHouseGlueExample {
def main(sysArgs: Array[String]): Unit = {
val args = GlueArgParser.getResolvedOptions(sysArgs, Seq("JOB_NAME").toArray)
val sc = new SparkContext()
val glueContext = new GlueContext(sc)
Job.init(args("JOB_NAME"), glueContext, args.asJava)
val readOptions = JsonOptions(Map(
"connectionName" -> "",
"secretId" -> "clickhouse/glue/credentials",
"database" -> "default",
"table" -> "example_table"
))
val source = glueContext.getSource(
connectionType = "marketplace.spark",
connectionOptions = readOptions,
transformationContext = "clickhouseSource"
)
val dyf = source.getDynamicFrame()
val writeOptions = JsonOptions(Map(
"connectionName" -> "",
"secretId" -> "clickhouse/glue/credentials",
"database" -> "default",
"table" -> "target_table"
))
glueContext.getSink(
connectionType = "marketplace.spark",
connectionOptions = writeOptions
).writeDynamicFrame(dyf)
Job.commit()
}
}
```
```python theme={null}
import sys
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
logger = glueContext.get_logger()
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
read_options = {
"connectionName": "",
"secretId": "clickhouse/glue/credentials",
"database": "default",
"table": "example_table"
}
source = glueContext.create_dynamic_frame.from_options(
connection_type="marketplace.spark",
connection_options=read_options,
transformation_ctx="clickhouse_source"
)
dyf = source
logger.info(f"Read {dyf.count()} rows from ClickHouse")
write_options = {
"connectionName": "",
"secretId": "clickhouse/glue/credentials",
"database": "default",
"table": "target_table"
}
glueContext.write_dynamic_frame.from_options(
frame=dyf,
connection_type="marketplace.spark",
connection_options=write_options,
transformation_ctx="clickhouse_sink"
)
job.commit()
```
For more details, please visit our [Spark documentation](/integrations/apache-spark).