Run Kafka Connect on Kubernetes

How to run Kafka Connect Worker on Kubernetes

Download the Kafka Source and Sink Connectors to plugins folder. Note: Some of the plugins are not in GZ format. You can convert them into GZ format so that they can be used with the Docker Add command easily.

Build docker image

docker build -t my-kafka-connect:0131v1 .

Run Kafka Connect Locally
Gather all variables noted in the configmap.yaml below and all kubernetes secrets such as keystore passwords, and create a file kc.env with key=value pairs.

For Kafka Connect Worker to connect to Kafka Broker, you need JKS files to establish SSL Communication. Have consumer-client.jks, producer-client.jks and composer-client.jks along with the client-truststore.jks files in jks folder.

docker run -p 8083:8083 — name kc-worker -v C:/Users/xyz/dev/wkspc/my-kafka-connect/jks:/etc/config/my-kc — env-file kc-dev.env my-kafka-connect:0131v1

Run Kafka Connect Worker on Kubernetes cluster
Create kubernetes secret containing the JKS files and volume mount them to etc/config/my-kc/ as shown in configmap.yaml and deployment.yaml files below.

Update Liveness and Readiness Probes to check on Port 8083.

By default, Kafka Connect runs on port 8083. Expose container port 8083 on host port 80.

This will configure Kafka Connect Worker on your Kubernetes installation whether using Konvoy or OpenShift.

You should be able to hit your ingress endpoint and verify your configuration.

https://my-kafka-connect.dev.k8s.xyz.com
https://my-kafka-connect.dev.k8s.xyz.com/connector-plugins
https://my-kafka-connect.dev.k8s.xyz.com/connectors