Using a connector with Helm-installed Kafka/Confluent

Ideally this should be configurable as part of the helm script, but unfortunately it is not as of now. One way to work around this is to build a new Docker from Confluent's Kafka Connect Docker image. Download the connector manually and extract the contents into a folder. Copy the contents of this to a path in the container. Something like below.

Contents of Dockerfile

FROM confluentinc/cp-kafka-connect:5.2.1
COPY <connector-directory> /usr/share/java

/usr/share/java is the default location where Kafka Connect looks for plugins. You could also use different location and provide the new location (plugin.path) during your helm installation.

Build this image and host it somewhere accessible. You will also have to provide/override the image and tag details during the helm installation.

Here is the path to the values.yaml file. You can find the image and plugin.path values here.

Just an add-on to Jegan's comment above: https://stackoverflow.com/a/56049585/6002912

You can choose to do the Dockerfile below. Recommended.

FROM confluentinc/cp-server-connect-operator:5.4.0.0

RUN confluent-hub install --no-prompt debezium/debezium-connector-postgresql:1.0.0

Or you can use a Docker's multi-stage build instead.

FROM confluentinc/cp-server-connect-operator:5.4.0.0

COPY --from=debezium/connect:1.0 \
    /kafka/connect/debezium-connector-postgres/ \
    /usr/share/confluent-hub-components/debezium-connector-postgres/

This will help you to save time on getting the right jar files for your plugins like debezium-connector-postgres.

From Confluent documentation: https://docs.confluent.io/current/connect/managing/extending.html#create-a-docker-image-containing-c-hub-connectors

The Kafka Connect pod should already have the confluent-hub installed. It is that pod you should run the commands on.