How to connect Apache Kafka with Amazon S3?
You can use Kafka Connect to do this integration, with the Kafka Connect S3 connector.
Kafka Connect is part of Apache Kafka, and the S3 connector is an open-source connector available either standalone or as part of Confluent Platform.
For general information and examples of Kafka Connect, this series of articles might help:
- https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-1/
- https://www.confluent.io/blog/blogthe-simplest-useful-kafka-connect-data-pipeline-in-the-world-or-thereabouts-part-2/
- https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-3/
Disclaimer: I work for Confluent, and wrote the above blog articles.
April 2020: I have recorded a video showing how to use the S3 sink: https://rmoff.dev/kafka-s3-video