Amazon Kinesis + Integration Tests

It might already be too late to give the solution but I will add what my team has done to replicate AWS resources locally as we use a lot of Kinesis, DynamoDb, S3 and cloudWatch.

We have created wrappers around Localstack -> https://github.com/localstack/localstack that allow us to spin up local instances of the necessary services as docker containers using docker-compose.

A typical docker-compose.yml file for us looks like:

version: '2'
services:
  localstack:
    image: "localstack/localstack"
    environment:
      - SERVICES=kinesis,dynamodb,cloudwatch
    ports:
      - "4568"
      - "4569"
      - "4582"

Then during the setup phase for the integration-tests, our wrapper fires up docker-compose up and runs the tests against the local infrastructure. Later during tear-down, the wrapper kills the containers.

I ran into the same issue and the only mock implementation I found so far was a nodejs one: https://github.com/mhart/kinesalite It does the trick - I managed to run my Java Kinesis client against it, just had to set the endpoint on the kinesis.properties:

kinesisEndpoint=http://localhost:4567

The downside is that it is not trivial to use it during build time tests - need to figure a way to start the mock kinesis before the test (using a maven plugin or something), didn't get to it yet..