Apache Kafka on Docker – stops after few hundred pools

I have a problem with running Kafka that is used as event store in Java application.

  • Kafka version: 0.10.1.0
  • Java lib version: 0.10.1.0
  • Docker version: current edge (17.06.0-ce, build 02c1d87) (same problem on stable release)
  • Java 1.8
  • Docker compose version: 3.3

Kafka stops suddenly after 147 requests. Any changes to configuration, docker setup, host machine etc do not result in different outcome.

  • Dockerfile or Registry? Which is the preferred strategy for distribution?
  • Docker installation failed
  • How to configure xvfb default installation on Jenkins startup?
  • What is the significance of the user in `/etc/sub{u,g}id`? (Help me grok user namespaces)
  • Where is the Docker JSON file logging driver writing files to?
  • Gradle Docker plugin connection timed out
  • Docker compose file:

    version: '3.3'
    services:
      queue:
        image: spotify/kafka
        ports:
          - "9092:9092"
          - "2181:2181"
        environment:
          - ADVERTISED_HOST=queue
          - ADVERTISED_PORT=9092
    #  kafka-manager:
    #    image: sheepkiller/kafka-manager
    #    ports:
    #      - "9000:9000"
    #    environment:
    #      - ZK_HOSTS=queue:2181
    #    depends_on:
    #      - queue
    #    links:
    #      - queue
      postgresql:
          image: postgres:9.4
          ports:
            - "5432:5432"
          environment:
            - POSTGRES_USER=db_user
            - POSTGRES_PASSWORD=db_pass
    

    To investigate this issue we have created simple test (problem started occurring in main application). Here is the setup:

    @Test
        public void testKafkaPublisherWithEventStore() {
            String topicId = UUID.randomUUID().toString();
            System.out.println("Topic ID: " + topicId);
            System.out.println("Entity ID: " + this.aggregateEntity.identity());
            KafkaPublisher publisher = new KafkaPublisher(
                    "generalTestTopic_" + topicId,
                    new SubscriberPriorityQueue(new SubscriberPriorityQueueComparator()),
                    this.props,
                    new TestEventConverter()
            );
            //Attach subscribers
            this.subscriberQueue = new SubscriberPriorityQueue(new SubscriberPriorityQueueComparator());
            TestEventStore eventStore = new TestEventStore(TestEntity.class, this.props, new TestEventConverter());
            this.subscriberQueue.add(new GeneralSubscriberQueueElement(eventStore, 1));
            this.subscriberQueue.forEach(publisher::attachSubscriber);
            int eventsCount = 0;
            int publishEventsTimes = 300;
            try {
                //Make sure listeners attached before publishing events
                Thread.sleep(3000);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
            System.out.println("Publishing new events");
            for (int j = 0; j < publishEventsTimes; j++) {
                eventsCount++;
                publisher.publish(new TestEvent1(this.aggregateEntity.identity(), 1, new Date()));
                eventsCount++;
                publisher.publish(new TestEvent2(this.aggregateEntity.identity(), 2, new Date()));
            }
            try {
                int counter = 0;
                int eventContainerSize = 0;
                do {
                    counter++;
                    Thread.sleep(100);
                    eventContainerSize = eventStore.get(this.aggregateEntity.identity()).events().size();
                    System.out.println("Event store caught " + eventContainerSize + " events after " + counter + " tries");
                } while(eventContainerSize < eventsCount);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
            assertEquals(eventsCount, eventStore.get(this.aggregateEntity.identity()).events().size());
        }
    

    On local setup, the problem occurs much later due to to many open files error, but setting even ridiculously high limits doesn’t help on docker.

    I don’t know what other details will be useful.

  • How to run a Redis server AND another application inside Docker?
  • random error in docker when mounting a volume directory.
  • Regarding docker daemon to spawn containers in a custom CIDR block
  • Liberty Docker Image is not running
  • Error setting up docker on Windows
  • Authorization error with AWS
  • Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.