LogStash container restarts on Kube pod

I have a Kubernetes cluster running on GCE, one of my PODs has two containers:

  • my app
  • Logstash

For some reason the container of LogStash inside the POD keeps restarting, see the result of “describe pod”:

  • Apache Access/Error Logs Not Populated in Docker Container with Supervisor D
  • Kafka in Docker not working
  • Rename file in docker container
  • How do I setup a docker image to dynamically pull app code from a repository?
  • Docker build MySQL dockerfile occur none:none
  • Docker not running on windows 10 error: Hardware assisted virtualization and data execution protection must be enabled in the BIOS
  •   logstash:
        Image:  logstash:1.5.6
        Limits:
          cpu:      500m
          memory:       2Gi
        State:      Running
          Started:      Fri, 08 Apr 2016 08:28:53 +0000
        Ready:      False
        Restart Count:  12
    Conditions:
      Type      Status
      Ready     False
    Events:
      FirstSeen             LastSeen            Count   From                    SubobjectPath               Reason      Message
      Fri, 08 Apr 2016 07:34:53 +0000   Fri, 08 Apr 2016 07:34:53 +0000 1   {kubelet kubernetes-minion-1m8z}    implicitly required container POD   pulled      Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
      Fri, 08 Apr 2016 07:34:53 +0000   Fri, 08 Apr 2016 07:34:53 +0000 1   {scheduler }                                    scheduled   Successfully assigned to kubernetes-minion-1m8z
      Fri, 08 Apr 2016 07:34:54 +0000   Fri, 08 Apr 2016 07:34:54 +0000 1   {kubelet kubernetes-minion-1m8z}    implicitly required container POD   created     Created with docker id f916e5aee8c7
      Fri, 08 Apr 2016 07:34:54 +0000   Fri, 08 Apr 2016 07:34:54 +0000 1   {kubelet kubernetes-minion-1m8z}    implicitly required container POD   started     Started with docker id f916e5aee8c7
      Fri, 08 Apr 2016 07:35:24 +0000   Fri, 08 Apr 2016 07:35:24 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       pulled      Successfully pulled image "logstash:1.5.6"
      Fri, 08 Apr 2016 07:35:25 +0000   Fri, 08 Apr 2016 07:35:25 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       created     Created with docker id 059218faf86d
      Fri, 08 Apr 2016 07:35:25 +0000   Fri, 08 Apr 2016 07:35:25 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       started     Started with docker id 059218faf86d
      Fri, 08 Apr 2016 07:35:34 +0000   Fri, 08 Apr 2016 07:35:34 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       created     Created with docker id 44e0800b2992
      Fri, 08 Apr 2016 07:35:34 +0000   Fri, 08 Apr 2016 07:35:34 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       started     Started with docker id 44e0800b2992
      Fri, 08 Apr 2016 07:35:44 +0000   Fri, 08 Apr 2016 07:35:44 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       created     Created with docker id 32df246c83dc
      Fri, 08 Apr 2016 07:35:44 +0000   Fri, 08 Apr 2016 07:35:44 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       started     Started with docker id 32df246c83dc
      Fri, 08 Apr 2016 07:35:54 +0000   Fri, 08 Apr 2016 07:35:54 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       created     Created with docker id 353d04eb9000
      Fri, 08 Apr 2016 07:35:54 +0000   Fri, 08 Apr 2016 07:35:54 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       started     Started with docker id 353d04eb9000
      Fri, 08 Apr 2016 07:36:04 +0000   Fri, 08 Apr 2016 07:36:04 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       started     Started with docker id 9d1fa98006a5
      Fri, 08 Apr 2016 07:36:04 +0000   Fri, 08 Apr 2016 07:36:04 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       created     Created with docker id 9d1fa98006a5
      Fri, 08 Apr 2016 07:36:14 +0000   Fri, 08 Apr 2016 07:36:14 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       created     Created with docker id 1c85ff51a31b
      Fri, 08 Apr 2016 07:36:14 +0000   Fri, 08 Apr 2016 07:36:14 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       started     Started with docker id 1c85ff51a31b
      Fri, 08 Apr 2016 07:36:24 +0000   Fri, 08 Apr 2016 07:36:24 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       created     Created with docker id b3ad00177295
      Fri, 08 Apr 2016 07:36:24 +0000   Fri, 08 Apr 2016 07:36:24 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       started     Started with docker id b3ad00177295
      Fri, 08 Apr 2016 07:36:34 +0000   Fri, 08 Apr 2016 07:36:34 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       started     Started with docker id 15da5f437b2c
    

    The configuration part for LogStash on K8 is like this:

       - name: logstash
          env:
            - name: ESNODES
              value: localhost
            - name: DELAY
              value: '240'
          image: logstash:1.5.6
          resources:
            limits:
              memory: 2Gi
              cpu: 500m
          volumeMounts:
            - name: logs
              mountPath: /elk
      volumes:
        - name: logs
          emptyDir: {}
    

    This makes accessing my application impossible.
    Any clue the reason for this?

  • Link docker-compose to local docker daemon
  • Limit memory on a docker container doesn't work
  • How to deploy app in this meteor docker image?
  • Access hosts zfs from docker container
  • no nodes available to schedule pods - Running Kubernetes Locally with No VM
  • Get all running Docker containers whose image is derived from a specific image
  • Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.