LogStash container restarts on Kube pod

I have a Kubernetes cluster running on GCE, one of my PODs has two containers:

  • my app
  • Logstash

For some reason the container of LogStash inside the POD keeps restarting, see the result of “describe pod”:

  • OS name for docker images
  • Private docker registry authentication in aws ecs optimized AMI is not successful
  • php docker container not loading php files
  • Configuring ELK with Kitematic - Elastic Search won't start - localhost not available
  • docker run passing command line params as config file
  • Using docker to compose a remote image with a local code base for *development*
  •   logstash:
        Image:  logstash:1.5.6
        Limits:
          cpu:      500m
          memory:       2Gi
        State:      Running
          Started:      Fri, 08 Apr 2016 08:28:53 +0000
        Ready:      False
        Restart Count:  12
    Conditions:
      Type      Status
      Ready     False
    Events:
      FirstSeen             LastSeen            Count   From                    SubobjectPath               Reason      Message
      Fri, 08 Apr 2016 07:34:53 +0000   Fri, 08 Apr 2016 07:34:53 +0000 1   {kubelet kubernetes-minion-1m8z}    implicitly required container POD   pulled      Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
      Fri, 08 Apr 2016 07:34:53 +0000   Fri, 08 Apr 2016 07:34:53 +0000 1   {scheduler }                                    scheduled   Successfully assigned to kubernetes-minion-1m8z
      Fri, 08 Apr 2016 07:34:54 +0000   Fri, 08 Apr 2016 07:34:54 +0000 1   {kubelet kubernetes-minion-1m8z}    implicitly required container POD   created     Created with docker id f916e5aee8c7
      Fri, 08 Apr 2016 07:34:54 +0000   Fri, 08 Apr 2016 07:34:54 +0000 1   {kubelet kubernetes-minion-1m8z}    implicitly required container POD   started     Started with docker id f916e5aee8c7
      Fri, 08 Apr 2016 07:35:24 +0000   Fri, 08 Apr 2016 07:35:24 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       pulled      Successfully pulled image "logstash:1.5.6"
      Fri, 08 Apr 2016 07:35:25 +0000   Fri, 08 Apr 2016 07:35:25 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       created     Created with docker id 059218faf86d
      Fri, 08 Apr 2016 07:35:25 +0000   Fri, 08 Apr 2016 07:35:25 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       started     Started with docker id 059218faf86d
      Fri, 08 Apr 2016 07:35:34 +0000   Fri, 08 Apr 2016 07:35:34 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       created     Created with docker id 44e0800b2992
      Fri, 08 Apr 2016 07:35:34 +0000   Fri, 08 Apr 2016 07:35:34 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       started     Started with docker id 44e0800b2992
      Fri, 08 Apr 2016 07:35:44 +0000   Fri, 08 Apr 2016 07:35:44 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       created     Created with docker id 32df246c83dc
      Fri, 08 Apr 2016 07:35:44 +0000   Fri, 08 Apr 2016 07:35:44 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       started     Started with docker id 32df246c83dc
      Fri, 08 Apr 2016 07:35:54 +0000   Fri, 08 Apr 2016 07:35:54 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       created     Created with docker id 353d04eb9000
      Fri, 08 Apr 2016 07:35:54 +0000   Fri, 08 Apr 2016 07:35:54 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       started     Started with docker id 353d04eb9000
      Fri, 08 Apr 2016 07:36:04 +0000   Fri, 08 Apr 2016 07:36:04 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       started     Started with docker id 9d1fa98006a5
      Fri, 08 Apr 2016 07:36:04 +0000   Fri, 08 Apr 2016 07:36:04 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       created     Created with docker id 9d1fa98006a5
      Fri, 08 Apr 2016 07:36:14 +0000   Fri, 08 Apr 2016 07:36:14 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       created     Created with docker id 1c85ff51a31b
      Fri, 08 Apr 2016 07:36:14 +0000   Fri, 08 Apr 2016 07:36:14 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       started     Started with docker id 1c85ff51a31b
      Fri, 08 Apr 2016 07:36:24 +0000   Fri, 08 Apr 2016 07:36:24 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       created     Created with docker id b3ad00177295
      Fri, 08 Apr 2016 07:36:24 +0000   Fri, 08 Apr 2016 07:36:24 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       started     Started with docker id b3ad00177295
      Fri, 08 Apr 2016 07:36:34 +0000   Fri, 08 Apr 2016 07:36:34 +0000 1   {kubelet kubernetes-minion-1m8z}    spec.containers{logstash}       started     Started with docker id 15da5f437b2c
    

    The configuration part for LogStash on K8 is like this:

       - name: logstash
          env:
            - name: ESNODES
              value: localhost
            - name: DELAY
              value: '240'
          image: logstash:1.5.6
          resources:
            limits:
              memory: 2Gi
              cpu: 500m
          volumeMounts:
            - name: logs
              mountPath: /elk
      volumes:
        - name: logs
          emptyDir: {}
    

    This makes accessing my application impossible.
    Any clue the reason for this?

  • Docker interactive mode and executing script
  • x509: certificate is valid for *.xyz.net, not docker.abc.xyz.net
  • REST API container create and port bindings
  • Why do we need run update before run install in Dockerfile?
  • Docker keeps pushing same hashes
  • Unable to find image latest locally
  • Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.