Kubernetes replication controller in CrashLoopBackOff state

I performed the following steps.

Created the replication controller with the following config file:

  • How to test and deploy a Docker production image?
  • Jenkins and SonarQube in same dockerfile
  • Sinatra app on AWS Beanstalk with docker and SQS
  • DevOps: automatically restarting a failed container
  • Disable ip v6 in docker container
  • Django - default port 0 instead of 3306 - Can't connect to MySQL server on '127.0.0.1' (61)
  • {
       "kind":"ReplicationController",
       "apiVersion":"v1",
       "metadata":{
          "name":"fsharp-service",
          "labels":{
             "app":"fsharp-service"
          }
       },
       "spec":{
          "replicas":1,
          "selector":{
             "app":"fsharp-service"
          },
          "template":{
             "metadata":{
                "labels":{
                   "app":"fsharp-service"
                }
             },
             "spec":{
                "containers":[
                   {
                      "name":"fsharp-service",
                      "image":"fsharp/fsharp:latest",
                      "ports":[
                         {
                            "name":"http-server",
                            "containerPort":3000
                         }
                      ]
                   }
                ]
             }
          }
       }
    }
    

    Run the command:

    kubectl create -f fsharp-controller.json
    

    Here is the output:

    $ kubectl get rc
    CONTROLLER       CONTAINER(S)     IMAGE(S)                             SELECTOR             REPLICAS
    cassandra        cassandra        gcr.io/google-samples/cassandra:v8   app=cassandra        3
    fsharp-service   fsharp-service   fsharp/fsharp:latest                 app=fsharp-service   1
    $ kubectl get pods
    NAME                   READY     REASON    RESTARTS   AGE
    cassandra              1/1       Running   0          28m
    cassandra-ch1br        1/1       Running   0          28m
    cassandra-xog49        1/1       Running   0          27m
    fsharp-service-7lrq8   0/1       Error     2          31s
    $ kubectl logs fsharp-service-7lrq8
    
    F# Interactive for F# 4.0 (Open Source Edition)
    Freely distributed under the Apache 2.0 Open Source License
    
    For help type #help;;
    
    $ kubectl get pods
    NAME                   READY     REASON             RESTARTS   AGE
    cassandra              1/1       Running            0          28m
    cassandra-ch1br        1/1       Running            0          28m
    cassandra-xog49        1/1       Running            0          28m
    fsharp-service-7lrq8   0/1       CrashLoopBackOff   3          1m
    $ kubectl describe po fsharp-service-7lrq8
    W0417 15:52:36.288492   11461 request.go:302] field selector: v1 - events - involvedObject.name - fsharp-service-7lrq8: need to check if this is versioned correctly.
    W0417 15:52:36.289196   11461 request.go:302] field selector: v1 - events - involvedObject.namespace - default: need to check if this is versioned correctly.
    W0417 15:52:36.289204   11461 request.go:302] field selector: v1 - events - involvedObject.uid - d4dab099-04ee-11e6-b7f9-0a11c670939b: need to check if this is versioned correctly.
    Name:               fsharp-service-7lrq8
    Image(s):           fsharp/fsharp:latest
    Node:               ip-172-20-0-228.us-west-2.compute.internal/172.20.0.228
    Labels:             app=fsharp-service
    Status:             Running
    Replication Controllers:    fsharp-service (1/1 replicas created)
    Containers:
      fsharp-service:
        Image:      fsharp/fsharp:latest
        State:      Waiting
          Reason:       CrashLoopBackOff
        Ready:      False
        Restart Count:  3
    Conditions:
      Type      Status
      Ready     False
    Events:
      FirstSeen             LastSeen            Count   From                            SubobjectPath           Reason      Message
      Sun, 17 Apr 2016 15:50:50 -0700   Sun, 17 Apr 2016 15:50:50 -0700 1   {default-scheduler }                                    Scheduled   Successfully assigned fsharp-service-7lrq8 to ip-172-20-0-228.us-west-2.compute.internal
      Sun, 17 Apr 2016 15:50:51 -0700   Sun, 17 Apr 2016 15:50:51 -0700 1   {kubelet ip-172-20-0-228.us-west-2.compute.internal}    spec.containers{fsharp-service} Created     Created container with docker id d44c288ea67b
      Sun, 17 Apr 2016 15:50:51 -0700   Sun, 17 Apr 2016 15:50:51 -0700 1   {kubelet ip-172-20-0-228.us-west-2.compute.internal}    spec.containers{fsharp-service} Started     Started container with docker id d44c288ea67b
      Sun, 17 Apr 2016 15:50:55 -0700   Sun, 17 Apr 2016 15:50:55 -0700 1   {kubelet ip-172-20-0-228.us-west-2.compute.internal}    spec.containers{fsharp-service} Started     Started container with docker id 688a3ed122d2
      Sun, 17 Apr 2016 15:50:55 -0700   Sun, 17 Apr 2016 15:50:55 -0700 1   {kubelet ip-172-20-0-228.us-west-2.compute.internal}    spec.containers{fsharp-service} Created     Created container with docker id 688a3ed122d2
      Sun, 17 Apr 2016 15:50:58 -0700   Sun, 17 Apr 2016 15:50:58 -0700 1   {kubelet ip-172-20-0-228.us-west-2.compute.internal}                    FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "fsharp-service" with CrashLoopBackOff: "Back-off 10s restarting failed container=fsharp-service pod=fsharp-service-7lrq8_default(d4dab099-04ee-11e6-b7f9-0a11c670939b)"
    
      Sun, 17 Apr 2016 15:51:15 -0700   Sun, 17 Apr 2016 15:51:15 -0700 1   {kubelet ip-172-20-0-228.us-west-2.compute.internal}    spec.containers{fsharp-service} Started     Started container with docker id c2e348e1722d
      Sun, 17 Apr 2016 15:51:15 -0700   Sun, 17 Apr 2016 15:51:15 -0700 1   {kubelet ip-172-20-0-228.us-west-2.compute.internal}    spec.containers{fsharp-service} Created     Created container with docker id c2e348e1722d
      Sun, 17 Apr 2016 15:51:17 -0700   Sun, 17 Apr 2016 15:51:31 -0700 2   {kubelet ip-172-20-0-228.us-west-2.compute.internal}                    FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "fsharp-service" with CrashLoopBackOff: "Back-off 20s restarting failed container=fsharp-service pod=fsharp-service-7lrq8_default(d4dab099-04ee-11e6-b7f9-0a11c670939b)"
    
      Sun, 17 Apr 2016 15:50:50 -0700   Sun, 17 Apr 2016 15:51:44 -0700 4   {kubelet ip-172-20-0-228.us-west-2.compute.internal}    spec.containers{fsharp-service} Pulling     pulling image "fsharp/fsharp:latest"
      Sun, 17 Apr 2016 15:51:45 -0700   Sun, 17 Apr 2016 15:51:45 -0700 1   {kubelet ip-172-20-0-228.us-west-2.compute.internal}    spec.containers{fsharp-service} Created     Created container with docker id edaea97fb379
      Sun, 17 Apr 2016 15:50:51 -0700   Sun, 17 Apr 2016 15:51:45 -0700 4   {kubelet ip-172-20-0-228.us-west-2.compute.internal}    spec.containers{fsharp-service} Pulled      Successfully pulled image "fsharp/fsharp:latest"
      Sun, 17 Apr 2016 15:51:46 -0700   Sun, 17 Apr 2016 15:51:46 -0700 1   {kubelet ip-172-20-0-228.us-west-2.compute.internal}    spec.containers{fsharp-service} Started     Started container with docker id edaea97fb379
      Sun, 17 Apr 2016 15:50:58 -0700   Sun, 17 Apr 2016 15:52:27 -0700 7   {kubelet ip-172-20-0-228.us-west-2.compute.internal}    spec.containers{fsharp-service} BackOff     Back-off restarting failed docker container
      Sun, 17 Apr 2016 15:51:48 -0700   Sun, 17 Apr 2016 15:52:27 -0700 4   {kubelet ip-172-20-0-228.us-west-2.compute.internal}                    FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "fsharp-service" with CrashLoopBackOff: "Back-off 40s restarting failed container=fsharp-service pod=fsharp-service-7lrq8_default(d4dab099-04ee-11e6-b7f9-0a11c670939b)"
    

    What is wrong?

    How can I find out the reason why the controller won’t start correctly?

    UPDATE.

    I have tried to change the simple “fsharp/fsharp:latest” image to another image where there would be a service listening to a port, this is how I want to use the container.

    The image is called “username/someservice:mytag” and has a service listening to the port 3000.

    I run the service as:

    mono Service.exe
    

    When I look at the logs I see this:

    $ kubectl logs -p fsharp-service-wjmpv
    Running on http://127.0.0.1:3000
    Press enter to exit
    

    So the container is in the same state even though the process shouldn’t exit:

    $ kubectl get pods
    NAME                   READY     REASON             RESTARTS   AGE
    fsharp-service-wjmpv   0/1       CrashLoopBackOff   9          25m
    

    I also tried to run the container from my image with the -i flag, to make the container not exit, but kubectl doesn’t seem to recognize -i flag :\

    Any thoughts?

  • Can Terraform set a variable from a remote_exec command?
  • Validating docker v0.8 on OS X, boot2docker ssh hangs, docker version generates EOF
  • dial tcp: lookup xxx.xxx.xxx.xxx: no such host
  • php-fpm process hight memory usage, and the host jammed
  • Prevent volume creation on docker run
  • Docker run error: could not create session key: too many links
  • 3 Solutions collect form web for “Kubernetes replication controller in CrashLoopBackOff state”

    You are launching a container that immediately exits. The kubelet notices, restarts it, and then it exits again. After this happens a few times, the kubelet slows down the rate at which it tries to launch the container (this is the CrashLoopBackOff state).

    The fsharp documentation says to run the container with the -i flag, which gives an interactive prompt. If you just do

    docker run fsharp/fsharp:latest
    

    you’ll notice that the container exits immediately and dumps you back into your local shell. This is the way in which you are trying to invoke the container in your cluster, and it is likewise exiting immediately.

    I would use kubectl logs to try to find out what has happened to your container, like so:

    kubectl logs -p fsharp-service-7lrq8
    

    The -p flag lets you get logs for the previous startup which is necessary in this case since the container is crashing.

    More information: http://kubernetes.io/docs/user-guide/kubectl/kubectl_logs/

    I have added the following line to my F# service (Unix specific code) to make sure process doesn’t exit:

    let signals = [| new UnixSignal (Signum.SIGINT); 
                     new UnixSignal (Signum.SIGTERM); 
                     new UnixSignal (Signum.SIGQUIT)
                  |]
    
    let which = UnixSignal.WaitAny (signals, -1);
    

    After that my replication controller is running normally.

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.