ECS Agent starting Docker container doesn't complete. Manual start successful

I followed the ECS Getting Started tutorial but the ECS Agent isn’t getting the container started. When I start the image manually on the same instance it starts fine.

The image is a Spring Boot web application with a single endpoint on / that returns the string “Hello world!!”. The container runs fine locally, and also runs fine on a CentOS EC2 instance I’ve created. The endpoint is available publicly when I run the docker image on the CentOS EC2 instance.

  • Docker Java Client API Connection Pool
  • Cannot connect to the Docker daemon on Windows 7
  • docker alpine postgres in compose not executing docker-entrypoint-initdb.d scripts
  • What is the best practices of using Docker for front end development on OS X and passthrough ENV from host to container
  • Is there a docker image which is similar (or the same) as AWS EC2 AMI?
  • how to map a local folder as the volume the docker container or image?
  • The ECS Instance has security groups created by the wizard and has port 80 open. I added port 22 for SSH access.

    When I SSH into the ECS instance to see the Docker logs for my container, and it looks like it’s hanging during the entrypoint execution.

    Here are the Docker logs for the hanging instance:

    [ec2-user@ip-10-0-0-156 ~]$ docker logs --tail 100 107d4cf04dd8
    
      .   ____          _            __ _ _
     /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
    ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
     \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
      '  |____| .__|_| |_|_| |_\__, | / / / /
     =========|_|==============|___/=/_/_/_/
     :: Spring Boot ::        (v1.4.2.RELEASE)
    
    2016-11-25 17:36:22.505  INFO 1 --- [           main] ecstest.Application                      : Starting Application on 107d4cf04dd8 with PID 1 (/ecstest-1.0-SNAPSHOT.jar started by root in /)
    2016-11-25 17:36:22.546  INFO 1 --- [           main] ecstest.Application                      : No active profile set, falling back to default profiles: default
    2016-11-25 17:36:23.059  INFO 1 --- [           main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@6d21714c: startup date [Fri Nov 25 17:36:23 UTC 2016]; root of context hierarchy
    2016-11-25 17:36:30.972  INFO 1 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat initialized with port(s): 8080 (http)
    2016-11-25 17:36:31.014  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service Tomcat
    2016-11-25 17:36:31.016  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet Engine: Apache Tomcat/8.5.6
    2016-11-25 17:36:31.464  INFO 1 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
    2016-11-25 17:36:31.464  INFO 1 --- [ost-startStop-1] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 8458 ms
    

    At first it seems like an application error in my container image, but when I stop the docker process and run the same image manually, the output is as expected and I can reach my endpoint from outside the instance as expected.

    [ec2-user@ip-10-0-1-124 ~]$ docker stop -t 1 4d2401d7db93 && docker run -p 80:8080 -d ############.dkr.ecr.us-west-2.amazonaws.com/ecstest
    4d2401d7db93
    db8cffa89995401d9314d7d70e954f09c7fde972a5e6a423615827d8c47b9d10
    [ec2-user@ip-10-0-1-124 ~]$ docker ps
    CONTAINER ID        IMAGE                                                  COMMAND                  CREATED             STATUS              PORTS                  NAMES
    db8cffa89995        ############.dkr.ecr.us-west-2.amazonaws.com/ecstest   "java -jar ecstest-1."   10 seconds ago      Up 9 seconds        0.0.0.0:80->8080/tcp   small_gates
    85bd18480c99        amazon/amazon-ecs-agent:latest                         "/agent"                 11 minutes ago      Up 11 minutes                              ecs-agent
    [ec2-user@ip-10-0-1-124 ~]$ docker logs --tail 1000 db8cffa89995
    
      .   ____          _            __ _ _
     /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
    ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
     \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
      '  |____| .__|_| |_|_| |_\__, | / / / /
     =========|_|==============|___/=/_/_/_/
     :: Spring Boot ::        (v1.4.2.RELEASE)
    
    2016-11-25 18:06:57.960  INFO 1 --- [           main] ecstest.Application                      : Starting Application on db8cffa89995 with PID 1 (/ecstest-1.0-SNAPSHOT.jar started by root in /)
    2016-11-25 18:06:58.004  INFO 1 --- [           main] ecstest.Application                      : No active profile set, falling back to default profiles: default
    2016-11-25 18:06:58.578  INFO 1 --- [           main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@6d21714c: startup date [Fri Nov 25 18:06:58 UTC 2016]; root of context hierarchy
    2016-11-25 18:07:05.784  INFO 1 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat initialized with port(s): 8080 (http)
    2016-11-25 18:07:05.866  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service Tomcat
    2016-11-25 18:07:05.876  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet Engine: Apache Tomcat/8.5.6
    2016-11-25 18:07:06.283  INFO 1 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
    2016-11-25 18:07:06.283  INFO 1 --- [ost-startStop-1] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 7753 ms
    2016-11-25 18:07:07.026  INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.ServletRegistrationBean  : Mapping servlet: 'dispatcherServlet' to [/]
    2016-11-25 18:07:07.031  INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'characterEncodingFilter' to: [/*]
    2016-11-25 18:07:07.031  INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
    2016-11-25 18:07:07.032  INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'httpPutFormContentFilter' to: [/*]
    2016-11-25 18:07:07.033  INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'requestContextFilter' to: [/*]
    2016-11-25 18:07:08.432  INFO 1 --- [           main] s.w.s.m.m.a.RequestMappingHandlerAdapter : Looking for @ControllerAdvice: org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@6d21714c: startup date [Fri Nov 25 18:06:58 UTC 2016]; root of context hierarchy
    2016-11-25 18:07:08.786  INFO 1 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[],methods=[GET]}" onto public java.lang.String ecstest.Application.get()
    2016-11-25 18:07:08.800  INFO 1 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
    2016-11-25 18:07:08.801  INFO 1 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
    2016-11-25 18:07:09.036  INFO 1 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
    2016-11-25 18:07:09.036  INFO 1 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
    2016-11-25 18:07:09.204  INFO 1 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
    2016-11-25 18:07:09.893  INFO 1 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
    2016-11-25 18:07:10.201  INFO 1 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8080 (http)
    2016-11-25 18:07:10.216  INFO 1 --- [           main] ecstest.Application                      : Started Application in 14.385 seconds (JVM running for 16.522)
    

    Any ideas why the ECS Agent isn’t getting my application started?

    Task Definition JSON

    {
      "attributes": null,
      "requiresAttributes": [
        {
          "value": null,
          "name": "com.amazonaws.ecs.capability.ecr-auth",
          "targetId": null,
          "targetType": null
        }
      ],
      "taskDefinitionArn": "arn:aws:ecs:us-west-2:############:task-definition/DcTaskDefinition:4",
      "networkMode": "bridge",
      "status": "ACTIVE",
      "revision": 4,
      "taskRoleArn": null,
      "containerDefinitions": [
        {
          "volumesFrom": [],
          "memory": 128,
          "extraHosts": null,
          "dnsServers": null,
          "disableNetworking": null,
          "dnsSearchDomains": null,
          "portMappings": [
            {
              "hostPort": 80,
              "containerPort": 8080,
              "protocol": "tcp"
            }
          ],
          "hostname": null,
          "essential": true,
          "entryPoint": null,
          "mountPoints": [],
          "name": "DcContainer",
          "ulimits": null,
          "dockerSecurityOptions": null,
          "environment": [],
          "links": null,
          "workingDirectory": null,
          "readonlyRootFilesystem": null,
          "image": "############.dkr.ecr.us-west-2.amazonaws.com/ecstest:latest",
          "command": null,
          "user": null,
          "dockerLabels": null,
          "logConfiguration": null,
          "cpu": 0,
          "privileged": null,
          "memoryReservation": null
        }
      ],
      "placementConstraints": [],
      "volumes": [],
      "family": "DcTaskDefinition"
    }
    

  • Is it proper to run a Docker image on a host with a different system?
  • fabric8 docker-maven-plugin capAdd fails with “unknown capability to add”
  • Why is my dockerfile not copying directories
  • Why can't the Docker (“master”) container see workers?
  • docker-compose - Expose linked service port
  • Docker machine using generic driver: x509: certificate signed by unknown authority when trying to add remote host to local machine
  • One Solution collect form web for “ECS Agent starting Docker container doesn't complete. Manual start successful”

    The memory key in the Task Definition JSON imposes a hard memory limit on the container. When a container tries to exceed that limit, the Docker daemon is supposed to kill it.
    I’m not sure whether this can cause your container to get “stuck”, however that’s the only important difference I see between how your container runs when ECS runs it and how it runs from the command line.
    So, I would try to set the memory value to at least 300, or use the memoryReservation key instead which imposes a soft memory limit.

    More information on the difference between hard and soft memory limits can be found in the official ECS documentation.

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.