ECS Agent starting Docker container doesn't complete. Manual start successful

I followed the ECS Getting Started tutorial but the ECS Agent isn’t getting the container started. When I start the image manually on the same instance it starts fine.

The image is a Spring Boot web application with a single endpoint on / that returns the string “Hello world!!”. The container runs fine locally, and also runs fine on a CentOS EC2 instance I’ve created. The endpoint is available publicly when I run the docker image on the CentOS EC2 instance.

  • Increase the disk size of docker (Error: Insufficient space)
  • Can't install Kubernetes on Vagrant
  • GO - Local import not working
  • Docker VS Native installation on a server. How to recover from downtime mysql/mariadb without data lloss
  • Error checking context is accessible: 'can't stat '.gvfs''. Please check permissions and try again
  • Understanding chroot
  • The ECS Instance has security groups created by the wizard and has port 80 open. I added port 22 for SSH access.

    When I SSH into the ECS instance to see the Docker logs for my container, and it looks like it’s hanging during the entrypoint execution.

    Here are the Docker logs for the hanging instance:

    [ec2-user@ip-10-0-0-156 ~]$ docker logs --tail 100 107d4cf04dd8
    
      .   ____          _            __ _ _
     /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
    ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
     \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
      '  |____| .__|_| |_|_| |_\__, | / / / /
     =========|_|==============|___/=/_/_/_/
     :: Spring Boot ::        (v1.4.2.RELEASE)
    
    2016-11-25 17:36:22.505  INFO 1 --- [           main] ecstest.Application                      : Starting Application on 107d4cf04dd8 with PID 1 (/ecstest-1.0-SNAPSHOT.jar started by root in /)
    2016-11-25 17:36:22.546  INFO 1 --- [           main] ecstest.Application                      : No active profile set, falling back to default profiles: default
    2016-11-25 17:36:23.059  INFO 1 --- [           main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@6d21714c: startup date [Fri Nov 25 17:36:23 UTC 2016]; root of context hierarchy
    2016-11-25 17:36:30.972  INFO 1 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat initialized with port(s): 8080 (http)
    2016-11-25 17:36:31.014  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service Tomcat
    2016-11-25 17:36:31.016  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet Engine: Apache Tomcat/8.5.6
    2016-11-25 17:36:31.464  INFO 1 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
    2016-11-25 17:36:31.464  INFO 1 --- [ost-startStop-1] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 8458 ms
    

    At first it seems like an application error in my container image, but when I stop the docker process and run the same image manually, the output is as expected and I can reach my endpoint from outside the instance as expected.

    [ec2-user@ip-10-0-1-124 ~]$ docker stop -t 1 4d2401d7db93 && docker run -p 80:8080 -d ############.dkr.ecr.us-west-2.amazonaws.com/ecstest
    4d2401d7db93
    db8cffa89995401d9314d7d70e954f09c7fde972a5e6a423615827d8c47b9d10
    [ec2-user@ip-10-0-1-124 ~]$ docker ps
    CONTAINER ID        IMAGE                                                  COMMAND                  CREATED             STATUS              PORTS                  NAMES
    db8cffa89995        ############.dkr.ecr.us-west-2.amazonaws.com/ecstest   "java -jar ecstest-1."   10 seconds ago      Up 9 seconds        0.0.0.0:80->8080/tcp   small_gates
    85bd18480c99        amazon/amazon-ecs-agent:latest                         "/agent"                 11 minutes ago      Up 11 minutes                              ecs-agent
    [ec2-user@ip-10-0-1-124 ~]$ docker logs --tail 1000 db8cffa89995
    
      .   ____          _            __ _ _
     /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
    ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
     \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
      '  |____| .__|_| |_|_| |_\__, | / / / /
     =========|_|==============|___/=/_/_/_/
     :: Spring Boot ::        (v1.4.2.RELEASE)
    
    2016-11-25 18:06:57.960  INFO 1 --- [           main] ecstest.Application                      : Starting Application on db8cffa89995 with PID 1 (/ecstest-1.0-SNAPSHOT.jar started by root in /)
    2016-11-25 18:06:58.004  INFO 1 --- [           main] ecstest.Application                      : No active profile set, falling back to default profiles: default
    2016-11-25 18:06:58.578  INFO 1 --- [           main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@6d21714c: startup date [Fri Nov 25 18:06:58 UTC 2016]; root of context hierarchy
    2016-11-25 18:07:05.784  INFO 1 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat initialized with port(s): 8080 (http)
    2016-11-25 18:07:05.866  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service Tomcat
    2016-11-25 18:07:05.876  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet Engine: Apache Tomcat/8.5.6
    2016-11-25 18:07:06.283  INFO 1 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
    2016-11-25 18:07:06.283  INFO 1 --- [ost-startStop-1] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 7753 ms
    2016-11-25 18:07:07.026  INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.ServletRegistrationBean  : Mapping servlet: 'dispatcherServlet' to [/]
    2016-11-25 18:07:07.031  INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'characterEncodingFilter' to: [/*]
    2016-11-25 18:07:07.031  INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
    2016-11-25 18:07:07.032  INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'httpPutFormContentFilter' to: [/*]
    2016-11-25 18:07:07.033  INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'requestContextFilter' to: [/*]
    2016-11-25 18:07:08.432  INFO 1 --- [           main] s.w.s.m.m.a.RequestMappingHandlerAdapter : Looking for @ControllerAdvice: org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@6d21714c: startup date [Fri Nov 25 18:06:58 UTC 2016]; root of context hierarchy
    2016-11-25 18:07:08.786  INFO 1 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[],methods=[GET]}" onto public java.lang.String ecstest.Application.get()
    2016-11-25 18:07:08.800  INFO 1 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
    2016-11-25 18:07:08.801  INFO 1 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
    2016-11-25 18:07:09.036  INFO 1 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
    2016-11-25 18:07:09.036  INFO 1 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
    2016-11-25 18:07:09.204  INFO 1 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
    2016-11-25 18:07:09.893  INFO 1 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
    2016-11-25 18:07:10.201  INFO 1 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8080 (http)
    2016-11-25 18:07:10.216  INFO 1 --- [           main] ecstest.Application                      : Started Application in 14.385 seconds (JVM running for 16.522)
    

    Any ideas why the ECS Agent isn’t getting my application started?

    Task Definition JSON

    {
      "attributes": null,
      "requiresAttributes": [
        {
          "value": null,
          "name": "com.amazonaws.ecs.capability.ecr-auth",
          "targetId": null,
          "targetType": null
        }
      ],
      "taskDefinitionArn": "arn:aws:ecs:us-west-2:############:task-definition/DcTaskDefinition:4",
      "networkMode": "bridge",
      "status": "ACTIVE",
      "revision": 4,
      "taskRoleArn": null,
      "containerDefinitions": [
        {
          "volumesFrom": [],
          "memory": 128,
          "extraHosts": null,
          "dnsServers": null,
          "disableNetworking": null,
          "dnsSearchDomains": null,
          "portMappings": [
            {
              "hostPort": 80,
              "containerPort": 8080,
              "protocol": "tcp"
            }
          ],
          "hostname": null,
          "essential": true,
          "entryPoint": null,
          "mountPoints": [],
          "name": "DcContainer",
          "ulimits": null,
          "dockerSecurityOptions": null,
          "environment": [],
          "links": null,
          "workingDirectory": null,
          "readonlyRootFilesystem": null,
          "image": "############.dkr.ecr.us-west-2.amazonaws.com/ecstest:latest",
          "command": null,
          "user": null,
          "dockerLabels": null,
          "logConfiguration": null,
          "cpu": 0,
          "privileged": null,
          "memoryReservation": null
        }
      ],
      "placementConstraints": [],
      "volumes": [],
      "family": "DcTaskDefinition"
    }
    

  • DNS Server in Docker Container
  • Where is information from dockerfile located?
  • docker hornetQ standalone … jnp issue
  • Docker - overlay network on rhel 7
  • MySQL 2002 Connection Refused - OSX/MySQL/Docker
  • Strategy for arangodb data backup in docker container
  • One Solution collect form web for “ECS Agent starting Docker container doesn't complete. Manual start successful”

    The memory key in the Task Definition JSON imposes a hard memory limit on the container. When a container tries to exceed that limit, the Docker daemon is supposed to kill it.
    I’m not sure whether this can cause your container to get “stuck”, however that’s the only important difference I see between how your container runs when ECS runs it and how it runs from the command line.
    So, I would try to set the memory value to at least 300, or use the memoryReservation key instead which imposes a soft memory limit.

    More information on the difference between hard and soft memory limits can be found in the official ECS documentation.

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.