Docker For Windows – Java app deployed on a container gets a connection refused error when calling another java app deployed on another container

I was using an old version of Docker Toolbox for Windows so I uninstalled it and installed the latest stable Docker for Windows.

I have two Java applications running on its own containers:

  • Java ee api gets 404 on payara production server
  • How to use Nomad with Nvidia Docker?
  • Is there any way to divide server resources between users of Java application?
  • How to mount openstack container in a docker container
  • How to work with files which belong to the subuser namespace under Linux?
  • Establish PSSession to Windows docker container from remote machine (not container host)
    • Application1: http://archimedes1:8761
    • Application2: http://archimedes2:8762

    Dockerfile used in both apps:

    FROM frolvlad/alpine-oraclejdk8:slim
    VOLUME /tmp
    ADD archimedes-0.0.1-SNAPSHOT.jar app.jar
    RUN sh -c 'touch /app.jar'
    ENV JAVA_OPTS="-Xms750m -Xmx750m"
    ENTRYPOINT [ "sh", "-c", "java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /app.jar" ]
    

    I modified my Windows /etc/hosts to make archimedes1\2 map 127.0.0.1 (with the previous Docker Toolbox it mapped 192.168.99.100):

    127.0.0.1           archimedes1
    127.0.0.1           archimedes2
    

    This is how I start the containers (notice the use of –add-host to make the containers aware of the host names):

    docker run -e "SPRING_PROFILES_ACTIVE=archimedes1" -p 8761:8761 --name archimedes1 --add-host archimedes1:127.0.0.1 --add-host archimedes2:127.0.0.1 -d storyteller/archimedes 
    docker run -e "SPRING_PROFILES_ACTIVE=archimedes2" -p 8762:8762 --name archimedes2 --add-host archimedes1:127.0.0.1 --add-host archimedes2:127.0.0.1 -d storyteller/archimedes
    

    This worked with Docker Toolbox but since I installed Docker for Windows the containers, when contacting with each other show a Connection Refused error. In this case archimedes2 tries to call archimedes1:

    2017-01-25 13:10:27.406 DEBUG 5 --- [t_archimedes1-2] c.n.d.shared.MonitoredConnectionManager  : Get connection: {}->http://archimedes1:8761, timeout = 200
    2017-01-25 13:10:27.406 DEBUG 5 --- [t_archimedes1-2] c.n.d.shared.NamedConnectionPool         : [{}->http://archimedes1:8761] total kept alive: 0, total issued: 0, total allocated: 0 out of 1000
    2017-01-25 13:10:27.406 DEBUG 5 --- [t_archimedes1-2] c.n.d.shared.NamedConnectionPool         : No free connections [{}->http://archimedes1:8761][null]
    2017-01-25 13:10:27.406 DEBUG 5 --- [t_archimedes1-2] c.n.d.shared.NamedConnectionPool         : Available capacity: 500 out of 500 [{}->http://archimedes1:8761][null]
    2017-01-25 13:10:27.406 DEBUG 5 --- [t_archimedes1-2] c.n.d.shared.NamedConnectionPool         : Creating new connection [{}->http://archimedes1:8761]
    2017-01-25 13:10:27.407 DEBUG 5 --- [t_archimedes1-2] c.n.d.shared.MonitoredConnectionManager  : Released connection is not reusable.
    2017-01-25 13:10:27.407 DEBUG 5 --- [t_archimedes1-2] c.n.d.shared.NamedConnectionPool         : Releasing connection [{}->http://archimedes1:8761][null]
    2017-01-25 13:10:27.407 DEBUG 5 --- [t_archimedes1-2] c.n.d.shared.NamedConnectionPool         : Notifying no-one, there are no waiting threads
    2017-01-25 13:10:27.407 ERROR 5 --- [t_archimedes1-2] c.n.e.cluster.ReplicationTaskProcessor   : Network level connection to peer archimedes1; retrying after delay
    
    com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException: Connection refused (Connection refused)
            at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:187) ~[jersey-apache-client4-1.19.1.jar!/:1.19.1]
            at com.netflix.eureka.cluster.DynamicGZIPContentEncodingFilter.handle(DynamicGZIPContentEncodingFilter.java:48) ~[eureka-core-1.4.12.jar!/:1.4.12]
            at com.netflix.discovery.EurekaIdentityHeaderFilter.handle(EurekaIdentityHeaderFilter.java:27) ~[eureka-client-1.4.12.jar!/:1.4.12]
            at com.sun.jersey.api.client.Client.handle(Client.java:652) ~[jersey-client-1.19.1.jar!/:1.19.1]
            at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) ~[jersey-client-1.19.1.jar!/:1.19.1]
            at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) ~[jersey-client-1.19.1.jar!/:1.19.1]
            at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:570) ~[jersey-client-1.19.1.jar!/:1.19.1]
            at com.netflix.eureka.transport.JerseyReplicationClient.submitBatchUpdates(JerseyReplicationClient.java:116) ~[eureka-core-1.4.12.jar!/:1.4.12]
            at com.netflix.eureka.cluster.ReplicationTaskProcessor.process(ReplicationTaskProcessor.java:71) ~[eureka-core-1.4.12.jar!/:1.4.12]
            at com.netflix.eureka.util.batcher.TaskExecutors$BatchWorkerRunnable.run(TaskExecutors.java:187) [eureka-core-1.4.12.jar!/:1.4.12]
            at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
    Caused by: java.net.ConnectException: Connection refused (Connection refused)
    

    I start a sh into the container and ping archimedes1 from archimedes2’s container and it DOES answer.

    -> docker exec -it archimedes2 sh
    / # ping archimedes1
    PING archimedes1 (127.0.0.1): 56 data bytes
    64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.072 ms
    

    However if I do a telnet it gets a connection refused:

    docker exec -it archimedes2 sh
    / # telnet archimedes1 8761
    telnet: can't connect to remote host (127.0.0.1): Connection refused
    

    If I do the telnet against the own container it works:

    C:\Users\jinga4x>docker exec -it archimedes2 sh
    / # telnet archimedes2 8762
    

    What’s going on here?

    UPDATE:

    I have also tested this: starting archimedes1 as a normal Java application on Windows and archimedes2 in the container.

    Archimedes1 can contact archimedes2 but archimedes2 gets the connection refused when it tries to connect archimedes1.

    UPDATE 2:

    This is my docker network inspect bridge info:

    [
        {
            "Name": "bridge",
            "Id": "546e7a5ef627c8d23e8ffdc05911fcae096167a359701fa4ee08ada0f7e1ae7f",
            "Created": "2017-01-25T11:09:27.651777Z",
            "Scope": "local",
            "Driver": "bridge",
            "EnableIPv6": false,
            "IPAM": {
                "Driver": "default",
                "Options": null,
                "Config": [
                    {
                        "Subnet": "172.17.0.0/16",
                        "Gateway": "172.17.0.1"
                    }
                ]
            },
            "Internal": false,
            "Attachable": false,
            "Containers": {
                "58634d0c4430895cf0dfbee294c3ea75ca38921441684d614a670421661eb628": {
                    "Name": "archimedes2",
                    "EndpointID": "b2f40396b4c0f8210ca667d93c7d787296f3dad2d0eb295c31d4f01bfe3b39e1",
                    "MacAddress": "02:42:ac:11:00:03",
                    "IPv4Address": "172.17.0.3/16",
                    "IPv6Address": ""
                },
                "85f70520ad900e729944dc768f8c6951e9221650269b5669a2d0269506a4c16b": {
                    "Name": "archimedes1",
                    "EndpointID": "651bf095eed639ecc61a24ffdaf2130bddd338f38f42f47a6c54b460c3a979ab",
                    "MacAddress": "02:42:ac:11:00:02",
                    "IPv4Address": "172.17.0.2/16",
                    "IPv6Address": ""
                }
            },
            "Options": {
                "com.docker.network.bridge.default_bridge": "true",
                "com.docker.network.bridge.enable_icc": "true",
                "com.docker.network.bridge.enable_ip_masquerade": "true",
                "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
                "com.docker.network.bridge.name": "docker0",
                "com.docker.network.driver.mtu": "1500"
            },
            "Labels": {}
        }
    ]
    

    archimedes1 container ifconfig:

    eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:00:02
              inet addr:172.17.0.2  Bcast:0.0.0.0  Mask:255.255.0.0
              inet6 addr: fe80::42:acff:fe11:2/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:133 errors:0 dropped:0 overruns:0 frame:0
              TX packets:105 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0
              RX bytes:13158 (12.8 KiB)  TX bytes:345156 (337.0 KiB)
    
    lo        Link encap:Local Loopback
              inet addr:127.0.0.1  Mask:255.0.0.0
              inet6 addr: ::1/128 Scope:Host
              UP LOOPBACK RUNNING  MTU:65536  Metric:1
              RX packets:64 errors:0 dropped:0 overruns:0 frame:0
              TX packets:64 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1
              RX bytes:3200 (3.1 KiB)  TX bytes:3200 (3.1 KiB)
    

    archimedes1 cat etc/hosts:

    127.0.0.1       localhost
    ::1     localhost ip6-localhost ip6-loopback
    fe00::0 ip6-localnet
    ff00::0 ip6-mcastprefix
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters
    127.0.0.1       archimedes1
    127.0.0.1       archimedes2
    172.17.0.2      a8045a473784
    

    UPDATE 3:

    The containers can access each other using the internal Private IPs, but not the public one (127.0.0.1).

    UPDATE 4:

    I started a proxy monitoring archimedes1:22233 and sending it’s traffic to archimedes1:8761. archimedes2 tries to comunicate through the proxy but no traffic reaches it.

  • how to connect Docker containers without a bridge?
  • Docker logs retrieval from MongoDB
  • Using COPY on dockerfile for apache build
  • Boot2docker : image don't save
  • Upload to s3 from docker container running inside ECS not working
  • Visual studio 2017 docker error running app
  • One Solution collect form web for “Docker For Windows – Java app deployed on a container gets a connection refused error when calling another java app deployed on another container”

    To get the correct IP address, you need to use docker-machine ip.

    For the new Docker for Windows for Windows 10, get the container IDs with docker ps and then look for IPAddress in the output of docker inspect $CID ($CID == the container ID to inspect).

    Background: Docker can’t run containers on Windows. Docker is a pure Linux software. It needs a running Linux kernel to work. The workaround used by the Docker for Windows guys is to install a virtual machine (VirtualBox) on your computer which simulates a whole PC – including network cards, hard disk drives, CPU and everything. Newer versions of the software use Hyper-V which is similar.

    Inside of that virtual PC, a complete Linux is installed plus Docker. The Windows tools will connect to that virtual PC to talk to the Docker running inside.

    Actually, the address 127.0.0.1 will not leave the container – it won’t even mean “the virtual PC inside of which Docker is running”. Ping of course works because every computer responds to the address 127.0.0.1 – that doesn’t mean it’s the same computer you think it should be.

    [EDIT] The file /etc/hosts looks wrong:

    10.0.75.1       archimedes1
    10.0.75.1       archimedes2
    

    Those are two different containers; they should have different IP addresses.

    I think you have a typo in your docker run. --add-host archimedes2:10.0.75.1 should be --add-host archimedes2:10.0.75.2

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.