All external DNS queries fail from within docker container

Current docker version: 1.13.1, build 092cba3.

Contents of /etc/resolv.conf:

  • How to enable clustering of RabbitMQ Docker containers
  • Docker - how to cache added directory?
  • Why is Docker useful when running builds on slave nodes?
  • Running pudb inside docker container
  • docker-compose environment variables used as path for volumes don't get resolved
  • How can I create a Jenkins Docker image that uses provided ssh keys for jenkins user?
  • search mycompany.local
    nameserver 127.0.0.11
    options ndots:0
    

    (real company name obfuscated).

    nslookup on the host itself is 100% fine, but from within container any external hostname look fails (can’t event run apt-get update).
    The same symptoms persist in all my hosts in the 4-node cluster.
    Note that internal service name resolution seems to be working between the containers.

    Running the same application directly on my laptop (on same office network) hostnames resolve fine.

    This is becoming a bit of a slow moving disaster.

    The cluster involved is still a pre-1.12 build, it that might have any bearing.

  • Build failed while appending line in source of docker container
  • port forwarding in docker-compose
  • php-fpm error “no input file specified” with Docker
  • Friendly URL between docker containers/images
  • Mesos slave won't start docker
  • docker login returns a 404 for artifactory
  • 2 Solutions collect form web for “All external DNS queries fail from within docker container”

    In Linux, the lo or localhost interface will have the address 127.0.0.1/8 (i.e. netmask 255.0.0.0). That netmask covers this entire range:

    127.0.0.0 - 127.255.255.255
    

    Since 127.0.0.11 falls into this range, connections to that address will attempt to route via the lo interface (inside the container) as a connected route. Unless your container has that address configured internally and has a DNS resolver listening on that address, this will result in a connection timeout.

    You can probably solve this by either routing 127.0.0.11 out the main interface of the container (e.g. eth0), or by changing the DNS resolver address so it is outside of 127.0.0.0/8.

    You can also set DNS server IP(s) explicitly.

    docker run --dns 1.2.3.4                  # set one server
    docker run --dns 1.2.3.4 --dns 5.6.7.8    # set multiple servers
    

    Or using docker-compose.yml:

    dns: 1.2.3.4
    
    dns:
      - 1.2.3.4
      - 5.6.7.8
    

    Here’s the setup I use:

    1. Install dnsmasq.
    2. Run echo interface=docker0 > /etc/dnsmasq.d/docker
    3. Restart dnsmasq.
    4. Add –dns 172.17.0.1 to you docker-run or to the Docker daemon (adding it to the DOCKER_OPTS variable in /etc/default/docker or editing the ExecStart directive in /lib/systemd/system/docker.service).
    5. Restart Docker.

    Now you have all your containers pointing to Dnsmasq as a DNS resolver. Another plus side is that in your entries in /etc/hosts are resolved as well.

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.