How to run HDFS cluster without DNS

I’m building a local HDFS dev environment (actually hadoop + mesos + zk + kafka) to ease development of Spark jobs and facilitate local integrated testing.
All other components are working fine but I’m having issues with HDFS. When the Data Node tries to connect to the name node, I get a DisallowedDataNodeException:

org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode

Most questions related to the same issue boil down to name resolution of the data node at the name node either static through the etc/hosts files or using dns. Static resolution is not an option with docker, as I don’t know the data nodes when the name node container is created. I would like to avoid creating and maintaining an additional DNS service. Ideally, I would like to wire everything using the --link feature of docker.

  • Command line shortcut to connect to a docker container [closed]
  • How to import CSV data into elasticsearch using docker?
  • docker cannot start container after reboot use '--restart-always'
  • Docker images across multiple disks
  • getting to the files system of the Alpine linux which runs Docker-for-mac
  • How to run and install Protractor E2E tests in Docker container for Laravel?
  • Is there a way to configure HDFS in such a way that it only uses IP addresses to work?

    I found this property and set to false, but it didn’t do the trick:

    dfs.namenode.datanode.registration.ip-hostname-check (default: true)

    Is there a way to have a multi-node local HDFS cluster working only using IP addresses and without using DNS?

  • docker ubuntu container filesystem
  • Connect to meteor mongo from local docker image on Mac
  • Self hosted kibana 5.1.1 showing blank pages after connected to AWS Elasticsearch 5.1.1
  • How do you handle nontrivial environment differences with docker?
  • boot2docker equivalent of docker exec
  • custom docker image is not running postgres
  • One Solution collect form web for “How to run HDFS cluster without DNS”

    I would look at reconfiguring your Docker image to use a different hosts file [1]. In particular:

    1. In the Dockerfile(s), do the switch-a-roo [1]
    2. Bring up the master node
    3. Bring up the data nodes, linked
    4. Before starting the datanode, copy over the /etc/hosts to the new location, /tmp/hosts
    5. Append the master node’s name and master node ip to the new hosts file

    Hope this works for you!

    [1] https://github.com/dotcloud/docker/issues/2267#issuecomment-40364340

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.