Indices Red/Dangling when upgrading Elasticsearch cluster from 2.x to 5.x in docker

I manage a small ELK stack which resides on one docker host with a 3 node elasticsearch cluster, a master, a client node, and a data node each running elasticsearch 2.4.x. Each of these nodes had the same host directory bind mounted as the elasticsearch data directory even though only the data node needed it.

While testing the upgrade path to 5.x I was running into a very strange issue. The cluster would come back up, but would not initialize any of the 2.x created indices, throwing errors:

  • Setting $PATH in docker image?
  • Connectivity issue to Cassandra cluster from Docker container (Java)
  • How can I show only the allocated resources in Docker container?
  • How to determine what containers use the docker volume?
  • Install R on windows nanoserver image using powershell
  • Only able to run DinD containers in detached mode
  • [o.e.g.DanglingIndicesState] [elastic-data] [[logstash-2017.02.01/pBco8d7dQAqmZoI37vUIOQ]] dangling index exists on local file system, but not in cluster metadata, auto import to cluster state

    The indices would never initialize and remain red. The stack would create new indices fine and if I deleted these indices the system would work perfectly fine, but that data loss is definitely sub-optimal if I was going to do this on a production system.

  • Reverse proxy between a unicorn server and Galera cluster
  • Linux Docker container on Windows Server 2016
  • docker push to the registry with changed Dockerfile and same tag
  • How to properly setup a data volume container?
  • How do i deploy from GitLab CI to Google Container Engine instance using Docker?
  • Can you have a Docker registry with credentials?
  • One Solution collect form web for “Indices Red/Dangling when upgrading Elasticsearch cluster from 2.x to 5.x in docker”

    The fact the the master and client nodes had been mounting the data directory turned out to be the cause of this issue. Elasticsearch 5.x enforces a default limit of 1 node to a data directory and while the master and client 2.x nodes had not been actively data managing, they had affected the folder structuring. I was able to get a clean upgrade with all green indices by first removing the bind mounts from master and client on the 2.x cluster and letting that sort itself out, then upgrading to 5.x. Hope this helps anybody else who runs into this issue.

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.