Indices Red/Dangling when upgrading Elasticsearch cluster from 2.x to 5.x in docker

I manage a small ELK stack which resides on one docker host with a 3 node elasticsearch cluster, a master, a client node, and a data node each running elasticsearch 2.4.x. Each of these nodes had the same host directory bind mounted as the elasticsearch data directory even though only the data node needed it.

While testing the upgrade path to 5.x I was running into a very strange issue. The cluster would come back up, but would not initialize any of the 2.x created indices, throwing errors:

  • What features do Docker for Mac/Windows provide over Docker Toolbox on my development box?
  • run a meteor app on docker - doesn't open in localhost
  • Error :Could not connect to Redis at redis:6379: Name or service not known
  • How to assign a name to running container in docker?
  • Howto run a Prestashop docker container with persistent data?
  • ebean-orm.xml not reloading after change
  • [o.e.g.DanglingIndicesState] [elastic-data] [[logstash-2017.02.01/pBco8d7dQAqmZoI37vUIOQ]] dangling index exists on local file system, but not in cluster metadata, auto import to cluster state

    The indices would never initialize and remain red. The stack would create new indices fine and if I deleted these indices the system would work perfectly fine, but that data loss is definitely sub-optimal if I was going to do this on a production system.

  • docker process not running in background
  • How to configure list of containers / apps for Docker to run?
  • Export Consul nodes from Key/Value store as Prometheus targets
  • How to create an serverless infrastructure with docker?
  • Mounting host volume for mongo data in docker compose container
  • How do I publish an ASP.NET Core application in a Docker container?
  • One Solution collect form web for “Indices Red/Dangling when upgrading Elasticsearch cluster from 2.x to 5.x in docker”

    The fact the the master and client nodes had been mounting the data directory turned out to be the cause of this issue. Elasticsearch 5.x enforces a default limit of 1 node to a data directory and while the master and client 2.x nodes had not been actively data managing, they had affected the folder structuring. I was able to get a clean upgrade with all green indices by first removing the bind mounts from master and client on the 2.x cluster and letting that sort itself out, then upgrading to 5.x. Hope this helps anybody else who runs into this issue.

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.