Passive replicas with Docker swarm mode
I’m trying to figure out a proper way to implement active/passive failover between replicas of a service with Docker swarm mode.
The service will hold a valuable in-memory state that cannot be lost, that’s why I need multiple replicas of it. The replicas will internally implement Raft, so that only the replica which is active (“leader”) at a given moment will accept requests from clients.
(If you’re unfamiliar with Raft: simply put, it is a distributed consensus algorithm, which helps implement active/passive fault-tolerant cluster of replicas. According to Raft, the active replica – the leader – replicates changes in its data to passive replicas – the followers. Only leader accepts requests from clients. If the leader fails, a new leader is elected among the followers).
As far as I understand, Docker will guarantee that a specified number of replicas are up and running, but it will balance incoming requests among all of the replicas, in the active/active manner.
How can I tell Docker to route requests only to the active replica, but still guarantee that all replicas are up?
One option is routing all requests through an additional
NGINX container, and updating its rules each time a new leader is elected. But that will be an additional hop, which I’d like to avoid.
I’m also trying to avoid external/overlapping tools such as
kubernetes, in order to keep the solution as simple as possible. (
HAProxy is not an option because I need a Linux/Windows portable solution). So currently I’m trying to understand if this can be done with
Docker swarm mode alone.
Another approach I came across is returning a failing health check from passive replicas. It does the trick with
kubernetes according to this answer, but I’m not sure it will work with Docker. How does swarm manager interpret failing health checks from task containers?
I’d appreciate any thought.