Reliable centralized logging in Kubernetes/Docker with Elastic Stack

We are setting up a Kubernetes cluster on CoreOS to run 50+ different applications (mostly Java apps), each of which produces logs potentially in its own format.

We’re looking to centralise logs from all containers with the Elastic Stack (formerly ELK stack), with some specific requirements:

  • Docker volume mount doesn't exist
  • How can I add a volume to an existing Docker container?
  • How to launch docker image with --entrypoint and arguments?
  • Copy a file from host OS to docker image
  • How to use docker inside docker container in a save way
  • Installing docker with devstack
    1. Reliability in the face of network, container or node failures.
    2. Exactly-once processing for each log statement. Even if a failure occurred, the solution must remember where it left off and it should continue dispatching logs from that point onwards, once healthy again.
    3. Configure log grokking patterns within the configuration of the pod/replication controller. We’d like to avoid configuring app-specific patterns in a centralised, common element (e.g. Logstash).

    The solution we have come up with is to use:

    1. Logspout to consume Docker logs – forwarding them to a local…
    2. rsyslog daemon (e.g. syslog://localhost:514), which would forward them to a…
    3. Logstash instance running in the cluster and exposed via a Kubernetes Service.
    4. Logstash would filter and transform the log files based on container ID, and would dispatch the result to Elasticsearch.

    Would this solution be reliable? This solution seems to cover all our requirements except for #3 (grokking patterns alongside apps).

    Do you have any suggestions? We are open to using Filebeat, fluentd or other components.

  • How to run command on stopped Docker container?
  • Golang Microservices can't communicate using Docker for Mac
  • docker on windows not working
  • Dockerfile vim installation does not work
  • How to configure list of containers / apps for Docker to run?
  • How to recreate Docker container?
  • Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.