Reliable centralized logging in Kubernetes/Docker with Elastic Stack

We are setting up a Kubernetes cluster on CoreOS to run 50+ different applications (mostly Java apps), each of which produces logs potentially in its own format.

We’re looking to centralise logs from all containers with the Elastic Stack (formerly ELK stack), with some specific requirements:

  • Init Layer in Docker
  • How should a registry be run in a docker swarm?
  • Unable to locate package language-pack-en
  • How to connect to mysql created by docker-compose
  • container is in waiting state, kubernetes, docker container
  • File not found in docker container
    1. Reliability in the face of network, container or node failures.
    2. Exactly-once processing for each log statement. Even if a failure occurred, the solution must remember where it left off and it should continue dispatching logs from that point onwards, once healthy again.
    3. Configure log grokking patterns within the configuration of the pod/replication controller. We’d like to avoid configuring app-specific patterns in a centralised, common element (e.g. Logstash).

    The solution we have come up with is to use:

    1. Logspout to consume Docker logs – forwarding them to a local…
    2. rsyslog daemon (e.g. syslog://localhost:514), which would forward them to a…
    3. Logstash instance running in the cluster and exposed via a Kubernetes Service.
    4. Logstash would filter and transform the log files based on container ID, and would dispatch the result to Elasticsearch.

    Would this solution be reliable? This solution seems to cover all our requirements except for #3 (grokking patterns alongside apps).

    Do you have any suggestions? We are open to using Filebeat, fluentd or other components.

  • Is there any way to create a link between file inside container docker and host?
  • Minimal configuration for Apache reverse proxy in Docker container
  • How many WordPress instances can i run on Google Compute engine in a single google cloud platform project.
  • Running alembic in a python container that depends upon a mysql container
  • Docker: How to use selenium server to do nightwatchJS test?
  • Ubuntu16 Docker-engine startup error: “no sockets found via socket activation: make sure the service was started by systemd”
  • Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.