How to aggregate logs for all docker containers in mesos

I have multiple microservices written in node and microservices are installed into the docker container and we are using Mesos+Marathon for clustering.

How can I aggregate the logs of all the containers(microservices) on different instance.?

  • What are the best practices for tagging Docker Hub versions
  • Exposing spring boot random server port using docker
  • Docker in AWS Elastic Beanstalk: executable file not found in $PATH"
  • Docker run - User group not working as expected?
  • Can not connect to elasticsearch container in docker
  • How can you make the docker container use the host machine's /etc/hosts file?
  • Write logs in PHP outside Docker container
  • Docker-compose does not install or run properly on boot2docker
  • java -jar lib/kvstore.jar ping - ConnectException when using IPAddress
  • Docker Compose not updating content
  • How to run “docker-machine create” in OS X?
  • It's possible to add an interface to a Docker container for speed up connectivity?
  • 3 Solutions collect form web for “How to aggregate logs for all docker containers in mesos”

    This is a broad question but I suggest you setup an Elastic Search, Logstash, Kibana stack (ELK)

    https://www.elastic.co/products/elasticsearch

    https://www.elastic.co/products/logstash

    https://www.elastic.co/products/kibana

    Then on each one of your containers you can run the logstash forwarder/shipper to send logs to your logstash frontend.

    Logs get stored in Elastic Search and then you search for them using Kibana or the Elastic Search API

    Hope it helps.

    We’re using Docker+Mesos as well and are shipping all logs to a log analytics service (it’s the service the company I work for offers, http://logz.io). There a couple of ways to achieve that:
    – Have a log shipper agent within each docker – an agent like rsyslog, nxlog, logstash, logstash-forwarder – that agent would ship data to a central logging solution
    – Create a Docker that is running the shipper agent (like rsyslog, nxlog, logstash, logstash-forwarder) and that agent reads logs from all Dockers on each machine and ships them to a central location – this is the path we’re taking

    I am also doing some Docker + Mesos + Marathon work so, I guess, I am going to face same doubt that you have.

    I don’t know if there’s any native solution yet. But there’s a blog by the folks at elastic.io on how they went about solving this issue.

    Here’s the link – Log aggregation for Docker containers in Mesos / Marathon cluster

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.