How to aggregate logs for all docker containers in mesos

I have multiple microservices written in node and microservices are installed into the docker container and we are using Mesos+Marathon for clustering.

How can I aggregate the logs of all the containers(microservices) on different instance.?

  • How does docker-compose link work?
  • Logging to logstash from python
  • Issue when I try to create a docker image [duplicate]
  • Node cannot join Swarm Cluster
  • How do I communicate from Docker Hub to Jenkins via web hook?
  • ssh pseudo-tty causes PTY allocation request failed on channel 0
  • Tagging docker images and adding variables in docker-compose
  • How to organize containers “horizontally” inside Kubernetes pods?
  • Docker --volumes-from permissions
  • Why is inter container communication in Docker the default?
  • How to connect to Cassandra in Docker
  • curl in Docker container fails b/c of ssl
  • 3 Solutions collect form web for “How to aggregate logs for all docker containers in mesos”

    This is a broad question but I suggest you setup an Elastic Search, Logstash, Kibana stack (ELK)

    https://www.elastic.co/products/elasticsearch

    https://www.elastic.co/products/logstash

    https://www.elastic.co/products/kibana

    Then on each one of your containers you can run the logstash forwarder/shipper to send logs to your logstash frontend.

    Logs get stored in Elastic Search and then you search for them using Kibana or the Elastic Search API

    Hope it helps.

    We’re using Docker+Mesos as well and are shipping all logs to a log analytics service (it’s the service the company I work for offers, http://logz.io). There a couple of ways to achieve that:
    – Have a log shipper agent within each docker – an agent like rsyslog, nxlog, logstash, logstash-forwarder – that agent would ship data to a central logging solution
    – Create a Docker that is running the shipper agent (like rsyslog, nxlog, logstash, logstash-forwarder) and that agent reads logs from all Dockers on each machine and ships them to a central location – this is the path we’re taking

    I am also doing some Docker + Mesos + Marathon work so, I guess, I am going to face same doubt that you have.

    I don’t know if there’s any native solution yet. But there’s a blog by the folks at elastic.io on how they went about solving this issue.

    Here’s the link – Log aggregation for Docker containers in Mesos / Marathon cluster

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.