Logs not being flushed to Elasticsearch container through Fluentd

I have a local setup running 2 conainers –

One for Elasticsearch (setup for development as detailed here – https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html). This I run as directed in the article using – docker run -p 9200:9200 -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" docker.elastic.co/elasticsearch/elasticsearch:5.4.1

  • In Influxdb, I cannot retrieve correct values in time range
  • Docker DNS for Service Discovery to resolve Windows Container´s address by name does not work consistently
  • Django channels and docker-compose error
  • Many Docker Containers and Thrift
  • docker cannot specify container connection
  • How to see where my website is running after creating a container using docker image?
  • Another as a Fluentd aggregator (using this base image – https://hub.docker.com/r/fluent/fluentd/). My fluent.conf for testing purposes is as follows :

    <source>
        @type forward
        port 24224
    </source>
    
    <match **>
        @type elasticsearch
        host 172.17.0.2    # Verified internal IP address of the ES container
        port 9200
        user elastic
        password changeme
        index_name fluentd
        buffer_type memory
        flush_interval 60
        retry_limit 17
        retry_wait 1.0
        include_tag_key true
        tag_key docker.test
        reconnect_on_error true
    </match>
    

    This I start with the command – docker run -p 24224:24224 -v /data:/fluentd/log vg/fluentd:latest

    When I run my processes (that generate logs), and run these 2 containers, I see the following towards the end of stdout for the Fluentd container –

    2017-06-15 12:16:33 +0000 [info]: Connection opened to Elasticsearch cluster => {:host=>"172.17.0.2", :port=>9200, :scheme=>"http", :user=>"elastic", :password=>"obfuscated"}

    However, beyond this, I see no logs. When I login to http://localhost:9200 I only see the Elasticsearch welcome message.

    I know the logs are reaching the Fluentd container, because when I change fluent.conf to redirect to a file, I see all the logs as expected. What am I doing wrong in my setup of Elasticsearch? How can I get to seeing all the indexes laid out correctly in my browser / through Kibana?

  • Docker nmap one container from inside another?
  • Whats exactly mean by build, ship and run any app, anywhere with Docker
  • Docker ubuntu 16.04 container 64-bit run 32-bit applications
  • How to build an image with Dockerfile in Kitematic?
  • Docker - running commands from all containers
  • Eureka clients in docker container
  • One Solution collect form web for “Logs not being flushed to Elasticsearch container through Fluentd”

    It seems that you are in the right track. Just check the indexes that were created in elasticsearch as follows:

    curl 'localhost:9200/_cat/indices?v'
    

    Docs:
    https://www.elastic.co/guide/en/elasticsearch/reference/1.4/_list_all_indexes.html

    There you can see each index name. So pick one and search within it:

    curl 'localhost:9200/INDEXNAME/_search'
    

    Docs: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html

    However I recommend you to use kibana in order to have a better human experience. Just start it and by default it searches for an elastic in localhost. In the interface’s config put the index name that you now know, and start to play with it.

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.