Tag: logstash

Dockerize Logstash, Redis setup

Have a Logstash process using file as input sending events to Redis and from there to second Logstash process and over to custom http process. So, Logstash –> Redis –> Logstash –> Http In order to dockerize the setup I was thinking of keeping Logstash –> Redis on one container and linking that with Redis […]

problems running logstash with -f flag in docker

I’m trying to run the logstash container in red hat 7 with the command: docker run -v /home/logstash/config:/conf -v /home/docker/logs:/logs logstash logstash -f /conf/logstash.conf –verbose and the response received is: {:timestamp=>”2016-05-05T10:21:20.765000+0000″, :message=>”translation missing: en.logstash.runner.configuration.file-not-found”, :level=>:error} {:timestamp=>”2016-05-05T10:21:20.770000+0000″, :message=>”starting agent”, :level=>:info} and the logstash container is not running. If I execute the folowing command: docker run -dit […]

How to get tomcat log from docker container running in atomic host

I am working on Log monitoring, I have requirement of getting tomcat application server logs(example: catalina.log) running in the docker container(container is running in atomic host) and passing it to Logstash server using rsyslog. I am able to get the docker container related logs, but not able to get the tomcat server and application logs […]

Filebeat is not forwarding logs

I followed all the steps mentioned here readthedocs but my filebeat is not sending logs to the url http://localhost:9200/_search?pretty. my filebeat.yml file is – https://ghostbin.com/paste/rrjeh I didn’t config any logstash file as mentioned. my registry file is filling after starting of filebeat and filebeat log (syslog) is showing no error regarding filebeat

Logstash cannot receive data with Kafka

I’m using the Docker ELK container from https://hub.docker.com/r/sebp/elk/, according to the logstash-logs everything is fine and running. Now I try to receive data from Kafka and write into ES with the following config: input { kafka { topic_id => “collectortopic” zk_connect => “172.17.0.2:2181” type => “kafka-input” } } output { elasticsearch { hosts => [“172.17.0.4:9200”] […]

How to configure Logstash to parse AWS ELB logs?

I want to parse AWS ELB logs [stored in a S3 bucket] from Logstash that is set up inside a dockerised ELK stack. I cloned this repo. Here are it’s docs. I added my logstash config file like this [and commented out all the others]: # AWS ELB configuration file ADD ./aws_elb_logs.conf /etc/logstash/conf.d/aws_elb_logs.conf The config […]

Log level as a field for Docker GELF logging driver

I want to get stdout logs from a docker container and send them to ELK stack. So far, I know that there is a GELF logging driver in Docker. However, I can’t figure out how I can parse ERROR, WARNING or DEBUG messages from the message and put them in a new field like log_level […]

Infinite logging with logstash through kafka

Overview I’m attempting to get Logstash working with Kafka, using docker containers. For my Kafka container, I’m using the spotify build. I am also using the official Logstash container. I want to achieve something like the following: Here are the configs I am using for Logstash: file_kafka.conf input { file { path => [“/data/*.log”] start_position […]

Two Logstash instances on same Docker container

Am wondering if there is a way two logstash processes with separate configurations can be run on a single Docker container. My setup has a Logstash process using file as input sending events to Redis and from there to second Logstash process and over to custom http process. So, Logstash –> Redis –> Logstash –> […]

Logstash crashes as soon as it starts

As stated in the title, Logstash crashes as soon as it starts. I’m running it through Docker. This is my output: logstash_1 | {:timestamp=>”2016-10-25T13:14:31.470000+0000″, :message=>”Reading config file”, :config_file=>”/etc/logstash/conf.d/logstash.conf”, :level=>:debug, :file=>”logstash/config/loader.rb”, :line=>”69″, :method=>”local_config”} logstash_1 | {:timestamp=>”2016-10-25T13:14:31.570000+0000″, :message=>”Plugin not defined in namespace, checking for plugin file”, :type=>”output”, :name=>”stdout”, :path=>”logstash/outputs/stdout”, :level=>:debug, :file=>”logstash/plugin.rb”, :line=>”86″, :method=>”lookup”} logstash_1 | {:timestamp=>”2016-10-25T13:14:31.584000+0000″, :message=>”starting […]

Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.