Monday, February 27, 2017

Unix like " tail -f " command for Elasticsearch (using Python)

elasticsearch-logo-tail-command-linux-blog-domenech-org



Humble implementation of a Unix like "tail" command for Elasticsearch using Python.

Tested with Logstash indexed content.

Install


$ git clone https://github.com/juan-domenech/elasticsearch-python.git
$ cd elasticsearch-python/
$ python elasticsearch-tail.py



Basic usage

The only mandatory parameter is --endpoint

Example:

$ python elasticsearch-tail.py --endpoint http://elk.example.com



By default the last 10 lines of log are displayed. You can change this behaviour with --docs or -n switch.

Example: To display the last 50 lines.

$ python elasticsearch-tail.py --endpoint http://elk.example.com -n 50



To have continuous output use -f or --nonstop

Example:

$ python elasticsearch-tail.py --endpoint http://elk.example.com -f



By default ES type = apache is used. You can select other types with --type

Examples:

$ python elasticsearch-tail.py --endpoint http://elk.example.com --type java

$ python elasticsearch-tail.py --endpoint http://elk.example.com --type apache


Advanced


By default the more recent Logstash Index is used. Optionally you can specify the desired index name using --index

Example:

$ python elasticsearch-tail.py --endpoint http://elk.example.com --index logstash-2016.08.08



When using --type java there are two other selectors available: --javalevel and --javaclass

Examples:

$ python elasticsearch-tail.py --endpoint http://elk.example.com --type java --javalevel ERROR

$ python elasticsearch-tail.py --endpoint http://elk.example.com --type java --javaclas error.handler.java.class



When using --type apache there are two other selectors available: --httpresponse and --httpmethod

Examples:

$ python elasticsearch-tail.py --endpoint http://elk.example.com --type apache --httpresponse 404

$ python elasticsearch-tail.py --endpoint http://elk.example.com --type apache --httpmethod POST



To display the native Elasticsearch timestamp of each event use --showheaders (Convenient when adding a |grep on the output to do additional filtering):

Example:

python elasticsearch-tail.py --endpoint http://elk.example.com --showheaders



To display events belonging to a particular host and ignore the rest use --hostname

Example:

$ python elasticsearch-tail.py --endpoint http://elk.example.com --hostname server1.example.com



Connection modes


The default protocol is HTTP (Port 80) but any other port can be specified under --endpoint

Example for HTTPS:

$ python elasticsearch-tail.py -f --endpoint https://elk.example.com 

or

$ python elasticsearch-tail.py -f --endpoint https://elk.example.com:443



Example for Elasticsearch native port:

$ python elasticsearch-tail.py -f --endpoint http://elk.example.com:9200




Known issues

You need elasticsearch Python module installed


If you don't have it, run 'sudo pip install elasticsearch' to install it.


TLS error: "elasticsearch.exceptions.ConnectionError: ConnectionError(('Connection failed.', CannotSendRequest())) caused by: ConnectionError(('Connection failed.', CannotSendRequest()))"


Update urllib3 running 'sudo pip install --upgrade urllib3' or use a non HTTPS Endpoint URL.


Error: "check_index: No index found! Exiting"


The current script only cares for indices with the "logstash-" prefix. If you use any other prefix to index your logs it will fail.
If you think this shouldn't be this way please let me know in the comments.



elasticsearch-logo-tail-command-linux-python-blog-domenech-org

Done for fun. Feel free to comment on bugs or additional desired features.

Thank you!

9 comments:

  1. Great Article. Thank you for sharing! Really an awesome post for every one.

    IEEE Final Year projects Project Centers in Chennai are consistently sought after. Final Year Students Projects take a shot at them to improve their aptitudes, while specialists like the enjoyment in interfering with innovation. For experts, it's an alternate ball game through and through. Smaller than expected IEEE Final Year project centers ground for all fragments of CSE & IT engineers hoping to assemble. Final Year Project Domains for IT It gives you tips and rules that is progressively critical to consider while choosing any final year project point.

    Spring Framework has already made serious inroads as an integrated technology stack for building user-facing applications. Spring Framework Corporate TRaining the authors explore the idea of using Java in Big Data platforms.
    Specifically, Spring Framework provides various tasks are geared around preparing data for further analysis and visualization. Spring Training in Chennai

    ReplyDelete