Monday, February 27, 2017

Unix like " tail -f " command for Elasticsearch (using Python)


Humble implementation of a Unix like "tail" command for Elasticsearch using Python.

Tested with Logstash indexed content.


$ git clone
$ cd elasticsearch-python/
$ python

Basic usage

The only mandatory parameter is --endpoint


$ python --endpoint

By default the last 10 lines of log are displayed. You can change this behaviour with --docs or -n switch.

Example: To display the last 50 lines.

$ python --endpoint -n 50

To have continuous output use -f or --nonstop


$ python --endpoint -f

By default ES type = apache is used. You can select other types with --type


$ python --endpoint --type java

$ python --endpoint --type apache


By default the more recent Logstash Index is used. Optionally you can specify the desired index name using --index


$ python --endpoint --index logstash-2016.08.08

When using --type java there are two other selectors available: --javalevel and --javaclass


$ python --endpoint --type java --javalevel ERROR

$ python --endpoint --type java --javaclas

When using --type apache there are two other selectors available: --httpresponse and --httpmethod


$ python --endpoint --type apache --httpresponse 404

$ python --endpoint --type apache --httpmethod POST

To display the native Elasticsearch timestamp of each event use --showheaders (Convenient when adding a |grep on the output to do additional filtering):


python --endpoint --showheaders

To display events belonging to a particular host and ignore the rest use --hostname


$ python --endpoint --hostname

Connection modes

The default protocol is HTTP (Port 80) but any other port can be specified under --endpoint

Example for HTTPS:

$ python -f --endpoint 


$ python -f --endpoint

Example for Elasticsearch native port:

$ python -f --endpoint

Known issues

You need elasticsearch Python module installed

If you don't have it, run 'sudo pip install elasticsearch' to install it.

TLS error: "elasticsearch.exceptions.ConnectionError: ConnectionError(('Connection failed.', CannotSendRequest())) caused by: ConnectionError(('Connection failed.', CannotSendRequest()))"

Update urllib3 running 'sudo pip install --upgrade urllib3' or use a non HTTPS Endpoint URL.

Error: "check_index: No index found! Exiting"

The current script only cares for indices with the "logstash-" prefix. If you use any other prefix to index your logs it will fail.
If you think this shouldn't be this way please let me know in the comments.


Done for fun. Feel free to comment on bugs or additional desired features.

Thank you!