Monday, February 27, 2017

Unix like " tail -f " command for Elasticsearch (using Python)

elasticsearch-logo-tail-command-linux-blog-domenech-org



Humble implementation of a Unix like "tail" command for Elasticsearch using Python.

Tested with Logstash indexed content.

Install


$ git clone https://github.com/juan-domenech/elasticsearch-python.git
$ cd elasticsearch-python/
$ python elasticsearch-tail.py



Basic usage

The only mandatory parameter is --endpoint

Example:

$ python elasticsearch-tail.py --endpoint http://elk.example.com



By default the last 10 lines of log are displayed. You can change this behaviour with --docs or -n switch.

Example: To display the last 50 lines.

$ python elasticsearch-tail.py --endpoint http://elk.example.com -n 50



To have continuous output use -f or --nonstop

Example:

$ python elasticsearch-tail.py --endpoint http://elk.example.com -f



By default ES type = apache is used. You can select other types with --type

Examples:

$ python elasticsearch-tail.py --endpoint http://elk.example.com --type java

$ python elasticsearch-tail.py --endpoint http://elk.example.com --type apache


Advanced


By default the more recent Logstash Index is used. Optionally you can specify the desired index name using --index

Example:

$ python elasticsearch-tail.py --endpoint http://elk.example.com --index logstash-2016.08.08



When using --type java there are two other selectors available: --javalevel and --javaclass

Examples:

$ python elasticsearch-tail.py --endpoint http://elk.example.com --type java --javalevel ERROR

$ python elasticsearch-tail.py --endpoint http://elk.example.com --type java --javaclas error.handler.java.class



When using --type apache there are two other selectors available: --httpresponse and --httpmethod

Examples:

$ python elasticsearch-tail.py --endpoint http://elk.example.com --type apache --httpresponse 404

$ python elasticsearch-tail.py --endpoint http://elk.example.com --type apache --httpmethod POST



To display the native Elasticsearch timestamp of each event use --showheaders (Convenient when adding a |grep on the output to do additional filtering):

Example:

python elasticsearch-tail.py --endpoint http://elk.example.com --showheaders



To display events belonging to a particular host and ignore the rest use --hostname

Example:

$ python elasticsearch-tail.py --endpoint http://elk.example.com --hostname server1.example.com



Connection modes


The default protocol is HTTP (Port 80) but any other port can be specified under --endpoint

Example for HTTPS:

$ python elasticsearch-tail.py -f --endpoint https://elk.example.com 

or

$ python elasticsearch-tail.py -f --endpoint https://elk.example.com:443



Example for Elasticsearch native port:

$ python elasticsearch-tail.py -f --endpoint http://elk.example.com:9200




Known issues

You need elasticsearch Python module installed


If you don't have it, run 'sudo pip install elasticsearch' to install it.


TLS error: "elasticsearch.exceptions.ConnectionError: ConnectionError(('Connection failed.', CannotSendRequest())) caused by: ConnectionError(('Connection failed.', CannotSendRequest()))"


Update urllib3 running 'sudo pip install --upgrade urllib3' or use a non HTTPS Endpoint URL.


Error: "check_index: No index found! Exiting"


The current script only cares for indices with the "logstash-" prefix. If you use any other prefix to index your logs it will fail.
If you think this shouldn't be this way please let me know in the comments.



elasticsearch-logo-tail-command-linux-python-blog-domenech-org

Done for fun. Feel free to comment on bugs or additional desired features.

Thank you!

6 comments: