Wednesday, October 20, 2021

Logstash: How to check if a field exists?

 


TL;DR: 


 if [yourField] or [yourField] =~ /.*/ {
    # Your code
 }


Why?


In Logstash [yourField] could come in one of these datatypes:
  • [yourField] = String
  • [yourField] = Null
  • [yourField] = Number
  • [yourField] = Boolean
i.e. [level] = "ERROR"

And also could have sub-fields and therefore these combinations:
  • [yourField][yourSub] = String
  • [yourField][yourSub] = Null
  • [yourField][yourSub] = Number
  • [yourField][yourSub] = Boolean
i.e. [request][time_milliseconds] = 73


If we use "if [yourField]" to check if the field exists, turns out that not always matches and if we use "if [yourField] =~ /.*/" not always matches.

Result:


if [yourField]

if [yourField] =~ /.*/




[yourField] = "foo"

Match

Match

[yourField] = ""

NO

Match

[yourField] = 0

Match

NO

[yourField] = True

Match

NO




[yourField][yourSub] = "foo"

Match

NO

[yourField][yourSub] = ""

Match

NO

[yourField][yourSub] = 0

Match

NO

[yourField][yourSub] = True

Match

NO


  • "if [yourField]" won't work when the field exists but is Null.
  • "if [yourField] =~ /.*/" won't work when is a number, boolean or has sub-fields.

The safer approach is:

if [yourField] or [yourField] =~ /.*/ {

    # Your code

}