As anyone who’s spent any time debugging ELK, or more accurately logstash will tell you, it’s a slow process. Everyone instantly leaps to share Grok Constructor or Grok Debugger, which are both super useful, but not always the point. There’s a lot more to debugging logstash than just getting regexs right (though that does take the other 90% of the time). Debugging actual logstash configs is a slow process, thanks to the JVM and the huuuuuuuuge cost of spinning one up every time. There’s Drip which claims to speed that up after the first run, but I didn’t find that to actually work.
Logstash is pretty nice about splitting things up. I have the following directory tree
[durazac]% tree testinglogstash
testinglogstash
├── 01.rules.conf
├── 10-nginx-access.conf
├── 99.rules.conf
└── patterns
├── common
├── dpkg
└── nginx
The 01 and 99 are p simple.
[durazac:testinglogstash]% cat 01.rules.conf
input {
file {
path => "/Users/bea/some_stupid_logfile.asn1"
}
}
[durazac:testinglogstash]% cat 99.rules.conf
output {
stdout {}
}
So I just have a logfile that gets read, and then it outputs to stdout. The theory with not reading from stdin, i that Drip won’t work if using the stdin input, butttttt that doesn’t work anyway, but in practice, ultimately it’s proven to be easier.
the 10-whatever.conf just has to go in the middle, now be aware it won’t have a type as there’s no filebeat adding that, so you can either add a mutate add_tags in a prior file, or just comment them out for the sake of testing.
So, don’t worry, here’s the trick, running logstash with
logstash -f ~/testinglogstash --log.level=info --config.reload.automatic
(or even --log.level=debug in there too). This means whenever you change any of the configs, in about 3 seconds it’ll try and reload it. Now, sometimes you mess up the config and you get an ugly rubby stack trace, or it’ll fully crash, but it’s still so much quicker than the 20-30 second spin up time of a Jrubby JVM.
The ingress log file, if you want to use it multiple times, requires a new inode for logstash to pick it up each time. hax hax hax… I found taking the original logfile, or a sample of it if need be and doing:
[durazac]% rm some_stupid_logfile.asn1 ; gshuf -n 5 ~/original_logfile >some_stupid_logfile.asn1
The gshuf (or just shuf if you’re not installing it from brew) is just to add different logfiles every time, which may or may not be desirable. Adding too many just involves a log of scrolling to if it worked.
Hopefully this is useful to someone but me! Endure!