I'm currently monitoring a large network with Hobbit and have been tasked with lowering the amount of false (or at least irrelevant) alarms. At the top of my list are the tests "http" and "conn", initiated by bbtest-net. This command checks ping, ssh, etc, and if for instance a ping times out, it immediately sets the status to red. One minute later, the bbretest command kicks in, checks all the newly reddened hosts, and finds it to be green again. This happens all the time, and it clutters up my log.
Is there any way for me to make Hobbit report a red status AFTER bbretest has been run the first time?
Best Solution
I think your best bet is to shun the stock Hobbit service tests and write your own one. It's not difficult.
It is a good idea that your test script will not go red unless several successive attempts fail.
You can disable the standard Hobbit ones and use your own instead. Having said that, the default behaviour of the "conn" test seems fairly reasonable (going red immediately if the server doesn't ping).
Unfortunately there's no option on the Hobbit alerting system to only alert if a problem persists for X minutes, that would be really useful - but I'm sure you could do that as well with a custom alerting script.