Make monit wait longer before thinking something is dead
You can check a specific service on a different interval than the default...
See SERVICE POLL TIME in the Monit documentation.
An example for your Resque program would be to check on a different number of cycles:
check process resque with pidfile /var/run/resque.pid every 5 cycles
or from the examples section:
Some servers are slow starters, like for example Java based Application Servers. So if we want to keep the poll-cycle low (i.e. < 60 seconds) but allow some services to take its time to start, the every statement is handy: check process dynamo with pidfile /etc/dynamo.pid every 2 cycles start program = "/etc/init.d/dynamo start" stop program = "/etc/init.d/dynamo stop" if failed port 8840 then alert
or you can leverage the cron-style checks.
check process resque with pidfile /var/run/resque.pid every 10 * * * *
or if you're experiencing a slow startup, you can extend the timeout in the service start command:
check process apache with pidfile /var/run/httpd.pid start program = "/etc/init.d/httpd start" with timeout 90 seconds
How do I delay the time Monit checks again, just for this process?
What you are trying to achieve could be done via "SERVICE POLL TIME" feature of monit
Monit documentation says
Services are checked in regular intervals given by the
set daemon n
statement. Checks are performed in the same order as they are written in the .monitrc file, except if dependencies are setup between services, in which case the services hierarchy may alternate the order of the checks.
One of the method to customize service poll is
- custom interval based on poll cycle length multiple
EVERY [number] CYCLES
check process resque with pidfile /your/app/root/tmp/pid/resque.pid every 2 cycles
Or should I solve this in another way?
I also did initial attempt to monitor resque jobs with monit because monit is a very lightweight daemon but eventually settled with GOD. I know , I know GOD is more resource hungry in comparison to monit but in case of resque we found it to be a good match.
You can also check if something has failed for X times straight:
if failed port 80 for 10 cycles then alert
Or for X times within Y polls:
if failed port 80 for 3 times within 5 cycles then alert
check filesystem rootfs with path /dev/hda1 if space usage > 80% for 5 times within 15 cycles then alert if space usage > 90% for 5 cycles then exec '/try/to/free/the/space'
A member of my team came up with a rather clever solution that allows monit to check frequently (every minute), but once it has attempted to restart the service (which takes ~10 minutes) it will wait a specified grace period before attempting to start again.
This prevents waiting too long between checks, which combined with slow start is a much larger impact to customers. It works by using an intermediate script that acts as flag to indicate monit is already taking action from the last failure.
check host bamboo with address bamboo.mysite.com if failed port 443 type tcpSSL protocol http and status = 200 and request /about.action for 3 cycles then exec "/bin/bash -c 'ps -ef | grep -v "$$" | grep -v "grep" | grep restartBamboo.sh >/dev/null 2>&1; if [ $? -ne 0 ]; then /opt/monit/scripts/restartBamboo.sh; fi'"
If bamboo (slow starting web app) is down for 3 minutes in a row, restart, BUT only if a restart script is not already running.
The the script that is called has a specified sleep that waits LONGER then the slowest start time for the service (in our case we expect to finish in ~10, so we sleep for 15)
#!/bin/bash echo "Retarting bambo by calling init.d" /etc/init.d/bamboo stop echo "Stopped completed, calling start" /etc/init.d/bamboo start echo "Done restarting bamboo, but it will run in background for sometime before available so, we are sleeping for 15 minutes" sleep 900 echo "done sleeping"
The current version of Monit (5.16) supports a timeout for the start scripts with the syntax:
<START | STOP | RESTART> [PROGRAM] = "program" [[AS] UID <number | string>] [[AS] GID <number | string>] [[WITH] TIMEOUT <number> SECOND(S)]
The docs state:
In the case of a process check, Monit will wait up to 30 seconds for the start/stop action to finish before giving up and report an error. You can override this timeout using the TIMEOUT option.
Which is what the "timeout" value will do.