Thursday, February 09, 2012

Remember to put a timeout on your heartbeat pings

I'm working a distributed piece of software which needs to be disabled when a tier is not reachable. To do that, I developed a Spring bean (a java service class) and I scheduled a method on it (with a @scheduled annotation). This method makes some heartbeat pings on each tier. The java service class contains a flag which indicates if all is OK. If an heartbeat fails, this boolean status is set to false. Of course, thanks to Spring, I easily injected this class into my web pages. If a user tries to access to the application and the health status is set to false, he is automatically redirected to a page which indicates that the service is closed. But, let's go back to pings.

I scheduled my pings to be executed each minute. That was for me a good value. But, a problem appeared during my first real tests. I realized that pings durations could be longer than one minute because the network connect() method blocked my thread. Of course, I forgot to manage short timeouts!

After a few googlings, I found that : http://blog.smartkey.co.uk/2011/09/adding-a-thread-timeout-to-methods-in-java/

This is a quite simple code. I copied it here :

public void doWork() {
//perform some long running task here...
}

public void doWorkWithTimeout(int timeoutSecs) {

//set the executor thread working
final Future future = executor.submit(new Runnable() {
public void run() {
try {
doWork();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
});

//check the outcome of the executor thread and limit the time allowed for it to complete
try {
future.get(timeoutSecs, TimeUnit.SECONDS);
} catch (Exception e) {
//ExecutionException: deliverer threw exception
//TimeoutException: didn't complete within downloadTimeoutSecs
//InterruptedException: the executor thread was interrupted

//interrupts the worker thread if necessary
future.cancel(true);

log.warn("encountered problem while doing some work", e);
}
}

Now, all is good for me and my heartbeat pings.