I've mentioned in a earlier blog post that this website uses the forever module to ensure that it keeps running, even in the event of an uncaught exception causing the entire app to halt. However, a Node.js application can still stop working, even if an exception is never thrown. For instance, the app could have an infinite loop, or experience a deadlock. In this case, forever will not help. The forever module will only restart the app if the app's process dies.

Fortunately for me, this website is hosted on a Linux server, where detecting and restarting an unresponsive app is easy. My solution is as follows:

1. Create one or more "alive" urls that are used to determine if the web app is alive and responsive.
2. Create a scheduled job that verifies that the website is alive every N seconds, and restarts it if not.

My "alive" url just returns the current date/time and an HTTP status code of 200 if the app is running. It also logs some interesting memory usage statistics, so that I can be aware of any memory leaks that might occur.

    app.get('/alive', function(req, res) {
        var message = 'alive at ' + new Date().toISOString();
        global.logger.info(message);
        global.logger.info('rss: ' + utils.formatNumber(process.memoryUsage().rss));
        global.logger.info('heapTotal: ' + utils.formatNumber(process.memoryUsage().heapTotal));
        global.logger.info('heapUsed: ' + utils.formatNumber(process.memoryUsage().heapUsed));

        res.send(message);
    });

I wrote a bash shell script, named "keep_alive" that determines if the app is alive. keep_alive writes to a log having a filename corresponding to the current date. This log will be useful, since it will allow me to troubleshoot website failures in the future. keep_alive uses the Linux "curl" command to hit the alive url and return the status. The curl command is set to timeout after 15 seconds. The "urls" array holds all the alive urls. If you need to monitor multiple urls, just add more elements. If any of the alive urls do not yield good responses within the timeout, the app will be restarted, by calling the "bounce" shell script.

keep_alive:

#/bin/bash

. ~/.bashrc

pushd /home/ericbtco

log="logs/keep_alive/`date +\"%Y-%m-%d\"`.log"
urls=(http://www.ericbt.com/alive http://www.bjterrell.com)

bounce=false

for url in "${urls[@]}"
do

  echo -n checking $url at `date`: >> $log

  if curl -X GET --max-time 15 -I $url | grep -q "200 OK"; then
    echo " alive" >> $log
  else
    echo " **** DEAD ****" >> $log

    bounce=true
    break
  fi

done

if $bounce; then
  echo "**** BOUNCING ****" >> $log
  . ./bounce >> $log
fi

echo "" >> $log

popd

The bounce shell script restarts the app by first stopping the process with the "forever stopall" command. The app may be running but unresponsive, so the script does not assume that the process has halted. Then the script runs the "run_ericbt" shell script to re-launch the app.

bounce:

#!/bin/bash

. ~/.bashrc

pushd /home/ericbtco

forever stopall

echo "restarting ericbt.com"
bash ./run_ericbt

echo "running jobs"

forever list

popd

The Linux cron command runs the keep_alive script once a minute. Here's the crontab entry:

*/1 * * * * /home/ericbtco/keep_alive

In addition to restarting the app when it becomes unresponsive, the app will also be restarted after a server reboot. About one minute after the reboot, cron will run "keep_running". The app will not respond, and will be restarted.