Adding custom states to god September 18th, 2008

God is a monitoring-framework written in Ruby, with Ruby config files.

For one of our projects I’m using it to monitor some of the application processes (mongrels, a stirling, a juggernaut server, an engine). Most of these have a normal lifecycle: they get initialized, they start, they crash/go down, and god restarts them.

the engine process, however, has a different life-cycle. It should start when god does, but when it crashes, it has to stay offline, and I should receive an email saying it crashed. Then I could go in to inspect what made it crash. Apparently this isn’t included in god by default. Luckily, ruby’s open class infrastructure allows you to add items to constant-arrays, so you can just add your own states.

ENGINE_ROOT=PROJECT_ROOT + '/engine/'

# Add the stopped state
God::Watch::VALID_STATES << :stopped

God.watch do |w|
  w.name = "engine"

  w.interval = 5.seconds # default

  w.start = "#{ENGINE_ROOT}/bin/engine start"
  w.stop  = "#{ENGINE_ROOT}/bin/engine start"
  w.pid_file = "#{ENGINE_ROOT}/log/engine.pid"

..snip..
  # Just mail us if the process dies
  w.transition(:up, :stopped) do |on|
    on.condition(:process_running) do |c|
      c.interval = 3
      c.notify = {:contacts => ['openminds'], :priority => 'URGENT'}
      c.running = false
    end
  end
..snip..
end

And that’s how easy that is ;-)

tags: l 1 comments »