Re: [SAGE] number of eggs in a basket

[David Parter <dparter@cs.wisc.edu>]

> I'd like to get some opinions regarding best practices for mission
> critical systems with multiple services. 

....

> I do not like having all my eggs in this one basket, but on the other
> hand distributing the services to several machines seems to complicate
> things and increase the likeliness of one of the services failing.
> 
> So... what are your comments/experiences?  How many eggs do you keep in
> your basket(s)?

We believe very strongly in only one service per server. 

I am not sure why you think that 
  "distributing the services to several machines seems to complicate
   things and increase the likeliness of one of the services failing."

I am speculating -- but I could see two causes for that:
  1) the services are so interdependent anyway, that separating them
     increases the complexity

     If that is the case, you need to see if you can simplify the
     software

  2) the basic server platform is unstable, and the OS crashes and/or
     hardware failures are likely, so with more than one server, there is
     an increased likelyhood of crashes

     If that is the case, you have a hardware/OS management problem that
     needs to get fixed!

With only one service per server, we can isolate problems, and never run
into the situation where one service can't be upgraded because something
else on the same server has an incompatable requirement, or where one
service really needs the server to reboot (ouch!) but that would take
out another service that is otherwise ok.

it also lets up work on reliability solutions on a per-service basis:
some services can be replicated, others need hot-standby, etc.

The other advantage of one-service-per-server is that you can often get
by with many smaller, less-expensive servers. This has many advantages:
   1) you can upgrade/add resources as needed for the appropriate
      service (one service might need a faster CPU, another more
      memory..)

   2) you can upgrade as needed: upgrading a few smaller servers is
      cheaper than upgrading ONE BIG SERVER

   3) you can afford to have a spare -- if you have 5 small servers, 4
      in use, and 1 spare, then you can probably get through a hardware
      failure. You probably can't afford a spare BIG SERVER

good luck,

     --david