Re: [SAGE] number of eggs in a basket

[Brad Knowles <brad@stop.mail-abuse.org>]

At 10:46 PM -0500 2005-01-07, Steve Simmons wrote:

>  With DNS, any machine that has a copy of the zone files and can do
>  virtual IP interfaces will work just fine.  When primary DNS dies,
>  you activate a virtual interface with the address of the DNS server
>  and manually start DNS.  Instant restoration of service, cheap and
>  easy.

	It depends on what you mean by "dies".  There are certain kinds 
of failures that may happen within the application which would be 
propagated to all the secondaries, and would not necessarily be 
detectable by the monitoring system.

	Also keep in mind that you will have ARP cache timeout issues 
when you activate the virtual IP address.  Better to have multiple 
service IP addresses that are advertised via servers which are 
designed for this kind of failure, and a monitoring/fail-over system 
that can help minimize the ARP cache timeout issues, etc....

	When you're talking about this sort of thing, you really want to 
use dedicated nameserver boxes if at all possible.  Being a 
nameserver can be a very heavy load on a machine, even if it seems 
that the machine isn't actually doing much work.  This can seriously 
impact the other services on the machine, if you try to run it on a 
shared box.

>  With web service, it depends strongly on whether your raw HT files
>  are on the local disk or a server, if you access thru a DB or not,
>  what kinds of certificates you need, what kind of software you might
>  have that is or isn't too expensive to duplicate, ya-da, ya-da.

	Any time you start talking about databases, NIS, etc... being 
shared across the network, you have to be real careful when you talk 
about putting NFS into that mix.  For one thing, Berkeley DB (and 
most other database systems) tend to use mmap() system calls for 
maximum performance, and mmap() cannot be used on NFS.

	This can cause problems in all sorts of unexpected areas.  For 
example, Cyrus can't be used on NFS, because it stores the mailbox 
meta-data indexes in Berkeley DB, which uses mmap().  You can't store 
diablo USENET news server indexes on NFS, because it also uses mmap() 
for speed (storing the articles on NFS is a different matter, but 
does have it's own issues).

	NIS, LDAP, and other network information database systems may 
also make use of Berkeley DB as the back-end, or otherwise make use 
of mmap().  Note that MySQL can also use Berkeley DB as the back-end 
(which is how they achieve full ACID compliance for their MaxSQL 
product).


	Anyway, it all comes down to this -- YMMV.  You need to do the 
cost/benefit analysis and figure out what works best for you.

>  Things get a bit more complex but still manageable when you start
>  putting multiple services on a single machine and still want to
>  fail them over.  In a case like that, I'd allocate a name and
>  address for each service: dns, www, dhcp, etc.

	That works up to the point where you need to reboot the box, at 
which point all these things go down at once.

	Server partitioning schemes (e.g., such as found on certain types 
of Sun Enterprise class servers for Solaris 9 and in Solaris 10 for a 
wider array of boxes) can help make this process easier by allowing 
you to reboot the virtual box while the physical hardware will 
hopefully remain operational.  Again, that only works to a point.

	The next step would be to go to blade systems, where you really 
do get entirely separate machines that all share a common chassis. 
Of course, that has its own failure modes.

>  With a situation like that, you put as many services on a pair of
>  machines as they can handle.  The services that can't be paired
>  you put on individual machines.

	That assumes that a pair of machines is sufficient to handle the 
load, and the second machine wouldn't melt if the primary died.  This 
may be a valid assumption in some cases, but not in others.

>  Most of us have or will soon have limited power and cooling capacity
>  in our server rooms.  One server per service can suck up a fair
>  amount of capacity.

	The bigger server-class machines can also suck up a lot of power 
and cooling.  You need to find out what is the right balance for you.

>  Server-class machines aren't cheap.  But if you can run a bunch of
>  services on a pair of machines, you don't need server-class machines.
>  Just machines fast enough for those services.

	They may be a lot more expensive to own and operate, as well as 
to upgrade.  You need to look at all the pieces in the cost/benefit 
equation and decide what works best for you.

>  Conversely, if you put those services on old PCs running Linux, they'll
>  have all the reliability of old PCs.  IMHO better to buy a couple of
>  good ones.

	Server-grade machines doesn't necessarily mean Sun boxes.  You 
can get server-grade PCs, too.  You just have to look harder.

>  Fewer machines to maintain and admin means lower cost.  Granted, it
>  may not be a *lot* lower, depending on the rest of your environment.
>  But it will be lower.

	Maybe.  It might raise your costs.  It all depends on your 
particular situation.

>  YMMV, offer only available to adults over 18, not valid where explicitly
>  prohibited by law, etc, etc.

	That's the key -- YMMV.

-- 
Brad Knowles, <brad@stop.mail-abuse.org>

"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety."

     -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
     Assembly to the Governor, November 11, 1755

   SAGE member since 1995.  See <http://www.sage.org/> for more info.