Re: [SAGE] number of eggs in a basket

[Jeff Fisher <jeff@jeffenstein.dyndns.org>]

On Fri, Jan 07, 2005 at 09:13:45AM -0800, Bevan C Bennett wrote:
>
> In what sort of crazy environment would you need to implement 1000 
> different services? Even if you're assuming maximum redundancy and 
> throwing 2 servers at each service (which, if their load is 'trivial' 
> isn't needed for load balancing) that's still 500 different services!
> 
> Normally I see more on the order of 12-20 different services, and I'm 
> having a hard time trying to come up with a plausible scenario that 
> would use more than 30 truly distinct services.

Depends entirely on the business.  I work for a small telco in a small
country, and we are already way past 30 in applications, not even counting
windows front-ends.  However, if you mean distinct services, then there is
only really two in the entire company; billing and HR/Financials;  everything
else is somehow related to these, and even these two islands are have many
interconnections.

For our parent company (bigger country, lots more clients),  it's even worse.
They have thousands of servers for just about as many applications.

> Even in large enterprises, it's primarily the number of clients that 
> scales up, not the number of services being provided, so I don't see 
> this as being a valid argument.

What I've seen more is the number of applications increasing, or newer
versions of software having bigger hardware requirements.  However, I've never
worked for a company with the Internet as it's primary business, so I don't
know what it's like there...

> ... However, having the services separate is 
> generally a much for efficient use of -you- (and your time), as others 
> have already explained.

We've found that consolidating saves us quite a bit on this.  Most
troubleshooting in this environment is application issues, which are generally
dealt with on development servers.  By the time it gets to a shared production
server, most applications are well understood, and relatively well behaved.

For production, the limitation on consolidation has more to do with service
levels and risk rather than horsepower.  What it saves us is not only hardware
costs, but SAN ports, LAN ports, maintenance, number of administrators, etc...
When a new application is ready for production or starts in development, we
don't (generally) need to purchase a new server; request facilities to run new
fibers and lan cables; request new network ports, san ports, power
connections; request new IP address and DNS entries; install the OS; modify
the vendor maintenance contract; setup a new server in the backup schedule
(only a new filesytem or database), etc...  The application can be running in
a week or two instead of two or three months.


> In all my places of employment so far, when I came in there were one or 
> two servers that were running every service all together.  Over time I 
> slowly acquire additional (often smaller) server hardware and separate 
> out the services onto dedicated hardware and, as I have done so, 
> debugging and maintainance has -always- gotten easier.  As an added 
> bonus, doing this properly helps to harden your network against attacks, 
> as each box has fewer possible points of entry, and an exploit in a 
> particular service will only effect that service.
> 

Hmmm... Where I've worked, there has always been a seperate group to debug
application problems.  When they do come to the sysadmins for help, I haven't
seen much interaction between two applications that aren't directly related.
Sometimes applications will take down the whole server, but they generally
have their own dedicated server because they don't behave.  System problems
are seperate from application problems, and application<->system problems can
almost always be reproduced on development servers without affecting the other
applications on the production servers.

Can't speak for security; I have done very little with internet-facing
services, and most I have seen haven't been very secure despite layering,
firewalls, etc...  So far as layering internal networks, well, you have to let
all the important stuff through the firewall anyway, so it only slows down the
viruses that you aren't patched against.  The developers seem to do a lot more
damage than any hackers, and they're supposed to be on the servers.

-- 
jeff@jeffenstein.dyndns.org
At the University of Texas, one of the physics professors would often
include bonus questions of a quasi-silly nature on his tests.  On one
test, he asked the question "What is the speed of light through 
Jello?"

The only student to get credit for answering the question was the one
who wrote:  "What flavor?"