PDA

View Full Version : Why knowing a little can damage a lot...



FoolsGold_MKII
July 10th, 2007, 06:41 AM
http://worsethanfailure.com/Articles/Im-Sure-You-Can-Deal.aspx

If you can't be bothered clicking the link (there are people like that), the body of the link is reproduced below:
----

Not too long ago, there was a small furor in the local media about a major disaster at The State's Technology Services Division. The details were a bit sketchy – mostly because The State was “unable to comment on an ongoing investigation” – but what was reported was that, for two full days, employees of The State were unable to logon to their computers or access email, and that this caused business within The State to grind to a halt.

As the “investigation” carried on, the media lost interest in the story and moved on to more newsworthy stories like who Paris Hilton was partying with last weekend. Fortunately for us, a certain employee of The State named J.N. works in the Technology Services Division and decided to share what really was behind those fateful days.

When employees of The State came in to work following a three day weekend, they found their workstations overloaded with "cannot logon" and "Exchange communication" error messages. The Network Services folks had it even worse: the server room was a sweltering 109° Fahrenheit and filled with dead or dying servers.

At first, everyone had assumed that the Primary A/C, the Secondary A/C, and the Tertiary A/C had all managed to fail at once. But after cycling the power, the A/Cs all fired up and brought the room back to a cool 64°. At the time, the “why” wasn’t so important: the network administrators had to figure out how to bring online the four Exchange Services, six Domain Controllers, a few Sun servers, and the entire State Tax Commission’s server farm. Out of all of the downed servers, those were the only ones that did not come back to life upon a restart.

They worked day and night to order new equipment, build new servers, and restore everything from back-up. Countless overtime hours and nearly two hundred thousand dollars in equipment costs later, they managed to bring everything back online. When the Exchange servers were finally restored, the following email finally made its way to everyone's inbox, conveniently answering the “why”


From: ----- -----------
To: IT Department
Re: A/C constantly running.

To whom it may concern,

I came in today (Monday) to finish up a project I was working
on before our big meeting with the State ----- Commission tomorrow,
and I noticed that there were three or four large air conditioners
running the entire time I was here. Since it's a three day weekend,
no one is around, why do we need to have the A/C running 24/7?

With all the power that all those big computers in that room use, I
doubt it is really eco-friendly to run those big units at the same
time. And all computers have cooling fans anyway, so why put the A/C
for the building in that room?

I got a keycard from [the facility manager’s] desk and shut off the
A/C units. I'm sure you guys can deal with it being warm for an hour
or two when you come in tomorrow morning.

In the future, let's try to be a little more conscientious of our
energy usage!

Thanks,
-----


As for the employee who sent it, he decided to take an early retirement.

lisati
July 10th, 2007, 06:48 AM
Ouch!

Dokatz
July 10th, 2007, 06:50 AM
That's pretty funny. What's even funnier, Is I clicked on and read the contents of the link before even glancing at your post...Which apparently was just a redundancy thing.

Good times man.

Motoxrdude
July 10th, 2007, 06:58 AM
Wow, im dumbfounded.

dustigroove
July 10th, 2007, 07:24 AM
I'd wager farce... but humorous none-the-less.

eentonig
July 10th, 2007, 07:26 AM
Reminds me of our 'cleaning lady story'

A firm I used to work in, suddenly started experiencing something strange. Every night around 7pm, a bunch of servers in a remote DC stopped working. This was an unmanned site around that time, so we always had to dispatch someone to restart the boxes. Within 45min, when the tech arrived, they were always working again.

After four days, we decided to sent someone earliern to see what was going on at the site and find an explanation for these events. So the guy was sitting on site around 5pm, in time to see the office personal leave the building. At 6pm, the cleaning services come in and start doing there thing. .... Around 7pm, he sees a woman entering the DC and he casually follows her. Out of a closet, she takes a vacuum cleaner, pull out a power cord to make room for the vacuum cleaner and starts vacuum cleaning the DC.

Problem solved:
Actions taken?
- Cleaning firm got a huge fine and was forced to provide a training to all their employees who worked on our sites.
- The nice woman was eplained that she needn't clean that room. And certainly not pull out power cords randomly.
- The DC got a badgereader.
- The responsible for the DC got fired and his replacements first job was to make sure all IT equipment was only powered from registered power sources. And not via some extension cord somewhere.

paintandswim09
July 10th, 2007, 07:28 AM
dum... just dum. That guy doesn't even merit a "b" at the end of his dumbs. By the way, is 109 F really that hot? My computer runs fairly cool, about 85-87 in the case and 89-91 on the CPU. Would 109 really kill a server? Are they that susceptible to heat? Running virus scans and games and unknowingly with one broken fan (out of 3, besides CPU fan), i got the CPU up to 50 c- 122 f, and the case at 40 c- 104 f. It ran fine. Oh well, still a good story.

~LoKe
July 10th, 2007, 07:36 AM
dum... just dum. That guy doesn't even merit a "b" at the end of his dumbs. By the way, is 109 F really that hot? My computer runs fairly cool, about 85-87 in the case and 89-91 on the CPU. Would 109 really kill a server? Are they that susceptible to heat? Running virus scans and games and unknowingly with one broken fan (out of 3, besides CPU fan), i got the CPU up to 50 c- 122 f, and the case at 40 c- 104 f. It ran fine. Oh well, still a good story.

The room itself was 109, I can only imagine the servers internally would be much hotter.

Warpnow
July 10th, 2007, 08:15 AM
If the EXTERNAL room temp was 109 the sever internals prolly approached 150-170 easily...I'd imagine even higher than that...that will kill the cpu pretty quick...