Solving the World's problems with common sense and a flamethrower.

Thursday, March 06, 2008

The Event - a visit from the Fuckup Fairy

Normally, when one of our lusers does something that's common-or-garden stupid, we laugh (at them), die a little inside at the plummeting IQ of the world, post a mildly amusing rant on our Blog or Usenet group of choice, and carry on.

But sometimes - just sometimes - a luser does something so dumb, so moronic, so spectacularly cretinous that simply finding the words for the rant becomes difficult.

When dealing with the full ramifications of The Event today, and the immediate aftermath, I thought tonight's post would be easy to write. My mind was awhirl with vitriolic comments - a cascade of inventive invective primed and ready to be unleashed upon an unsuspecting Web in all its diabolical fury. It hasn't ended up quite that way - simply because the sheer enormity of The Event was so staggering, so unbelievable that by the time I sat here at the PC, with its cursor winking expectantly, I found it hard to summon the words to describe it.

Still, nothing ventured and all that, so here we go...

Like many managed services providers, we host our client systems in large datacentres. The datacentre involved in The Event is a large one in London. You may have heard of it - however to protect the guilty and disguise it, I shall call it BlueCoach.

Two of our largest clients also have their backup devices in Bluecoach. These clients both have large tape stackers connected to backup management servers. Each client has their own rack in their own cage, and the cages themselves are situated some considerable distance apart. Each client performs daily, weekly and monthly backups, often totalling many Terabytes of data.

So - an email from BlueCoach arrives - the loading bay's locked on one of the tape stackers. I try and unlock it remotely - and it won't unlock. One quick check with our Guru later, we've established that the software is saying it's unlocked, while the hardware has decided to do its own thing, and is resolutely refusing to relinquish its tapes.

What do you expect - it's Slackup Exec running on 'Doze. AHS, ASS.

OK, no problem - we can drop the backup server, get the Tape Guy to powercycle the stacker, change the tapes, then he can restart the backup server from there. Simple.

Only the Tape Guy has gone home, and in his place is...well, I'm still lost for a description. Amoeba-man? A Neanderthal? A lobotomised three-toed Sloth? Gordon Brown? I don't know. But whatever it was, it sure as hell shouldn't be anywhere near a datacentre. Or computers. Or humans, for that matter.

But I digress. Anyway, I fire off an email to BlueCoach saying the following:
  • I am going to remotely shut down the backup management server $foo-backup01;
  • Please would you go to the tape library and physically power-off the library, then turn it back on, unlock it and change the tapes.
  • Once you have done so, please restart the backup management server $foo-backup01 - it's a 1u server, in the same rack, and obviously its LEDs will be off.
Well...After three hours and two phone calls from the knuckle-dragging, subhuman fsckwit charged with carrying out the 'Intelligent Hands' request (probably the best oxymoron I've heard in years), he still hasn't found the server.

I get the feeling he'd be happier lolling around somewhere, learning how to find his arse with both hands, map and flashlight.

This pathetic pseudo-simian couldn't FIND server $foo-backup01, despite precise information even down to the rack name and its position in said rack. Finally, just as I'm losing the will to live (or, more accurately, losing the will to let HIM live), he sends me a triumphant email thusly:
  • Request completed. The server was called $bar-backup01
At this moment, I discovered the Sphinctosecond - "that brief arse-tightening sensation which inevitably accompanies the realisation that Things Have Just Become Not What They Should Be".

He'd rebooted $bar-backup01.
A live, running backup server.
For our largest client.
22 hours into a multi-terabyte full monthly backup that had been requested by their Auditors!

Even some 6 hours on, I'm still struggling to find any reasoning for what 'Homo Moronicus' has managed to achieve. He'd managed to go to completely the wrong rack, in completely the wrong cage - a significant distance from the correct one - and then reboot a live device when he'd been TOLD that the device he was looking for was powered down?

The mind boggles. Surely at some point, some under-utilised neuron must have given a brief fizzle and he'd have questioned the name of the server (suffice to say that my descriptions are much less detailed than our corporate naming protocol)?

Could he even READ?

What sort of bottom-feeding, scum sucking, microcephalic excuse for a proto-human would perform an act of such inconceivable imbecility? I appreciate that 'datacentre troll' isn't exactly the most intellectually challenging role - after all, you really only need one finger to turn the boxes off and on again - but come on, surely even a retarded chimp might have taken a brief second away from masturbation and dung-flinging to realise what he was about to do might need verification!

Well - the incident reports are filed, and I sincerely hope - in fact it is my most fervent wish that this thing, this creature, this floating blob of something unpleasant polluting the gene pool, receives a P45-shaped LART. Thanks for fucking up my day, dickhead. Don't let the door hit you in the brain-cell on the way out.


Anonymous said...

Okay . I understand Neil. Now just let it all out okay. You let it all out. Deep breaths and easy breathing.

hahah I havent laughed so hard at one of your Tech Rants in a while. I missed that vitriol. You used to only reserve it for Moronic MDs and Financial Directors of pissant tech support companies who couldnt find a profit line in a 10k sale !

Love it . good to see your back in form there.

Anonymous said...

Oh man! I sat on the pc for ages last night refreshing your blog awaiting this!

I'm so glad I remembered to check back in because the silver lining to your pain is my happy grin as I head up to bed. (well, it's my silver lining anyhow) ;)

ShapeThrower said...

That's a rant of quite stupendous proportions but then again the monumental nature of said muppet-tude deserves nothing less ....
Made me chuckle on this grey and dismal Monday morning.

Anonymous said...

I came to this from El Reg (thank you for posting the link, it would have been too cruel of you to leave it out!). I am not a techie, I am a partial techie (archaelogist by training/office manager for a dev. co.) - my husband insists on me learning how to do and fix things rather than letting him do and fix 'em! I was gobsmacked (a rare occurrence believe me) when I read this; how is this level of stupidity even possible and the idiot is still alive? Evolution/natural selection should have removed his genes from the pool some considerable time ago! A cattle prod is definitely required.

Dungeekin said...

Hi, and thanks for your comment!

Please keep reading, and I hope you'll find other examples of the rampant Cluelessness infecting our Nation.....


Anonymous said...

That kind of behaviour is way past cattle prod time. That calls for a nice aluminium baseball bat, exercised for a significant period of time against several limbs and other body parts.

Oh, and I'm also here because of El Reg.