The Event - a visit from the Fuckup Fairy

Normally, when one of our lusers does something that's common-or-garden stupid, we laugh (at them), die a little inside at the plummeting IQ of the world, post a mildly amusing rant on our Blog or Usenet group of choice, and carry on.

But sometimes - just sometimes - a luser does something so dumb, so moronic, so spectacularly cretinous that simply finding the words for the rant becomes difficult.

When dealing with the full ramifications of The Event today, and the immediate aftermath, I thought tonight's post would be easy to write. My mind was awhirl with vitriolic comments - a cascade of inventive invective primed and ready to be unleashed upon an unsuspecting Web in all its diabolical fury. It hasn't ended up quite that way - simply because the sheer enormity of The Event was so staggering, so unbelievable that by the time I sat here at the PC, with its cursor winking expectantly, I found it hard to summon the words to describe it.

Still, nothing ventured and all that, so here we go...

Like many managed services providers, we host our client systems in large datacentres. The datacentre involved in The Event is a large one in London. You may have heard of it - however to protect the guilty and disguise it, I shall call it BlueCoach.

Two of our largest clients also have their backup devices in Bluecoach. These clients both have large tape stackers connected to backup management servers. Each client has their own rack in their own cage, and the cages themselves are situated some considerable distance apart. Each client performs daily, weekly and monthly backups, often totalling many Terabytes of data.

So - an email from BlueCoach arrives - the loading bay's locked on one of the tape stackers. I try and unlock it remotely - and it won't unlock. One quick check with our Guru later, we've established that the software is saying it's unlocked, while the hardware has decided to do its own thing, and is resolutely refusing to relinquish its tapes.

What do you expect - it's Slackup Exec running on 'Doze. AHS, ASS.

OK, no problem - we can drop the backup server, get the Tape Guy to powercycle the stacker, change the tapes, then he can restart the backup server from there. Simple.

Only the Tape Guy has gone home, and in his place is...well, I'm still lost for a description. Amoeba-man? A Neanderthal? A lobotomised three-toed Sloth? Gordon Brown? I don't know. But whatever it was, it sure as hell shouldn't be anywhere near a datacentre. Or computers. Or humans, for that matter.

But I digress. Anyway, I fire off an email to BlueCoach saying the following:

• I am going to remotely shut down the backup management server $foo-backup01;

• Please would you go to the tape library and physically power-off the library, then turn it back on, unlock it and change the tapes.

• Once you have done so, please restart the backup management server $foo-backup01 - it's a 1u server, in the same rack, and obviously its LEDs will be off.

Well...After three hours and two phone calls from the knuckle-dragging, subhuman fsckwit charged with carrying out the 'Intelligent Hands' request (probably the best oxymoron I've heard in years), he still hasn't found the server.

I get the feeling he'd be happier lolling around somewhere, learning how to find his arse with both hands, map and flashlight.

This pathetic pseudo-simian couldn't FIND server $foo-backup01, despite precise information even down to the rack name and its position in said rack. Finally, just as I'm losing the will to live (or, more accurately, losing the will to let HIM live), he sends me a triumphant email thusly:

• Request completed. The server was called $bar-backup01
  

At this moment, I discovered the Sphinctosecond - "that brief arse-tightening sensation which inevitably accompanies the realisation that Things Have Just Become Not What They Should Be".

He'd rebooted $bar-backup01.
...
A live, running backup server.
....
For our largest client.
.....
22 hours into a multi-terabyte full monthly backup that had been requested by their Auditors!

Even some 6 hours on, I'm still struggling to find any reasoning for what 'Homo Moronicus' has managed to achieve. He'd managed to go to completely the wrong rack, in completely the wrong cage - a significant distance from the correct one - and then reboot a live device when he'd been TOLD that the device he was looking for was powered down?

The mind boggles. Surely at some point, some under-utilised neuron must have given a brief fizzle and he'd have questioned the name of the server (suffice to say that my descriptions are much less detailed than our corporate naming protocol)?

Could he even READ?

What sort of bottom-feeding, scum sucking, microcephalic excuse for a proto-human would perform an act of such inconceivable imbecility? I appreciate that 'datacentre troll' isn't exactly the most intellectually challenging role - after all, you really only need one finger to turn the boxes off and on again - but come on, surely even a retarded chimp might have taken a brief second away from masturbation and dung-flinging to realise what he was about to do might need verification!

Well - the incident reports are filed, and I sincerely hope - in fact it is my most fervent wish that this thing, this creature, this floating blob of something unpleasant polluting the gene pool, receives a P45-shaped LART. Thanks for fucking up my day, dickhead. Don't let the door hit you in the brain-cell on the way out.