Blog's control panel: | Home | Tags | Index | Rss 2.0

15 minutes of pure terror and astonishment

Sun, 06 May 2007 | Permalink | Tags: ,

I'm posting this as a sort of death sign for future reference, to remind me how much, not just good, but perfect cabling could save you a lot of grief. . Posted under "fun" 'cause it's the sort of things that you either laugh or cry hard at. A transcript of this evil morning is also included.

What a better way to start Sunday's than a 5am sms about a faulty switch. Got up, went checking, and it was indeed stone dead. We got more than a switch for redundancy so I resolved that it was better to wait for a more decent time in the day than phoning an half asleep guy to get a reboot (no matter what they claim, a large part of those doing shifts are sleepy most of the times and not very reliable - not that coughunderstaffedcough sysadmin departments are much better).
Fast forward to midday. Some bits might be unbelievable but anybody who's hanged around long enough knows that these sort of things indeed happen. Phoning the datacentre:

Me: Hi, I'm bla bla bla from bla bla bla, phoning to get a switch rebooted.

Techguy: Bla bla bla. I'll phone you back when I'm in front of the rack.

... 15 mins later:

Techguy: Hi, phoning from blabla for that switch blablabla

Me: Hi, yeah, that switch is stone dead and I'm monitoring it, so just power cycle it and everything should be alright. There's no power button (and believe me you want one, dont buy stuff that forces you to pull cables) so you have to pull the power cord. Just to make sure, it's the third U from the bottom, labelled blah, with 3 cables in blab bla bla.

Techguy: Ok, got, I'm on it, leaving the phone for a moment as I have to lay down.

Me: Uhm, ok.

And here he's gone for a good 5 mins... I can hear noises and start getting worried

Techguy: Hello, cable unplugged, but there's a bit of a problem.... I cant get it back in 'cause there are other network cables in the way from the machine underneath it.

Me: eeeeeerrr (wondering hth he unplugged it in the first place, most likely by pulling it from the cord itself rather than from the plug into the PSU. And most important, why not unplugging the other end on the power extension?!)

Techguy: Can I unplug this black cable going to this router on the back of the rack?

Me: eeeerr, I'd rather prefer if you didn't unplug anything, what other options do we have?

Techguy: ok, I can go and look for Marianne, she's got small hands, I'll phone you back in 10, 15 minutes.

Me: You what?! Hang on there, I need the switch back. That cable you mentioned is the one going to the out of bound router, so I guess fine, unplug that. Just be careful 'cause that server is the firewall.

Do I need to say what happened next? Apparently without the black cable in there was still not enough space... Fast forward 5 mins after he managed to put back all the network cables he unplugged...

Me: (trying to not shout) Ok, everything seems back to normal, thanks.

Techguy: Actually, there's a small problem...

Me: ....

Techguy: To put back the cables I had to push the server a bit and it got out of the rails so I cant close the rack. I'm gonna fetch some tools and come back to you in 10 mins.

Me: Eeeer, and leave the rack open unguarded?!?

Techguy: Yeah, well, it'll be 2 minutes.

Me: You said 10 2 seconds ago, but I guess it cant remain in that state so if that's the only option go for it.

In the end he got back and fixed it.

I spoke to my boss and considered to report the guy or do something about it but then it's hard to prove what happened and it's Sunday after all, so we decided to bump the thing to Monday.




SpikeLab.org is a Filippo Spike Morelli copyright 2005-2008
This work is licensed under Creative Commons Att-SA License.