Topic: IT sucks sometimes

[scottws]

So last week on Friday at 12:00pm my boss, as part of a mass e-mail to the employees of our Wisconsin office, gets an e-mail from the director of operations that the Iola office is undergoing maintenance on power equipment the following day from 7:00am - 12:00pm and that everyone needs to turn their computer off before they leave on Friday.  Keep in mind that we are in Ohio, not Wisconsin and this is the first we are hearing of this.

My boss quickly calls the guy and says basically "WTF?!  You can't give us less than 24 hours notice for something like this!  We need to have someone available on-site and need a few hours to shut down all of our server and network equipment."  So he successfully gets them to postpone to the following Saturday (today).  He also schedules a meeting with the entire IT staff so we can go over the planned shutdown procedure and staff responsibilities.

Before that meeting takes place he finds out from the head of the maintenance department that the Iola office has a whole-building UPS and also a backup generator.  Power runs through that and then into the building.  They need to do maintenance on both the UPS and the generator; however, there is a switch that reroutes power and bypasses the backup equipment and goes straight to the building.  The maintenance guy assures us that if anything, it will be a few millisecond blip in power not really an outage.

So we go to a standby state just in case some of the equipment doesn't handle the blip well (though it should as it is all protected by UPS' in the data center) rather than planning a complete shutdown.

So come 7:00am today, my BlackBerry blows the fuck up.  Basically everything we are monitoring in Wisconsin is in a down state.  This can happen if the network link between here and there goes down, so I tried to connect to their VPN.  Nope, that's down too.

My boss calls the head of maintenance, who is in Wisconsin overseeing this whole thing.  He says that the switchover was smooth and the lights didn't even flicker.  My boss asks him to check the data center.

No signs of power at all.

Fuck.

[Pugnate]

Woah that sucks. I have some family members who work in IT, and imagine how efficient it all is when you are working in a third world country, or working in place like the Middle East, where all the people in charge are illiterate but in power because they are the locals, and the people doing the actual work are expats etc, trying to cover up all the mistakes and unreasonable demands.

Apparently it is one clusterfuck after another.

[idolminds]

Wow, that sounds terrible.

[Xessive]

Man, I feel for you, Scott. Pug summed up the Middle-East situation quite well. Despite the added pressure of management incompetence the IT field can get pretty gnarly wherever you go.

[scottws]

It actually turned out that we got pretty lucky.  When the power came back on, 95% of the networking equipment and server hardware came up fine.  We just had to turn on an external RAID device and reboot its attached server and reboot one of a cluster's nodes and then move the cluster over to that node and then reboot the other node.  The biggest trouble we had was with our VMware environment.  It took two reboots of one of our hosts before it could see all the storage on the SAN and we have a VMware consultant still working on bringing up a virtual file server related to that problem.

It turned out that only about 1/3 of the building still had power after the cutover.  Idiots.