Sometimes things don’t go as expected

February 27, 2012

And those things can sometimes happen with a bright flash of lightning and a sonic boom of thunder. Although, God usually finds a way to work them out. But that would be getting ahead of myself.

Sunday was a pretty relaxed day, as well as kind of short. We were tired from the long flight so our minds weren’t 100% yet, but we got a few things done. After breakfast we did a quick tour of the facility, did some brainstorming about how we were going to tackle everything, and then wen’t to church services.

Church.jpgChurch was rather cool to see. It was actually an unusual day for services since they had some weekend event that many of the adults do with the seniors to help prepare them for graduating and heading back to America; since most of them have for a good portion of their life. Everything in the church service was done by the kids, except for the actual message. They lead worship, they collected the offering, they did some of the prayers, all in all it was very fun to see.

The rest of the day was spent down at what is called the Lower Station, which is basically the hospital and a few other buildings. We worked through the existing PBX (phone system) to try and figure out exactly how to hook everything up. We had spent most of the previous weekend configuring the box for a very confusing protocol called SS7, which the telephone company had told RVA the phone line was using. Not so much. It was using just plain old ISDN (i.e. what we use here in the U.S.). We called it an early night so we could get up early Monday morning and start working at 6am.

Welcome to Monday. This is where things got interesting. We go back down to the Lower Station to hookup our new PBX router. We plug the telephone line in. Works great. We plug the line to the existing PBX in. Doesn’t work. Not at all. Nothing. Nada. Complete fail. Come 8am we call it quits since the hospital is opening up and will need their phone lines with plans to try again after lunch (during their break time).

After lunch we head back down. Still nothing. We’re down to the last few minutes of time we have to work on it and on a whim we try a cable that is, to put it simply, wired backwards and shouldn’t work. It works! The stupid device is wired backwards! Welcome to Africa! After putting everything back together we came back to RVA to discuss the actual switchover plans. This is our first proof that we can actually make this work so we are quite excited. We’re ecstatic. We’re overjoyed. We can do this. We can pull it off. *flash* *boom* *SIZZLE-SPARK* *screaming girls out front* Uh oh.

That sizzle and spark was the server rack in the IT office we were discussing things in, which is now without power. The screaming girls out front are perfectly fine but in complete shock and hysterics from seeing lightning up close and personal striking the building that was about 30 feet away from them. Which, by the way, happens to house the main IT equipment and servers. And the battery backup unit powering both rooms (which is now oddly enough, not powering either).

In short, the lightning fried the 10,000 watt UPS. Completely. It’s dead, gone, won’t turn on. After about 30 minutes of tinkering with that thing trying to see if we can get it to work, without success, they start pulling smaller UPS units from all over campus to get the servers back online. This also requires us to start making custom cables to hookup all the equipment to these new UPS units that were not designed to handle this type of equipment. Come 5pm Jeff from RVA and myself need to head back down to the Lower Station to do the actual phone system swap, with only about 75% of the servers back up and running.

This is where God starts shining. Okay, well he did shine quite well in the lightning strike, but that was more of a “hide behind that rock and see a flash of my back as I run past” kind of shining. Jeff and I go down, plug in the cables, the lights go green and we make our first test call… works! Second test call… also works! a dozen more calls, each one works perfectly. Finally, something is going our way again! Hey even better, the Internet guys show up to work on getting the Internet connection up and running which has only been working 60% of the time since we’ve been here. *cell phone rings* “You got all of the servers back up? Great” “Not great?” “What kind of data corruption?” Then from the Internet guys: “We’re done working on the Internet for now, it now works 0% of the time but we’ll keep working on it tomorrow morning” *sigh* Back to RVA…

Jeff goes to help continue the repair efforts while I start researching a few things with the PBX we will need to configure the next day. Around 8 I finish my research and head up to the larger IT room to see how they are doing. The mail server is the one that is corrupted and none of the mail services will start. Some of the config files are now also filled with garbage and need to be rebuilt by hand. One of their guys has been working on it for the past 4 hours. It happens to be a Linux box using much of the same software we use at HDC for mail so I join up with him. Finally after another 3 and a half hours we get it up and running.

Everything is finally back up and running, we are breaking for the night to sleep. We also managed to get some internet going through their old provider who happened to not shut off the line yet *shhh*.

So things don’t always go as we expect, or even hope. But God always seems to have his hand in it. The lightning strike was incredibly bad luck for RVA, especially in the middle of installing a new phone system. But the fact that it happened while we were here was a tremendous blessing to them as they had an extra 4 hands to work through all the problems and get everything back up for the next day’s classes. The phone system stuff had us extremely worried, but when it came time to sink or swim, we actually found we were floating.

We still have lots to do, the part of the PBX we installed is just a router. But it lets us bring the phones up one at a time on the new system instead of trying to go whole-hog and switch everything over at once. So we can spend tomorrow doing the changeover at a more manageable pace and have time to do some serious testing. As well as help them make plans on buying a new UPS! The company they bought the UPS from is actually coming out to look at it tomorrow, so hopefully they will decide it should be covered under some kind of warranty, since it is only 2 years old.

I just wish I had been outside to see the lightning strike!