Track a Tech

Reviewing the WNDAP360 Access Point

The WNDAP360 is on of Netgear’s newest Access Points in their Wireless lineup. It features dual radios for simultaneous 2.4GHz and 5GHz operation. It has internal antennas, but does include 2 connectors for 2.4GHz external antennas.  No support for 5GHz external antennas. It has gigabit LAN.  Why do i even mention this?  Because many 802.11n wireless access points do not.  They only include a 10/100 Ethernet connection which means that even if you get the theoretical 300Mbs speeds they advertise, you are limited to 100Mb anyway.  Though as I said, the WNDAP360 does include gigabit so that is a plus.  It also uses standard 802.11af PoE so it will work with your standard PoE switches.

This AP has a nice look with it’s clean-white cover and silver base. It includes a detachable mounting bracket which means you can mount the bracket to the wall without using any stupid paper templates and hoping your screws end up in the right place. Once the mounting bracket is installed the AP just snaps into it. So aesthetically it looks very nice. The problem its looks is the size. This thing is a monster. They do a good job of taking pictures of it in a way that hide the fact that this thing is 10×10″ and 2″ tall. In the picture I put a CD on top to give you an idea of just how massive this is.

In theory (meaning it probably works but I didn’t buy it so I can’t speak directly to how it actually performs) you can buy a rack-mount network device (the WMS5316) that lets you centrally manage and monitor up to 16 of these APs for only $600. The feature set included in the description is pretty impressive. It will do automatic monitoring and adjustments to ensure the best service to all the wireless clients. For me the big thing would be that it would let me look at all the connected clients from one central spot instead of going to each AP individually.

Some people complain about the interface for configuring the AP but I found it pretty easy to work with. I had the AP setup with out RADIUS server providing access on both 2.4GHz and 5GHz in about 5 minutes.  If you are a home user, yes you are going to be confused by all these terms. But if you are a business user you should probably recognize most of the configuration terms already. You can configure up to 8 SSIDs per frequency band which means potentially up to 16 total. But in reality you will probably have some SSIDs that are in both 2.4GHz and 5GHz. But allowing you to configure separately is a nice feature. It means if I want to setup a dedicated 5GHz network I don’t have to waste an SSID in the 2.4GHz band.

Unfortunately that is where the good news stops. The case is made of extremely cheap plastic. Like I thought it was going to break when I picked it up with one hand and felt the top and bottom flex in my grip. Out of the box the edges between the top and bottom pieces don’t line up properly so the lip on the bottom piece sticks out past the edge of the top piece. I’m not saying it should be made of metal, I know metal is not good for wireless. But at-least use some plastic that doesn’t look and feel cheap. If you are making a business grade product make it look like it belongs in a business.

Reception is bad. I mean like, “oops we forgot to mention the 5GHz doesn’t actually work” bad. I tested this unit against our existing D-Link DAP-2553 units for comparison.  Across the board the WNDAP360 had worse reception/range than the DAP-2553, but in the 5GHz it was just laughable (well it would have been if it had been a free unit). There were two major issues with the 5GHz band. First was just getting it to keep a solid connection. I tried different positions just to be sure I wasn’t in a bad spot, but they all worked the same. For example, I sat at a desk that was 15 feet away from the AP with nothing but open air in between (not even a cubicle wall). The speed would range from 2Mbps up to 125Mbps. I’m talking actual throughput speed, not what is reported (I used iperf for this testing), though what was reported also varied dramatically. During these variations the actual signal strength remained the same, so it’s not like it was getting interference – and yes, I tried different channels. It seems like something internal (software) was deciding to change the available bandwidth speed and jacking up the connection.

Problem two I think might be related to the same issue. Somehow the AP was “locking” the device to it for too long. For example, I would walk with my laptop down the hallway with a ping command running so I could see when I would start losing signal (or rather data). I would hit a certain point and all packets would drop instantly. I’m not talking the packets started getting iffy and then went away I mean I would go from 0% packet loss with like 1.2ms ping times, take one more step and have it jump to 100% packet loss instantly. Take a step back and I’m back to 0% packet loss with great ping times. Keep walking another 10-15 feet and still 100% packet loss and my WiFi adapter isn’t switching to another AP (with slightly stronger signal), but again because something in the WNDAP360 is internally deciding to drop packets/throttle the connection I have no connection even though I have “okay” signal. While this may be a software issue, I was using the latest firmware available at the time: 2.1.1.

I did the same test with our DAP-2553 units and the laptop roamed like expected (and I could get farther). As I got further and further away from the AP my signal would drop AND I would start to get high ping times and occasional packet loss until it finally just “lost everything” because I was too far away (at which point my WiFi icon starts showing that I am no longer connected anyway). When going back to my “normal” setup so that the laptop would roam as my signal just started to drop off and I would just start to get packet loss my laptop would roam to the other AP and I would have about a 1.5 second delay in my network activity. So again, something in the software logic of the WNDAP360 is all jacked up and even though the “physical” connection is good its dropping packets when it shouldn’t be. (Note: in 2.4GHz I did not notice this problem, it seemed to work as expected).

Summary

I’m returning our unit. Nice idea, I liked the concept, I generally speaking liked the look.  It’s big but I could get over that.  It has dual-radios so I can work in both bands at the same time. But it just doesn’t work.  5GHz is completely unreliable so that means it is a $280 2.4GHz device.  For that price I can buy two of the DAP-2553 units.

Pros

• Dual-radio simultaneous support for 2.4GHz and 5GHz
• Easy to use web interface
• Up to 8 SSIDs per radio (up to 16 total between the two radios/bands)
• Standard PoE support
• Central management available
• Cheap price: $280

Cons

• 5GHz band is unreliable
• Cheap construction
• Way oversized
• Overpriced since the 5GHz band doesn’t work

Moving Apple’s PasswordServer to Linux

So awhile ago I started working on a way to move our Open Directory database off of Mac and onto a Linux box running OpenLDAP. But I wanted to find a way to do it without losing any functionality. So far it’s going pretty good. I have another post I’m working on that details all the steps to accomplish this. I’m kind of waiting until I make the move to post everything. One big hiccup I ran into is that without Apple’s PasswordServer most applications only support cleartext passwords, which is kind of a pain. Especially since some Apple applications (such as iChat) complain if you are using cleartext passwords, even if it is over an SSL link. So I wanted to find a way to provide the advanced password services that Apple provides.

Thus was born the Linux Password Server (or lpws). What I discovered is that Apple’s PasswordServer is simply a bridge between a client computer that wants to do SASL authentication and a centralized server that has access to the cleartext passwords.  The protocol itself is simple text with the SASL data encoded in hex strings. While I don’t have everything working yet, all the authentication seems to work just fine. I have been able to generate an RSA private/public key pair which it uses to verify the authenticity of the host the client is talking to. I can store that public key in the authAuthority record of a user and add the server to the Password Server list in the LDAP database. From then on any requests to authenticate the user will work just as if you were still using Apple’s own services. All the authentication schemes seem to work just fine.

Since the password server just uses SASL to perform the actual authentication there was actually very little code to write for the server itself. I had to write the usual “client tracking” code and socket code you would need to provide basic TCP services and then implement a command processor to deal with the different commands that can be sent by the client. After that it was just pass along the data sent from the client into the SASL library. There were two big hurdles I ran into to get to the point I’m at now.

The first was the WEBDAV-DIGEST mechanism. This mechanism doesn’t exist on Linux, or anywhere else that I could find.  After some time deciphering a bit of the protocol I found that it was basically just the DIGEST-MD5 mechanism with a slight twist. Instead of the authentication server generating the nonce and all that stuff, the “intermediate” server (i.e. iChat, Address Book, mail, etc.) generates the nonce and gives it to the authentication server. I suppose there is a slight security consideration with doing that as it is kind of close to a replay attack, but from my understanding of DIGEST-MD5 there are still enough security measures in place. So basically I had to duplicate a modify a few lines of code in the DIGEST-MD5 mechanism and rename it to WEBDAV-DIGEST.

The second hurdle was DHX.  Nearly everything uses DIGEST-MD5 or WEBDAV-DIGEST to authenticate the user.  I found only two cases where DHX is used.  The first is when logging into a network account at the login window. Oddly enough, if DHX failed then it would fall back to DIGEST-MD5.  This caused a significant delay in the login process so it was unacceptable to me. The second place was when changing a password via the “Connect to server” window (and possibly via System Preferences, I didn’t try that yet). When changing a password the client first authenticates you via DHX and then issues a CHANGEPASS command to set the new password. Why they chose to use a different mechanism just to authenticate you before changing your password I don’t know. But again, there did not exist an SASL DHX mechanism that I could use. So I had to write one. That took awhile to figure out but I finally managed to get it working after a few days of pouring over various resources online.

So what works now?  Well you can authenticate users via the Linux Password Server. The services uses SASL so as long as SASL works on the server then it can authenticate. For my use, I wrote a tiny auxprop plugin that loads the cleartext password from LDAP and gives it to SASL to authenticate the user with. This all assumes you are using OpenLDAP and storing cleartext passwords (in as safe a way as possible). In theory the passwords could be stored somewhere else like Apple does, but this would require some customization to OpenLDAP so that when a user tries to do basic authentication they will be authenticated via the password server instead of the userPassword property. I didn’t really care for that option.

What doesn’t work? The CHANGEPASS command I mentioned above is not yet implemented but I don’t expect it to be any harder than the DHX was. I would expect it to be a fair amount easier actually. The only other command I have seen so far in all my testing is a GETPOLICY command during loginwindow logins. Apple returns a space separated list of abilities that the user has (for example, isAdmin=1/0). So this should be pretty easy to simulate by just returning a blank list or something. Right now it gives back an error response and the login proceeds normally. I think normally the loginwindow would check to see if the policy allows local logins, needs to change password, etc. There are also about 30+ more commands buried in Apple’s implementation that I haven’t even seen used yet. But those may not actually ever be used in the setup I will be running so we’ll have to wait and see.

If you are interested in taking a look at whats going on head over to Google Code and take a look at the project: http://code.google.com/p/lpws/

If you would like to help out just drop a note to me over there in the project. I’d be happy to have any extra help, especially with some of the documentation. Otherwise feel free to download the code and take it for a spin.

You might, by the way, be wondering why I would ever want to do this anyway. The answer is simple. Apple’s tools work great if you play by their rules. If you don’t want to customize anything or do anything a little different you are all good. As soon as you start making custom changes (like installing your own SSL certificate so it’s actually valid instead of a self-signed one) or modifying the way the Mail service filters spam; that is when you run into trouble. I don’t play by their rules 100% of the time. I can’t. They work great if you are a company with maybe a dozen employees but I have to deal with over 60 user accounts and about 150 client machines.

The other reason I want to do this is redundancy. I want to put all this authentication stuff on a VM, but I can’t do that with Apple. In order to virtualize I have to run it on Apple hardware. Only problem with that is they got rid of their server hardware.  The Mac Mini is great as a client device but a joke as server.  One ethernet? One power supply? Yea right. The Mac Pro is not much better. I gain a secondary ethernet port but still have a single power supply. But even if I did go with a Mac Pro, to get proper redundancy I would need at-least 2 Mac Pros in the rack (yea there goes 12U of space) and to buy a separate copy of VMWare Essentials Plus (at another $1,500 thank you). So now I’m sitting at $7,500 just to run a single server? no thanks. I’d rather do it on Linux where it’s free and works with everything I’ve already got installed. Does that mean I will still continue down this path if I can’t get the Mac clients to integrate nicely with Linux server? nope. That is still #1 priority, I need to have easy management. Time will tell.

Anime Review: Last Exile, Squid Girl, Kaze no Stigma

Last Exile

Claus and Lavie are childhood friends who fly their fathers’ vanship (think mini-airplane). They lost their parents when they were both just little kids and have had nothing but each other for years.  They are message couriers flying around the country delivering letters so they can pay to keep their little house and their vanship flying. Flying is the only thing they love. They pretty much ignore the war between their country, Anatore and the neighboring country Dusis. They just want to fly. The war is controlled by a small group of powerful people called the Guild. The Guild supplies the engines that power the larger ships and allow them to fly, and through flying to do battle with each other.

Clause and Lavie may not have much, but they are happy because they get to fly together. Their life is simple. Until they get an expected job delivering a little girl named Alvis. Things start to go downhill for all three of them as they quickly discover that the Guild is after Alvis and will stop at nothing to get her. Clause can’t stand to just dump Alvis off and leave her so he and Lavie take it on themselves to make sure Alvis is safe, even after reaching their destination. Only the captain of the Sylvana (where Alvis was to be delivered) seems to know why the Guild is after Alvis, and that reason seems to be that Alvis is the only one who can save the world from the Guild.

This was an exciting story with some really neat thought put into the design of the various ships flying through the air. There is a second season with all new characters but I have not yet started watching it so I can’t say much about it. There was a good story behind the animation instead of throwing a bunch of half-naked girls on screen to cover for the lack of story.

Season 2 was different. It really was less a season two and more a spin-off. I’m not finished watching it but I’ve gotten most of the way through. The story is pretty good but not as good as Season 1. The first half of the season was spent trying to figure out what was going on. Some parts of the story seemed to indicate they were back on Earth (assumed to be Earth) and other parts indicated that they were still on the other planet that Season 1 took place on. To be honest I’m still not sure where they are.

The Vanships you grew to love in Season 1 are pretty much gone. There are all kinds of different Vanships but they are all these tiny little things that make no sense. Season 1 was filled with ships that needed the Guild to power them so they could fly. Season 2 is filled with ships that are powered by themselves. The story up to now seems to pretty much be about one of the countries trying to service as a government in exile. If you haven’t been able to tell already while Season 2 was decent it hasn’t left any lasting memories.

As for content, Season 1 was very clean. Season 2 was a different story. The very first scene was the main star pulling off most of her clothes and falling out of a plane. It has nothing to do with the rest of the show. She is sleepwalking and they comment about how this happens all the time and I’ve yet to see it happen again. So basically they put it in to get a rating. It’s worth skipping. The opening sequence is also very inappropriate and worth skipping. Again I got the feeling that the only reason sequence was made that way was for the rating. If you skip these two pieces you should enjoy the show.

Summary: Solid story with great animation. Thoroughly enjoyed watching and have already watched again.
Seasons: 2
Season length: 26 episodes
Episode length: 23 minutes
Content: Very clean in Season 1. Dirty opening sequence in Season 2 but otherwise seems clean.

Squid Girl

This was a great show. The story revolves around “Squid Girl” who is from the sea. The “sea” is upset at humanity for polluting it and destroying the sanctity of the sea. Umm yea okay, whatever. It’s kind of a strange premise but doesn’t really matter. It doesn’t matter past the first few minutes. So anyway the sea sends Squid Girl (Ika-chan) to overthrow humanity and control the surface world so that the sea can remain clean.

So skipping the premise and moving on to the actual story… Ika-chan is a complete failure at her mission. Yea I know, big surprise. She starts trying to take over the Lemon, which is a kind of diner on the beach run by the sisters Chizuru and Eiko, along with their little brother, Takeru, who helps out from time to time. Not able to take over the Lemon right away Ika realizes that her plan to take over the surface would could take awhile – and she will need a place to stay!

The rest of the story follows Ika as she tries (and fails) to take over small parts of the surface world. Many of her adventures are outrageous as she slowly learns what it is to be human. Over time she even seems to begin to understand humans and realize that not all humans are bad. The show is filled with lots of humor, lots of friends and lots of crazy antics by Ika-chan.

Summary: Lots of comedic adventures as we follow Ika’s attempt to take over the surface world!
Seasons: 2
Season length: 12 episodes
Episode length: 23 minutes
Content: Completely clean, I don’t recall anything in the entire show.

Kaze no Stigma

Kazuma is from the Kannagi family – a family of fire users. He’s the one person in the whole family that can’t use fire. As a child one of his distant relatives, Ayano, beat him in a dual because of his inability to use fire and his father banished him from the family – never to be allowed to return because of the disgrace. While out surviving on his own he forms a contract with the wind spirit to become a wind user. Because his power to control wind comes from the spirit itself his power is beyond that of other users.

Returning home he finds his family still refuses to let him be a part of the family, and that is just fine by him as he doesn’t want anything to do with his family either. His relative Ayano has become the one who will inherit the families sword, Enraiha. Unfortunately she isn’t yet up to the task and needs help. And that help seems to keep coming from Kazuma, whom Ayano finds to be one of the most annoying jerks she has ever met. That coupled with the fact the family considers him a traitor makes for some tense situations. Those tense situations become even more fired up when Ayano begins to have feelings for the fire-wielding Kazuma.

Kazuma just wants the money that comes along with helping Ayano perform the tasks set before her that come with wielding Enraiha. Time and time-again Kazuma ends up coming to her rescue and helping her out just enough to keep her from losing. The situation gets complicated when members of the Kannagi family start mysteriously dying – and the evidence points to a wind user. Can Ayano overcome her mixed feelings for Kazuma and trust in him or will she draw her sword against her new friend?

Summary: A show with an interesting use of the elements, good story with a decent amount of comedic relief to break it up.
Seasons: 1
Season length: 24 episodes
Episode length: 23 minutes
Content: Mostly clean. If I recall there are some underwear shots as Ayano is jumping around wielding her sword but I don’t recall anything extreme.

Kenya 2012 Summary

So I wanted to give a little summary of everything that happened.  If you are interested in some more of the details and hear some more of the story there are about 4 or 5 posts that I wrote during my trip.  As I write this it has been about a week since I got back to the States and have had some time to adjust back to life here and think through a little more all that happened while I was in Kenya. The whole point of the trip, in-case you don’t know already, was to move RVA (missionary kid boarding school in Kenya) from their existing shared phone system to a new phone system that would no longer be shared.

Gosh. So much happened while I was over there it is hard to believe it all happened in 4-5 days.  In that short week I did more fast-paced work than I normally do in 4 months at HDC. It isn’t that we don’t deal with the same kinds of stuff (though we have yet to have a lightning strike take out half our network *knock on wood* at HDC), it’s just that they are more spread out over time.  They don’t all happen right on top of each other.  The only time I can think of being that rushed was when we went live with Arena. We came out of installing our new phone system and then had just over a month and a half to prepare for and install Arena.

So my trip. The travel itself was good. Long, but good. Nothing bad happened on the plane flight. No delays, no missed flights, no lost luggage.  Once we landed in Kenya, however, we got stuck in Customs and ended up paying about $400 in duties on the equipment we brought on, which was a first for RVA. In 6 years they’ve never had people coming in have to pay duties on anything brought in. “Just tell them your with RVA and they’ll let you right through.” So we tried that. “Never heard of them.” *sigh* That whole debacle added about an hour to our time in the airport. Oh. And as we are going through passport control we notice a big sign stating that it will cost us $50 to enter the country. It kind of left me wondering what would happen if somebody like me had flown 18 hours to Kenya and then was told they couldn’t enter the country because they didn’t have $50. “Thanks for the visit, come back when you have the cash!”

It was at this point that I first met Hany (one of our team that flew out of Egypt) and Jeff (from RVA). Both were really great guys. Hany proved himself to know quite a bit about electronics and just had a good grasp of “how things work” in general. At this point it was around 10:30 or 11pm local time so we hit the road for about an hour drive to the RVA campus. The drive was… exciting. We gave Jeff, who was driving, a hard time though I don’t think any of us truly felt afraid for our lives – but the roads there sure gave us quite a few thrills and scares. Most of the road has no lines and no dividers and people pass all the time wether it’s safe to pass or not. And there are lots of random curves that make you think you are about to have a head-on collision until the last second.

Sunday started with church at about 10am or so. It was a great service. It had it’s rough spots and hiccups (which were apparently not normal) but was still great to stand there worshiping God with people from over a dozen different countries. After church and lunch we made a trip down to the hospital (about a mile away) where the telephone line comes in.  Most of the afternoon was spent there trying to work out how the current system works and figure out how to interface the new phone system with the telephone company as well as the old phone system (which the hospital was staying on, only the school switched). Monday morning was also a quick trip back down to make perform some tests with the old system to make sure we can interface.

Monday after lunch is when the fun started. Just as we were about to break up and tackle all the various tasks we had ahead of us there was a massive lightning strike. Again the first that anybody at RVA can remember actually hitting a building (and causing damage). The toll as of when we left (and I’m sure they have found more since we left) was that the lightning took out their 10,000watt UPS (which runs all the servers), 9 out of 15 network switches, over a dozen lab computers, and 3 staff/faculty computers. Amazingly (i.e. God) none of the actual servers were hit. The hard power off caused some issues with the mail server that will take awhile to work out, but for the most everything critical to their operations survived.

So the rest of Monday and all of Tuesday was spent rebuilding their network. It was so damaged that we couldn’t even really work on the phone system. Jeremy, and myself helped Jeff get their network back up and running while Hany and Fady worked on tracing out phone wires and doing the re-wiring in preparation of installing the phone system. Those guys deserve some serious props for sitting in front of these little 1ft by 1ft boxes with over 300 wires in them tracing things out, moving wires around, labeling and cleaning up stuff so that we could jump right into installing the new hardware once the network was up.  I think all said and done over the week they spent probably more than 25 hours sitting in front of those boxes. One we got the network up Tuesday night, Tom (a college intern from the States) another Jeff from RVA and myself worked until around 5am to get their staff/faculty computers back up and the computer classroom so they could hold classes in the morning. While we were doing this, by the way, Hany and the first Jeff were still wiring in the boxes.

We all took a short nap and then Wednesday morning (late morning) began working on bringing the actual phones up on the new system. We basically worked the rest of Wednesday until around 11pm and then all day Thursday again patching phones in and testing them. We had crossed lines to untangle, lines that were shorted together that we had to move around, and just tracing random phones that got missed the first time around. We were able to quit a little early on Thursday, about 8 or 9pm, so that we could all spend a little time with our host families whom we had basically not seen or talked to since we arrived. Generally we would have gone home around 7 or 8pm each night but because of the lightning strike we had so little time we never got home at a decent hour.

So our last night there I got to spend about an hour or so chatting with my host family and getting to know them (and them me) a little better. I wish I could have gotten to know them better, but I’m glad that God worked it all out in the end. I think we were all a little worried that our trip to install the phone system might turn out to be a wash. We were all very glad, however, that we were there when the lightning strike happened. They could have gotten things back up, but it would have taken longer. As it was they went from a team of basically 2 people all the way up to a team of 6 with us there.

Friday we got to spend the morning and part of the afternoon doing a Safari, which was a lot of fun.  We all had a blast getting to see the various animals. From there we had a short stop back at RVA to collect all our stuff and then heading into Nairobi to spend the night, as our flight out was pretty early in the morning.

All in all, we managed to get everything critical done. Out of about 170 phones we got all but about 10 online. Of those 10 it is mostly just a matter of them needing to trace out a few phone lines to figure out where they should be patched. They are able to call from RVA down to the hospital. The hospital can (sort-of) call back to RVA. We didn’t have time to reprogram the old system so they have to dial one of 8 extensions that we captured and then they can dial again for the actual RVA extension. They are hoping to have somebody out to program the old system in the next day or two so they can just dial direct, or at-least dial a single extension instead of 8. We only gave them a 5-minute overview of the phone system software so we have been doing some simple e-mail support to answer questions as they have come up.

I have already said this once, but I have given it a lot more thought since I originally said in my last post that I would enjoy going back. I still would enjoy going back. The truth is I had a great time at RVA. It was tough, long hours, nothing but rushing around. But I would love to go back and do more I.T. work with them and, hopefully, have a little more time to get to know everybody. It was truly a blessing to me to have gone. I’m glad my friend Jeremy didn’t take my initial “no” as final and instead asked me again when the timing of the trip changed. I think God worked in me as much as he was able to work through me during this trip. I’m not sure when or where my next trip will be but I know I will be doing more traveling in my future. Not just because there is a huge need for I.T. “missionaries” but because I think God has a lot to teach me and because there are things I have decided I want to do. I don’t want to just sit on my couch any more. I want to see what there is in the world. I want to see what God can do through me.

Leaving On a Jet Plane…

(Updated 3-3-2012, added more info about Safari trip)

It’s Saturday morning local time.  Fady and Hany flew out at 4am.  Jeremy and I fly out in a few hours, at 8:30am. What a whirlwind week. We managed to get almost everything done and functional.  There are probably about a dozen phone lines out of 170 that don’t work at all and another dozen that cannot get their DSL to connect (there are about 70 family houses, each uses DSL to get internet from the main IT office). I think we were tracing down wiring problems until about 2 minutes before we got in the car to head to the guest house near the airport.

We managed to go on a short Safari into Naraku (I think that is how it is spelled) reservation. Got to see a lot of animals, nearly all of which I’ve forgotten the names of. When we got back to RVA to pick up all our bags and head out again there was a small disaster that had to be fixed. One of the temporary replacement switches lost some of its configuration when they installed the new UPS because we forgot to save so we had to track down and re-set those settings. Since we were waiting we also fixed a couple more phone lines that we found that were crossed or missing.

The people at RVA were wonderful. Very welcoming and helpful to us as we walked around campus trying to track down issues. We ended up being far more disruptive than we would normally have been because of the lightning strike – we figured everything was down anyway, might as well just do a hard switch on the phones and take them down too – but everybody was happy to let us disrupt them for a few minutes to test wires or track down crossed circuits.

The Safari trip was fantastic. We got to see a lot of really cool animals and had a great couple of hours to just hang out with the other guys on the team and get to know each other a bit more. I’ve included a couple pictures from the Safari to give an idea of what we saw out there. As fun as the Safari was, I’m rather torn because I think I would have rather spent the day at RVA just hanging, visiting and exploring the campus a bit more.

I only got to chat with my host family once, on Thursday night, and that was only for about 30 or 45 minutes. The rest of the week I really only saw them in passing. I would have liked to be able to spend a bit more time getting to know them as well as the other I.T. guys at RVA. It would have been fun to be able to go around in the daytime and take a few pictures of the campus to remember the place a bit more. Or even to be able to see Nairobi in the daytime. As it is we arrived late at night and left early in the morning, so I really never got to see the city itself in daylight.

My stay at Amsterdam was also short and in the dark. By the time we landed and off the plane it was already getting dark. Jeremy and I went into town for about an hour and walked one of the streets and took a few pictures of the buildings – to which can only be described as amazing. It is truly amazing what they have achieved in progress yet still kept many, maybe all, of the original buildings. I have not yet been to London, but Amsterdam looks a lot like I imagine London to look (at least what I saw of it in the dark). Lots of tall, majestic looking buildings that make you think they must be hundreds of years old.

So I’m Leaving on a Jet plane
Don’t know when I’ll be back again…

… But I think I will.

When it rains, it pours. Literally.

It happens figuratively too. As I write this up for posting in the morning when I can get Internet access, I am finishing up one very long day. It started on Tuesday. I don’t remember much of Tuesday except that it rained a lot and we had to pause our phone system work to help get the network operational. The network was in such shambles from the lightning strike that we couldn’t even work on the simplest tasks for the phone system.

Out of 15 primary network switches, 9 sustained damage and will need to be replaced. 8 of those switches we were able to find temporary replacements by stealing from other buildings and making use of some older equipment. The remaining one should be replaced now but we just don’t have anything we can put in its place.

Pretty much up until dinner time I helped Jeff walk the entire campus tracking down which switches were damaged and which were still good and could be used for other more critical needs. It was very discouraging work to see how much equipment was lost, but I was also able to see parts of the compound that I would have never seen otherwise. So that part of it was nice.

After dinner I helped Tom (an intern working here for a semester during his college time in the US) and a different Jeff (Jeff H.) swap out the central network switches to stabilize the network. Things were so bad with the switches that every time we got a device to work we would come back 30 minutes later to find that yet another port on the damaged switch had died and needed to be moved and reconfigured.

The three of us finished that around midnight and then spent the next couple hours trying to bring up the computer classroom so they could have class in the morning (at 7:45). Out of 26 computers in the classroom I think we swapped out 9 of them that didn’t work any more. We pulled from the computer lab across the hall and even some of those didn’t work either. We didn’t bother to check the rest of them as that can be done later, so that count will likely go up once they get the time to check them out.

While we were working on getting the network to function Hany and Jeff S. (from RVA) were cross-connecting existing phones into the new phone system. They finished about a half hour after we did. That is to say, they finished the set of 100 or so phones they were working on. There is another 70 that terminate in a different location.

AAfter chatting with them about how they were doing and having them give me a run-down of what Jeremy and I would need to do in the morning. Jeremy and I met at 6am and began a new day by linking the phones that Hany and Jeff S. had just finished connecting into the new phone system. We finished that about breakfast time and shortly after I went back to the house I’m staying at to take a short nap (I never really made it home the night before) while the other guys started tracking down all the various phone glitches people were running into.

Back up for lunch (and maybe an hour and a half of actual sleep) I joined the guys tracing down software issues, programming issues and later in the evening some wiring issues. We all finished up about 10:30 and headed off to bed, which gives us a decent night sleep tonight.  Actually Jeff S. is in the other location finishing up the 40 out of 70 phone lines they were able to trace out during the day and getting them attached into the ATA devices. In the morning we will begin tracing out the 30 they can’t find so far. We are looking for 30 pairs of wire in a bundle of about 500 pairs.

I’m hoping Jeff S. and myself can get back down to the Lower Station today to finish programming the old phone system to talk to the new one. There are two things we still need to figure out. First is that there is a special phone that needs to be able to dial a specific code and then get an outside line to anything they want (I have found the documentation for doing this, I just need to do it). Secondly, we need to find a way for people on the old PBX to call people on the new PBX (people on the new PBX can already call the people on the old system).

To do that, we basically have two choices. Find a way to program the old system so they can dial a special * code and then dial the extension at the RVA campus. This is the preferred way of doing it. The second method would be to simply take an analog phone interface and hook it up to 8 phone lines out of the old PBX. The downside to this is that people would have to dial one of 8 phone extensions and then dial again to get somebody at RVA. The first option is of course preferable, but we are not sure we can do it with the time we have.

And did I mention that it rained a lot today? I mean a lot. Like if this was Victorville we would have called it a flash flood rain. Water was just pouring off the top of the buildings in solid sheets. In the week I’ve been here, I haven’t seen this much rain in years. And this isn’t even their rainy season here. The rain started right after lunch and abut 3 hours after it started it was clear blue skies again. Sadly, I didn’t make it back to the house to pull my laundry off the clothes line before the rain kicked up, oh well.

We have two days left.  We are confident we will be done in time, just maybe not leaving them with the system as fully configured as we would have liked. We will just keep ploughing forward and get as much done as we can and leave the rest in God’s hands to help them along after we leave. Again, we are confident. The single item I’m worried about is getting the old PBX to talk to the new one. When we started partially switching at HDC for testing we spent days just figuring this part out. I am guessing I will have about 2 hours tomorrow and thats it. So hopefully some door opens up.

I’ve only been here 4 days and I am already realizing that I will miss this place and the people here. To be honest, the last 2 days I hardly spoke to anybody because we were all locked in small rooms trying to fix things. Even so I’ve seen how much people (for the most part) are here because they want to be here, not because they have to be here. The teachers, students and all the support staff are not here simply because it is a paycheck. They want to make a difference.

Sometimes things don’t go as expected

And those things can sometimes happen with a bright flash of lightning and a sonic boom of thunder. Although, God usually finds a way to work them out. But that would be getting ahead of myself.

Sunday was a pretty relaxed day, as well as kind of short. We were tired from the long flight so our minds weren’t 100% yet, but we got a few things done. After breakfast we did a quick tour of the facility, did some brainstorming about how we were going to tackle everything, and then wen’t to church services.

Church was rather cool to see. It was actually an unusual day for services since they had some weekend event that many of the adults do with the seniors to help prepare them for graduating and heading back to America; since most of them have for a good portion of their life. Everything in the church service was done by the kids, except for the actual message. They lead worship, they collected the offering, they did some of the prayers, all in all it was very fun to see.

The rest of the day was spent down at what is called the Lower Station, which is basically the hospital and a few other buildings. We worked through the existing PBX (phone system) to try and figure out exactly how to hook everything up. We had spent most of the previous weekend configuring the box for a very confusing protocol called SS7, which the telephone company had told RVA the phone line was using. Not so much. It was using just plain old ISDN (i.e. what we use here in the U.S.). We called it an early night so we could get up early Monday morning and start working at 6am.

Welcome to Monday. This is where things got interesting. We go back down to the Lower Station to hookup our new PBX router. We plug the telephone line in. Works great. We plug the line to the existing PBX in. Doesn’t work. Not at all. Nothing. Nada. Complete fail. Come 8am we call it quits since the hospital is opening up and will need their phone lines with plans to try again after lunch (during their break time).

After lunch we head back down. Still nothing. We’re down to the last few minutes of time we have to work on it and on a whim we try a cable that is, to put it simply, wired backwards and shouldn’t work. It works! The stupid device is wired backwards! Welcome to Africa! After putting everything back together we came back to RVA to discuss the actual switchover plans. This is our first proof that we can actually make this work so we are quite excited. We’re ecstatic. We’re overjoyed. We can do this. We can pull it off. *flash* *boom* *SIZZLE-SPARK* *screaming girls out front* Uh oh.

That sizzle and spark was the server rack in the IT office we were discussing things in, which is now without power. The screaming girls out front are perfectly fine but in complete shock and hysterics from seeing lightning up close and personal striking the building that was about 30 feet away from them. Which, by the way, happens to house the main IT equipment and servers. And the battery backup unit powering both rooms (which is now oddly enough, not powering either).

In short, the lightning fried the 10,000 watt UPS. Completely. It’s dead, gone, won’t turn on. After about 30 minutes of tinkering with that thing trying to see if we can get it to work, without success, they start pulling smaller UPS units from all over campus to get the servers back online. This also requires us to start making custom cables to hookup all the equipment to these new UPS units that were not designed to handle this type of equipment. Come 5pm Jeff from RVA and myself need to head back down to the Lower Station to do the actual phone system swap, with only about 75% of the servers back up and running.

This is where God starts shining. Okay, well he did shine quite well in the lightning strike, but that was more of a “hide behind that rock and see a flash of my back as I run past” kind of shining. Jeff and I go down, plug in the cables, the lights go green and we make our first test call… works! Second test call… also works! a dozen more calls, each one works perfectly. Finally, something is going our way again! Hey even better, the Internet guys show up to work on getting the Internet connection up and running which has only been working 60% of the time since we’ve been here. *cell phone rings* “You got all of the servers back up? Great” “Not great?” “What kind of data corruption?” Then from the Internet guys: “We’re done working on the Internet for now, it now works 0% of the time but we’ll keep working on it tomorrow morning” *sigh* Back to RVA…

Jeff goes to help continue the repair efforts while I start researching a few things with the PBX we will need to configure the next day. Around 8 I finish my research and head up to the larger IT room to see how they are doing. The mail server is the one that is corrupted and none of the mail services will start. Some of the config files are now also filled with garbage and need to be rebuilt by hand. One of their guys has been working on it for the past 4 hours. It happens to be a Linux box using much of the same software we use at HDC for mail so I join up with him. Finally after another 3 and a half hours we get it up and running.

Everything is finally back up and running, we are breaking for the night to sleep. We also managed to get some internet going through their old provider who happened to not shut off the line yet *shhh*.

So things don’t always go as we expect, or even hope. But God always seems to have his hand in it. The lightning strike was incredibly bad luck for RVA, especially in the middle of installing a new phone system. But the fact that it happened while we were here was a tremendous blessing to them as they had an extra 4 hands to work through all the problems and get everything back up for the next day’s classes. The phone system stuff had us extremely worried, but when it came time to sink or swim, we actually found we were floating.

We still have lots to do, the part of the PBX we installed is just a router. But it lets us bring the phones up one at a time on the new system instead of trying to go whole-hog and switch everything over at once. So we can spend tomorrow doing the changeover at a more manageable pace and have time to do some serious testing. As well as help them make plans on buying a new UPS! The company they bought the UPS from is actually coming out to look at it tomorrow, so hopefully they will decide it should be covered under some kind of warranty, since it is only 2 years old.

I just wish I had been outside to see the lightning strike!

The Eagle has Landed

It’s hard to believe that I have to fly all the way to Amsterdam in order to listen to Dutch music. I’m on an airplane flying at 600mph over the United States. The in-seat personal entertainment TV has a music selection with includes “KLM fm”, “CD jukebox”, “Audio books”, and “Dutch music”. My friend Matt would be crushed that it’s so easy to listen to that music if all you have to do is buy a plane ticket. It is also quite amazing to see that it is -57 degrees at the altitude we are flying at. That is cold, even for us used to the wild weather in the desert (wonder what the chill factor is like at 600mph?). We are “only” 5 miles high. I know in reality that is probably a lot, but when you think of the fact that we can drive 5 miles in about as many minutes, kind of makes you realize how low the planes really cruise at.

Getting on the plane went well and was extremely quick. Jeremy and I spent about a half hour at his house redistributing the weight of our luggage so we could stay under the 50lb per bag, 2 bag each limit. We got everything weight in at the airline scales at 200.5lbs; and then the lady was kind enough to ask if we wanted to check our carry-on as well since we get 3 checked bags each. Free. (due apparently to Jeremy’s skymiles card). *sigh* oh well. We got everything we needed on the plane. The best part is all 4 bags actually arrived in Kenya!

Minor miracle as we were about to climb into Jeremy’s car to drive over to LAX. We ordered 10 ATA units for the project, one of which is to be a spare. When we unboxed and tested them last Sunday 2 had damaged connectors and one was completely DOA. The replacements were scheduled to arrive “sometime today (Friday)”. As we were getting in the car Jeremy got a text from somebody in the office letting him know the replacement ATAs had just arrived. While we drove over to pick them up and swap out luggage they tested them for us to make sure they turned on and were not damaged.

So Jeremy and I spent about 18 long hours in the air.  We had a 10 hour flight from LAX to Amsterdam and then a 1 hour 20 minute layover before we boarded the next plane. All said and done we had about 15 minutes to stretch our legs. From there it was another 8 hour flight down to Kenya. In reality everything went great. We didn’t have any trouble and honestly there really wasn’t anything different about the flight other than having to show my passport every few minutes instead of just my drivers license. That is until we got to Nairobi airport.

So, Jeff, the guy that picked us up and is the point-man for our visit here at RVA, said that in the 6 years he has been here customs has never charged anybody when bringing equipment into the country. They just say it is for RVA and “oh okay, go on ahead then.” So of course, Jeremy and I have to break that record. The guy has never heard of RVA.  We of course have no paperwork on any of the equipment to show how much it actually costs. So what is the obvious thing for the customs agent to do? (actually maybe they are called duty agents, I’m not sure which is for incoming and which is for outgoing) Why of course, he makes up some numbers in his head as to how much he thinks those items cost and then calculates the amount of tax we have to pay to bring that equipment into the country… On the order of $370-something. Ouch. Thankfully Fady was standing outside and happened to have a wad of cash in his pocket.

So then the drive out to RVA. It’s about an hour drive. And some of the most terrifying things I have ever seen. Never mind the fact that everybody drives on the “wrong” side of the road. Never mind that at the check-gate leaving the airport there were armed guards with AK-47s (or whatever) walking around peering in the windows. Never mind that everybody drives like they are from Vegas. Let me be sure to mention one thing. There are no lines on the roads and for more than half the drive no dividers either. Oh and the road does the small curves constantly so you think you are about to have a head-on with somebody until at the last second the road veers away from them. By the way, did I mention that it was night-time when we arrived and very dark? We also passed more than one semi truck stopped (that is parked for the night) right smack dab in the middle of the road with all the lights turned off. I think I forgot to mention that there are traffic signals, so that was helpful with all the circular loops at every single intersection… Except for the fact that nobody pays attention to them and everybody runs the red lights without even slowing down. I’ll never complain about Los Angeles again.

On the bright side, we finally got to RVA (alive I might add). They were hoping to have a single residence for us to all share. Oops no such luck. They do, however, have 4 different families that each have a spare bedroom (or maybe it was they had a kid they could move in with another kid for a few days). So we are each staying at a different house with bleary-eyed people we each met for about 30 seconds before going back to bed. We still haven’t addressed the water issue (as in the “it’s safe to drink from the tap” statement), but malaria mosquitos don’t seem to be much of a problem because of the elevation (2,500km, or about 7,500 feet). And God is good to me on my first trip, I get a real bed to sleep in!

So with that ends day number 1. Starting at 5am Friday PST, ending at 1:30pm Saturday PST (or 12:30am Sunday morning for me). Tomorrow we will eat breakfast, go to church and then roll up our sleeves. Now the fun really begins!

Ready or Not…

It’s early… Really early…  God isn’t awake yet.

Okay, maybe 5:30am isn’t that early, but it sure feels like it to me.  I’m all packed and ready to go, I hope.  At this point it is all in God’s hands (was it ever not?) so I’m just waiting for my ride and trying to keep myself calm and relaxed for the upcoming 18 hours of plane flights.

The latest information for the project is that the Hospital has decided (by the fact they never got back to us) that they are not switching phone systems just yet.  In one aspect that means less work for us as there are now 350 less extensions to worry about.  In another aspect that means more work as we have to figure out how to interface with the old system and make both phone systems talk to each other as well as the outside world at the same time (through a single wire to the telephone company). We also will need to do our best to make sure that everything is setup in such a way that if their existing PBX fails we can setup all the equipment here in the states and ship it to them and have them drop it in place with a minimal amount of effort and training.

Another challenge is the fact that in Africa (and many other parts of the world) the telephone company uses something called SS7 to communicate over digital telephone lines. In the US, we use something called PRI that has no configurable parameters, it’s just plug and play. SS7 on the other hand has about 9 configurable parameters, all(most?) of which need to be set correctly for the communication to work. I found a book on Amazon called something like Introduction to SS7, it was 1,500 pages.  Oi. In lieu of buying that book and shipping it here overnight I collected various resources and online books from the Internet to help with our setup.

Between Jeremy and myself (the two of us flying out of the states) we have 4 suitcases of equipment, each weighing in at nearly 50 pounds apiece. Plus a few extra things coming with the guys who are flying out of Egypt and meeting us there.

… Here I Go!

Apple finally lets me close files!

With the 10.7.3 update apple has “fixed” the non-local-HFS save issue.  I say fixed in quotes since they haven’t really fixed it but they did make it manageable now.

Recap of the problem: Basically if you open a file on a non-local-HFS volume (i.e. a file server or SD card), your Autosave/Versions aware application would automatically save your changes without storing a previous version, thus erasing your original content. In 10.7.2 they added a dialog box warning you they were about to erase your content, but still didn’t give you any (easy) way around it.

New in 10.7.3: There is a new option in that previously oh-so-unhelpful dialog box called Revert. Clicking this button will revert to the original and then close the window for you.  Know what that sounds like?  Sounds like the Don’t Save button we have been trained to use for 20+ years. I give Apple credit for trying something new, but I applaud them for listening to the masses of people complaining about this new feature.

It should be noted that the Do not show this message again checkbox only works when you click the OK button. Turning on that checkbox and clicking Revert ignores that checkbox and the dialog will come up again next time.

It isn’t a 100% perfect solution, but teaching people that “Revert” means “Don’t Save” is far easier than teaching people to learn how to follow a new 6 step process to close a file without saving it and losing your original content. Thank you Apple for listening!