fouldsy.com

Ted got picked me up at stupid o’clock this morning to be at Yute Air for 7.30a.m to fly out to Atmautluak with their new Open Directory server for deployment. They actually had us on the plane and in the air right around 8.30a.m as planned which was novel. It really was a little puddle-jumper flight across the melting tundra:

A very short, sub-10 minute flight brought us in to the Atmautluak, a pretty small village west of Bethel:

There was quite a bit more snow down than around Bethel, was quite weird. At least the airline agent took the X-serve on a sled to the school whilst we took the 10 minute or so walk there, as I did not fancy carrying a 50lbs server box through the snow! There are signs of melting though, and these dogs are soon going to be swimming if they’re not moved to slightly higher ground:

After some minor networking issues, we brought their new server online without too many problems, got it replicating back to the district office in Bethel and then began migrating some of their staff. We managed to get a good half a dozen staff running against the new server which was most of their teaching staff and our goal. Even more surprisingly, Yute were seemingly on time to pick us up at 4.30p.m, so we grabbed a ride out to the airport. It’s a tough life sometimes working in Alaska – this was Ted + I in the back of a sled whilst Brent clung on to the snowmobile (the sled was a lot safer!):

On the plane, a rather curious navigation device that managed to maintain it’s position throughout the flight back to Bethel:

But, at least we were back in Bethel by 5p.m, a good end to the day. Good ol’ Yute Air finally managed to redeem themselves though – a couple of boxes we were bringing back for the media center never made if off the plane (even though we were the only passengers on it, the pilot helped us load them on, and in fact, had to move one of the boxes to get our bags off in Bethel…) and instead went completely the opposite direction down to Eek and Quinhagak That’s our last Open Directory deployment for the school year though, we’ve now got four deployed outside the district office, and a good spread of users running off them. Will give us some valuable testing before rolling out more users across those sites after the summer and then deploying the technology to more sites across the district.

I decided my glasses were boring after seeing a pair Andrea was walking around with:

Spiffy, but don’t clear up my vision quite as well. Just another Monday morning in the office…

This morning brought more weather holds and AT&T called off the trip out here. I was quite relieved as had to get GCI cable sorted and head to the chiropractor over lunchtime, so hadn’t wanted to travel today anyway. At least our engineer working on our core networking made it out okay this afternoon and got straight to work, so I’m hoping things go smoothly over the weekend and by Monday/Tuesday our spiffy VMware environment will finally be ready to roll.

Anyways, I got cable TV canceled today, and internet is apparently switched over though curiously still works here. My phone has been disconnected so hoping it’s switched too. Is weird sitting in a bare apartment with everything in boxes and a laptop in the middle of the floor as my only source of entertainment and connection to anyone Have rounded up a couple of troops for moving tomorrow, and hoping donating some beers tonight brings a few more!

Weather holds called off my trips to Napaskiak and Nunapitchuk today. The Alaska Air jet circled Bethel for an hour before returning back to Anchorage – after an hour and a half delay before taking off and then 3 hours in the air only to return to Anchorage, the guys on the plane must have really been fuming! Yute Air were grounded anyways too. Tomorrow we were meant to head to Kwethluk and Akiuk, so will see what the plan is when AT&T actually make it out here. I have another engineer coming out to deploy some new Cisco 3560 switches and work on restructuring our core networking for the VMware environment, so I might try to switch and stick in the district office tomorrow. Am booked out to Atmautlauk next Thursday on another Open Directory install, so is all go at the moment!

I have pretty much everything packed up in my apartment ready for Saturday, didn’t realise how much stuff I’ve accumulated over the last 8 months. Utilities should already be on or getting switched tomorrow, and have rounded up a couple of people to help on Saturday so all being well shouldn’t be too bad getting everything moved. Very excited now, can’t wait to get moved in. All my photography gear was shipped out today as well, so should get here the start of next week!

It’s weird how much nicer it’s been the past week walking to work with the sun shining as opposed to it being pitch black until 10a.m! This was the sunrise right outside my apartment this morning:

One of the lakes I cross as a shortcut to work was too slushy and wet to try walking over, but at least the slough was frozen enough to cross otherwise it adds 10+ minutes to my walk. Somehow, a slushy lake I couldn’t walk across didn’t concern me considering I was meant to be driving down the river to Oscarville an hour later… But, we got there okay and spent a few hours binding most of their computers to the Open Directory and migrating some users, and then caught a little of culture week they had going on. This lady was teaching some kids to make mittens:

There was also a group learning to prepare fish, I’m guessing that the other groups that have been heading out jigging will bring back fish these guys will then end up preparing:

After heading out of Oscarville, we swung by Napaskiak just across the river to set up some software ready for running some remote tests from Bethel tomorrow. I’m heading back down to Napaskiak on Thursday with AT&T troubleshooting some networking and VTC issues, but based on the river when we were coming back in to Bethel, we could end up flying down:

We we back in the office a little after 2p.m, but it ended up feeling like such a long day having hit two villages and having spent almost 4 hours just in Oscarville. Was a fairly productive day though, and we have a couple of engineers heading out on Friday to upgrade our core network infrastructure over the weekend which I’m very excited about as will allow us to start to fully utilize our virtual environment.

Making the most of the river before it becomes, er, a river, am heading down to Oscarville again tomorrow to follow up from their Open Directory deployment a couple of weeks ago. The weather is warming up, haven’t worn snow pants in 3 days (weird how these things make you excited when you live up here…), so by the end of the week the river might be getting a little too slushy on top for safe driving. The latest prediction from National Weather Service isn’t for spring break-up until May 12th-19th, but that’s for the ice to actually break free and start moving, it could easily become unsafe to travel within a week. Which is fun since I also volunteered (?) to head to Napaskiak on Thursday with Tom and engineers from AT&T to troubleshoot 8-month long VTC ‘issues’. Somehow my planned trip to Atmautluak on Thursday to deploy an Open Directory for them gave the signal I was free for a Napaskiak trip instead… Would be flying to Atmau, so may as well put it off until next week and drive down to Naps whilst we can.

Is weird I hadn’t left the office in 7 months, and I’ll have done 4 village trips by the end of the week, a 5th in a month if we get out to Atmau next week Not complaining, is good fun, and has taken an awful lot of work to get to the stage where I have some stability on the backbone systems to be able to step away and concentrate on the important areas, out in the villages where the kids are.

Over lunchtime, Greg gave me a ride to the post office to pick up my Walmart bush order which was cool. I have a toaster now. Most of the other stuff is for my new apartment, so am too lazy to unbox everything since I need to actually start packing the next few nights if I’m wanting to move on Saturday.

Today, Ted S, Brent and I packed up and headed for Kwethluk to deploy their new Open Directory server. First we brought up internal DNS and configured recursion back to the central DNS server in the district office, and then configured the new X-serve as an OD replica. It all went pretty smoothly, and then we spent the afternoon migrating quite a few members of staff from local user accounts to new mobile OD accounts which was slow, but effective.

Kwethluk is a smart village, and a good size school, and was nice to get out the office again and feeling like we were really accomplishing something. Getting a good number of users running off the new server was good too, and we’re planning on returning to Oscarville to bring some more of their users online as well.

Driving along the ice road to Kwethluk, a mile or so out of town we came across a rather interesting marker:

The rear of the sign read ‘April Fools’, so nice to know there’s a little good humor going on

Heading back to Bethel this afternoon, the ice road started getting busy with a number of cars on the road. Still felt a little weird being on the river doing 40-50mph in a truck!

When we got back in to Bethel I saw Dean and arranged to meet up tomorrow evening to check over new apartment and get the keys for moving in. Will try to head to city board and utilities to get water + sewer and electricity set up tomorrow (along with paying the quite outrageous deposits…) and plan for telephone to be switched over and see if I can grab DSL with them too. Am wanting to ditch cable completely as new apartment has Dish Network for TV, and GCI is already expensive for internet service really, although good, and will have extra charges if I’m not taking cable TV and stuff with them. Nuts to that!

“Er, I’ve never driven the road to Oscarville, but I’m pretty sure this is taking us off the river to Napaskiak, Ted”…

We got there in the end, and spent a frustrating number of hours rebuilding what should have their new Open Directory replica due to yet another IP conflict. 6+ hours later and no lunch, we finally finished, but only let us set up one teacher bound to the directory. Was nice to get out the office to a village though:

Oscarville itself was about as expected, an awful lot smaller than Tunt. I think there’s only about 40 people or so live there. Didn’t make me miss living in the villages, though did make me wonder how different things may had been to only have been 10 minutes out of Bethel and with a couple of other villages less than that distance away too.

Coming back, less than 200 yards out of Oscarville Ted dumped us on a little snow drift After shovelling a snow out the way of the wheels and with Ken and I pushing, the truck popped right out and off we went again. Thankfully we’re not heading to Kwethluk tomorrow based on Ted’s driving, though has at least given me plenty of ammunition for winding him up for a few days at least!

On Tuesday I threw a sick day. I felt like crap after all the business of Cama’i, and had picked up a nice chesty cough. Having no heat all Monday evening didn’t help. But, I picked myself up and was back in work today. I did have our Apple engineer threatening to wrestle me to the ground and give me a double wet-willy (he’s a big bloke, I wouldn’t have fought back) after spending an hour troubleshooting mobile Open Directory account synchronization problems off-site when the personal firewall is up. Bizarrely, log in sync worked, just not on demand, scheduled, or log out. A smile about April Fool’s didn’t cut it, but he saw the funny side of it, and was actually quite interesting and gives us more to poke around with as to when the firewall is actually brought up and if the log in sync runs differently as a result. Tomorrow I’m being allowed out the office again to head a few miles downriver to Oscarville to deploy their new Open Directory replica which should be good fun.

This evening was the K300 Kids Race, a 20 mile round trip race up river to the Hoffman fish camp and back, with a 30 minute layover for hotdogs, beans, and juice. I snagged a ride up in a truck with Bev Hoffman (I promised my chiropractor I wouldn’t bounce and shake along the river on a *snowmobile*, didn’t mention anything about a *truck* ), and got some good photos over the layover whilst grabbing a hot dog and juice to boot. All the kids had fun, and I’ll try to get some of my photos up here tomorrow. Have processed most of them and passed a few on to Casie for the K300 site, but too tired to edit them down for here tonight.

But, was quite weird tearing along the river in a truck, first time I’ve headed on to the ice road other than on a snowmobile. Was good fun, even with our little spin! A few miles down to Oscarville tomorrow should be cool too.

There’s a lot of confusion of APC’s PCNS software for VMware. Basically, ignore the generic instructions provided on the CD This was tested on multiple ESX 3.5 Update 4 hosts and worked perfectly connected to a pair 2200′s. From what we could tell, the pay-for version is basically just PCNS 2.2.3 but with compatible VMware components. Nothing on the CD tells you this, and the instructions just add to the confusion. These setup steps figured out in conjunction with Kurt Bunker from GCS and posted here for reference.

Prior to configuring the PCNS software you need to configure the SNMP card with the following (note that the admin password and passphrase need to be the same on multiple APC units):

1. IP address
2. Administrative account password
3. Administrator Passphrase (minimum 15 char, Max 32)
4. Set APC SNMP settings for shutdown: UPS tab -> Configuration -> Shutdown -> Low Battery Duration

With the CD inserted, from the command line of your ESX host:

mount /dev/cdrom
cd /mnt/cdrom
./install.sh

During the install:

Instance '1'
Default install directory
Hit enter to automatically install JRE

Once install finished, open require firewall ports *before* running PCNSConfig.sh – you can only run the script once – even if you cancel out, it won’t run again unless you run the full uninstall script. If you’re unhappy about opening all these firewall rules, browse the forums to see exactly what they’re doing. We found they all needed to be open to function correctly.

esxcfg-firewall -o 80,tcp,out,"APC PowerChute Port 80"
esxcfg-firewall -o 2161,tcp,out,"APC PowerChute Port 2161"
esxcfg-firewall -o 2161,tcp,in,"APC PowerChute Port 2161"
esxcfg-firewall -o 2161,udp,out,"APC PowerChute Port 2161"
esxcfg-firewall -o 2161,udp,in,"APC PowerChute Port 2161"
esxcfg-firewall -o 3052,tcp,out,"APC PowerChute Port 3052"
esxcfg-firewall -o 3052,tcp,in,"APC PowerChute Port 3052"
esxcfg-firewall -o 3052,udp,out,"APC PowerChute Port 3052"
esxcfg-firewall -o 3052,udp,in,"APC PowerChute Port 3052"
esxcfg-firewall -o 6547,tcp,out,"APC PowerChute Port 6547"
esxcfg-firewall -o 6547,tcp,in,"APC PowerChute Port 6547"
esxcfg-firewall -o 6547,udp,out,"APC PowerChute Port 6547"
esxcfg-firewall -o 6547,udp,in,"APC PowerChute Port 6547"
esxcfg-firewall -o 6548,tcp,out,"APC PowerChute Port 6548"
esxcfg-firewall -o 6548,tcp,in,"APC PowerChute Port 6548"
esxcfg-firewall -o 6548,udp,out,"APC PowerChute Port 6548"
esxcfg-firewall -o 6548,udp,in,"APC PowerChute Port 6548"

With the ports open, configure the PCNS software:

cd /opt/APC/PowerChute/group1
./PCNSConfig.sh

During the configuration:

Select option '3' - Configure for multiple Smart-UPS devices
Enter IP address, port 80 (default), username, password, authentication passphrase
'Yes' to register settings
Enter IP of second card - username, password + phrase set already based on previous details
'No' to register another card
'Yes' to start the PCNS service

Note: you can start/stop/check status of thePCNS service in /etc/rc.d/init.d at any time such as /etc/rc.d/init.d/PowerChute start/status/stop

Can then load up http://esx-hostname:3052 in web browser and:

Configure Events - scroll down to UPS: On Battery
Click fourth column from the end
Check box for 'Yes, I want to shut down the system' and enter 60 seconds in box below
Select 'Configure Shutdown' from sidebar
Uncheck the box to 'Turn off the UPS after shutdown finishes'

Depending on what else you’ve tried to configure, these instructions might need tweaking to remove previous components, and the JRE might be picky. But, we pulled a UPS without the shutdown signal being sent, and then correctly initiated a shutdown when the second UPS became low on battery. Oh, and make sure you set your power management options in the BIOS correctly – as the UPS initiates a clean shutdown, when the power is restored, make sure the server is set to always power back on On our R805′s, ‘Last’ is great in the event the power just drops whilst the server is running, but since it was shut down cleanly, will not power on with the UPS back online, needs to be set to ‘Always’.