fouldsy.com

Even though I only have three weeks left working at LKSD, I’m determined to make the most of it. There’s a big project building up for the summer with a large scale Open Directory deployment going on district-wide, with 40 or so Xserves that have come in to the department along with three Promise RAID arrays, fibre channel switching and other network gear. I’ve taken the opportunity to put myself up for travel duty to install Xserves at our village sites.

So, on Monday I flew in + out of Kwethluk, Tuesday I hit up Toksook Bay and flew back to Bethel via Newtok, then Wednesday I flew down to Kwigillingok and then on to Kipnuk where I camped out overnight at the school before coming home on Thursday. After getting things squared in the office through Thursday afternoon and Friday, I then helped figure out next week’s installs So, this Monday I fly to Mekoryuk (where it wouldn’t be awful I got stuck overnight…) and then Tuesday I’m scheduled for Kasigluk to do installs at Akiuk and Akula, before then heading to Atmautluak on Wednesday. Phew!

It was really nice to get back to Nelson Island again and spend the day in Toksook Bay, and then getting down to Kwigillingok and Kipnuk on the coast was great after being out here 4 years and Tunt being fairly close to them. It was quite, quite different to what I expected. It was a lot more desolate than I thought it would be – pretty much every day direction was completely flat and devoid of shrubbery or elevation change. The climate was also totally different. Kwig had 4ft high snowdrifts alongside the boardwalks where they’d plowed the way, and Kipnuk had boardwalks impossible to walk on at times because of the snow! I rode a snowmachine back to the airport on Thursday and really got cold in howling winds, before then driving around in Bethel not two hours later with sunglasses on, window down on the truck, and no snow to be seen at all! I’m quite intrigued as to what Mekoryuk will look like, as Nelson Island was still covered in snow.

On Monday I implemented some routing and switching changes with the help of AT&T and our awesome network support guys from Integrated Logic. This finally let our iSCSI traffic for our Equallogic SAN units talk to each other between the district office and our off site location, Gladys Jung, across a 100Mb fiber line. To say I was a happy monkey upon seeing the LUN replications firing up is an understatement! Tomorrow I plan on bringing one of the LUN’s online, connecting to the DR VMware ESX host and seeing if I can actually bring the virtual machine online. At that stage, I will definately be beyond happy. We started planning our VMware infrastructure at the end of September, and has been a long project to get to the finish line, but totally worth it in terms of the set up we are now running, especially for the middle of no-where Alaska!

Today I bought the bullet and plumped for OS X and iLife updates, taking my MacBook from OS X 10.4 to 10.5 and iLife ’06 to ’09. Was literally 3 years ago last week I got the little guy, and I have to say, after a 2Gb RAM upgrade pretty much right away for under $80 and then a hard drive replacement, also around $80, after 18 months of abuse (literally – more than a dozen major airline flights, bush airlines, and 2 month road trip!), it’s been flawless and still powers everything I need without leaving me wanting. It’s probably the most impressive piece of computer equipment I’ve ever bought (I would say electronics in general, bit I do love my Canon EOS 50D!). Given I’ve seen hundreds of new machines passing through the office and workshops this past week for imaging and inventory, I know what the current crop of MacBooks run at, and for general day to use, this 3 year old baby is holding it’s own still!

The last couple of days have been busy with VMware and PowerSchool. We’ve had a consultant in working with us, and it was the perfect opportunity to migrate our student information system (SIS), PowerSchool, in to our VMware environment. I was planning on leaving PowerSchool as one of the our last physical servers to move in to the virtual environment, but given we had the experience on site to do it and I was confident in how VMware has been running, I thought we may as well give it a go. If nothing else, could simply roll back to the single physical server.

But, it all actually went fairly smoothly. I built up a template for Win 2k3 Enterprise with the base config and software, then deployed to 4 new virtual machines. One of these is running Oracle for the backend database, two are running the PowerSchool application node, with one being designated for general staff + parent logins, and the other for teachers and grading, and a final server dedicated to serving images, scripting and PHP reports. I have also snap-shot’d the database VM and one of the application nodes to be used for testing reports and in training sessions. Add in the new SIF ZIS which will be being deployed by the state over the summer, and that gives 7 virtual servers for PowerSchool, a far cry from the reliance and strain on a single physical server.

I’m really happy with how the migration went, as it really showed the power our VMware environment provides in terms of flexibility and resources. It also takes a huge weight off my shoulders, as we’ve never been able to successfully recover from a simulated failure using the backups due to the complexity of the integration between components, so with using straight vRanger Pro snapshots of the entire virtual machines, I can recover in minutes. I can also easily duplicate entire servers for testing updates, new releases (such as the upcoming PowerSchool 6), or for training purposes. Given PowerSchool is such a core system alongside FileMaker, both of which now run in our VMware environment, my management work load and stress levels should hopefully ease up considerably!

We still have a little work to do tomorrow – I’d like to automate a snapshot of the Oracle VM to a test VM that can be used by staff for building reports or whatever, though due to the way the database is tied in to the host IP, will need a little scripting. I’d also like to duplicate one of the application nodes and set it aside for testing the upgrade to PowerSchool 6. Is all positive stuff though, and giving me a lot of confidence in systems moving forward.

This is why I run a 30″ Apple Cinema display. Days like this make me think I either need another one, or another job:

18 Apple Remote Desktop connections plus a Microsoft Remote Desktop Connection I had to quickly work on one chain of our site servers. As Ted pointed out – a computer can multi-task, but can Iain? Will be glad when it’s the weekend and the schools have all broken up so it will quiet down for the summer.

There’s a lot of confusion of APC’s PCNS software for VMware. Basically, ignore the generic instructions provided on the CD This was tested on multiple ESX 3.5 Update 4 hosts and worked perfectly connected to a pair 2200′s. From what we could tell, the pay-for version is basically just PCNS 2.2.3 but with compatible VMware components. Nothing on the CD tells you this, and the instructions just add to the confusion. These setup steps figured out in conjunction with Kurt Bunker from GCS and posted here for reference.

Prior to configuring the PCNS software you need to configure the SNMP card with the following (note that the admin password and passphrase need to be the same on multiple APC units):

1. IP address
2. Administrative account password
3. Administrator Passphrase (minimum 15 char, Max 32)
4. Set APC SNMP settings for shutdown: UPS tab -> Configuration -> Shutdown -> Low Battery Duration

With the CD inserted, from the command line of your ESX host:

mount /dev/cdrom
cd /mnt/cdrom
./install.sh

During the install:

Instance '1'
Default install directory
Hit enter to automatically install JRE

Once install finished, open require firewall ports *before* running PCNSConfig.sh – you can only run the script once – even if you cancel out, it won’t run again unless you run the full uninstall script. If you’re unhappy about opening all these firewall rules, browse the forums to see exactly what they’re doing. We found they all needed to be open to function correctly.

esxcfg-firewall -o 80,tcp,out,"APC PowerChute Port 80"
esxcfg-firewall -o 2161,tcp,out,"APC PowerChute Port 2161"
esxcfg-firewall -o 2161,tcp,in,"APC PowerChute Port 2161"
esxcfg-firewall -o 2161,udp,out,"APC PowerChute Port 2161"
esxcfg-firewall -o 2161,udp,in,"APC PowerChute Port 2161"
esxcfg-firewall -o 3052,tcp,out,"APC PowerChute Port 3052"
esxcfg-firewall -o 3052,tcp,in,"APC PowerChute Port 3052"
esxcfg-firewall -o 3052,udp,out,"APC PowerChute Port 3052"
esxcfg-firewall -o 3052,udp,in,"APC PowerChute Port 3052"
esxcfg-firewall -o 6547,tcp,out,"APC PowerChute Port 6547"
esxcfg-firewall -o 6547,tcp,in,"APC PowerChute Port 6547"
esxcfg-firewall -o 6547,udp,out,"APC PowerChute Port 6547"
esxcfg-firewall -o 6547,udp,in,"APC PowerChute Port 6547"
esxcfg-firewall -o 6548,tcp,out,"APC PowerChute Port 6548"
esxcfg-firewall -o 6548,tcp,in,"APC PowerChute Port 6548"
esxcfg-firewall -o 6548,udp,out,"APC PowerChute Port 6548"
esxcfg-firewall -o 6548,udp,in,"APC PowerChute Port 6548"

With the ports open, configure the PCNS software:

cd /opt/APC/PowerChute/group1
./PCNSConfig.sh

During the configuration:

Select option '3' - Configure for multiple Smart-UPS devices
Enter IP address, port 80 (default), username, password, authentication passphrase
'Yes' to register settings
Enter IP of second card - username, password + phrase set already based on previous details
'No' to register another card
'Yes' to start the PCNS service

Note: you can start/stop/check status of thePCNS service in /etc/rc.d/init.d at any time such as /etc/rc.d/init.d/PowerChute start/status/stop

Can then load up http://esx-hostname:3052 in web browser and:

Configure Events - scroll down to UPS: On Battery
Click fourth column from the end
Check box for 'Yes, I want to shut down the system' and enter 60 seconds in box below
Select 'Configure Shutdown' from sidebar
Uncheck the box to 'Turn off the UPS after shutdown finishes'

Depending on what else you’ve tried to configure, these instructions might need tweaking to remove previous components, and the JRE might be picky. But, we pulled a UPS without the shutdown signal being sent, and then correctly initiated a shutdown when the second UPS became low on battery. Oh, and make sure you set your power management options in the BIOS correctly – as the UPS initiates a clean shutdown, when the power is restored, make sure the server is set to always power back on On our R805′s, ‘Last’ is great in the event the power just drops whilst the server is running, but since it was shut down cleanly, will not power on with the UPS back online, needs to be set to ‘Always’.

Last week was busy. I broke 60 hours by the time I got home around 7p.m Sunday. We had an engineer out with our VMware deployment which didn’t exactly go to plan due to networking issues outside of our control which was a little frustrating, but totally expected to be honest. The handling and resolution of those issues within the department weren’t ideal though, which at the end of a long week, didn’t help improve matters It took a lot of phone calls and soul searching to make it back in to the office this morning.

On the plus side, the VMware environment delivers everything it’s meant to – we tested power supply redundancy, NIC redundancy, controller redundancy and hard drive redundancy on all the servers and SAN units; set up APC PCNS software for a multi-master setup with dual UPS’; had replication between the two Equallogic PS5000E SAN’s working; got the core Cisco 3560G configured and running everything nicely; and tested HA and DRS functioned correctly and responded to failures as expected. We now have a 3-node ESX cluster of Dell R805′s running through Virtual Center on a EQL PS5000E, with another PS5000E as a replica and a single Dell 2970 as standalone ESX host for disaster recovery. Due to our networking issues, the replica EQL box and ESX host could not be put offsite as planned, so that will be phase two (by which point a second 2970 can be re-deployed giving us a pair of off-site servers in the event of a failure at the core). Having spent so long getting everything planned out and now up + running, it’s really quite sad to be crippled by a single 100Mb uplink for all LAN traffic to/from the virtual environment so can’t really use it for much (too complicated to explain the logic behind a $700 10/100 ProCurve at our backbone I wasn’t fully aware of…).

But, I finally made it to the chiropractor this afternoon, which was definately needed after last week After a checkup, x-rays and review of past medical records, was hooked up to a TENS system to relax the muscles along hot pads before a bit of a massage, before actually getting on to the chiropractic adjustment. After that, I then lay down on a weird setup with a rolling base to decompress the spine. All in all, just a little different to treatment back in England! Have another couple of sessions this week, and figure I may as well make the most of it with the medical insurance covering so much of it.

Although this morning started out with a nice tax refund being deposited in to my account, Friday 13th hit, and hit hard. Aside from the school district’s bank not handling direct deposits overnight and so ending up without being paid (still hasn’t posted to my account), technical gremlins plagued the day.

Around 9a.m I got a call that our streaming media server wasn’t providing content. Couldn’t connect via remote desktop, so went in to the server room to be greeted by three health warning lights across the RAID array. Diagnostics claim 3 out of 4 drives in the array had failed. The odds on that happening are pretty slim shall we say. This is the same server that was hit with errors last summer before I started with temperatures breaking 105F in the server room on a number of occasions. I’m thinking it’s more likely the controller has failed than three simultaneous drive failures. Regardless, without any spare parts or usable servers, I quickly brought online a Win 2k3 VM, moved 60Gb worth of content from the backups and brought it online. I could have been doing this on a new 3-node R805 cluster designed specifically for VMware, but am lacking support for a $10k Cisco 3750 switch given people are too used to expecting a $500 ProCurve to adequately perform as a core backbone. Right now I have $120k worth of gear doing nothing as a result. That’s another ongoing issue getting comical if it wasn’t so serious.

I spent quite a bit of time out the office this week in other sites seeing how their set ups are functioning, and honestly, the excuse of “it’s Bethel” just doesn’t wash anymore. No disrespect at all to Joe and Dean, but if a couple of guys at KYUK with ‘Networking for Dummies’ can knock 10 bells out of the district setup, there’s something clearly wrong. It’s not even complacency in some areas, just plain laziness and being hopelessly out in the deep end.

FirstClass continues to be the bane of my life, especially as it’s not even my responsibility. Once again it had issues today. After being yelled at for questioning the validity of restarting the entire server in the middle of the afternoon once again, I gave up. The idea that 25 years ago, ‘bits needed to drain out memory’ or even 5 years ago ‘log files would fill up and lock the whole system’ may have been valid, but on a server with 16Gb RAM that never breaks 3Gb in use and daily log files that don’t break half a meg, slow delivery of mail and loading inboxes is a different issue and it gets embarrassing being involved in it’s management sometimes.

I ended up leaving at 4p.m, will see if I get up on Monday morning and go back in. I’ve come to realise that work is not the be-all and end-all. If it brings you to tears on a Sunday evening at the thought of going back in on a Monday morning, I can happily walk away knowing I gave everything I could and with no regrets. The most frustrating part is that outside of work I’m actually very happy + stable, looking to the future, and simply enjoying life. To leave Bethel would be sad, but I’m learning to put myself first rather than carry on in a situation I’m not happy with.

My Canoscan 8800F finally made it through the mail system yesterday As I had quite a bit of studying to do last night for my VCP exam earlier today, I didn’t get much chance to play, and have then been out most of this evening with a guy in town from Phoenix Learning working on our student information system (PowerSchool) as he was wanting to bounce ideas for developing an iPhone app around.

So, without getting my hands dirty too much with regards to post-processing, here’s a few choice frames from the 35mm films I’ve developed over the last couple of weeks. First up, one of Audrey from Christmas Day back in England:

Not sure if this is a Kuskokwim 300 or Bogus Creek 150 musher pulling out the chute at the start of the race:

And a dog team getting ready for the Bogus Creek 150, with one guy obviously having a lot to say for himself:

I was quite impressed that a) I managed to develop film okay myself and b) once actually easily viewable were pretty decent. Not sure if it’s my photographic skills, the Olympus OM-1 having not been used in 10 years or cleaned in longer, or my developing skills (or lack thereof), but there was a distinct under-exposure and graininess on pretty much everything. Shooting ISO400 in black & white I would expect some grain, it could also be not reading that manual thing that came with the scanner Hopefully I’ll have some time at the weekend to really try things out, but there’s a birthday party on Saturday and then Superbowl party on Sunday, so that unfortunately means drinking may be involved…

After hanging around for 45 minutes this morning whilst the ‘test center’ got ready (read – a corner in the college library, was not impressed), I finally got to sit the VCP exam and passed with a score of 83 So, I’m now a VMware Certified Professional (VCP) in VMware Infrastructure 3 (VI3).

Since the school board gave approval for $110k+ worth of equipment to migrate our entire datacenter to a virtual environment including full off-site SAN replication along with a pair of redundant ESX hosts in the event of a major failure as a complete disaster recovery location (without SRM), it’s all coming together nicely and should makes things a whole lot easier come the summertime.

Since I was quite happy with how the server room reconfigure turned out, thought I’d grab a quick photo earlier on today:

Not a bad setup out of the DO now – 14 Win 2k3 servers, 8 Apple X-Serves, a VMWare ESX 3.5 box, the obligatory Linux server, a couple of 7.2Tb DAS fiber channel arrays, and an off-site 2Tb NAS for storing backups. Throw in 27-ish remote sites with at least one X-Serve in all of them except Platinum and that’s a good number of servers to be looking after Still seemed like I had too many meetings today, buggered up my plans to dodge out of work early.