Manage your Macs with a Munki!

If you are not familiar with Munki you need to go take a look right now. It is possibly one of the easiest software management tools I have ever seen. It works on the same principal as Apple’s own Software Update system. You build a catalog of packages that you want installed on computers and it makes sure that software is installed and kept up to date. What munki provides beyond that is the ability to have each computer have it’s on manifest of software. A manifest tells it what software to install and what catalogs to pull from (i.e. production, testing, development, etc.).

Recently I spent some time getting our setup ironed out and better designed, and here is what I came up with. I’m not going to go through a step-by-step process for everything I did as this is a more open-ended install setup. Managing computers like this requires a bit more knowledge of the product itself. So you really should take a look at some of the munki documentation, particularly the getting started guide, as that will give you some understanding of the terms I will be using.

Setting up the munki repository

First off, I built a VM to run all this in. It is a simple Fedora 18 install with no GUI.  You can use any *nix flavor you want, but I am most familiar with Fedora. The server is named munki and has a few DNS aliases pointing to it as well, which I will get to in a moment. First off I installed netatalk to provide a few AFP shares, but for this use I primarily use it for a single share: the munki repository. I need to be able to manage the repository from my desktop Mac as that is where the command line tools are installed. Sadly there are no tools for managing the munki repository on *nix, only Mac. Since *nix does not have (good) support for things like Apple’s .pkg format, or .dmg even, it would be more trouble than it is probably worth to try and go for a pure *nix setup.

Anyway, I created a /var/www/munki/html folder that will store the repository and shared it with netatalk to just our “staff admin” users. This folder is also shared by apache under the hostname munki.mydomain.com. Next mount the share on the Mac and initialize the repository (or in my case, copy my existing repository to this location). For this I use MunkiAdmin, which provides a nice GUI for working with the munki repository. Now with your repository created you can start building up your packages, catalogs, manifests, etc. I can’t help you with that specifically, but later on I will give you an idea of how we structure our manifests.

One thing to note with this is that I put the /var/www/munki/html folder as it’s own hard disk volume in the VM.  Currently my repository is 5.7GB, so to give myself some growing room I created the hard disk as an LVM of 15GB. Doing an LVM means I can increase the size of the hard disk without rebuilding everything.

Setting up your web interface

The same guy who wrote munki also wrote munkiwebadmin, which is a web interface to munki. Primarily it is for monitoring and seeing status information, but you can do some basic manifest editing. That is handy for moving something from one manifest to the other, but since you can’t upload new packages you will probably do most of your true admin stuff in MunkiAdmin. However, I found munkiwebadmin (yes I know, sorry for the confusing names here) to be most powerful in its reporting abilities. I followed the install instructions here with a few minor changes. I created my virtual environment in /var/www/munki, so the final path became /var/www/munki/munkiwebadmin_env. You will also be pointing your repository to /var/www/munki/html so it can find everything. In Apache I setup a new virtual host of munkiadmin.mydomain.com that will handle the WSGI stuff and access to the website and obviously also setup a DNS alias to point to the server.

For the install I used MySQL instead of the default SQLite. The reason for this is that I also installed phpMyAdmin so that I can get CSV exports of the inventory data from munkiwebadmin. We only keep computers for 5 years and then they get sold off, usually to staff at the going eBay price minus a little. Using this list lets me sort the computers by how old they are and then take the bottom 1/5 of the list as the computers to be replaced this year. I can do the same in an SQLite database, just not as easily since that requires logging in and running some command line commands and then copying files across the network.

I also had to make 2 small changes to the source code with you can find here and here. Hopefully by the time anybody works on this these two issues will be fixed. Basically those two changes make it so the manifest list does not include the files created by netatalk and also addresses an issue when using MySQL as your database.

Finally, I built a package that contains the 3 scripts from munkiwebadmin. For building custom packages I use an app called Packages, it is simple and quick. The reason I build a separate package (which I called munkiscripts version 1.0) is that it lets me easily go back and update just the scripts. For example if I decide that I want to have munki do some other pre/post-flight operations I can push out a new munkiscripts package with those updated scripts. In fact I have considered doing this to have the postflight script scan the console log for disk I/O error messages and notify me and potential hard drive failures. Anyway, all I have to do is bump the version number and import it into munki and munki will install the new scripts on the next run. So yes, you will want to add this package to your repository.

Setting up your own SUS server

The last thing I wanted to do was have this server manage all software updates. Currently I have a Mac server doing this, and wasting a ton of disk space at the same time. There is an “application” called reposado that is basically a few linux scripts that handle downloading software updates from Apple and mirroring them locally on your server. I basically followed the getting started instructions. I created a new set of folders for reposado in /var/www/reposado/html and /var/www/reposado/metadata and pointed the configure command at those 2 folders.

I also created a 3rd hard drive in the VM and mounted it in /var/www/reposado. While the munki repository you could have as part of your room VM disk, I recommend the SUS volume be a separate disk for one very specific reason.  It’s big and can be downloaded again. At the time of writing this my entire SUS repository is 100G and will only get bigger as time moves on. The disk that backs this volume is set to not be backed up in VMware. If I lose it, I just run the sync command again.

So anyway back to reposado, basically follow the instructions and create an apache share for /var/www/reposado/html. In my case I created yet another virtual host of reposado.mydomain.com to handle these requests and setup the DNS alias. I tend to do things this way as it makes it easier to migrate things from one server to another. For example if I later decide to move reposado off the munki server I just have to update the DNS entry, not every client. Once reposado is setup, run the sync tool and go get a coffee, or lunch, or if you have a slow internet connection maybe a weekend. Like I said, 100GB.

Configuring Clients

We have all this great stuff happening on our server, but nothing is talking to our server just yet.  Three things need to happen to get all your clients integrated happily.

First we need to get munki installed on each client workstation. The software needs to be installed (which is pretty straight forward) and it then needs to be configured to point to your repository. You can do the latter any way you want. You can run a terminal command to set the properties, managed preferences, whatever. I use managed preferences so that I can easily change settings per-client. Munki install docs talk about various ways to do this. Your repository URL will be something like http://munki.mydomain.com.

Second we need to run munki so that it downloads all initial software, including the scripts. This will either happen after about an hour or you can use a terminal command to have it check now. I recommend doing a “check now” on a few clients just to be sure everything is working the way you expect, that way you are not waiting for hours to find out something was configured wrong.

Finally you need to update the software update catalog URL used by Apple. Reposado has some examples of URLs you will be using, yours will be something along the line of http://reposado.mydomain.com/content/catalogs/others/index-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog for Mountain Lion (others will be similar). I use a modified script that runs as a login-hook I found here. Scroll down a few comments to one by the user feyd, that is the one you want (at-least as a starting point).

Reboot just to make sure everything is picked up. Wait a little bit and you should start seeing data populate in munkiwebadmin. Also you can verify that your clients are pulling from your SUS because the Software Update app (or App Store) will tell you where it got the update from, so it should have your reposado URL listed.

Example manifest structure

So the way munki handles everything is by manifest. You can either set a specific manifest or let munki use a default one (computer name, serial number, “site-default”, etc.). But each manifest references packages to be installed/updated/remove as well as catalogs to pull those packages from. It can take some work to get things sorted out in an order that works well. I can’t say what I have is best but it works for me.

First, the reason I ended up with this layout. We have basically 4 different computer groups. Realistically 95% of our workstations are in 2 of those groups. We have a standard install group that every computer gets (this would include computers that are head-less, or otherwise non-user centric). Then we have a staff group that is basically office computers which have people sitting at them. On top of that we have 2 speciality groups: Communications/Graphics/Video Department and Teaching Pastors. Each of those last 2 exist for one very specific reason. The first gets the latest version of the Adobe suite while the others get an older version (saves us some money doing it this way). The latter group get a different set of installed modules for some Bible software we use, so the staff gets the packaged version and those 4 computers (currently) get a manually installed version.

Now, a quick bit of understanding for how manifests work. A manifest references catalogs to determine which version of the package to install. So you might have a production catalog and a testing catalog. You put the upgrade in the testing, roll it out to your “early adopters” group and then when you are satisfied there are no problems move it to your production catalog. A manifest can also include other manifests, however, if an included manifest references any catalogs those catalogs will be search first. So for example, my early adopters manifest is called infotech and everybody else uses staff. The infotech manifest includes staff. infotech also references the testing catalog. If the staff manifest references the production catalog then the production catalog will always be checked first, even if the software is mentioned directly in the infotech catalog.

What I have adopted to resolve this is a “split manifest” system. So these are the manifests I currently have:

• standard_software – Lists all the software, but no catalogs.
• standard – Includes standard_software and references the production catalog.
• office_software – Lists all the software for office computers, but no catalogs.
• staff – Includes the office_software and standard_software manifests, references production catalog.
• infotech – Includes office_software and standard_software manifests, references testing and production catalogs.
• graphics – Includes office_software and standard_software manifests, references graphics and production catalogs. (graphics catalog has the newer version of the Adobe suite)
• teaching – Includes office_software and standard_software manifests, references teaching and production catalogs. (teaching catalog has the different version of the Bible software)

Using managed preferences, it is fairly easy and straightforward to assign computers to various manifests and change them back and forth, though bear in mind munki may get confused by changing people’s manifests dramatically. i.e. if I install a computer as a teaching computer and then switch it to a staff computer, it will stick with the advanced version of the Bible software unless I manually remove it and then munki will re-install the correct version.

So basically ,when I say “split manifest” above what I mean is one manifest to hold the package names and another manifest that the computers point to. So all the *_software manifests are what contain the software, but the computers point to standard, staff, infotech, and graphics. That allows me to easily create a graphics_testing manifest that one computer points to which includes the same child manifests but references the testing catalog first so it gets the updated version.

Apple Software Updates

One other thing munki can do is install Apple Software Updates as well. This is useful if your users do not have admin accounts on the computer, and thus are unable to install the updates themselves. I have not yet rolled this out. It seems like it is not 100% stable from watching the discussion mailing lists. I do not mean stuff is broken and computers break, just that the update process is based upon a “apple does their own thing, we think we have figured it out” principal. So occasionally Apple releases an update that behaves strangely because it doesn’t fit into the mold of “we have figured it out” just yet and the mold has to be slightly modified. I think the worse I have seen is updates that keep showing up over and over again.

There is lots of documentation for how to turn this on (and off) so it may be worth your attention.  I will be looking at it again once I have more time, but for us it is not a critical issue since most of our users have admin access on their computers so they install their own updates.  As we move forward and (possibly) take that away this will be a bigger issue so we will look into it more.

Migrate Snow Leopard Server to Lion or Mountain Lion

I have a Snow Leopard server running, obviously, 10.6.8.  It does too many things.  It serves AFP, OD, iCal, AddressBook, Wiki, RADIUS, Software Updates, and a few other things.  That is just too much stuff on one server that goes down every time I do a little maintenance. What I wanted to do is setup a new Mac Mini server on Mountain Lion and migrate the collaboration services (iCal, AddressBook and Wiki) from the old server to the new one. Easier said than done. It seems all the public documentation is for how to migrate the entire server, not piecemeal.

I did some digging and found some useful scripts in the Server.app.  Fire up Terminal and take a look in the /Applications/Server.app/Contents/ServerRoot/System/Library/ServerSetup/MigrationExtras folder. There are various Python, Perl and Ruby scripts to migrate various service information.  A few of the ones you might be interested in:

• FTP
• VPN
• Web Config
• Wiki
• Mail
• Calendar (also does Address Book/Contacts)
• RADIUS

I will give an example of what I did. I found simple instructions on Apple’s website for how to migrate the Wiki information from 10.6 to 10.7/10.8. I also needed to migrate the Calendar/Contacts information.  I did this as a dry run to test and make sure these steps would work so I can test the new server and then I will setup the server clean again and do the final process and go live.

First I mounted the time machine volume for the old Snow Leopard server.  From the command line I ran the following:

• cd /Applications/Server.app/Contents/ServerRoot/System/Library/ServerSetup/MigrationExtras
• sudo ./70_calendarmigrator.py –sourceRoot /Volumes/Time\ Machine\ Backups/Backups.backupdb/augustine/Latest/Augustine\ HD/ –sourceVersion “10.6.8″
• sudo chmod -R -a# 0 /Library/Server/Calendar\ and\ Contacts/Data

This ran for about 5 minutes before it finished. You should probably make sure your Calendar and Contacts services are not running, I realized Calendar was running half way through and turned it off. Luckily everything still seemed to work. When the process finished it starts up the Calendar and Contacts services. There were a few things more I had to do to get everything working properly. So I will give an overview of the whole process.

1. Use the 70_calendarmigrator.py script to migrate the settings and data.
2. Use the Server app to select the appropriate Certificate for the Calendar service (it was set to none which caused proxy errors).
3. Use the Server app to give users (or groups) permissions to use the Calendar and Contacts services (by default nobody has access even though they already have calendars).
4. Restart the entire server. Technically you can just restart all the various services, but after a migration like this I would just restart the server to make sure all is well.
5. Update DNS to point to the new server. (For testing I edited the /etc/hosts file of a client to simulate the DNS update and everything just worked)

Finally, a few caveats I ran into trying to get things to work:

1. I ran into an issue with an “CalDAVAccountRefreshQueueableOperation error 500″ in iCal client and “Data Corruption Detected” in the caldav error.log. Once I checked the error.log I found that there was a Geo-location based reminder on my calendar (actually 4) that was causing issues for some reason. I’m not sure why. I had to use the psql command line tool to log into the PostgreSQL database and delete the offending entries from calendar_object table.
2. My user works fine in iCal but has an issue in the WebCal interface. I have 2 calendars, but the WebCal interface shows duplicates on the sidebar, though it does not duplicate the actual calendar data. Furthermore, when I try to edit the calendar it says I don’t have permission. This may be related to the issue I ran into above or it may not. Because this was a test server I believe I had already logged into the WebCal client before I imported the data so I may have really fouled things up on my own username. I checked a few other users and none of them have any problems. I was able to delete one of the “ghost” calendars, and I can create events on the real calendar of that pair, but the other 2 calendars (one real, one ghost) neither one shows that I can delete and I’m a little. I tried a full server restart but that did not fix. Again, I think this is just because I used a previously used system before instead of a clean system.

Overall the process was super easy and only took me about 2 hours to figure out and, transfer the data, fix a few issues and be back up on running on the sandbox server. The key to the whole process was finding those migration scripts so I could migrate individual service data.

Update 1/2/2013

Here are some tips for getting things fixed if you run into similar Calendar issues.  Just for the record, only my personal user account had issues with the Geo-location TODO items. Other users have Geo-location TODO items and they seem to work fine; what the difference is I don’t know. Anyway some tips on how to get into PostgreSQL to fix this.  First, to actually login to PostgreSQL run this command:

psql -h /Library/Server/PostgreSQL\ For\ Server\ Services/Socket -U _postgres -d caldav

To find the offending records (in my case, they all had the street name of “Jenkins” so I was able to search on that):

select resource_id from calendar_object where icalendar_text like ‘%Jenkins%’;

You should see it spit back a handful of records. If you see a large number of rows returned, you may want to try and fine-tune your query a bit. Once you have the query down, change the “select resource_id” to “delete” like this to delete those same rows:

delete from calendar_object where icalendar_text like ‘%Jenkins%’;

Once this is done I recommend restarting the Calendar service just to be safe.

Disk speed testing the new Retina MBP and Thunderbolt

My friend just received his new MBP Retina.  Because my older MBP also has an (aftermarket) SSD in it, we decided to do some speed test comparisons. The 2011 MBP has a Samsung SSD, but it is the older model. The current model is supposed to have about a 30-50% speed boost over the model I have, however I do not have a new one to test with. We used Kona’s AJA System Test to run these disk tests as it provides a really nice interface to do these tests with.

The first set of tests was just running against each laptop’s internal SSD drive.

As you can see the newest model outperformed the older model, which is to be expected. But the new model seriously pulled 304MB/s and 301MB/s read and write speeds respectively. The older MBP with the Samsung ran at 212MB/s and 182MB/s respectively. So not only did the new drive run faster, but it was able to read and write at the same speed while the Samsung wrote slightly slower.

The second test consisted of booting each MBP in turn into target disk mode and using a Thunderbolt cable to run the speed tests. So the Late 2012 MBP Retina results are with the new MBP in target disk mode with the test executing on the older MBP. The Early 2011 MBP results are with the older MBP in target mode and the test executing on the Retina MBP.

Again the newer and (presumedly) better drive performed better. One odd thing to note, and we ran this test several times to make sure, is we got better write speeds (224MB/s) than we did read speeds (178MB/s) over thunderbolt when the Retina MBP was in target mode. While not stellar performance, the early 2011 MBP drive still achieved a decent 149MB/s and 104MB/s read and write speeds respectively.

The obvious question is why the huge difference in speeds from internal to Thunderbolt, and the answer would be to remember that unlike a storage array, the MBP is doing a software-based interface to provide the hard drive over Thunderbolt (or firewire). This means there is extra overhead. Apple didn’t design their target disk mode for speed, but for (in my opinion) “oh shoot, I fouled up my computer and can’t boot it anymore but I need my data”. It would also be interesting to see a speed test run over Thunderbolt with 2 brand new Retina MBPs. We might see closer speed to the internal read/write speeds.

One thing to remember is that these speeds are not too shabby, even the slowest speeds of 149/104MB/s is still pretty dang fast for a consumer device. We are not talking about high-end RAID controllers here, just booting an off-the-shelf MBP in target disk mode and pretending it is an external hard drive. Also noteworthy is the fact that we ran the tests executing on the Retina MBP both with and without a Display Port device hooked up and noticed no difference in either the speed test or the responsiveness of the display.

Finally, even at 300MB/s, remember that this Thunderbolt test comes out to only 1.79Gbps, or roughly 18% of the possible speed of the Thunderbolt bus. So if you were to buy a “thunderbolt docking station” (such as this, this, or this), utilizing the Gigabit network, USB 3 interface, audio and eSata, you are still only hitting (at peak) 9Gbps, and you are getting 300MB/s to that eSata drive. I really hope we continue to see more of these docking stations because they will make things like the Macbook Air a much more interesting option for people like me. I don’t often need all those extra connections, but with how small the Belkin one is, for example, I can throw it and a short Thunderbolt cable into my bag and carry an entire army of connectors in that one little box.

Oh, the conclusion? The new Retina MBP not only looks slick, has a great display, but it is FAST.

Are your Time Machine backups really working?

Apple built a great system into Time Machine to tell the user if their machine is not backing up.  First is the menubar icon. It turns into an exclamation mark if the last backup failed and clicking on it gives the user quick access to know what the problem is (in a very generic way). After 10 days of backups not running it actually pops up an alert window that tells you your machine has not been backed up in 10 days. This works great for me and my personal laptop.  It has my data on it so I pay attention to if it is backed up or not. But when dealing with a corporate environment, users aren’t as concerned about their data. That is until they accidentally deleted something and need to get it back. Then they just expect the backups have been working and want to know why you were not doing your job.

In an environment where I need to maintain over 125 computers, about 50 of which run Time Machine, whose screens I never see, I need a better way to monitor whats going on with Time Machine. If you are using Time Machine in a corporate environment you probably (hopefully) are using a central server that all your machines backup to. This makes the job of monitoring these backups easier since they are all in one place. We wrote a shell script that will go through the backup directory and mount each sparsebundle in order to examine the contents and determine how many backups, how big, and when the most recent backup was performed for each sparsebundle and then e-mail the results. It also includes any warnings or errors at the top, such as “so and so has not been backed up for 14 days.”.

In order to allow the script to run we setup a cron-job as root user and use the MAILTO= statement in the cron file to indicate who gets the e-mail (you can specify multiple addresses with “,”). The main script, checkTMBackups.sh, does all the leg work of mounting the various disk images and parsing the information in them. It includes a script called functions.sh which provides some special logging functionality. I did this because I needed a way to log “normal” messages and “critical” messages at the same time, but then output the critical messages all grouped together before any of the normal messages. This makes it easier to spot (potential) problems.

checkTMBackups.sh

#!/bin/sh
#

#
# Source the helper functions in.
#
source "/usr/local/bin/functions.sh"
SetLogPrefix "Time Machine"

#
# Change these values to suite your needs.
#
BackupTMAgeAlert="14"
TMPath="/Volumes/HDCBackups"

#
# You shouldn't need to edit anything below this line.
#
ATTACH_PARMS="-readonly -noverify -noautofsck -noautoopen -quiet"
PATH="$PATH:/usr/sbin:/sbin"

#
# Check the volume with the given name in $1
#
function CheckVolume
{
	#
	# Check if volume is in use, try for 5 minutes.
	#
	i=0
	while [ $i -lt 10 ]; do
		LSOF=`lsof | grep "$1/bands"`
		if [ -z "$LSOF" ]; then break; fi
		sleep 30
		i=$[$i + 1]
	done
	if [ $i -eq 10 ]; then
		LogAlert "$BackupName: Volume in use, cannot mount."
		return 1
	fi

	#
	# Get short name
	#
	BackupName=`echo "$1" | cut -f1 -d. | cut -f1 -d_`

	#
	# Try to mount the volume quietly.
	#
	mkdir -p /tmp/mount
	hdiutil attach -mountpoint /tmp/mount $ATTACH_PARMS "$TMPath/$1"
	RESULT="$?"
	if [ $RESULT != 0 ]; then
		LogAlert "$BackupName: Could not mount volume ($RESULT)."
		rmdir /tmp/mount
		return 1
	fi

	#
	# Check if the backup has finished one cycle yet.
	#
	VolumeName=`ls -1 /tmp/mount/Backups.backupdb | grep -v "^\." | head -n1`
	if [ ! -e "/tmp/mount/Backups.backupdb/$VolumeName/Latest" ]; then
		ls "/tmp/mount/Backups.backupdb/$VolumeName"
		LogAlert "$BackupName: Has not finished full backup cycle yet."
		hdiutil detach -quiet /tmp/mount
		rmdir /tmp/mount
		return 1
	fi

	#
	# Get the name of the first drive backed up and then check
	# when it was last backed up.
	#
	LastMod=`stat -L -f "%m" "/tmp/mount/Backups.backupdb/$VolumeName/Latest"`
	CurDate=`date "+%s"`
	DaysSinceBackup=$[$[$CurDate / 86400] - $[$LastMod / 86400]]

	#
	# Check when it was first backed up.
	#
	FirstName=`ls -1 "/tmp/mount/Backups.backupdb/$VolumeName" | grep -v "^\." | head -n1`
	LastMod=`stat -L -f "%m" "/tmp/mount/Backups.backupdb/$VolumeName/$FirstName"`
	DaysSinceFirstBackup=$[$[$CurDate / 86400] - $[$LastMod / 86400]]

	#
	# Get the number of backups that are around.
	#
	NumberOfBackups=`ls -1 "/tmp/mount/Backups.backupdb/$VolumeName" | grep -v inProgress | grep -cv Latest`

	#
	# Determine space used and total.
	#
	SizeAllowed=`df -H | grep /tmp/mount | awk '{print $2}'`
	SizeOfBackup=`df -H | grep /tmp/mount | awk '{print $3}'`
	BackupUsed=$[$[`df | grep /tmp/mount | awk '{print $3}'` * 100] / `df | grep /tmp/mount | awk '{print $2}'`]

	#
	# Unmount and detach the image.
	#
	hdiutil detach -quiet /tmp/mount
	rmdir /tmp/mount

	#
	# Log the information about this backup.
	#
	if [ $DaysSinceBackup -gt $BackupTMAgeAlert ]; then
		LogAlert "$BackupName has not been backed up in $DaysSinceBackup days."
	fi
	LogMessage "$BackupName has $NumberOfBackups backups. $SizeOfBackup of $SizeAllowed (${BackupUsed}%). First/last backup was $DaysSinceFirstBackup/$DaysSinceBackup days ago."

	return 0;
}

#
# Stop server admin and check all volumes.
#
for f in "$TMPath"/*.sparsebundle; do
	backup=${f:$[${#TMPath} + 1]}
	CheckVolume "$backup"
done

#
# Determine total usage for time machine.
#
SizeAvail=`df -H | grep "$TMPath" | awk '{print $4}'`
BackupUsed=`du -skch "$TMPath"/*.sparsebundle | tail -n1 | awk '{print $1}'`
LogMessage "Total backup space used for time machine $BackupUsed ($SizeAvail available)."

DumpAlertLog
DumpLog

functions.sh

#!/bin/sh
#
# This file provides common functions I use. I make no guarentees that
# any of it will work.
#
# Copyright (c) 2010 Daniel Hazelbaker
#
# Version 1.0 - 2010/02/19
#

######################################################################
#
# Functions to provide logging information.
#
######################################################################

Log_Messages=""
Log_Alerts=""
Log_Prefix=""
Log_AlertPrefix="*** CRITICAL -"

#
# Set the prefix used when logging messages.
#
function SetLogPrefix
{
	Log_Prefix="$1"
}

#
# Set the prefix used when logging alerts.
#
function SetLogAlertPrefix
{
	Log_AlertPrefix="$1"
}

#
# Log a simple message.
#
function LogMessage
{
	local msg

	if [ -n "$Log_Prefix" ]; then
		msg="["`date "+%F %T"`" $Log_Prefix] $1"
	else
		msg="["`date "+%F %T"`"] $1"
	fi

	LogMessageRaw "$msg"
}
function LogMessageRaw
{
	if [ -n "$1" ]; then
	        if [ "$Log_Messages"X == "X" ]; then
		        Log_Messages="$msg"
		else
			Log_Messages="$Log_Messages
$msg"
		fi
	fi
}

#
# Display the log messages
#
function DumpLog
{
	if [ "$Log_Messages"X != "X" ]; then
		echo "$Log_Messages"
		echo ""
	fi
}

#
# Log a critical alert
#
function LogAlert
{
	local msg=""

	if [ -n "$Log_Prefix" ]; then
		msg="["`date "+%F %T"`" $Log_Prefix]"
	else
		msg="["`date "+%F %T"`"]"
	fi
	if [ -n "$Log_AlertPrefix" ]; then
		msg="$msg $Log_AlertPrefix"
	fi
	msg="$msg $1"

	LogAlertRaw "$msg"
}
function LogAlertRaw
{
	if [ -n "$1" ]; then
		if [ "$Log_Alerts"X == "X" ]; then
			Log_Alerts=$msg
		else
			Log_Alerts="$Log_Alerts
$msg"
		fi
	fi
}

#
# Display the alert messages
#
function DumpAlertLog
{
	if [ "$Log_Alerts"X != "X" ]; then
		echo "$Log_Alerts" >&2
		echo "" >&2
	fi
}

Cron job file

MAILTO=”daniel@mailinator.com,rharman@mailinator.com”
0 1 * * Sun,Wed /usr/local/bin/checkTMBackups.sh 2>&1

The above cron job will send e-mails to both daniel and rharman each time the script runs. An e-mail will be sent wether or not any problems were detected. The script runs at 1am every Sunday and Wednesday.  Both scripts should be installed in /usr/local/bin folder (you may need to create this path on your system). To edit root’s cronjob list you can use the command

EDITOR=nano sudo crontab -e

The “EDITOR=nano” part tells it to use the editor called nano, otherwise you will be stuck with vi which is a pain if you are not used to it. sudo tells it to run as root (it will ask for your login password) and crontab -e instructs it to edit the users crontab. Note: crontabs work on Snow Leopard for sure. I can’t say for sure if they work on Lion or Mountain Lion yet. If they do not you will have to use launchd to configure, but hopefully they just work.

You will occasionally (or if you have a lot of users as I do, one or two of them each run will come up with the warning) get warnings about not being able to mount the sparsebundle. This message can occur for two reasons. The first is if the sparsebundle is in use by a client (i.e. the backup is happening right now). Most of our machines get left on overnight so this isn’t uncommon. The script will actually try to work through this problem by re-trying 10 times at 30 second intervals. The second cause for this message is if the sparsebundle needs a file system check. Because we mount the FS read-only it will abort if the file system is not clean because it is not allowed to fix the problems. These messages I usually ignore unless I notice them 2 or three e-mails in a row and then I follow up and look into it.

Download Scripts

Moving Apple’s PasswordServer to Linux

So awhile ago I started working on a way to move our Open Directory database off of Mac and onto a Linux box running OpenLDAP. But I wanted to find a way to do it without losing any functionality. So far it’s going pretty good. I have another post I’m working on that details all the steps to accomplish this. I’m kind of waiting until I make the move to post everything. One big hiccup I ran into is that without Apple’s PasswordServer most applications only support cleartext passwords, which is kind of a pain. Especially since some Apple applications (such as iChat) complain if you are using cleartext passwords, even if it is over an SSL link. So I wanted to find a way to provide the advanced password services that Apple provides.

Thus was born the Linux Password Server (or lpws). What I discovered is that Apple’s PasswordServer is simply a bridge between a client computer that wants to do SASL authentication and a centralized server that has access to the cleartext passwords.  The protocol itself is simple text with the SASL data encoded in hex strings. While I don’t have everything working yet, all the authentication seems to work just fine. I have been able to generate an RSA private/public key pair which it uses to verify the authenticity of the host the client is talking to. I can store that public key in the authAuthority record of a user and add the server to the Password Server list in the LDAP database. From then on any requests to authenticate the user will work just as if you were still using Apple’s own services. All the authentication schemes seem to work just fine.

Since the password server just uses SASL to perform the actual authentication there was actually very little code to write for the server itself. I had to write the usual “client tracking” code and socket code you would need to provide basic TCP services and then implement a command processor to deal with the different commands that can be sent by the client. After that it was just pass along the data sent from the client into the SASL library. There were two big hurdles I ran into to get to the point I’m at now.

The first was the WEBDAV-DIGEST mechanism. This mechanism doesn’t exist on Linux, or anywhere else that I could find.  After some time deciphering a bit of the protocol I found that it was basically just the DIGEST-MD5 mechanism with a slight twist. Instead of the authentication server generating the nonce and all that stuff, the “intermediate” server (i.e. iChat, Address Book, mail, etc.) generates the nonce and gives it to the authentication server. I suppose there is a slight security consideration with doing that as it is kind of close to a replay attack, but from my understanding of DIGEST-MD5 there are still enough security measures in place. So basically I had to duplicate a modify a few lines of code in the DIGEST-MD5 mechanism and rename it to WEBDAV-DIGEST.

The second hurdle was DHX.  Nearly everything uses DIGEST-MD5 or WEBDAV-DIGEST to authenticate the user.  I found only two cases where DHX is used.  The first is when logging into a network account at the login window. Oddly enough, if DHX failed then it would fall back to DIGEST-MD5.  This caused a significant delay in the login process so it was unacceptable to me. The second place was when changing a password via the “Connect to server” window (and possibly via System Preferences, I didn’t try that yet). When changing a password the client first authenticates you via DHX and then issues a CHANGEPASS command to set the new password. Why they chose to use a different mechanism just to authenticate you before changing your password I don’t know. But again, there did not exist an SASL DHX mechanism that I could use. So I had to write one. That took awhile to figure out but I finally managed to get it working after a few days of pouring over various resources online.

So what works now?  Well you can authenticate users via the Linux Password Server. The services uses SASL so as long as SASL works on the server then it can authenticate. For my use, I wrote a tiny auxprop plugin that loads the cleartext password from LDAP and gives it to SASL to authenticate the user with. This all assumes you are using OpenLDAP and storing cleartext passwords (in as safe a way as possible). In theory the passwords could be stored somewhere else like Apple does, but this would require some customization to OpenLDAP so that when a user tries to do basic authentication they will be authenticated via the password server instead of the userPassword property. I didn’t really care for that option.

What doesn’t work? The CHANGEPASS command I mentioned above is not yet implemented but I don’t expect it to be any harder than the DHX was. I would expect it to be a fair amount easier actually. The only other command I have seen so far in all my testing is a GETPOLICY command during loginwindow logins. Apple returns a space separated list of abilities that the user has (for example, isAdmin=1/0). So this should be pretty easy to simulate by just returning a blank list or something. Right now it gives back an error response and the login proceeds normally. I think normally the loginwindow would check to see if the policy allows local logins, needs to change password, etc. There are also about 30+ more commands buried in Apple’s implementation that I haven’t even seen used yet. But those may not actually ever be used in the setup I will be running so we’ll have to wait and see.

If you are interested in taking a look at whats going on head over to Google Code and take a look at the project: http://code.google.com/p/lpws/

If you would like to help out just drop a note to me over there in the project. I’d be happy to have any extra help, especially with some of the documentation. Otherwise feel free to download the code and take it for a spin.

You might, by the way, be wondering why I would ever want to do this anyway. The answer is simple. Apple’s tools work great if you play by their rules. If you don’t want to customize anything or do anything a little different you are all good. As soon as you start making custom changes (like installing your own SSL certificate so it’s actually valid instead of a self-signed one) or modifying the way the Mail service filters spam; that is when you run into trouble. I don’t play by their rules 100% of the time. I can’t. They work great if you are a company with maybe a dozen employees but I have to deal with over 60 user accounts and about 150 client machines.

The other reason I want to do this is redundancy. I want to put all this authentication stuff on a VM, but I can’t do that with Apple. In order to virtualize I have to run it on Apple hardware. Only problem with that is they got rid of their server hardware.  The Mac Mini is great as a client device but a joke as server.  One ethernet? One power supply? Yea right. The Mac Pro is not much better. I gain a secondary ethernet port but still have a single power supply. But even if I did go with a Mac Pro, to get proper redundancy I would need at-least 2 Mac Pros in the rack (yea there goes 12U of space) and to buy a separate copy of VMWare Essentials Plus (at another $1,500 thank you). So now I’m sitting at $7,500 just to run a single server? no thanks. I’d rather do it on Linux where it’s free and works with everything I’ve already got installed. Does that mean I will still continue down this path if I can’t get the Mac clients to integrate nicely with Linux server? nope. That is still #1 priority, I need to have easy management. Time will tell.

Ongoing data-loss with Lion

So I got an official response from the bug reporting team over at Apple about the Lion data-loss bug with non-local volumes (see here and here).

There response was short and extremely unhelpful.

Currently, if want to close and undo any changes, you must hit Cancel then Revert to Last Opened/Saved, or use Versions to revert. Then, when you want to close again, you should just hit “Close”.

I sent them a very frustrated response.

I’m sorry, but I can’t consider this to be anything other than a Microsoft level answer (which is not a compliment, by the way). We have 80 users who all work on a central Mac OS X File Server. And I am supposed to tell them that in order to work with ANY file on the server they have to remember to “hit Cancel then Revert to Last Opened/Saved and then Close again” a few hundred times a day as they work with their files on the server?

And while I’m telling them this I am suppose to continue to maintain that Apple is a better product than Windows, despite the fact that an Apple client OS cannot work with files on an Apple server OS without erasing their content?

Take 10 people who, for the past 20 years, have had the ability to close a document and click “Do not Save”. Don’t give them any instructions on how to do things “the new way” and see how many of them manage to overwrite their original content. The user interface is very confusing and a very poor design to not provide users with a way, visually, to know what to do. They have the option to “close and overwrite” or “cancel and leave open”. That is all they will see and all they will understand.

I am extremely disappointed with this kind of answer from Apple to a severe data-loss bug.

Sadly, at this point, if this is their official response I don’t expect to see this ever resolved. I fully expect Apple’s official stance to be, “That is the way it is now so get used to it.” They have made those kinds of changes before where people don’t like them (including me at times), but you get used to it because you don’t have a choice. However, none of those have resulted in a data loss. Changing the way iCal, Mail, Finder, Spotlight, etc. look does not cause you to lose data if you click the wrong (and confusing) button. This change does.

Automatically forget wireless networks

Our wireless network uses WPA Enterprise, which requires users to enter both their username and their password.  We do this for two reasons. First we like the extra security.  Second, we like to be able to track down who was doing what on the network if an issue ever comes up.

We also have a number of “general use” laptops that people can use for various purposes, powerpoint presentations, weekend services, working away from their desk, etc. As such, on these laptops we don’t want the system to remember and automatically reconnect to the wireless network with the previous users identity credentials.  Here is the documentation for what we did to make this possible.

First, you will want to go into the System Preferences, Network, and go to the Advanced settings on the AirPort adapter. The Remember networks this computer has joined should be turned off. You know that little checkbox that asks if you want to remember this network that is checked by default? This makes it unchecked by default. Not perfect, but much more helpful. Second the Disconnect when logging out should be turned on. This will, as it says, disconnect from the airport when the user logs out. Just a helpful cleanup.

Next we use a LogoutHook script to reset a few things on the laptop. There are actually 2 scripts, but one just calls the other. I decided to create a single LogoutHook.sh script that calls all the other logout hook scripts, in case I decide to add more in the future. You want both these scripts owned by root, the easiest way to do this is via Terminal and use sudo nano filename to edit the file.

You will want to create the following script in /usr/local/bin/LogoutHook.sh

#!/bin/sh
#

/usr/bin/logger -i "Running Logout Hooks..."

/usr/local/bin/airport-cleanup.sh

/usr/bin/logger -i "Finished with Logout Hooks..."

You will then need to mark the script as executable: sudo chmod a+x /usr/local/bin/LogoutHook.sh

Next create the cleanup script in /usr/local/bin/airport-cleanup.sh

#!/bin/sh
#
# This script is designed to be run as root via the LogoutHook system.
# It's purpose is to ensure that the airport system has been disconnected
# and all EAP entries have been removed. It also restores the keychain
# to a vanilla keychain if one exists.
#

AIRPORT="/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport"

#
# Disconnect from the airport network and remove all preferred networks.
#
$AIRPORT -z
/usr/sbin/networksetup -removeallpreferredwirelessnetworks AirPort

#
# Delete any EAP 802.1x profiles.
#
rm -f /Users/*/Library/Preferences/com.apple.eap.profiles.plist
rm -f /Users/*/Library/Preferences/ByHost/com.apple.eap.bindings.*

Again, mark the script as executable: sudo chmod a+x /usr/local/bin/airport-cleanup.sh

Finally, you need to register the LogoutHook script in the system:
sudo defaults write com.apple.loginwindow LogoutHook “/usr/local/bin/LogoutHook.sh”

Now, every time a user logs out it should “de-configure” the airport settings.  If somebody manually checks the Remember this network checkbox when connecting to the wireless network, it will save the password in their keychain.  This will not cause an automatic reconnect or anything, it just leaves it around. So far this script setup has worked well for us.

Apple TV Plays Digital Audio Out of Sync

So I updated my Apple TV to iOS 4.4 (aka iOS 5) the other day and immediately ran into a problem.  All of my movies/TV shows that had a Dolby Digital audio track were playing with the audio out of sync by about 1/2 second.  The 2.0 channel audio played perfectly, and these same files played correctly just a few hours before. I also tried using AirPlay to send the videos to the Apple TV with the same results.  Oddly enough, when I went to Netflix and played Psych (has a Dolby Digital audio track) it played perfectly in-sync.

After some troubleshooting and trying various random things, including 2 full resets, I finally tracked down the problem.  If the screen-saver type is set to Random then the digital audio tracks play with a 1/2 second delay.  If I pick a specific screen saver, everything plays correctly.  So for now I have selected a specific screen-saver instead of letting it randomly pick one.

One more random bug in Apple’s latest updates.

Lion 10.7.2 Update

Well Apple released a 10.7.2 the other day.  It quietly “fixed” the data-loss bug that occurs on network (and other non-internal storage).  And by fixed I mean they mocked us entirely. So now doing the same test, I open an image off the file server in Preview and crop it, print it, and then try to close it without saving. Preview will now helpfully tell me that if I close the file I will not have access to the “older versions” (original) of the document. Great, I can click “Don’t Save” now right? Wrong. The two buttons available are Cancel and Close. If you still want to close without saving you have to “Revert to Last Saved” and then close the file. Oh wait, when I do that it STILL gives me the same warning even though I have reverted my changes; so I will still have users confused about what is going on.

While I appreciate Apple’s attempt to fix the problem, I think they still don’t realize how big an issue this is. They seem to think this is a minor issue that will only affect a few people and therefore a half-hearted attempt to fix the problem is in order. The problem is, they implemented a fundamental new way to save files in the core OS that they cannot easily change without breaking applications. Unfortunately this means that we will probably never get what we want which is the ability to disable Auto Save instead of these patchwork solutions to try and work around the real problem.

I hate to be this critical of Apple, but all this change will do is increase my help desk load with people calling me asking (1) how they are supposed to close without saving since there is no Don’t Save button or (2) what happened to their original file because they clicked Close anyway since there was really no other choice. What that dialog needs is Cancel, Close Without Saving, or Save and Close buttons. While I appreciate their attempt to “fix” the data-loss bug, this doesn’t really fix it. It just attempts to push the blame off of Apple and onto our users “because they don’t know how to properly use the computer” or on us “because you didn’t train them properly”.

Thanks for the attempt, but that is kind of like letting somebody get in a car, drive an hour to a movie theatre and then when they try to turn off the ignition warn them that if they turn off the car they won’t be able to turn it back on unless they drive home first. Yea, thanks for the warning, but I still need to get our of my car.

iOS 5 New Features

As a member of the iOS developer program, I have had early access to iOS 5 beta for some time now. I first installed it with Beta 3 in mid-july on my personal iPhone.  Being that I am a part-time developer and don’t have thousands of dollars to buy multiple iPod touches, iPads and iPhones to keep various iOS versions on as Apple recommends (bull-*cough*-crap), I only have my iPhone to use. This means I live on the edge a bit and have to deal with glitches in the system, but it also means I learn very quickly what new features are available, since I am using it all day long.

Most of the information I am presenting here I am writing while still in the Beta stage, so some things may have changed since I wrote them. But being a good, honest Apple Developer Program member (okay, mostly honest), I didn’t want to post this until iOS 5 was officially out. This is also not a comprehensive list. I am only including things that I think people will be interested to know about and may not initially realize.

Background Syncing

The first feature I noticed right off the bat is that iOS 5 performs background syncing with iTunes. Remember all those times you’ve started a long sync process on your device and then wanted to look something up on it real quick (maybe a text message, or address book entry, whatever) and couldn’t because of that stupid “Slide to cancel” message while it was syncing? No more! You now get the normal “Slide to unlock” message and can use your device while iTunes is still syncing. For example, I can browse the web while iTunes is syncing to my phone. I can send text messages, play a game, etc.

There are some things that don’t work so well. For example, currently iTunes is syncing my entire music library because it was a full device restore (3,420 songs takes awhile to sync). Trying to run the Music application while iTunes is syncing my music library causes it to hang and I have to “force quit” it. All in all, though, I like the new feature. I don’t often use my phone while it is syncing, but I have more than once had to cancel a sync so I could get to some bit of information and then resume the sync.

By the way, there is now a small spinning sync icon in the status bar on the iOS device telling you that it is actively syncing. I personally haven’t tried pulling the sync cable while it is syncing, but I wouldn’t recommend doing it on purpose.

WiFi Syncing

WiFi syncing is not news to most people since it has been talked about and shows up right in the iTunes window as an option. It does, however, have some cool features and there are some things to know about it before you decide to just turn it on and forget about it.

First off, WiFi syncing is slower than USB. To be honest, I expected WiFi to be faster. Yes I know, USB 2.0 is 480Mbps and WiFi is only 150Mbps. But watching my phone sync via USB and the speed it takes to sync movies, for example, makes it feel like it isn’t even close to the full speed of USB 2.0. And since the USB sync always felt so slow I figured a good 802.11n network connection should be able to do better. No, it can’t. WiFi is noticeably slower than USB.

The ability to have my phone sync to my computer while still in my pocket, however, is very cool. And the ability to have it sync automatically overnight is even more cool. For me, though, I sync my phone to my desktop computer at work. My only other option is my laptop which I don’t leave open and running overnight anyway. So the automatic nightly sync does me no good. Also, leaving the WiFi syncing enabled seems to keep my phone connected to WiFi even when locked (which it didn’t before) which seems to put a noticeable drain on my battery. Usually by the end of the day my phone is only down to around 80% battery. The 3 days I had WiFi syncing enabled, by 5pm when I left work my battery was down closer to 40%. This may be because it detects my desktop (which it is “paired” to) on the network and therefore keeps the WiFi active so that I can do a WiFi sync (which can be initiated from iTunes or the iPhone). So if you sync to your home computer you may not have the same problem of having your phone connected to your work’s WiFi for 8 hours.

One bit of good news on the WiFi sync is the ability to seamlessly switch from WiFi sync to USB sync. If I start a WiFi sync session on my phone and then plug it into my computer mid-sync it almost instantly switches over to USB and the speed improves dramatically. So if you start a sync on WiFi and then realize it has a lot of data to transfer, just plug it in via USB and let it keep going.

Hourly Weather

Maybe other people already knew about this, but I found it by accident. The supplied Weather app now has hourly weather forecasts. When looking at the weather screen simply tap on the screen and it will show you the hourly information which you can then scroll through.

As I said, I found it by accident so you may not know it is there. I can’t say I would use it a lot, but if I was planning a day at the beach or a picnic or something else it would be nice to get to this more detailed weather report. You can also, by the way, sync the little spinning sync icon in the top-left next to the WiFi icon. I took this screenshot while my phone was syncing to iTunes.

Notification Center

Yes, another item you already know about. But I just wanted to give you a quick bit of information on it that again you may not ever know about unless you happened to accidentally trigger it or specifically wanted to know if it worked and tried it (like I did). The notification center can be pulled down to sliding your finger down from the status bar right? Well it also works with full-screen apps.  How often have you wanted to check on the time on your phone while playing a full-screen game but didn’t want to exit out of the game to do so?

Well now you can, and very easily. Simply perform the slide-down gesture you normally would and you will see a little gray handle appear in the top-center of the screen. Perform your slide-down gesture again and the notification center will appear, along with your standard status bar. Simply slide the notification screen back up and you can resume your game. The reason for this double-swipe gesture is obvious as soon as you play a game that requires a lot of swiping, such as Harbor 3D (great game by the way). This game requires you to draw a path for the ship to follow in order to enter the harbor safely. Well sometimes when I get to a boat too quickly and it is near the top-edge of the screen the “little gray handle” will appear. My original gesture is passed on to the game as well so that does not cause any problems. But this way I do not accidentally trigger the notification center since I then have to grab on the handle itself to fully bring it in.

You will also find under the System Preferences on your device that you can set the notification behavior on a per-app basis. For example, by default text message are I believe now banners in the notification center (they appear briefly and then go away until you pull down the notification center window). Calendar alerts are the same old “in the way” alerts they have always been which require you to dismiss them before continuing on. This granularity allows you to configure your notifications in a way that suits you and works the way you want to work.