Update on TAR Backups

It’s been a little while since I posted anything – the summer was incredibly busy. I’m finally getting caught up on a few things, and have some time to update the blog.

When Ubuntu 11.04 came out, I enthusiastically upgraded, only to have the upgrade process break my installation thanks to an error partway through. I had to do a wipe and reinstall – which I should have just done from the beginning. The result was that I had to restore my data from my backups, which didn’t work so well. I’m not sure if it was a version problem with the tar utility or what, but all of my incremental backup files were corrupted, and I ended up losing about two months’ worth of data, only some of which was recoverable. The script never created any errors, which makes the whole thing bizarre.

The good thing is that the operating system upgrade and upgrades to Ubuntu backup utilities made it possible to connect directly to a network share instead of going through the clumsy mount process. Therefore, I am using Deja Dup to back up my data to my network drive over the air, and I’m using Ubuntu One to back up my Documents folder to the cloud.

Lessons learned:

  1. Always perform operating system upgrades after the semester is over.
  2. Always perform a full backup using your backup utility and a file copy backup before upgrading.
  3. Always wipe and install. Don’t rely on the built-in updater.
  4. Always back up the most important files to multiple locations, one of which is offsite.

Incremental Network Backups with tar

When Kelsey and I were both using OS X, we had automated incremental backups over our wireless network using an AirPort Extreme and a Western Digital MyBook 500 GB HDD through Time Machine. After I moved to Ubuntu, however, I needed to find a different solution for network backups. I wanted to make sure that the network backups for Kelsey’s MacBook continued uninterrupted, but I wanted to be able to back up my system to the same network drive. In order to do that, I had to enable SMB (Windows) file sharing at the AirPort Extreme level, and then mount the drive locally. Even though my configuration might not match others, the following instructions are useful for anyone trying to do network backups to an SMB share.

Most of the backup utilities available for Ubuntu can’t use SMB shares natively. You need to mount the drive locally – i.e., map it to a local folder. However, this creates problems with most backup utilities, since they use rsync, and when rsync is copying files to what it thinks is a local folder, it switches from bit-comparison to copy-and-replace, which is slow over a network (particularly a wireless network). Additionally, rsync needs to read the entirety of the target file to figure out what bits to switch, which also significantly slows down incremental network backups (again, particularly over wireless).

My solution to this was to use tar for backups. The advantage here is that tar has the ability to do incremental backups out of the box without needing to read the backup archive. It keeps a file that contains info about all of the files that you backed up, so it can pull incremental files without even needing to be connected to the backup share.

Although you can set tar to gzip or bzip2 your backup files to make the sizes smaller, for my needs, I determined that the amount of CPU time required to do the compression wasn’t worth the 2-3% average compression over all of my files. Part of that is that many of the files I am compressing are already compressed – e.g., MP3s, PDFs, etc – or where compression is difficult – e.g., full-rez 10MP JPEG photos. If you want super small files, you could tar them and then send them off to 7zip, which will make teeny tiny files but will take forever to run. I have plenty of space on my backup drive, so I decided to save myself the cycles and just stick with a raw .tar file.

Incremental backups using tar can be automated via cron, but I choose to run the backups manually so that I don’t inadvertently close my laptop lid and stop the process mid-run and have to deal with that in the shell script. The content of my shell script for backups is below. It doesn’t include everything on my system (although it could). I’m only concerned with backing up my data directories.

# Creates incremental backups for specified directories using tar, then copies them to network storage
backupDir() {
tar -g /mnt/backup/$1.snar -cvf /mnt/backup/$1-$2.tar /home/kevin/$1 >> $3
}

# Mounts network storage drive in preparation for copying backed up files
timestamp=date +%Y%m%d%H%M%S
logfile="/mnt/backup/$timestamp.log"
umount /mnt/backup >> $logfile
if mount -t cifs //server/backup/whitemac /mnt/backup -o username=guest,password=
then
backupDir Documents $timestamp $logfile
backupDir iso $timestamp $logfile
backupDir Music $timestamp $logfile
backupDir Pictures $timestamp $logfile
backupDir Sites $timestamp $logfile
backupDir Videos $timestamp $logfile
umount /mnt/backup >> $logfile
fi

OS X Leopard on VirtualBox on Ubuntu on a MacBook

I finished building out my three Windows XP VMs and my Ubuntu Server VM, and I just finished building the final missing piece – an OS X VM for Mac-based website and service testing. The performance isn’t great, but it works. It was a little tricky to install, though, so here are the instructions:

  1. Create a new VM in VirtualBox and set it up as OS X / OS X Server (even though it won’t actually be a server install). This setting will enable all of the required configuration options needed for OS X to run properly. Take the defaults (or scale up) but don’t scale down.
  2. If you have the space, set your virtual drive to a fixed size instead of auto-expand.
  3. You can use the DVD in the drive or use an ISO. I used the DVD, but the ISO is probably faster.
  4. When you go to do the install, you will get to the point where it asks for the drive, and nothing will show up. Go to Utilities > Disk Utility and format the drive as Mac OS X Extended, Journaled. When you go back to the installer, you can now select the drive.
  5. From here, the installation should proceed without incident, but it will likely take several hours to complete. Patching will also take a few hours, so make sure to carve out enough time.

Note that I installed it on Mac hardware, which (as far as I can tell) makes a difference. Those not using Mac hardware may need to use a Hackintosh build instead, although I’d be interested to know if there are success stories installing on non-Mac hardware with a native install disc.

WordPress Multisite & ScholarPress

After getting ready to install the fourth instance of WordPress on my host, I decided to give the whole multisite thing a go. It actually wasn’t all that difficult to set up, although it requires editing an .htaccess file and wp-config.php, so you have to be comfortable with that. I have SSH access to my server, so it’s easy enough to just SSH in and use VIM to edit the files on the server directly.

One of the motivations for moving to multisite was to stand up a ScholarPress installation at http://courses.kevinfodness.com. Another motivation was to be able to have http://www.kelseyandkevin.com on the same WordPress installation as this site, so I could keep upgrades / plugins / etc consolidated, and not update things in a ton of different locations. Well, both of these things failed.

ScholarPress runs on BuddyPress, which is a network-only plugin. If you’re running it on an individual site, you don’t notice any difference from any other plugin. However, if you are running it on a multisite installation, you have to have the plugin enabled for all of your sites or none of your sites. There is no way to say, “I want BuddyPress enabled on these sites but not these sites,” within the same multisite installation. Therefore, I had to create http://courses.kevinfodness.com as a separate WordPress installation to isolate the BuddyPress install. I’ve got it up and running with ScholarPress, so we’ll see how that goes for managing my TA responsibilities.

WordPress multisite (for some reason) only lets you manage subdomains or subdirectories off of a primary domain – so I can add all of the subdomains off of kevinfodness.com that I want, but I can’t add http://www.kelseyandkevin.com. (Technically, I can, but it’s hacky and requires editing the database directly, including for all of the subdomains I would want to add.) So, it looks like one install for this site, one for the courses site, and one for K&K, with the option to add subdomains off of K&K and this site whenever I want.

In concept, multisite is great, but they’ve got a few kinks to work out. I’m sure there are plenty of other folks like me that want to manage multiple domains on the same installation.

To Hell with VMWare Server

I had planned on doing a whole writeup on how to move from VMWare Fusion 2 on the Mac to VMWare Server 2 on Ubuntu, but instead, I’m writing about how I got fed up with the process and moved to VirtualBox OSE. In many ways, this was actually the superior move – VirtualBox is open source, whereas VMWare Server (while free) is not open source. Additionally, VirtualBox has much better integration with the OS than VMWare Server has. For instance, VirtualBox on Ubuntu is managed through a thick client, whereas VMWare Server is managed through a (rather clunky) web interface.

The two things that really turned me off VMWare Server were the lack of an OS-native client and the terrible console support. In order to get console support working at all, I had to downgrade my edition of Firefox from 3.6 to 3.5, and then the console was all but unusable. Attempts to connect via RDP and VNC were an utter failure, so I decided to move to VirtualBox instead.

I tried migrating my VMs from VMWare Server to VirtualBox – which is supposedly possible – but ended up failing. To be honest, I didn’t spend a lot of time doing this, so it might be possible. However, I had built my VMs on VMWare Fusion 1, upgraded them to VMWare Fusion 2, then moved them to VMWare Server 2 on Ubuntu using the VMWare Converter, and decided that after all that moving around, they probably weren’t that stable and/or efficient any longer. Therefore, I decided on rebuilding them from scratch under VirtualBox. The only annoying thing is spending the time building out three XP VMs and installing all of the patches and software all over again. The plus side is that I’m going to build out an Ubuntu VM as a webserver, so I can keep my configuration isolated, and leverage snapshots when making configuration changes.