Fri, 13 Dec 2013
I periodically need to set up a USB hard drive so that its files can be shared between different operating systems. I recently tried to update my practices. This is a record of my findings.
The short version of my conclusions is: If you need good portability between Windows, OSX, and Linux (and you're only writing data from Windows or Linux), use NTFS. If you don't need filesystem metadata like ownership or permissions and your files and disks aren't too large, FAT32 might work for you.
FAT32
FAT32 is often the filesystem of choice for flash drives and smaller media. Windows, OSX, and Linux all have native support for it. It can span filesystems up to 2TB in its default configuration and up to 16TB or so if you tweak its block size. It cannot support files larger than 4GB, nor does it include support for file ownership and permissions. Its support for filenames longer than eight characters plus a three character extension is something of a hack. It's not case-sensitive (though it does preserve case).
Basically, FAT32's biggest strength is its cross-platform support. In most other areas it falls down when compared to more modern filesystems.
NTFS
NTFS is a bit nicer than FAT32 in many ways. It supports filesystems up to 16EB in size, and you can fill all of that space with a single file, if you want. (In other words, there's effectively no limit on file size other than the size of the containing filesystem.) Filenames can be up to 255 characters long, just like other modern filesystems. NTFS supports POSIX-compatible file ownership and permissions, hard and soft links, case-sensitivity, and sparse files, all of which which make it a lot more interoperable with Unix than FAT32.
Its main drawback is that it's proprietary and what support exists has been reverse-engineered. Windows supports NTFS, since it came from Microsoft originally, back to Windows 2000 and Windows NT 3.5. Max OSX has had native read-only support since 10.3 (Panther). Linux can read and write NTFS volumes via NTFS-3G, which runs in userspace (via FUSE). NTFS-3G doesn't give tremendous performance when accessing SATA or SAS disks, but modern hardware is more than capable of keeping up with USB-attached disks. (At least for USB 2.0; I haven't done comparisons with USB 3.0 hosts and disks.)
exFAT
exFAT is an attempt to extend the aging FAT family (FAT12/FAT16/FAT32) to support larger files. Its size limits are somewhat crazy: maximum recommended filesystem size is 512TB (though it can theoretically go up to 64ZB); maximum file size is 127PB. Like FAT32, it does not support file ownership or permissions, hard or soft links, or sparse files, and it's case-preserving but not case-sensitive. Many of the unsupported features aren't completely necessary in a lot of USB drive use-cases, so their absence isn't an immediate strike against it.
The main problem with exFAT is that it's not just proprietary (like NTFS) but patented. It's supported natively from Windows Vista on (and Windows XP can be patched for support) and in OSX 1.6.5 (Snow Leopard) and later, but Linux support is currently very shaky and difficult to distribute because of the patents. Even if Linux support were not a factor, the fact that only relatively new OSes have support for exFAT would disqualify it from consideration in many situations.
UDF
UDF is something of a dark horse in this space. It was originally designed as a general-purpose, portable filesystem. It's only really seen adoption, though, as the filesystem used on DVDs and Blu-Ray disks. Because it's used for DVDs, though, it's natively supported by all three of the major desktop OSes: Windows (back to Windows 95), MacOS (back to OS 9), and Linux (back to kernel 2.2).
Like FAT32, UDF can only have 232 blocks in its filesystem, so it only goes up to 2TB filesystems with 512-byte blocks (which is what hard drives would use; the UDF spec requires that the UDF block size match the disk block size). Like NTFS, though, its only limit on file size is the size of the containing filesystem. Also like NTFS, it supports POSIX ownership, permissions, and metadata; ACLs; and hard and soft links, as well as being case-sensitive. It doesn't support sparse files.
It would seem that UDF would be the best choice for a portable filesystem: it's natively supported in practically every OS and it supports most of the features you'd want in a modern filesystem. Unfortunately, OS bugs hamper its usefulness.
Although Linux will read UDF filesystems with blocksizes other than 512 bytes (even though that's outside the official specification), Windows is pickier and will only recognize conforming layouts with 512-byte blocks. That immediately limits Windows to 2TB disks or smaller when using UDF. Also, Windows expects UDF to span the entire disk, with no partitions, so you can't even work around the size limitation by making multiple sub-2TB partitions.
Linux, on the other hand, has had problems with UDF with 512-byte blocks. It handles filesystems with 2KB blocks just fine, but you need to be running relatively new kernels for the 512-byte blocks (required for Windows compatibility) to work. (Two problems I've run into are disks being reported full when they're not and Linux not seeing UDF volume labels.)
The Linux problems have been fixed, but only just this year. Losing compatibility with all older Linux systems knocks out one of the biggest advantages that UDF has over NTFS. In my case, I have RHEL 5 and 6 systems that aren't going to get these fixes for a really long time. (Maybe by RHEL 5's 2017 EOL, assuming RHEL 7 includes all of the fixes.)
ext[234]/HFS+
There are also the native disk formats for Linux (the ext2/3/4 series) and MacOS (HFS/HFS+). While there's at least some support for accessing them from other systems, that support is generally less robust than the NTFS-3G project's work, so if you're going to require third-party support for cross-platform access you might as well use NTFS.
Other filesystems
There are a lot of other filesystems out there. Linux has native support for quite a lot of them. OSX and Windows don't. In cases where installing third-party drivers on OSX or Windows is an accepted requirement, some other filesystem might be a good choice. In my opinion, though, NTFS, FAT32, maybe UDF, and possibly exFAT cover the filesystem portability use-cases pretty well by themselves; it would have to be a very specific set of circumstances for a different filesystem to be a better choice than one of those four.
Fri, 09 Nov 2012
I recently started a new job and my new manager wanted me to give him a periodically-updated task list showing what I was working on, its progress, and any deadlines I had. I like Emacs' Org Mode, so I set out to set up something in Org Mode that worked for both of us. This is my story.
In general, I want to have a main directory for my files and, within that directory, use a separate file for each distinct project I work on. My manager wanted to easily see project progress via milestones, so each level-1 heading represents a milestone, while the deeper headings are more fluid and are more for my benefit than his. It was pretty easy to set up HTML publishing from the Org file directory to a web server, so my manager could browse through the details of my org files.
My manager also, however, wanted to see an overview page with sections for current, future, and recently-past projects, with detail lines for the milestones on current projects. That proved to be the trickiest thing to implement: since I dislike repeating myself, I wanted as much of the "current project status" part of the overview to be automatically generated from the individual project files, including any milestone deadlines but ignoring all of the more minor details.
Org Mode is big, so I spent a while reading through the manual about
custom agenda views and such until I stumbled on the C-c / d key
sequence, which folds all the file's subheadings and displays any
deadlines on the level-1 headings. In combination with C-c C-e v Spc,
which exports only the visible portions of the current buffer to a new
org-mode buffer, I was able to create an org-mode formatted project
overview. Thus, my general overview page (named index.org so it becomes
index.html during publishing) looks something like this:
* Active Projects
** [[file:project-1.org][Project 1 Description]]
DEADLINE: <2012-11-16 Fri>
#+INCLUDE: "project-1.org_overview" :minlevel 2
** [[file:project-2.org][Project 2 Description]]
DEADLINE: <2012-11-30 Fri>
#+INCLUDE: "project-2.org_overview" :minlevel 2
* Future Projects
** [[file:project-a.org][Project A Description]]
** [[file:project-b.org][Project B Description]]
* Recently Completed Projects
** [2012-11-01] [[file:project-x.org][Project X Description]]
I added a few CSS tweaks at the top of the index.org file to better fit its purpose. Mostly, they just put related things closer together and deemphasize some of the extra information on the page:
#+OPTIONS: H:2 num:nil toc:nil tasks:t
#+STYLE: <style>
#+STYLE: .outline-3 h3 {
#+STYLE: margin-bottom: 0;
#+STYLE: }
#+STYLE: .outline-text-3 p {
#+STYLE: margin: 0;
#+STYLE: }
#+STYLE: .outline-4 {
#+STYLE: line-height: 0.5em;
#+STYLE: }
#+STYLE: .outline-text-4 {
#+STYLE: font-size: 0.75em;
#+STYLE: }
#+STYLE: .outline-4 h4 {
#+STYLE: margin-bottom: 0;
#+STYLE: }
#+STYLE: </style>
I also wrote the following two elisp functions. The first one generates the overview file for a given buffer, while the second looks through a buffer for included overview files and regenerates all of them.
(defun org-export-overview (file)
(save-window-excursion
(find-file file)
(org-check-deadlines org-deadline-warning-days)
(org-export-visible ?\ 0) ; calls switch-to-buffer-other-window
(write-file (concat file "_overview"))
(kill-buffer)))
(defun org-export-overviews-for (buffer-or-name)
(let ((plan-dir (file-name-directory (buffer-file-name (get-buffer buffer-or-name)))))
(with-current-buffer buffer-or-name
(save-excursion
(goto-char 1)
(while (re-search-forward "^#\\+INCLUDE: +\"\\(.*\\)_overview\"" nil t)
(org-export-overview (concat plan-dir (match-string 1))))))))
Finally, I wrote a simple function that goes through all the stuff I need to publish my org files and I call that hourly, so my manager can see my current project status at pretty much any time with no manual steps on my part:
(defun pmg-org-publish ()
(org-save-all-org-buffers)
(org-export-overviews-for "index.org")
(org-publish-all))
(run-at-time "00:45" 3600 'pmg-org-publish)
All I need to do is periodically update index.org as I add or complete projects, and the rest of my work is done, as it normally is, in the project files. Thanks, Org Mode!
Fri, 07 Oct 2011
A question that perturbed me when I lived in Owings Mills was why there were so many roads named "Dolfield". There are three different Dolfield Roads, one South Dolfield Road, and three Dolfield Boulevards. I finally got around to researching the question, and now I share the answer (to the degreee that I have one) with everyone.
This is a map of Dolfield Road, past and present:
The thick, dark blue line is where Dolfield Road once ran. At one point, it connected Painters Mill Road and Deer Park Road.
The thinner, light blue line shows where Dolfield Road, in all its parts, runs today. When Interstate 795 was built, Dolfield Road was cut in two, as happened to many roads during the construation of the Interstate system. At the same time, part of the eastern portion of Dolfield Road was diverted to make room for the ramps from Owings Mills Boulevard to Interstate 795. The section of Dolfield Road south of Owings Mills Boulevard was renamed South Dolfield Road, for reasons I cannot find.
At some point--I cannot tell when, exactly--the chunk of Dolfield Road that extends from present-day Red Run Boulevard to Pleasant Hill Road was closed, though I was unable to find out why. The small section between Red Run Boulevard and Interstate 795 now serves as an access road to a SHA facility, and the rest of the closed section has been turned into a hiking/biking trail (colored in light green on the map). Finally, the section of Dolfield Road just west of Pleasant Hill Road was realigned when Dolfield Boulevard was built.
Here's a map of Dolfield Boulevard:
At some point in the past, the county planned to build a road named Dolfield Boulevard that would connect Owings Mills New Town with the eastern area of Reisterstown, both areas that were slated for new development. The road was planned to have two lanes in each direction with a raised median dividing the road. The dark purple line on the above map shows where it was to have gone, as far as I can tell. I'm not sure whether the original plans included anything further north, though it might have made sense to connect it to Owings Mills Boulevard.
Unfortunately for the county, a number of people living in the proposed path of the new road put up sufficient objections that the project stalled. Portions along Reisterstown Road and Academy Avenue were built and, in both cases, partly divided and partly not--the light purple lines are the divided sections and the pink lines are the undivided sections. Furthermore, the southern end of Dolfield Boulevard was built without incident as part of the development of Owings Mills New Town, though part of Pleasant Hill Road was rechristened Dolfield Boulevard but left undivided.
From what I can tell, the county never entirely abandoned its plans for Dolfield Boulevard, but they seems to be shelved indefinitely. Separately from those plans, they are currently considering adding an interchange between Dolfield Boulevard and Interstate 795 (which would have been really convenient for me if it had been in place when I lived there).
So there are three sections of Dolfield Road (and a South Dolfield Road) because Interstate 795 and other factors caused a former road to be split into pieces, and there are three sections of Dolfield Boulevard because that's as far as the county has gotten on the plans for a new road. This ends today's episode of "I'm a geek and will research anything that catches my fancy".
Wed, 21 Sep 2011
This is a thing, apparently: "Take knife from pocket, post pic on the internet."
My main knife, and the one that best matches the intent of the
meme, I think, is a Myerchin Lightknife L377P, which they
apparently don't sell anymore. It's a very nice, light, compact knife,
with a multipurpose half-straight half-serrated blade; a marlinspike,
which is useful for knot- and ropework; and a small red LED, which is nice
for seeing things without ruining your night vision. I use the
marlinspike a lot less these days, because I don't do as much decorative
knotwork, but it's still occasionally useful.
The knife was a Christmas gift from my dad and stepmother a number of years ago. (Possibly 2004 or so.) I'd been thinking about replacing my at-the-time pocket knife, something from Spyderco, with a lighter, more compact knife, when I read a Cool Tools review of the Myerchin L377P. It sounded like it would do everything I wanted and more, so I made a Christmas request and then got what I asked for.
The Myerchin isn't the only knife that meets the letter of the meme's
wording, though. I also have a credit-card-shaped multitool from
Brookstone in my wallet. I use it mostly for its tweezers, but its pen,
needle, scissors, and knife blade are also occasionally useful.
A knife I also usually have on me is my
Victorinox Swiss Army CyberTool. Technically, it lives in my
laptop bag, not my pocket, but I usually have the bag with me. It's very
useful for dealing with all sorts of small-scale hardware, because it's
got the screwdriver with interchangeable phillips, flat, and torx bits,
plus the small pliers (which also include a wire cutter).
This was a birthday gift from my parents back in 2000 or so. It replaced my previous everyday knife, a Swiss Army Automobile Special, which had also been a birthday gift from my parents. Victorinox had just come out with the CyberTool and I'd been working for a couple of years in a hardware and software tech support position, so I let my parents know I was really interested in the new knife. More than ten years later, it's still serving me well.
The last knife that I usually have on me (also in my laptop bag) is a
Columbia River Knife & Tool Rollock, which doesn't appear to be
for sale anymore. It's got an interesting opening mechanism; instead of
unfolding, the blade flips up a little and slides out of the body, locking
into place at its fullest extent. It's mostly in my laptop bag because
it's very slim, so it doesn't take up much extra room and it's handy if
I've forgotten my Myerchin somewhere. Because it's got a completely
straight blade, it's less of a pain to sharpen than the Myerchin, so it's
usually the sharpest knife I have on me, which sometimes means I use it
instead of the Myerchin even if I have both with me.
This one was a gift from my youngest sister at her wedding as thanks for being a groomsman and playing trumpet in the wedding.
Mon, 02 May 2011
Just today I found about GTFS, a standardized data format from Google for describing public transit schedules. I subsequently discovered that the MTA has a GTFS feed.
Back when I didn't have a car, I used the MTA daily and had a set of scripts to extract their schedule data from the website and import it into trainsched on my Palm Pilot. I asked them for more structured data at one point but was rebuffed, so I stuck with my scripts that had to be updated every time they changed their schedule pages.
I don't have a Palm Pilot any more (and trainsched doesn't even run on the PalmOS emulator on my WebOS phone), but I thought it would be nice to update my old scripts to use the new GTFS feeds. This should even work with other GTFS feeds; there's nothing really MTA-specific going on.
The updated scripts are here: GTFS to trainsched converter.
Fri, 02 Jan 2009
I previously considered different backup schemes. Writing that entry crystallized my basic ideas about what I was going to do about the backups. I then proceeded to implement them. This entry is a detailed description of what I did.
Backup Overview
I ended up buying two 80 GB hard drives and a ThermalTake BlacX external enclosure. The overall plan is to do backups to one of the drives on a daily, automated basis, and the other on a periodic, maybe monthly basis. Most of the time, the periodic backup drive will live at my office and will serve as my offsite backup.
I want to have a backup history in the same way that a tape rotation
scheme would give me. That way, if I don't notice there's something wrong
with a file for a time, there's still a good chance I can retrieve it. I
also want things stored space-efficiently, so backing up unchanged files
doesn't take up additional space. This is accomplished pretty easily with
rsync; I do one full backup, and then subsequent backups use rsync's
--link-dest option pointing to the most recent complete backup; any
files that haven't changed are simply hardlinked together, so the two
directory entries point to the same physical location on the disk.
For the backup history, I decided to go with a variant of the Towers of Hanoi backup rotation. Instead of using a preset number of pegs, as I would have to do if I were using physical tapes, I can just calculate the numbers on the fly, effectively behaving as if I had an infinite number of tapes. This rotation gives me roughly exponential backoff for my history; I can look at backups from yesterday, two days ago, four days ago, eight days ago, and so on.
Finally, I decided to encrypt the drives. This lets me cart them around with confidence that if they get lost or stolen, anyone who ends up with them won't have my personal files. I used LUKS to encrypt the disks, and used both a file on my computer and a password as keys. The file makes it easier to mount the drives without manual intervention (important if my desktop reboots), while the password lets me get at the data if the key file isn't available (important if the main disk dies and I need my backups).
Backup Procedure
Set up fstab.
The first thing I did was to set up /etc/fstab for the disks. Since I only have one enclosure, I have to swap disks, so only one will ever be connected at the same time. Thus, I mount whichever's in at the moment on /backups. Likewise, I will associate each one with the dm-crypt name of "backups", so the device will be /dev/mapper/backups. Thus, I added the following line to /etc/fstab:
/dev/mapper/backups /backups auto defaults 0 0
Set up rsync filters.
I'm using rsync to do the backups, but in some cases I don't want
everything in the source directories to be backed up. Thus, I tell
rsync to look at a filter file for each directory so it knows what to
skip and what to keep. rsync will go through the filters for each file
or directory it considers, and will take the first action that matches.
If nothing matched, the file is copied. If a directory is ignored, none
of the files in that directory are considered at all, so I had to include
a few explicit directory chains.
In /var, I only want to back up a few things, so the final pattern ignores anything that isn't explicitly included.
+ /backups
+ /backups/**
+ /lib
+ /lib/bzr
+ /lib/bzr/**
+ /lib/svn
+ /lib/svn/**
- *
For my home directory, I include everything, with a few exceptions. For instance, most of my music directory can be reripped from CD if needed, so I don't need to take up space backing up those files. On the other hand, I have some files that I either purchased online or downloaded and wouldn't be able to easily replace if they were lost, so I do back them up. Here's an excerpt from my home filter file:
+ /movies/Star_Wars_Holiday_Special
+ /movies/Star_Wars_Holiday_Special/**
- /movies
+ /music
+ /music/Bonerama
+ /music/Bonerama/Bringing_It_Home
+ /music/Bonerama/Bringing_It_Home/**
+ /music/Jonathan_Coulton
+ /music/Jonathan_Coulton/Thing_a_Week_Three
+ /music/Jonathan_Coulton/Thing_a_Week_Three/03-Code_Monkey*
+ /music/Nine_Inch_Nails
+ /music/Nine_Inch_Nails/Ghosts_I-IV
+ /music/Nine_Inch_Nails/Ghosts_I-IV/**
+ /music/Nine_Inch_Nails/The_Slip
+ /music/Nine_Inch_Nails/The_Slip/**
+ /music/Obelix
+ /music/Obelix/**
+ /music/Solo_String_Project
+ /music/Solo_String_Project/**
- /music/**
- /tmp
Initialize disks.
I wrote a script to initialize the disks for me: init-backup-disk. It takes two parameters: the name of the device for the backup disk, and the file to use as a key for the partition. If the key file doesn't exist, it will be created.
After a few sanity checks, the script starts doing things. It starts by
checking that the disk is good with badblocks. If it encounters any
errors, it stops there and the drive needs to be sent in for warranty
replacement. Following that, it goes into the most time-consuming part of
the initialization: writing random data to the disk. (badblocks already
wrote random data, but its PRNG is somewhat simplistic; /dev/urandom is a
much better source of pseudo-random numbers.) Without this step, it would
be obvious which bits of the disk had encrypted data on them. I use
pv to give a progress meter and time estimate. On my computer,
badblocks took a little over two hours and /dev/urandom took about eight
hours for each 80GB disk.
# Check and randomize disk.
badblocks -b 512 -s -w -t random -v $disk || exit 2
</dev/urandom pv -s $(fdisk -l $disk |
perl -nle 'm{^Disk '${disk}': [0-9.]+ [KMGT]B, (\d+) bytes$} and print $1') |
dd bs=512 conv=sync,noerror of=$disk
The next step is to format the encrypted partition. I use sfdisk to
create a single partition that spans the entire drive, followed by
cryptsetup to do the format. I explicitly specify the cipher in order
to use ESSIV, which makes certain attacks more difficult. The
--batch-mode option keeps it from asking for confirmation before
writing. The second call to sfdisk just tells the kernel to reread the
disk's partitions so it will pick up the UUID that cryptsetup created.
# Add and format the LUKS partition.
echo , | sfdisk --Linux $disk
cryptsetup luksFormat --cipher aes-cbc-essiv:sha256 --batch-mode ${disk}1 $keyfile
sfdisk -R $disk; sleep 5
Next, I open, format, and mount the partition. JFS is the filesystem that's been nicest to me, of all the journaling filesystems I've tried. (In the future, it might be nice to use ZFS for the backup disks--I'd get better detection of disk errors, at least--but I don't think it would play entirely well with running over an encrypted volume, and they haven't integrated encryption into ZFS yet, as far as I can tell.)
# Open LUKS partition, format and mount the encrypted volume.
cryptsetup --key-file $keyfile luksOpen ${disk}1 backups
mkfs -t jfs -q /dev/mapper/backups
mount /backups
Now I run the initial backup. Each backup version is just a separate numbered directory in the partition, so the first one is '1'. I'm backing up /etc, some of /var, my and my wife's home directories, and any databases I have. My wife's stuff isn't directly backed up here because she's on a different computer; I have to initiate her backup from there. The script, in its first requirement for user interaction, will wait until I tell it that's done.
# Do the initial backup.
mkdir -vp /backups/1/{etc,var,phil,postgresql,mysql,rivana}
chmod a-r /backups/1
chown rivana /backups/1/rivana
chown postgres /backups/1/postgresql
rsync -avP --filter 'merge /etc/backups/etc-filter' /etc/ /backups/1/etc/
rsync -avP --filter 'merge /etc/backups/var-filter' /var/ /backups/1/var/
rsync -avP --filter 'merge /etc/backups/phil-filter' /home/phil/ /backups/1/phil/
su -c 'pg_dumpall -v >/backups/1/postgresql/dump' postgres
mysqldump -v --all-databases >/backups/1/mysql/dump
echo -n "Hit enter when rivana is backed up. "
read foo
Now that the backup is done, the script unmounts and deactivates the partition.
# Deactivate the encrypted volume.
umount /backups
cryptsetup luksClose backups
And I get prompted for the password that will unlock the partition if the key file isn't available.
# Add user password.
cryptsetup --key-file $keyfile --verify-passphrase luksAddKey ${disk}1
Finally, the script displays the UUID for the partition, which is needed for later use.
# Display the partition's UUID.
echo -n 'UUID: '
cryptsetup luksUUID ${disk}1
Set up crypttab.
Debian, at least, has an /etc/crypttab file that lists encrypted partitions to be enabled at boot time. I put the onsite backup disk in there so it'll be automatically mounted if the computer reboots. This plus a backup cronjob make the backup process completely automated.
backups /dev/disk/by-uuid/<onsite UUID> <key file> luks
Do local backups.
I have a simple script to do the daily backups: perform-backup. It's
basically the same as the initial backup, but with the --link-dest
option as I mentioned previously.
last_num=$(ls -t /backups | head -1)
((num=$last_num+1))
mkdir -p /backups/$num/{etc,var,phil,postgresql,mysql,rivana}
chown rivana /backups/$num/rivana
chmod a-r /backups/$num
rsync -a --filter 'merge /etc/backups/etc-filter' --link-dest=/backups/$last_num/etc /etc/ /backups/$num/etc/
rsync -a --filter 'merge /etc/backups/var-filter' --link-dest=/backups/$last_num/var /var/ /backups/$num/var/
rsync -a --filter 'merge /etc/backups/phil-filter' --link-dest=/backups/$last_num/phil /home/phil/ /backups/$num/phil/
chown postgres /backups/$num/postgresql
su -c "pg_dumpall >/backups/$num/postgresql/dump" postgres
mysqldump --all-databases >/backups/$num/mysql/dump
Do Becca's backup
My wife has her own computer but, fortunately, simpler backup requirements. I have ssh public key authentication set up so she can ssh to my computer without a password, which makes the backups work properly in an automated fashion.
The initial backup is a simple rsync one-liner.
rsync -avP ~/ mithrandir:/backups/1/rivana/
Subsequent backups are a simple script:
#!/bin/sh
num=$(ssh mithrandir ls -t /backups | head -1)
last_num=$(ssh mithrandir ls -t /backups | head -2 | tail -1)
rsync -a --link-dest=/backups/${last_num}/rivana ${HOME}/ mithrandir:/backups/${num}/rivana/
Backup rotation.
The Towers of Hanoi rotation is effected by a script that goes through and deletes any directories that don't need to be there: purge-backups. I won't quote it here because I don't think it's all that interesting. It just finds the largest power of two less than or equal to the current number and then works its way down from that to enumerate all of the directories, deleting everything else.
Offsite backups.
Every so often I'll bring the offsite disk home and back up to it. The script for that is insert-offsite-backup. It unmounts the onsite disk, waits for me to insert the offsite disk, runs a backup, unmounts the offsite disk, waits for me to reinsert the onsite disk, then remounts that disk. It needs to be told what my offsite UUID is, but it picks up all the other settings from /etc/crypttab.
The backup takes about half an hour, so I have ample time to manually run the backup script on Becca's computer.
Mon, 22 Dec 2008
I had a dream last night that the apartment beneath ours caught on fire, we had to rush out of the building, and my computer and all of its data was destroyed.
I've been pondering a formal backup system for a while now. (My current system involves making sure important files are in a version control system and exist on at least my laptop and desktop. This is pretty ad-hoc, inconsistently updated, and not entirely comprehensive.) I'm taking my dream as impetus to actually set something up. This post is to help me organize my thoughts and see if anyone has any comments or suggestions.
My Requirements
I want to have a full rotating versioned backup system, where I have complete daily backups for a recent time span (say a week or so) and more sporadic backups back to as much as a year in the past. Ideally, the backups should be stored in a space-efficient manner so unchanged files don't take up more space than a single copy would require. The backups should have off-site redundancy. They should be relatively easy to use; they should be fully automated on a day-to-day basis, with notification when things go wrong. Ease of setup would be nice but not necessary.
My Data
I currently have about 720 GB of data in my home directory, plus a few hundred MB elsewhere on the computer that I'd want to back up. I also have about 11GB in a bzr repository, but all of that should remain duplicated in my home directory. Most of the data in my home directory is in media files that I can either replace (rerip CDs, etc.) or live without; only 25 GB of it is stuff that I must back up. (A further 130 GB is stuff that would be nice to back up, but I can just burn it to DVD and consider those my backups; the data is essentially static.)
JWZ Backups
The easiest approach is the JWZ backup solution. For all of my data, that would be two 1 TB external hard drives, for about $220. If I restrict myself to the "must backup" data, I could make do with two 60 GB external hard drives for about $80. In either case, I'd keep one drive at the office and swap them periodically.
The advantage of this approach is that I control everything. I can put encrypted volumes on the drives, so if they get lost or stolen, my data isn't usable to other people. I can use rsync with hardlinks between datestamped directories to get versioned backups with efficient disk usage. The drawbacks are a modest initial monetary outlay and the need to coordinate shuttling drives back and forth.
Amazon S3
Another approach is to use Amazon S3 to store my data. It's offsite by definition (and stored among multiple data centers; if I write data to it, I can reasonably trust that I'll get that data back). It's not too expensive: at $0.17/GB-month, my minimal backup will cost about $3.85/month. Throw in transfer costs and churn, and I doubt I'd exceed $6/month. (The initial upload would be $2.56. A full restore would cost me $4.36.) With S3, I would only back up the minimal data; the 130 GB of optional backups would cost an additional $20/month, which would exceed the cost of the full do-it-myself hard drive backups in one year.
The complication to S3 is that it's just a web-based data storage service; you need additional software to make a reasonable backup solution.
Jungle Disk
From everything I've read, Jungle Disk is currently the best software for storing filesystem data on S3. It runs on Windows, Mac OSX, and Linux, and exports your S3 buckets as a WebDAV disk, which you can then mount and treat like an ordinary (unlimited capacity) disk drive. All data is encrypted before it's sent to S3.
I like this approach. Since it looks like a disk, I can use the same rsync setup I would with my own disks, and since the data is encrypted, I don't need to worry too much about it being transported over the Internet and stored on someone else's servers. The main drawback is that it's proprietary software. In addition to my principled preference of open source software to proprietary, there's also the issue that, especially because the data's encrypted, this software would be my only access to my backups. If something went wrong and I couldn't get support from the company (e.g. they went out of business), I'd be out of luck.
The software costs $20. Assuming $5/month on S3, it would take one year for this approach to cost more than the minimal get-my-own-disks approach.
Other S3 software
I haven't seen anything else that will let me back up to S3 and keep versioned backups in a space-efficient manner. Most of the S3 backup software I've seen doesn't do versions, and the few that do don't appear to do it space-efficiently. As always, I have the option of writing my own, but that would take a fair amount of time and effort, and I'd be likely to give up partway through, continuing to leave myself without good backups.
Conclusion
Barring any better suggestions from others, I'm leaning towards the two smallish hard drives. They'd pay for themselves after a year of use, and I get complete control of my data (for better or worse). I like the idea of using S3, but it's more expensive in the long run, and I'm not completely happy with any of the software I've found to use with it.
