Thu, 04 Dec 2014

The Best Android Apps of 2014

3:05PM | Geekery | #

(According to /r/Android.)

Google recently came out with a "Best Apps of 2014" list. It was not well received, with many people feeling that the majority of apps present were there because Google was making money from them, not because they actually deserved to be on such a list. Consequently, the Android subreddit attempted to compile its own list. This post is my attempt to collect the most highly-voted submissions on that /r/Android post.

What follows are the 35 top apps, based on Reddit's "best" sorting algorithm. If I have time, I'll add more to the list later.

SuperSU

Gives you root on your system.

JuiceSSH

SSH client with optional cloud syncing and other features.

SeriesGuide

TV episode (and movie) watching management. Tracks things you've seen, tells you about upcoming things. Syncs with trakt or tvtag.

Pocket Casts

Podcasting app. "Buy it for the hilarious changelogs, stay for the awesome Podcast App."

QuickPic

Album app. Browse, display, and select images on your device.

Nova Launcher

Alternate home screen. Very customizable.

AirDroid

Manage your device from your desktop or laptop web browser over the network. You can transfer files, send and receive SMS messages, see notifications, and generally fully control your Android device.

Pushbullet

Easily send information between your device and desktop computer. Serves as both a cross-device notification system (e.g. see your Android notifications on your desktop) and a data sharing system (e.g. send a URL from your laptop to your phone).

CamScanner

Take photos of documents with your phone and turn them into PDFs.

Tasker

General-purpose automation for your Android device.

Moon+ Reader

Ebook reader.

Strava

Fitness app for tracking your running and bicycling.

Today Calendar

Material-designed calendar app.

Textra

Material-designed SMS app.

ES File Explorer

File management app.

Duet

Game. The gameplay's a little difficult to describe. You control two objects that rotate in sync around the same point and you have to move them to avoid obstacles.

Sleep as Android

Sleep tracker and sleep-cycle-aware alarm clock.

Nine

Synchronizes device data with a Microsoft Exchange Server.

Timely

Very customizable, multi-featured, and good looking clock app. Has a clock, alarm clock, timer, and stopwatch.

Reddit Sync

Reddit client with a card UI.

TextSecure

Encrypted messaging app.

Cerberus

Security program. Lets you track, manage, and even wipe your device remotely in case it's lost or stolen.

Waze

Traffic-aware GPS routing and navigation.

FolderSync

Sync local folders to various cloud storage providers.

IFTTT

Short for "If This Then That". Android client for the IFTTT web service. IFTTT hooks into a lot of other sites (and your Android device) and lets you set up triggers so if something happens in one place, it causes something else to happen in another. (e.g. if it's going to rain, it can have your phone pop up a notification to take an umbrella when you leave the house.)

TeamViewer

Kind of the reverse of AirDroid. Lets you manage your desktop computer from your phone.

Llama

Android device automation, similar to Tasker. (Or Tasker is similar to Llama. Plugins for either will work with both.)

Here

Offline maps. Not available in the Play Store, so you have to sideload it.

Plex

Client for the Plex Media Server.

AcDisplay

Shows notifications while the phone is locked. Detects when you pull your phone out of your pocket and turns on the screen to show your notifications. Similar in concept to the Moto X's Active Display.

RedReader

Reddit client.

Morning Routine

Alarm clock that lets you define a sequence of steps necessary to turn off the alarm. The idea is that you encode your entire morning routine into it, which makes sure you're awake by the end and makes sure you do everything you're supposed to.

Citymapper

Journey planning app, including transit, auto, foot, bike, and taxi. Only for specifically-supported cities.

Sunrise Calendar

Calendar app.

Xposed Framework

Framework for installing Xposed modules. Each module patches some aspect of the running system in order to change it. There are modules for all sorts of things, from working around Android bugs to adding cosmetic tweaks to making stock Android behave like a custom ROM. Root required. Must be sideloaded, since it's not in the Play store.

Solid Explorer

File manager.

Fri, 13 Dec 2013

Portable Filesystems for Portable Disk Drives

3:53PM | Geekery | #

I periodically need to set up a USB hard drive so that its files can be shared between different operating systems. I recently tried to update my practices. This is a record of my findings.

The short version of my conclusions is: If you need good portability between Windows, OSX, and Linux (and you're only writing data from Windows or Linux), use NTFS. If you don't need filesystem metadata like ownership or permissions and your files and disks aren't too large, FAT32 might work for you.

FAT32

FAT32 is often the filesystem of choice for flash drives and smaller media. Windows, OSX, and Linux all have native support for it. It can span filesystems up to 2TB in its default configuration and up to 16TB or so if you tweak its block size. It cannot support files larger than 4GB, nor does it include support for file ownership and permissions. Its support for filenames longer than eight characters plus a three character extension is something of a hack. It's not case-sensitive (though it does preserve case).

Basically, FAT32's biggest strength is its cross-platform support. In most other areas it falls down when compared to more modern filesystems.

NTFS

NTFS is a bit nicer than FAT32 in many ways. It supports filesystems up to 16EB in size, and you can fill all of that space with a single file, if you want. (In other words, there's effectively no limit on file size other than the size of the containing filesystem.) Filenames can be up to 255 characters long, just like other modern filesystems. NTFS supports POSIX-compatible file ownership and permissions, hard and soft links, case-sensitivity, and sparse files, all of which which make it a lot more interoperable with Unix than FAT32.

Its main drawback is that it's proprietary and what support exists has been reverse-engineered. Windows supports NTFS, since it came from Microsoft originally, back to Windows 2000 and Windows NT 3.5. Max OSX has had native read-only support since 10.3 (Panther). Linux can read and write NTFS volumes via NTFS-3G, which runs in userspace (via FUSE). NTFS-3G doesn't give tremendous performance when accessing SATA or SAS disks, but modern hardware is more than capable of keeping up with USB-attached disks. (At least for USB 2.0; I haven't done comparisons with USB 3.0 hosts and disks.)

exFAT

exFAT is an attempt to extend the aging FAT family (FAT12/FAT16/FAT32) to support larger files. Its size limits are somewhat crazy: maximum recommended filesystem size is 512TB (though it can theoretically go up to 64ZB); maximum file size is 127PB. Like FAT32, it does not support file ownership or permissions, hard or soft links, or sparse files, and it's case-preserving but not case-sensitive. Many of the unsupported features aren't completely necessary in a lot of USB drive use-cases, so their absence isn't an immediate strike against it.

The main problem with exFAT is that it's not just proprietary (like NTFS) but patented. It's supported natively from Windows Vista on (and Windows XP can be patched for support) and in OSX 1.6.5 (Snow Leopard) and later, but Linux support is currently very shaky and difficult to distribute because of the patents. Even if Linux support were not a factor, the fact that only relatively new OSes have support for exFAT would disqualify it from consideration in many situations.

UDF

UDF is something of a dark horse in this space. It was originally designed as a general-purpose, portable filesystem. It's only really seen adoption, though, as the filesystem used on DVDs and Blu-Ray disks. Because it's used for DVDs, though, it's natively supported by all three of the major desktop OSes: Windows (back to Windows 95), MacOS (back to OS 9), and Linux (back to kernel 2.2).

Like FAT32, UDF can only have 232 blocks in its filesystem, so it only goes up to 2TB filesystems with 512-byte blocks (which is what hard drives would use; the UDF spec requires that the UDF block size match the disk block size). Like NTFS, though, its only limit on file size is the size of the containing filesystem. Also like NTFS, it supports POSIX ownership, permissions, and metadata; ACLs; and hard and soft links, as well as being case-sensitive. It doesn't support sparse files.

It would seem that UDF would be the best choice for a portable filesystem: it's natively supported in practically every OS and it supports most of the features you'd want in a modern filesystem. Unfortunately, OS bugs hamper its usefulness.

Although Linux will read UDF filesystems with blocksizes other than 512 bytes (even though that's outside the official specification), Windows is pickier and will only recognize conforming layouts with 512-byte blocks. That immediately limits Windows to 2TB disks or smaller when using UDF. Also, Windows expects UDF to span the entire disk, with no partitions, so you can't even work around the size limitation by making multiple sub-2TB partitions.

Linux, on the other hand, has had problems with UDF with 512-byte blocks. It handles filesystems with 2KB blocks just fine, but you need to be running relatively new kernels for the 512-byte blocks (required for Windows compatibility) to work. (Two problems I've run into are disks being reported full when they're not and Linux not seeing UDF volume labels.)

The Linux problems have been fixed, but only just this year. Losing compatibility with all older Linux systems knocks out one of the biggest advantages that UDF has over NTFS. In my case, I have RHEL 5 and 6 systems that aren't going to get these fixes for a really long time. (Maybe by RHEL 5's 2017 EOL, assuming RHEL 7 includes all of the fixes.)

ext[234]/HFS+

There are also the native disk formats for Linux (the ext2/3/4 series) and MacOS (HFS/HFS+). While there's at least some support for accessing them from other systems, that support is generally less robust than the NTFS-3G project's work, so if you're going to require third-party support for cross-platform access you might as well use NTFS.

Other filesystems

There are a lot of other filesystems out there. Linux has native support for quite a lot of them. OSX and Windows don't. In cases where installing third-party drivers on OSX or Windows is an accepted requirement, some other filesystem might be a good choice. In my opinion, though, NTFS, FAT32, maybe UDF, and possibly exFAT cover the filesystem portability use-cases pretty well by themselves; it would have to be a very specific set of circumstances for a different filesystem to be a better choice than one of those four.

Fri, 09 Nov 2012

Org-Mode and Project Overviews

9:56AM | Geekery | #

I recently started a new job and my new manager wanted me to give him a periodically-updated task list showing what I was working on, its progress, and any deadlines I had. I like Emacs' Org Mode, so I set out to set up something in Org Mode that worked for both of us. This is my story.

In general, I want to have a main directory for my files and, within that directory, use a separate file for each distinct project I work on. My manager wanted to easily see project progress via milestones, so each level-1 heading represents a milestone, while the deeper headings are more fluid and are more for my benefit than his. It was pretty easy to set up HTML publishing from the Org file directory to a web server, so my manager could browse through the details of my org files.

My manager also, however, wanted to see an overview page with sections for current, future, and recently-past projects, with detail lines for the milestones on current projects. That proved to be the trickiest thing to implement: since I dislike repeating myself, I wanted as much of the "current project status" part of the overview to be automatically generated from the individual project files, including any milestone deadlines but ignoring all of the more minor details.

Org Mode is big, so I spent a while reading through the manual about custom agenda views and such until I stumbled on the C-c / d key sequence, which folds all the file's subheadings and displays any deadlines on the level-1 headings. In combination with C-c C-e v Spc, which exports only the visible portions of the current buffer to a new org-mode buffer, I was able to create an org-mode formatted project overview. Thus, my general overview page (named index.org so it becomes index.html during publishing) looks something like this:

* Active Projects

** [[file:project-1.org][Project 1 Description]]
   DEADLINE: <2012-11-16 Fri>
#+INCLUDE: "project-1.org_overview" :minlevel 2

** [[file:project-2.org][Project 2 Description]]
   DEADLINE: <2012-11-30 Fri>
#+INCLUDE: "project-2.org_overview" :minlevel 2

* Future Projects

** [[file:project-a.org][Project A Description]]
** [[file:project-b.org][Project B Description]]

* Recently Completed Projects

** [2012-11-01] [[file:project-x.org][Project X Description]]

I added a few CSS tweaks at the top of the index.org file to better fit its purpose. Mostly, they just put related things closer together and deemphasize some of the extra information on the page:

#+OPTIONS:   H:2 num:nil toc:nil tasks:t
#+STYLE: <style>
#+STYLE: .outline-3 h3 {
#+STYLE:   margin-bottom: 0;
#+STYLE: }
#+STYLE: .outline-text-3 p {
#+STYLE:   margin: 0;
#+STYLE: }
#+STYLE: .outline-4 {
#+STYLE:   line-height: 0.5em;
#+STYLE: }
#+STYLE: .outline-text-4 {
#+STYLE:   font-size: 0.75em;
#+STYLE: }
#+STYLE: .outline-4 h4 {
#+STYLE:   margin-bottom: 0;
#+STYLE: }
#+STYLE: </style>

I also wrote the following two elisp functions. The first one generates the overview file for a given buffer, while the second looks through a buffer for included overview files and regenerates all of them.

(defun org-export-overview (file)
  (save-window-excursion
    (find-file file)
    (org-check-deadlines org-deadline-warning-days)
    (org-export-visible ?\  0) ; calls switch-to-buffer-other-window
    (write-file (concat file "_overview"))
    (kill-buffer)))

(defun org-export-overviews-for (buffer-or-name)
  (let ((plan-dir (file-name-directory (buffer-file-name (get-buffer buffer-or-name)))))
    (with-current-buffer buffer-or-name
      (save-excursion
        (goto-char 1)
        (while (re-search-forward "^#\\+INCLUDE: +\"\\(.*\\)_overview\"" nil t)
          (org-export-overview (concat plan-dir (match-string 1))))))))

Finally, I wrote a simple function that goes through all the stuff I need to publish my org files and I call that hourly, so my manager can see my current project status at pretty much any time with no manual steps on my part:

(defun pmg-org-publish ()
  (org-save-all-org-buffers)
  (org-export-overviews-for "index.org")
  (org-publish-all))

(run-at-time "00:45" 3600 'pmg-org-publish)

All I need to do is periodically update index.org as I add or complete projects, and the rest of my work is done, as it normally is, in the project files. Thanks, Org Mode!

Fri, 07 Oct 2011

Why are there so many Dolfield Roads?

6:34PM | Geekery | #

A question that perturbed me when I lived in Owings Mills was why there were so many roads named "Dolfield". There are three different Dolfield Roads, one South Dolfield Road, and three Dolfield Boulevards. I finally got around to researching the question, and now I share the answer (to the degreee that I have one) with everyone.

This is a map of Dolfield Road, past and present:

The thick, dark blue line is where Dolfield Road once ran. At one point, it connected Painters Mill Road and Deer Park Road.

The thinner, light blue line shows where Dolfield Road, in all its parts, runs today. When Interstate 795 was built, Dolfield Road was cut in two, as happened to many roads during the construation of the Interstate system. At the same time, part of the eastern portion of Dolfield Road was diverted to make room for the ramps from Owings Mills Boulevard to Interstate 795. The section of Dolfield Road south of Owings Mills Boulevard was renamed South Dolfield Road, for reasons I cannot find.

At some point--I cannot tell when, exactly--the chunk of Dolfield Road that extends from present-day Red Run Boulevard to Pleasant Hill Road was closed, though I was unable to find out why. The small section between Red Run Boulevard and Interstate 795 now serves as an access road to a SHA facility, and the rest of the closed section has been turned into a hiking/biking trail (colored in light green on the map). Finally, the section of Dolfield Road just west of Pleasant Hill Road was realigned when Dolfield Boulevard was built.

Here's a map of Dolfield Boulevard:

At some point in the past, the county planned to build a road named Dolfield Boulevard that would connect Owings Mills New Town with the eastern area of Reisterstown, both areas that were slated for new development. The road was planned to have two lanes in each direction with a raised median dividing the road. The dark purple line on the above map shows where it was to have gone, as far as I can tell. I'm not sure whether the original plans included anything further north, though it might have made sense to connect it to Owings Mills Boulevard.

Unfortunately for the county, a number of people living in the proposed path of the new road put up sufficient objections that the project stalled. Portions along Reisterstown Road and Academy Avenue were built and, in both cases, partly divided and partly not--the light purple lines are the divided sections and the pink lines are the undivided sections. Furthermore, the southern end of Dolfield Boulevard was built without incident as part of the development of Owings Mills New Town, though part of Pleasant Hill Road was rechristened Dolfield Boulevard but left undivided.

From what I can tell, the county never entirely abandoned its plans for Dolfield Boulevard, but they seems to be shelved indefinitely. Separately from those plans, they are currently considering adding an interchange between Dolfield Boulevard and Interstate 795 (which would have been really convenient for me if it had been in place when I lived there).

So there are three sections of Dolfield Road (and a South Dolfield Road) because Interstate 795 and other factors caused a former road to be split into pieces, and there are three sections of Dolfield Boulevard because that's as far as the county has gotten on the plans for a new road. This ends today's episode of "I'm a geek and will research anything that catches my fancy".

Wed, 21 Sep 2011

Knife Meme

11:55AM | Geekery | #

This is a thing, apparently: "Take knife from pocket, post pic on the internet."

Myerchin Lightknife My main knife, and the one that best matches the intent of the meme, I think, is a Myerchin Lightknife L377P, which they apparently don't sell anymore. It's a very nice, light, compact knife, with a multipurpose half-straight half-serrated blade; a marlinspike, which is useful for knot- and ropework; and a small red LED, which is nice for seeing things without ruining your night vision. I use the marlinspike a lot less these days, because I don't do as much decorative knotwork, but it's still occasionally useful.

The knife was a Christmas gift from my dad and stepmother a number of years ago. (Possibly 2004 or so.) I'd been thinking about replacing my at-the-time pocket knife, something from Spyderco, with a lighter, more compact knife, when I read a Cool Tools review of the Myerchin L377P. It sounded like it would do everything I wanted and more, so I made a Christmas request and then got what I asked for.

Brookstone Credit Card Multi-tool The Myerchin isn't the only knife that meets the letter of the meme's wording, though. I also have a credit-card-shaped multitool from Brookstone in my wallet. I use it mostly for its tweezers, but its pen, needle, scissors, and knife blade are also occasionally useful.

Victorinox CyberTool A knife I also usually have on me is my Victorinox Swiss Army CyberTool. Technically, it lives in my laptop bag, not my pocket, but I usually have the bag with me. It's very useful for dealing with all sorts of small-scale hardware, because it's got the screwdriver with interchangeable phillips, flat, and torx bits, plus the small pliers (which also include a wire cutter).

This was a birthday gift from my parents back in 2000 or so. It replaced my previous everyday knife, a Swiss Army Automobile Special, which had also been a birthday gift from my parents. Victorinox had just come out with the CyberTool and I'd been working for a couple of years in a hardware and software tech support position, so I let my parents know I was really interested in the new knife. More than ten years later, it's still serving me well.

CRKT Rollock The last knife that I usually have on me (also in my laptop bag) is a Columbia River Knife & Tool Rollock, which doesn't appear to be for sale anymore. It's got an interesting opening mechanism; instead of unfolding, the blade flips up a little and slides out of the body, locking into place at its fullest extent. It's mostly in my laptop bag because it's very slim, so it doesn't take up much extra room and it's handy if I've forgotten my Myerchin somewhere. Because it's got a completely straight blade, it's less of a pain to sharpen than the Myerchin, so it's usually the sharpest knife I have on me, which sometimes means I use it instead of the Myerchin even if I have both with me.

This one was a gift from my youngest sister at her wedding as thanks for being a groomsman and playing trumpet in the wedding.

Mon, 02 May 2011

The MTA and GTFS

8:50PM | Geekery | #

Just today I found about GTFS, a standardized data format from Google for describing public transit schedules. I subsequently discovered that the MTA has a GTFS feed.

Back when I didn't have a car, I used the MTA daily and had a set of scripts to extract their schedule data from the website and import it into trainsched on my Palm Pilot. I asked them for more structured data at one point but was rebuffed, so I stuck with my scripts that had to be updated every time they changed their schedule pages.

I don't have a Palm Pilot any more (and trainsched doesn't even run on the PalmOS emulator on my WebOS phone), but I thought it would be nice to update my old scripts to use the new GTFS feeds. This should even work with other GTFS feeds; there's nothing really MTA-specific going on.

The updated scripts are here: GTFS to trainsched converter.

Fri, 02 Jan 2009

Backup Implementation

1:03PM | Geekery | #

I previously considered different backup schemes. Writing that entry crystallized my basic ideas about what I was going to do about the backups. I then proceeded to implement them. This entry is a detailed description of what I did.

Backup Overview

I ended up buying two 80 GB hard drives and a ThermalTake BlacX external enclosure. The overall plan is to do backups to one of the drives on a daily, automated basis, and the other on a periodic, maybe monthly basis. Most of the time, the periodic backup drive will live at my office and will serve as my offsite backup.

I want to have a backup history in the same way that a tape rotation scheme would give me. That way, if I don't notice there's something wrong with a file for a time, there's still a good chance I can retrieve it. I also want things stored space-efficiently, so backing up unchanged files doesn't take up additional space. This is accomplished pretty easily with rsync; I do one full backup, and then subsequent backups use rsync's --link-dest option pointing to the most recent complete backup; any files that haven't changed are simply hardlinked together, so the two directory entries point to the same physical location on the disk.

For the backup history, I decided to go with a variant of the Towers of Hanoi backup rotation. Instead of using a preset number of pegs, as I would have to do if I were using physical tapes, I can just calculate the numbers on the fly, effectively behaving as if I had an infinite number of tapes. This rotation gives me roughly exponential backoff for my history; I can look at backups from yesterday, two days ago, four days ago, eight days ago, and so on.

Finally, I decided to encrypt the drives. This lets me cart them around with confidence that if they get lost or stolen, anyone who ends up with them won't have my personal files. I used LUKS to encrypt the disks, and used both a file on my computer and a password as keys. The file makes it easier to mount the drives without manual intervention (important if my desktop reboots), while the password lets me get at the data if the key file isn't available (important if the main disk dies and I need my backups).

Backup Procedure

Set up fstab.

The first thing I did was to set up /etc/fstab for the disks. Since I only have one enclosure, I have to swap disks, so only one will ever be connected at the same time. Thus, I mount whichever's in at the moment on /backups. Likewise, I will associate each one with the dm-crypt name of "backups", so the device will be /dev/mapper/backups. Thus, I added the following line to /etc/fstab:

/dev/mapper/backups  /backups  auto  defaults  0  0

Set up rsync filters.

I'm using rsync to do the backups, but in some cases I don't want everything in the source directories to be backed up. Thus, I tell rsync to look at a filter file for each directory so it knows what to skip and what to keep. rsync will go through the filters for each file or directory it considers, and will take the first action that matches. If nothing matched, the file is copied. If a directory is ignored, none of the files in that directory are considered at all, so I had to include a few explicit directory chains.

In /var, I only want to back up a few things, so the final pattern ignores anything that isn't explicitly included.

+ /backups
+ /backups/**
+ /lib
+ /lib/bzr
+ /lib/bzr/**
+ /lib/svn
+ /lib/svn/**
- *

For my home directory, I include everything, with a few exceptions. For instance, most of my music directory can be reripped from CD if needed, so I don't need to take up space backing up those files. On the other hand, I have some files that I either purchased online or downloaded and wouldn't be able to easily replace if they were lost, so I do back them up. Here's an excerpt from my home filter file:

+ /movies/Star_Wars_Holiday_Special
+ /movies/Star_Wars_Holiday_Special/**
- /movies

+ /music
+ /music/Bonerama
+ /music/Bonerama/Bringing_It_Home
+ /music/Bonerama/Bringing_It_Home/**
+ /music/Jonathan_Coulton
+ /music/Jonathan_Coulton/Thing_a_Week_Three
+ /music/Jonathan_Coulton/Thing_a_Week_Three/03-Code_Monkey*
+ /music/Nine_Inch_Nails
+ /music/Nine_Inch_Nails/Ghosts_I-IV
+ /music/Nine_Inch_Nails/Ghosts_I-IV/**
+ /music/Nine_Inch_Nails/The_Slip
+ /music/Nine_Inch_Nails/The_Slip/**
+ /music/Obelix
+ /music/Obelix/**
+ /music/Solo_String_Project
+ /music/Solo_String_Project/**
- /music/**

- /tmp

Initialize disks.

I wrote a script to initialize the disks for me: init-backup-disk. It takes two parameters: the name of the device for the backup disk, and the file to use as a key for the partition. If the key file doesn't exist, it will be created.

After a few sanity checks, the script starts doing things. It starts by checking that the disk is good with badblocks. If it encounters any errors, it stops there and the drive needs to be sent in for warranty replacement. Following that, it goes into the most time-consuming part of the initialization: writing random data to the disk. (badblocks already wrote random data, but its PRNG is somewhat simplistic; /dev/urandom is a much better source of pseudo-random numbers.) Without this step, it would be obvious which bits of the disk had encrypted data on them. I use pv to give a progress meter and time estimate. On my computer, badblocks took a little over two hours and /dev/urandom took about eight hours for each 80GB disk.

# Check and randomize disk.
badblocks -b 512 -s -w -t random -v $disk || exit 2
</dev/urandom pv -s $(fdisk -l $disk |
    perl -nle 'm{^Disk '${disk}': [0-9.]+ [KMGT]B, (\d+) bytes$} and print $1') |
  dd bs=512 conv=sync,noerror of=$disk

The next step is to format the encrypted partition. I use sfdisk to create a single partition that spans the entire drive, followed by cryptsetup to do the format. I explicitly specify the cipher in order to use ESSIV, which makes certain attacks more difficult. The --batch-mode option keeps it from asking for confirmation before writing. The second call to sfdisk just tells the kernel to reread the disk's partitions so it will pick up the UUID that cryptsetup created.

# Add and format the LUKS partition.
echo , | sfdisk --Linux $disk
cryptsetup luksFormat --cipher aes-cbc-essiv:sha256 --batch-mode ${disk}1 $keyfile
sfdisk -R $disk; sleep 5

Next, I open, format, and mount the partition. JFS is the filesystem that's been nicest to me, of all the journaling filesystems I've tried. (In the future, it might be nice to use ZFS for the backup disks--I'd get better detection of disk errors, at least--but I don't think it would play entirely well with running over an encrypted volume, and they haven't integrated encryption into ZFS yet, as far as I can tell.)

# Open LUKS partition, format and mount the encrypted volume.
cryptsetup --key-file $keyfile luksOpen ${disk}1 backups
mkfs -t jfs -q /dev/mapper/backups
mount /backups

Now I run the initial backup. Each backup version is just a separate numbered directory in the partition, so the first one is '1'. I'm backing up /etc, some of /var, my and my wife's home directories, and any databases I have. My wife's stuff isn't directly backed up here because she's on a different computer; I have to initiate her backup from there. The script, in its first requirement for user interaction, will wait until I tell it that's done.

# Do the initial backup.
mkdir -vp /backups/1/{etc,var,phil,postgresql,mysql,rivana}
chmod a-r /backups/1
chown rivana /backups/1/rivana
chown postgres /backups/1/postgresql
rsync -avP --filter 'merge /etc/backups/etc-filter' /etc/ /backups/1/etc/
rsync -avP --filter 'merge /etc/backups/var-filter' /var/ /backups/1/var/
rsync -avP --filter 'merge /etc/backups/phil-filter' /home/phil/ /backups/1/phil/
su -c 'pg_dumpall -v >/backups/1/postgresql/dump' postgres
mysqldump -v --all-databases >/backups/1/mysql/dump
echo -n "Hit enter when rivana is backed up.  "
read foo

Now that the backup is done, the script unmounts and deactivates the partition.

# Deactivate the encrypted volume.
umount /backups
cryptsetup luksClose backups

And I get prompted for the password that will unlock the partition if the key file isn't available.

# Add user password.
cryptsetup --key-file $keyfile --verify-passphrase luksAddKey ${disk}1

Finally, the script displays the UUID for the partition, which is needed for later use.

# Display the partition's UUID.
echo -n 'UUID: '
cryptsetup luksUUID ${disk}1

Set up crypttab.

Debian, at least, has an /etc/crypttab file that lists encrypted partitions to be enabled at boot time. I put the onsite backup disk in there so it'll be automatically mounted if the computer reboots. This plus a backup cronjob make the backup process completely automated.

backups  /dev/disk/by-uuid/<onsite UUID>  <key file>  luks

Do local backups.

I have a simple script to do the daily backups: perform-backup. It's basically the same as the initial backup, but with the --link-dest option as I mentioned previously.

last_num=$(ls -t /backups | head -1)
((num=$last_num+1))

mkdir -p /backups/$num/{etc,var,phil,postgresql,mysql,rivana}
chown rivana /backups/$num/rivana
chmod a-r /backups/$num
rsync -a --filter 'merge /etc/backups/etc-filter' --link-dest=/backups/$last_num/etc /etc/ /backups/$num/etc/
rsync -a --filter 'merge /etc/backups/var-filter' --link-dest=/backups/$last_num/var /var/ /backups/$num/var/
rsync -a --filter 'merge /etc/backups/phil-filter' --link-dest=/backups/$last_num/phil /home/phil/ /backups/$num/phil/
chown postgres /backups/$num/postgresql
su -c "pg_dumpall >/backups/$num/postgresql/dump" postgres
mysqldump --all-databases >/backups/$num/mysql/dump

Do Becca's backup

My wife has her own computer but, fortunately, simpler backup requirements. I have ssh public key authentication set up so she can ssh to my computer without a password, which makes the backups work properly in an automated fashion.

The initial backup is a simple rsync one-liner.

rsync -avP ~/ mithrandir:/backups/1/rivana/

Subsequent backups are a simple script:

#!/bin/sh

num=$(ssh mithrandir ls -t /backups | head -1)
last_num=$(ssh mithrandir ls -t /backups | head -2 | tail -1)

rsync -a --link-dest=/backups/${last_num}/rivana ${HOME}/ mithrandir:/backups/${num}/rivana/

Backup rotation.

The Towers of Hanoi rotation is effected by a script that goes through and deletes any directories that don't need to be there: purge-backups. I won't quote it here because I don't think it's all that interesting. It just finds the largest power of two less than or equal to the current number and then works its way down from that to enumerate all of the directories, deleting everything else.

Offsite backups.

Every so often I'll bring the offsite disk home and back up to it. The script for that is insert-offsite-backup. It unmounts the onsite disk, waits for me to insert the offsite disk, runs a backup, unmounts the offsite disk, waits for me to reinsert the onsite disk, then remounts that disk. It needs to be told what my offsite UUID is, but it picks up all the other settings from /etc/crypttab.

The backup takes about half an hour, so I have ample time to manually run the backup script on Becca's computer.

Phil! Gold

Back to main page.