Fri, 13 Dec 2013
Portable Filesystems for Portable Disk Drives
I periodically need to set up a USB hard drive so that its files can be shared between different operating systems. I recently tried to update my practices. This is a record of my findings.
The short version of my conclusions is: If you need good portability between Windows, OSX, and Linux (and you're only writing data from Windows or Linux), use NTFS. If you don't need filesystem metadata like ownership or permissions and your files and disks aren't too large, FAT32 might work for you.
FAT32 is often the filesystem of choice for flash drives and smaller media. Windows, OSX, and Linux all have native support for it. It can span filesystems up to 2TB in its default configuration and up to 16TB or so if you tweak its block size. It cannot support files larger than 4GB, nor does it include support for file ownership and permissions. Its support for filenames longer than eight characters plus a three character extension is something of a hack. It's not case-sensitive (though it does preserve case).
Basically, FAT32's biggest strength is its cross-platform support. In most other areas it falls down when compared to more modern filesystems.
NTFS is a bit nicer than FAT32 in many ways. It supports filesystems up to 16EB in size, and you can fill all of that space with a single file, if you want. (In other words, there's effectively no limit on file size other than the size of the containing filesystem.) Filenames can be up to 255 characters long, just like other modern filesystems. NTFS supports POSIX-compatible file ownership and permissions, hard and soft links, case-sensitivity, and sparse files, all of which which make it a lot more interoperable with Unix than FAT32.
Its main drawback is that it's proprietary and what support exists has been reverse-engineered. Windows supports NTFS, since it came from Microsoft originally, back to Windows 2000 and Windows NT 3.5. Max OSX has had native read-only support since 10.3 (Panther). Linux can read and write NTFS volumes via NTFS-3G, which runs in userspace (via FUSE). NTFS-3G doesn't give tremendous performance when accessing SATA or SAS disks, but modern hardware is more than capable of keeping up with USB-attached disks. (At least for USB 2.0; I haven't done comparisons with USB 3.0 hosts and disks.)
exFAT is an attempt to extend the aging FAT family (FAT12/FAT16/FAT32) to support larger files. Its size limits are somewhat crazy: maximum recommended filesystem size is 512TB (though it can theoretically go up to 64ZB); maximum file size is 127PB. Like FAT32, it does not support file ownership or permissions, hard or soft links, or sparse files, and it's case-preserving but not case-sensitive. Many of the unsupported features aren't completely necessary in a lot of USB drive use-cases, so their absence isn't an immediate strike against it.
The main problem with exFAT is that it's not just proprietary (like NTFS) but patented. It's supported natively from Windows Vista on (and Windows XP can be patched for support) and in OSX 1.6.5 (Snow Leopard) and later, but Linux support is currently very shaky and difficult to distribute because of the patents. Even if Linux support were not a factor, the fact that only relatively new OSes have support for exFAT would disqualify it from consideration in many situations.
UDF is something of a dark horse in this space. It was originally designed as a general-purpose, portable filesystem. It's only really seen adoption, though, as the filesystem used on DVDs and Blu-Ray disks. Because it's used for DVDs, though, it's natively supported by all three of the major desktop OSes: Windows (back to Windows 95), MacOS (back to OS 9), and Linux (back to kernel 2.2).
Like FAT32, UDF can only have 232 blocks in its filesystem, so it only goes up to 2TB filesystems with 512-byte blocks (which is what hard drives would use; the UDF spec requires that the UDF block size match the disk block size). Like NTFS, though, its only limit on file size is the size of the containing filesystem. Also like NTFS, it supports POSIX ownership, permissions, and metadata; ACLs; and hard and soft links, as well as being case-sensitive. It doesn't support sparse files.
It would seem that UDF would be the best choice for a portable filesystem: it's natively supported in practically every OS and it supports most of the features you'd want in a modern filesystem. Unfortunately, OS bugs hamper its usefulness.
Although Linux will read UDF filesystems with blocksizes other than 512 bytes (even though that's outside the official specification), Windows is pickier and will only recognize conforming layouts with 512-byte blocks. That immediately limits Windows to 2TB disks or smaller when using UDF. Also, Windows expects UDF to span the entire disk, with no partitions, so you can't even work around the size limitation by making multiple sub-2TB partitions.
Linux, on the other hand, has had problems with UDF with 512-byte blocks. It handles filesystems with 2KB blocks just fine, but you need to be running relatively new kernels for the 512-byte blocks (required for Windows compatibility) to work. (Two problems I've run into are disks being reported full when they're not and Linux not seeing UDF volume labels.)
The Linux problems have been fixed, but only just this year. Losing compatibility with all older Linux systems knocks out one of the biggest advantages that UDF has over NTFS. In my case, I have RHEL 5 and 6 systems that aren't going to get these fixes for a really long time. (Maybe by RHEL 5's 2017 EOL, assuming RHEL 7 includes all of the fixes.)
There are also the native disk formats for Linux (the ext2/3/4 series) and MacOS (HFS/HFS+). While there's at least some support for accessing them from other systems, that support is generally less robust than the NTFS-3G project's work, so if you're going to require third-party support for cross-platform access you might as well use NTFS.
There are a lot of other filesystems out there. Linux has native support for quite a lot of them. OSX and Windows don't. In cases where installing third-party drivers on OSX or Windows is an accepted requirement, some other filesystem might be a good choice. In my opinion, though, NTFS, FAT32, maybe UDF, and possibly exFAT cover the filesystem portability use-cases pretty well by themselves; it would have to be a very specific set of circumstances for a different filesystem to be a better choice than one of those four.
Tue, 10 Dec 2013
Dottie Gold-Humphreys, October 1999 - 9 December 2013
Less than five months after getting married and moving into an apartment together, my wife, Becca, and I got a pair of cats. They'd been with a friend of ours for about ten years, longer even than she'd known her husband. She and her husband loved them dearly despite their cat allergies, but their two-year-old was turning out to be allergic and was terrorizing the cats (as only a two-year-old can do) to boot, so they were reluctantly looking for a new home for the cats. Thus it was that on January 31st, 2009, Becca and I took Nico and Dottie to our apartment to live with us.
Once the cats got used to the apartment, they settled into distinct roles. Nico was used to getting his way with everything and tended to crowd Dottie out when there was something he wanted. We were always worried about her getting enough food because even though we gave them separate bowls she usually wouldn't eat until he'd finished. Nico would often eat so quickly that he'd immediately regurgitate some of his food, which Dottie would promptly eat. It was a little gross, but we felt oddly comforted by it, because we knew Dottie was actually eating something. (We had Nico on a diet because he was 20 pounds when he should have been about 15. Dottie always seemed a little scrawny at her 7 pounds.)
Nico was always the cat that would greet newcomers to the house and aggressively demand attention, but Dottie was Becca's special baby. Dottie always seemed prim and dainty, especially when compared to the more boisterous Nico.
Becca delights in nicknames for people, cats, cars, and probably other entities too. Nico and Dottie were no exception and went through a plethora of names, including Small Cat, Muffin, Princess, and Sneezy for Dottie. "Princess Baby" was probably the most long-lived nickname for her (as was "Fatty" for Nico).
In February 2011, Nico developed kidney cancer and that April we made the difficult decision that it was time to let him go. Dottie mourned Nico for a time--at least it seemed that way. She was even more withdrawn than usual and I think she missed his presence in some sense or other. Eventually, though, she blossomed in a way that she had never done around him. She became more confident around the house; she hid under things a lot less; and she was a lot more comfortable around other people. For all that we loved Nico, I think Dottie was a lot happier by herself.
Both Dottie and Nico had had renal problems for pretty much their entire lives. When we got them, we continued their diet of kidney-friendly food (high in calories, low in protein) and we had bloodwork run on them periodically to make sure they were doing okay. The bloodwork measured levels of substances that the kidneys should be removing from the blood. For both cats, those levels were always higher than what a cat should normally have, but they never worsened into areas that would be problematic.
Until a couple of months ago. I forget now what particular symptom prompted the vet visit, but while we were there they ran her bloodwork and her creatinine levels had spiked, indicating a decrease in kidney function. At the vet's recommendation, we started administering subcutaneous fluids to help flush out her system and compensate for the lost kidney capacity. We had done this with Nico, and he had been easy to handle. He seemed to enjoy the attention, even though we were sticking him with needles on a regular basis. (One vet tech told us that he was the only cat she'd seen who didn't react at all when she took his temperature rectally.) Dottie was a lot less happy about the fluids. Although she eventually became more accustomed to them, she never liked the process and it took both of us to hold her while we were administering them.
The fluids plus a few pills to also supplement her kidney function held her for a month or so and seemed sustainable; we'd be doing them for the rest of her life, but it seemed likely that they'd be able to sustain her for some time to come while preserving her quality of life.
In November we moved into a new house. I was worried that Dottie would be disoriented by the change and would be unhappy with the new house, especially since there were stairs and she was starting to look a little arthritic in her back legs. She took to the house surprisingly well, however. She seemed happy to go up and down the stairs and explore everything in the place. Unfortunately, we discovered fleas several days after moving in. Most of the house has hardwood floors, but we fogged the entire place (spending several hours with Dottie in the old apartment) and put flea powder down on the carpets. We kept Dottie in our bedroom while the flea powder was down.
After about a week at the new house (and about three weeks ago), Dottie stopped eating. She'd gone through phases of not eating in the past and we had an appetite stimulant on hand for times when it went on for more than a couple of days. Correspondingly, after a couple of days, we gave one to her. It didn't appear to have any effect. We thought she might not have gotten it (sometimes she managed to make us think she took her pills but would spit them out later), so we waited three days (the minimum dose window) and gave her another. When that one also had no effect, we started syringe-feeding her and we took her to the vet as soon as we could. The vet found that she was very anemic--another side effect of the kidney dysfunction--and prescribed an injection that would fill in for the red-blood-cell-creation function of the kidneys.
After about a week on the injection, Dottie started recovering. She started becoming more energetic and she started eating again. We were administering a lot of treatments at that point, but it seemed likely that once she recovered more we could stop some of them and go back to just the handful of maintenance treatments.
In the last couple of days, though, her appetite dropped off again. We attributed this to one of the supplemental (and bad-tasting) drugs we were giving her, so we decided to just stop that one. It was for nausea relief and she was already on a couple other things with the same effects. Sunday night she seemed to be wheezing a bit but was otherwise about the same as she had been, so we just figured we'd mention the wheezing when we took in for her scheduled vet visit on Wednesday.
On Monday morning Becca and I woke up, showered, got dressed, gave Dottie her pills, syringe-fed her a bit, wished her goodbye, and went to work. When we came home that evening, she was dead.
Our vet is of the opinion that, given everything that we know, she probably went peacefully. That gives a little consolation, as does the fact that we'd been deliberately spending time with her--carrying her out to the couch to sit with us as we ate, talked, and watched TV--but we never really said goodbye, because we really didn't expect this so soon. We thought the treatments we were giving her would hold her for at least a few weeks, by which point we'd be able to tell whether she'd be able to recover from the most immediately-threatening symptoms like the anemia. We figured we'd be in store for a difficult decision about what would be best for her at that point, but we really expected her to live until then.
So this is the best I can do for now: Goodbye, Dottie. You were our princess baby and you were a special and unique cat. We will miss you and you will remain special to us for the rest of our lives.