I have a lot of data from a lot of years: about 5TB with around 8 million files. It’s very redundant, lots of copies of the same stuff. Many of the files are tiny, e.g., 100,000 1-2KB files in Maildir.

Most of the data are now on medium-sized external disks (2-8TB each) accessed via USB or Thunderbolt. It’s time to get everything onto a small set of usable disks (I’ve tried this before and I didn’t get very far).

One of the things that slows me down is that no matter how I set up the copy (cp, rsync, Finder), after a few minutes, the copy slows to a crawl. These are reasonably fast disks on USB3.0 or Thunderbolt2.0. The r/w speed on the disk should be around 150MB/s, and the connection is 5Gb/s, but I’d often see read speeds around 0.5MB/s. Ooof. You’re not going to move a terabyte at that speed.

And now I think I know what’s happening: the directories get disorganized. I am frustrated that I can’t figure out what this means, but I discovered that after running DiskWarrior on the offending drive, it’s now copying at 50-100MB/s (I’m using iostat to watch the r/w speeds). A big, big win.

Of course, APFS makes this useful knowledge nearly obsolete. Ah, the story of my life, learning useful stuff just as it becomes a kind of vintage affectation.

Gotta go, I’m going to write some shell scripts to make my terminal prompt look cool.

You thought I was kidding?