In a Hurry?
You probably don’t need to read this. Unless you want to.
Process, Process, Process
So, how do you solve the problem of testing a product like SuperDuper?
As you might expect, it’s pretty difficult. Of course, you can pin individual cases of UI and copy behavior (there are over 2,000 individual tests against the SuperDuper code base, which is about 250K lines of Swift, Objective-C and C): those are run every time I make a change to the code base…to protect against regression of things that are known.
Theory vs Practice
But this type of testing only gets you so far. I can model various situations that I’m trying to cover, but models just simulate reality (even if the cases are real).
Real backups are more complicated, and quirkier.
They’re often to different drive types. Those drives may be formatted in different ways, have their own layouts, failure modes, fragmentation levels, and connections.
So, not only do I run a few systems of different eras as test harnesses with various drives attached (simultaneously running Time Machine, too), my main development system (currently a first generation Mac studio) has a huge number of different drives attached.
These include (but are not limited to—I also have thumb-drives set up for specific tests, and assorted other SSDs and HDDs):
- 2x OWC Express 1M2 80G, one with an OWC SSD, one with a SanDisk SSD
- 1x LaCie D2 Thunderbolt 2 drive, connected via an Apple TB3/4->TB2 adapter, chained to
- 1x LaCie 2Big Thunderbolt 2 drive (RAID)
- 1x LaCie 2Big Desk/Dock (RAID)
- 1x SanDisk Pro USB-C SSD
- 1x SanDisk Professional TB4 SSD
- 1x SanDisk USB Transport with its own PRO-BLADE
- 1x SanDisk PRO-BLADE Station with 4 PRO-BLADE SSDs (JBOD)
- 1x OWC 4M2, configured as JBOD, with 2xM.2 SSDs
- 1x Synology 2415+ configured as iSCSI via ATTO’s initiator
- 4x SMB mounts from the Synology
Note that some of these drives are connected to an Anker TB5 dock, to make sure that’s covered, too; some are connected to an Apple Studio Display, to get a hub in there; many are daisy chained, so I deal with that, too.
Yeah, that’s a lot of drives (and a significant investment), and doesn’t count the separate Mac systems with their own external storage that are running tests all the time.
Those drives don’t do a lot on their own, of course: SuperDuper is configured to run copies against them, of various types, all day long (and all night).
If there are any failures, I’m notified immediately and can analyze the logs to determine the cause, how it was handled (since failures can happen to users), and whether it was my issue or a problem with hardware.
In addition, while all of the copies are checked separately for accuracy and under-copying, a number of tests also check for idempotency, to ensure there’s no thrashing or incorrect/over copying, and then there are a series of timed tests that make sure there hasn’t been a performance regression.
And on top of that, there are the aforementioned 2,000 automated tests in the full SuperDuper 4 test suite, with more manual tests as well (which I need to figure out how to automate one of these days).
Even So…
But even with all that, I can’t truly replicate every situation that can occur in the field. And even with all that I can screw up and miss things.
That’s where other testers come in. As I’ve mentioned previously, there were a large number of external testers during SuperDuper 4’s development, rolled in over time to ensure a series of “fresh testers” (and test cases) were always encountering the code.
When outside users run into a problem, I have to figure out what happened beyond just “Hey, Dave, last night’s copy failed”. So SuperDuper does extensive logging (if you want to be driven mad, feel free to look at the ocean of logging with log show in Terminal): when a user submits an issue, it includes both SuperDuper’s visible log, and a filtered-for-relevance couple of seconds before and after the failure from the system log, so that I have enough information to analyze the error.
When appropriate, verified problems turn into another test case, to make sure they’re both fixed and won’t break again. And then a test build is sent to the user to ensure the situation is covered in its original setting.
Bugs in the field are pretty rare things (and are often the result of configuration differences that I haven’t created in-house). But given a report, it’s nice to be able to fix them…and prevent them from happening again.
In Sum
In the end, there’s no magic, and there are no shortcuts: copy testing is done by making (and comparing) tons and tons of copies, small and large, under both “random” and “known” situations; UI testing is done by pinning behavior and ensuring things react as expected.
Combine that with (hopefully) proper design, extensive experience, and careful implementation, and you end up with SuperDuper 4 which, despite a number of small updates post-release, is (thankfully) proving to be very reliable in the field…the goal of any design, development, and testing process.