Thank god for ZFS.
I was messing around with Lidarr. After realizing that my "Artist - Album Name" folder structure was fundamentally incompatible I abandoned my attempt to import those files. But despite removing "Media/Music" as a root folder they were still present in the "Unmatched Files" section. There thousands of files that were never going to be imported and if that screen was to be of any use at all it would be need to be cleared. There was a a "Select All" and a "Delete" button among others but no explicit warning that real files would be removed rather than simply entries in a database of "potential files to be managed".
Normally programs are wildly inconsistent in their use of "remove" and "delete" and seem to simply expect you to trust them. Despite some web searches I couldn't confirm what would actually happen when I hit the "Delete" button - I did the reasonable thing and picked something that I would probably remove on my own anyway... and nothing happened. The folder was still there and I could see the cover photo preview (you might see where this is going).
I selected the rest and deleted them from Lidarr. It took a while but everything still looked okay until I went into one of the actual folders: no MP3/FLAC. If there were any extra files (lyrics, images, etc.) they were preserved along with the folder itself but the audio files were ALL gone.
This has happened to me before but I wasn't ready: when I was setting up my Jellyfin server a few years ago I set the transcode directory (temp files only) to be the root of my media server by mistake and promptly moved it elsewhere. What I couldn't have known is that Jellyfin removes all the files in the transcode folder before it moves it somewhere else. So by selecting my entire Media collection and then selecting somewhere more reasonable I had started a background deletion process of EVERYTHING. Many, many hours of finding music, tagging, organizing, etc. all gone along with TV and Movies all equally organized. It was a tough pill to swallow - I had spent so long thinking about and testing different backup solutions that I never got around to making a full 1:1 copy and just leaving it alone on a shelf.
Lesson learned three times over. In this case the solution was trivial: go back to an hourly snapshot (/$POOL/$DATASET/.zfs/$SNAPSHOT)from earlier in the evening and then rsync -hPa (I always remember this as hectoPascal) to recursively copy back all the data exactly as it was to the Music folder.
I've been burned by programs being brazen in their file deletions so many times that I've given up trusting them at all. With ZFS it's a mild inconvenience at worst and amusing at best. If I didn't notice tonight I would have noticed within a few days or a week and my local snapshots would cover that with ease. I now have enough storage to handle several YEARS worth of archives so while it would certainly be messier to recover from I would still be able to. Doing a "rollback" doesn't make much sense unless you're using it on very specific datasets (such as an OS in part or in whole) where ALL the data can change. Since my Media dataset is shared by other non-Music data this wasn't a good option. Copying from .zfs is an excellent solution.