Drop #203 (2023-02-20): Synchronicity
Discover more from hrbrmstr's Daily Drop
Drop #203 (2023-02-20): Synchronicity
Maestral; osync; Unison
Programming note: It turns out that last week's debilitation was due to everyone's least favourite spike protein. In other news, the ECMWF forecasts suggest we're going to be hit with a wallop of a ❄️ storm later this week. Between the continued recovery and the pending snopocalypse, I'm once more working to get a few of these Drops into the scheduler to ensure no disruption in delivery.
Be it known, dear reader, that I said Bye, Bye, Bye to the first pass at this edition's title and intro to avoid Driving Myself (and you) Crazy. So, Here We Go, in today's Digital Get Down Drop, where we take a look at how to ensure your files stay N Sync, so they're never Gone: Girlfriend, This I Promise You1.
Maestral
I can fondly remember, back in 2008, when the world (and I) went bonkers over Dropbox (ah, those were simpler times). Dropbox links were everywhere (most without, sadly, the dl=1 query parameter being set).
When the Snowden leak happened, I dropped them like a hot potato and moved on, but the world did not. While I rarely see Dropbox links shared today, this may be due to Dropbox functionality being baked into some other clients, such as Facebook, which makes it “easy” for members to share Dropbox files with each other.
Another possible reason is that both Microsoft and Apple worked hard to unseat Dropbox's sync dominance by beefing up their own, built-in cloud storage/sync offerings.
Yet another reason could be that many of the 700 million users are just active or passive users of the Dropbox API since — as Apple's iCloud continually demonstrates — “sync is hard”.
I removed “critical” files from Dropbox, but some shared folders are still in use by just enough other folks that I kept my account intact. On the rare occasion I need to fiddle with some files or folders, I tend to use the web interface, as I have no desire to infect my system with their client.
It turns out, I am not alone in my local-app-install wariness.
Maestral [GH] is an alternative Dropbox client for macOS and Linux, which used the Dropbox API to perform the same core functions as Dropbox-proper. It provides solid command line tools, supports gitignore patterns to exclude local files from syncing and allows syncing multiple Dropbox accounts. The motivation for Maestral came from Dropbox's (temporary) decision to make life more difficult for some Linux users.
Maestral syncs files/folders just like the Dropbox client. Well, not just like it, since the official client can take advantage of hidden APIs to only transfer changed chunks of files. Maestral only has the official API to work with, so it will be a tad slower — perhaps much slower, for large files/folders — in operation.
Along with having auditable open source (i.e., you can fully know what it is doing, unlike Dropbox's own client), the crew behind Maestral have some other selling points:
Smaller App bundle than the official macOS Dropbox app (40 MB vs. 420 MB).
Universal binary, which runs natively on Apple Silicon.
Less memory usage: 100 MB for a medium-sized Dropbox on macOS vs. 500 MB. The memory usage will depend on the size of your synced Dropbox folder, and can be further reduced when running the daemon without a GUI.
Does not count towards the three devices limit for basic Dropbox accounts.
It makes use of many well-known Python packages:
Communication between sync daemon and frontends uses Pyro5.
The command line interface is built with click and uses interactive prompts by survey.
The macOS GUI is built using toga and the macOS app bundle is built using briefcase, both part of the beeware project for writing cross-platform Python applications.
Credential storage uses system keychains via keyring.
watchdog allows us to receive local file system events.
If you still use Dropbox, or want to maximize the use of your old, free account(s), Maestal should definitely be on your radar.
osync
osync [GH] is “a two-way file sync script running on bash Linux, BSD, Android, MacOSX, Cygwin, MSYS2, Win10 bash and virtually any system supporting bash. File synchronization is bidirectional, and can be run manually, as scheduled tasks, or triggered on file changes in daemon mode. It is a command line tool rsync wrapper with many additional features baked in.”
Because it is based on the venerable rsync, It has many inherent capabilities, but has some crunchy goodness of its own, including:
Fault tolerance with resume scenarios
Email alerts
Logging facility
Soft deletion and multiple backups handling
Before / after command execution
Time control
Directory monitoring
Running on schedule or as daemon
Batch runner for multiple sync tasks with rerun option for failed sync tasks
It is very conductor / worker based in that you can sync local → local or local → remote (i.e., “local” is always the conductor).
Apart from tons of other guardrails and features, one special one is that osync has been designed to not delete any data, but rather make backups of conflicting files or soft deletes.
There is plenty of accessible documentation, and I'm considering switching from the utility in the next section to osync in the near future, so there may be a 'splainin Drop for that later this year to let you know how that goes.
Unison
Unison is a bidirectional file synchronization tool that runs on Unix-like operating systems (including Linux, macOS, and more arcane operating systems) and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host). Said collections can be modified separately, and then brought up to date by propagating the changes in each replica to the other.
Rather than just copy/paste the well-honed-over-20-years-README, I'll let you dig into that for all the features and just provide some extra commentary.
Unison is fully bidirectional and allows the user to detect and reconcile conflicts when both copies of a file have been changed since the last synchronization. It does not run continuously in the background, so you control the sync ops by manual or scheduled invocation.
A nit I used to have about Unison was the need to keep all Unison installations at the same version number. That is no longer the case, and immensely simplifies the upgrade process.
To give you an idea of just how long Unison has been around, you can peruse the 2005 article on it — one I may have read back in the day, to get going with the installation.
If you do read the README, I suspect you'll pick up on the edge it definitely cuts. We're in a time when some FOSS maintainers are (rightfully) disgruntled regarding the relationship freeloading service providers, businesses, and normal users have with the code creators.
I'm not going to write a tome (in this issue) on my opine regarding this situation. Suffice it to say, I'm concerned enough about Unison to consider said switch to osync, and taking stock of other tools and libraries I depend on to see if I should be concerned about any of those as well.
Still, Unison is mature, battle tested, versatile, and incredibly useful. If you haven't jumped into the personal sync fray, Unison might be a good starting point, despite my aforementioned concerns.
FIN
Don’t forget, guest contributor Noam Ross covered SyncThing back in Drop 192.
Many thanks for all the get-well wishes over the past week! ☮
I fully acknowledge that I deserve all the hate mail you drop my way, but I’m not sure puns from “The Police” would have been any better. https://www.eonline.com/news/1132643/ranking-the-20-best-nsync-songs-ever