atproto/bsky Deep Dive; Ethernet Is Old; Stock Dystopia

Let me begin with a quick update to note that the situation which required a hiatus the other week has had a near-term positive resolution. It’s elder healthcare related, which can be a rough subject for some folks, so I’ll exclude the details from here and re-point folks to my spouse’s update blog for any details.

To get back into the swing of things, I’ve got three diverse sections in the hopes that you’ll find at least one useful/appealing.

atproto/bsky Deep Dive

The Authenticated Transfer (AT) Protocol (blog; GH) is the core technology behind Bluesky, the (eventually) decentralized social network initiative we mentioned earlier this week. It aims to create a federated and open-source protocol for social media platforms. The creators of it believe it is/will be the best way to “power the next generation of social applications, offering a new foundation for public conversation and social networking”. I’ll be using “AT” as a shorthand for it in the remainder of this section.

Talking about an evolving protocol underlying a service that is attempting to “force” organic growth — which means many readers do not have an account on it yet and that the API spec is subject to wanton change — may seem odd, but hear me out before jumping to the next sections.

Coming from a position where I knew, liked, and used the Twitter API enough to write a mini-book on it, my initial reaction to the AT protocol was “gosh, that’s chatty/verbose”. However, thanks to, of all things, the AT Protocol SDK for Python, I have an appreciation for both the design goals of the API and the implementation of that Python module. There are enough folks with enough financial interest in this Fediverse alternative to ensure that Bluesky/AT will be around for some time. As such, it will eventually be important to grok how it works, if only to be able to mine/orchestrate it the way you may have done with the birdsite during its heyday.

There are four basic “pillars” of AT:

The first is “identity”. Users (until I’m more convinced of Bluesky’s intentions I’ll use this terrible word instead of “humans” since I have no idea whether they value us or not) are identified by domain names on the AT Protocol. I initially only joined Bluesky (as noted earlier this week) to claim my “handle”, which was hrbrmrmstr.bsky.social. I followed the directions on the site to turn that into hrbrmstr.dev. You’ll generally see an @ in front of these names, but the bare domain name itself is the actual identifier, along with the URI form of it: at://hrbrmstr.dev. DNS ownership seems to be how they want to validate these items (if you’re not just using a Bluesky handle). They map the human friendly URLs to “decentralized identifiers” (DIDs). You can see my mappings via both the methods below:

$ dig +short _atproto.hrbrmstr.dev TXT
$ curl --silent "https://hrbrmstr.dev/.well-known/atproto-did"

So, ultimately, I’m at://did:plc%253Ahgyzg2hn6zxpqokmp5c2xrdo on Bluesky’s server. We’re already at a far more complex place than Twitter ever was when it comes to platform identifies.

The next pillar is “data repositories”. User data is exchanged in signed data repositories. These repositories are collections of records that include posts, comments, likes, follows, media blobs, etc. They’re stored in the fancy-sounding Merkle Search Trees, which is a state-based “Conflict-free Replicated Data Type” (CRDT). It boils down to being a data structure that is easily and safely replicated across multiple computers in a network, allowing applications to update any replica independently, concurrently, and without coordinating with other replicas.

While that may sound complex, the nodes of this tree map to AT URI paths. My “root” is the aforementioned at://hrbrmstr.dev, my post collection is at://hrbrmstr.dev/app.bsky.feed.post, and an individual post is of the form at://hrbrmstr.dev/app.bsky.feed.post/xxxxxxxxx. As you poke at various items returned via the AT API, you’ll see these URI structures quite a bit.

We won’t get into all the crazy new acronyms that underpin AT in this post, as most readers have likely not seen them before and this section is already pretty huge. When you start digging into AT yourself, keep this link to InterPlanetary Linked Data (IPLD) handy, as it will help quite a bit. If you recognize that “InerPlanetary” term it is likely in the context of the “InterPlanetary File System” (IPFS). There was quite a bit of IPFS hype a few years ago, and it seems to be gaining traction with attackers more than anywhere else.

The penultimate pillar is “federation”. The AT Protocol syncs the repositories in a federated networking model. Federation ideally ensures the network is convenient to use and reliably available. Repository data is synchronized between servers over standard web technologies (HTTP and WebSockets).

While you can run your own AT server, there is no real federation at the time of this post, so we’ll move on to the last pillar which is made up of AT’s “core services”:

• Personal Data Servers (PDS): in this case, that’s just Bluesky’s repository of posts, likes, follows, etc.

• Big Graph Services (BGS): this handles “big-world” networking, which involves crawling the network, gathering as much data as it can, and outputting it in one big stream for other services to use. It can be considered analogous to a firehose provider or a super-powered relay node (similar to Twitter’s firehose but way more chatty given that you need to fetch and decode way more items given the Merkle tree structure and encodings used).

• App Views (basically what we “see” when we use Bluesky as non-developers)

I highly recommended taking a look at the aforementioned AT Protocol SDK for Python. It has many examples, and you can talk to the “firehose” without authentication, so you can actually see what these theoretical structures look like. Or, you can just watch it stream on by you.

One reason the firehose is public is that Bluesky doesn’t (for now) want to be the one solely in charge of how you prioritize what content you wish to see. That is, they will and do supply some basic “algorithmic feeds” (one of which is “reverse chronological”). But, they also let anyone build their own algorithmic feeds(GH). Flipboard has one, and I link to it since they cover a bit of “why” and “how”.

If you do have an account, I demonstrate use of that python module (via R’s {reticulate}) over here. With said account, you can also use SkyFeed as an alternative web client and algorithmic feed generator. Speaking of alternative clients, I really like this Raycast Bluesky extension for quick access to the service.

You can follow along with Bluesky’s growth via Bluesky Statistics, and I’ll let folks know when Bluesky graces me with a set of invites (every two weeks is what I’ve been told).

As noted in the post from earlier this week, I am still quite wary of Bluesky’s real intentions. That doesn’t mean I, or we, should ignore the tech that underpins it.

I also guess this means I have to re-up the “21 Recipes” for AT.

Ethernet Is Old

Ethernet, a transformative technology invented by Bob Metcalfe, celebrated its 50th anniversary in May of this year. This technology has revolutionized connectivity and enabled the free flow of data across the globe. Ethernet’s success can be attributed to its simple design, adaptability, and focus on interoperability. It has evolved to support multiple speeds and media, catering to various industries and network types, while maintaining strict standards to ensure seamless performance and high-quality user experiences.

Ethernet’s resilience lies in its adaptability across multiple interconnects, options, and applications. It is not constrained by cable or connection, allowing it to be used over a wide range of copper or fiber cables in homes, offices, factory floors, automobiles, and as the backbone of cellular phone networks. This adaptability is a result of the combined efforts of engineers worldwide who work together to maintain and innovate the underlying standards.

There is an entire Voices of Ethernet series that tells the story of the technology’s journey from Xerox PARC in 1973 to the foundation technology of the Internet today. It’s worth a watch/listen.

I remember bridging a major Fortune 500’s Ethernet-based internet DMZ to the then, core token ring network and also having to run both token ring and Ethernet cabling back in the day. I’ll gladly trade stripping coax and crimping BNC connectors for crimping twisted pair cables any day.

Stock Dystopia

The folks over at Stocky Photos embarked on the mission of applying AI to generate images as real as possible. In the pursuit of the perfect image, we ended up creating a bank of over 20,000 images that will leave you in awe1.

The section headers in today’s Drop are all from their HD collection, which is free to use (though they’ll gladly accept your coin, if you feel so inclined, for larger image versions).

It’s a cool (albeit, somewhat dystopian at times) alternative to Unsplash or painstakingly trying to prompt one of the generative image sites.

FIN

I may be re-using the Bluesky section sooner than later this coming week. When I do, I will add a fourth section to that Drop, so you have three fresh sections to peruse. ☮