2022-09-20.01

Spyglass; HTTP Toolkit; chartr

Keen observers will note the slight name change to "Drop" vs "Finds". "Finds" always felt awkward, and "Drop" feels much more appropriate. I'll get proper domains setup once I get caught back up at work/home.

Spyglass

Photo by Nika Benedictova on Unsplash

Internet content and search is busted. Said facts are two reasons I continue to be a paid subscriber to Kagi. Their search results continue to be above par, and their radical company transparency is refreshing.

But, one can do even better than Kagi if one is willing to roll up sleeves and deploy some infrastructure (don't worry, you can do so on your laptop).

Meet Spyglass [GH]: a "personal search engine, crawl & index websites/files you want with a simple set of rules". This is their pitch:

Spyglass is a solution to the following common issues when searching the web:

• Do you add terms such as reddit or wiki to your searches to narrow it down?

• Do you get frustrated with overzealous autocorrect on your search terms?

• Do you get frustrated with the terrible search some wikis/sites offer?

• Do you scroll past dozens of SEO spam pages to find the recipe/review/blog post you were looking for?

• Do you skip over a full-page of ads before getting to your actual search results?

• Do you have private websites / data / documents that you'd like to search through?

Kagi + Postlight can take care of some of those usability issues, but cannot help you in the local resource search department.

The kids today tell me y'all like to watch vs read, so here's their video pitch:

Like Kagi, Spyglass has the concept of "lenses", which is just a small JSON file with some metadata and a list of links + rules about you want Spyglass to crawl regularly. For example, you could add R Weekly itself and all the source blogs and content links it has (they're just JSON files in GH) and have a highly curated set of resources for searching for all things RStats.

You do not have to create lenses manually, as Spyglass has more batteries included in the form of netrunner, which is both a lens-creator assistant and a tool for making web archives from site content (so you can be your own Wayback Machine and preserve content history). There's also a built-in list of community lenses to choose from to get you started. With the xkcd lens installed, it's super quick to find a relevant comic for any given topic, such as "statistics":

It's all open source (Rust, ofc), so you can dive in and contribute as well as curate your own collection.

HTTP Toolkit

If you work with HTTP websites, APIs, or data, you've likely had need to fire up some person-in-the-middle proxy — such as Burpsuite, mitmproxy, or others — to inspect HTTP sessions to either learn about resources or debug something.

Well, there's a new tool in town, aiming to usurp your favorite tool. HTTP Toolkit [GH] is both a fully open source (literally the entire company's codebase) and optionally paid product that has done an outstanding job re-imagining the interception experience.

The section header image shows an example of inspecting the background data call to one of the resources mentioned in yesterday's newsletter. Rather than have to set up special TLS certificates and reconfigure local browsers, HTTP Toolkit ships with the ability to launch pre-configured custom browsers/terminals. You can point anything to the proxy, though, so it fully behaves like other tools.

Unlike Burpsuite and mitmproxy, you have to drop some coin for advanced features (or deploy their free stack), such as saving to HTTP archives (HARs) and making mocks for HTTP APIs.

It's definitely a tool I'll keep around, but I've caught a mean case of subscription overload and am working on reducing what I pay for vs adding to the money drain. So, I'll keep using my other tools for the more advanced use cases. Organizations with dev shops may want to consider the premium options, since they are rather good and a bit cheaper than some alternatives.

chartr

This last drop is pretty quick, as it's just asking you to hit up chartr and subscribe to their stories/newsletters. They're solid data storytellers, and regularly drop informative (+ usually, fun) "data-driven insights into business, tech, entertainment and society".

The section header image is from their August 26th edition, which includes said story on Peloton's fiscal woes.

The chartr folks are also good about identifying sponsored content (folks gotta make a livin' somehow), something other sites are increasingly doing so poorly.

Disclaimer: I receive nothing from anyone for this mention. I just 💙 the resource and suspect you will too.

FIN

I'll close today's edition with a bit of shameless self-promotion. If you have need to make maps of the internet, I've built a new Hilbert heatmap generator in Rust you may want to poke at. That GH repo link also has pre-built binaries in the releases section, so you can try it out without having a local Rust toolchain handy. ☮