2022-05-12.01

Hello, Observable!; Scraping TikTok; Warp's Commands.dev

Hello, Observable!

The universe of data visualization ("datavis" from now on to save me some keystrokes) owes a major debt of gratitude to Mike Bostock (@mbostock), inventor of D3, a JavaScript library for manipulating documents based on data. It is almost impossible to go a day on the internet without encountering some datavis that uses this foundational tooklit. Mike also made bl.ocks to help make it easier for folks to experiment with D3 and show off their m@d ski11z. While D3 + bl.ocks was a useful and powerful combination, it was a bit cumbersome (to me) to have to work with a GitHub gist context.

To help even more individuals and teams harness the power of interactive datavis, Mike started Observable, a collaborative notebook platform that comes with batteries included to make it fairly straightforward to dive into ad hoc data analysis or produce complete reports.

Wrapping your head around Observable may take some doing, depending on your mental model of how notebooks and datavis. Luckily, some talented folks, such as Scott Franz are here to help you grok this new, reactive-based environment, with their Hello, Observable! notebook, which is also in a non-Observable platform context.

Scott's introduction walks you through the fundamentals of the Observable platform with many (many) examples, and also shows how to bring Observable with you to your own hosting environment.

One of the best parts of walkthroughs like Scott's is the fact that you can look at the code that was used to make it right in the notebook and riff off of it for your own creations.

If you've been Observable-curious, this resource should help scratch that itch.

Scraping TikTok

While I am not on TikTok, it is similar to D3 (see above) in that it is also nigh impossible to go a day on the internet without running into one of the short, viral posts from that platform.

TikTok content can be both silly and serious, and it is a large, global social network, so there are good reasons to look at it with the eyes of a data analyst, whether it be mining trends or keeping an eye out for more nefarious uses of the platform.

While TikTok has some developer resources, you're somewhat on your own if you want to use TikTok itself as a dataset. Well, you were until Andrew Nord built the TikTok Scraper & Downloader. With it, you can:

• Save post metadata to the JSON/CSV files

• Download media with and without the watermark and save to the ZIP file

• Download single video without the watermark from the CLI

• Sign URL to make custom request to the TikTok API

• Extract metadata from the User, Hashtag and Single Video pages

• Save previous progress and download only new videos that weren't downloaded before. This feature only works from the CLI and only if download flag is on.

• View and manage previously downloaded posts history in the CLI

• Scrape and download user, hashtag, music feeds and single videos specified in the file in batch mode

From personal, cursory use, it definitely does what it says on the tin.

I found this resource via Johanna Wild's Bellingcat post, where I also learned about their Investigate TikTok Like A Pro! guide (which I likely glossed over back in 2020 b/c "TikTok"), and the TikTok hashtag analysis toolset, based on Andrew's work.

Now, all we need is for someone to make a TikTok showing how to analyze TikTok.

Warp’s Commands.dev

Back in April, I introduced the new Warp terminal "platform". I've been using it daily since then, and even have an interview (not the job kind) with the Warp team coming up in June.

The Warp team recently debuted Commands.dev, a templated and searchable catalog of popular terminal commands. The website serves up the Warp workflows commands catalog available to the Warp community.

Now, most hardcore terminal denizens make use of shell functions, scripts, and aliases to encapsulate useful idioms and command presets. As the Warp team notes, getting aliases and functions to a productive state requires an upfront investment that’s justifiable for devs who spend most of their workday in the terminal, but less so for beginners and casual users.

Anyone can contribute a workflow, and if this does truly become a broader commmunity resource, I can envision wiring it up to VS Code, Sublime Text, iTerm, Alfred, and even RStudio.

FIN

I am in disbelief that I have made eighteen "TikTok" references in this post. O_O

If you make said video, created or found some cool Observable notebooks, or have some Warp tips or crazy cool commands, def drop a link to them in the comments. ☮

---