Discover more from hrbrmstr's Daily Drop

A digest of all the interesting data, packages, blogs and papers covering lots of programming languages, CLI utilities, cybersecurity, data visualization, data science, web-scraping and more!

2022-05-06.01

Moar CLI Data Wrangling; The Laws of Simplicity; The gif Gallery

boB Rudis

May 6, 2022

jApologies for two missed days. This is my penultimate week at my current employer and Wednesday was the day the establishment announced it, so it was more than a tad stressful. Yesterday was rather selfish as it was the perfect weather here in southern Maine for a massively long road cycle, which ate up the time usually devoted to these missives. Rest assured there will be five newsletters this week come hook or crook!

Moar CLI Data Wrangling

A large percentage of my computing life has been and still is at the command line as I’m nigh obsessed with the power contained in that thin (or blocky) blinking cursor. One challenge that still remains is how to turn the output of various commands into something that resembles data. Sure, many new CLI tools have delimited or JSON output as an option, but most of the old, reliable ones still do not, and many of us have spent years accumulating *sh functions/scripts to handle the output. While a previous newsletter edition showed one tool that is frighteningly good at turning CLI output into "data", one can never have too many tools. With that in mind, today's CLI installment features jc ("JSON Convert"), a "CLI tool and python library that converts the output of popular command-line tools and file-types to JSON or Dictionaries. This allows piping of output to tools like jq and simplifying automation scripts."

It, too, has an astonishing number of parsers:

--acpi: acpi command parser
--airport: airport -I command parser
--airport-s: airport -s command parser
--arp: arp command parser
--asciitable: ASCII and Unicode table parser
--asciitable-m: multi-line ASCII and Unicode table parser
--blkid: blkid command parser
--cksum: cksum and sum command parser
--crontab: crontab command and file parser
--crontab-u: crontab file parser with user support
--csv: CSV file parser
--csv-s: CSV file streaming parser
--date: date command parser
--df: df command parser
--dig: dig command parser
--dir: dir command parser
--dmidecode: dmidecode command parser
--dpkg-l: dpkg -l command parser
--du: du command parser
--env: env command parser
--file: file command parser
--finger: finger command parser
--free: free command parser
--fstab: /etc/fstab file parser
--git-log: git log command parser
--group: /etc/group file parser
--gshadow: /etc/gshadow file parser
--hash: hash command parser
--hashsum: hashsum command parser (md5sum, shasum, etc.)
--hciconfig: hciconfig command parser
--history: history command parser
--hosts: /etc/hosts file parser
--id: id command parser
--ifconfig: ifconfig command parser
--ini: INI file parser
--iostat: iostat command parser
--iostat-s: iostat command streaming parser
--iptables: iptables command parser
--iw-scan: iw dev [device] scan command parser
--jar-manifest: MANIFEST.MF file parser
--jobs: jobs command parser
--kv: Key/Value file parser
--last: last and lastb command parser
--ls: ls command parser
--ls-s: ls command streaming parser
--lsblk: lsblk command parser
--lsmod: lsmod command parser
--lsof: lsof command parser
--lsusb: lsusb command parser
--mount: mount command parser
--mpstat: mpstat command parser
--mpstat-s: mpstat command streaming parser
--netstat: netstat command parser
--nmcli: nmcli command parser
--ntpq: ntpq -p command parser
--passwd: /etc/passwd file parser
--pidstat: pidstat command parser
--pidstat-s: pidstat command streaming parser
--ping: ping and ping6 command parser
--ping-s: ping and ping6 command streaming parser
--pip-list: pip list command parser
--pip-show: pip show command parser
--ps: ps command parser
--route: route command parser
--rpm-qi: rpm -qi command parser
--rsync: rsync command parser
--rsync-s: rsync command streaming parser
--sfdisk: sfdisk command parser
--shadow: /etc/shadow file parser
--ss: ss command parser
--stat: stat command parser
--stat-s: stat command streaming parser
--sysctl: sysctl command parser
--systemctl: systemctl command parser
--systemctl-lj: systemctl list-jobs command parser
--systemctl-ls: systemctl list-sockets command parser
--systemctl-luf: systemctl list-unit-files command parser
--systeminfo: systeminfo command parser
--time: /usr/bin/time command parser
--timedatectl: timedatectl status command parser
--tracepath: tracepath and tracepath6 command parser
--traceroute: traceroute and traceroute6 command parser
--ufw: ufw status command parser
--ufw-appinfo: ufw app info [application] command parser
--uname: uname -a command parser
--update-alt-gs: update-alternatives --get-selections command parser
--update-alt-q: update-alternatives --query command parser
--upower: upower command parser
--uptime: uptime command parser
--vmstat: vmstat command parser
--vmstat-s: vmstat command streaming parser
--w: w command parser
--wc: wc command parser
--who: who command parser
--xml: XML file parser
--xrandr: xrandr command parser
--yaml: YAML file parser
--zipinfo: zipinfo command parser

You can install it with a simple pip3 install jc (you can brew install jc on macOS with Homebrew), but if you're keen to "try it before you buy it", there's a web version — which may take a few seconds to warm up, being a Heroku app — where you can paste command output and get your conversion.

One CLI tool I like to use jc with is dig DNS lookup tool. Try running this command to see just how many DNS TXT records Optimizely has:

$ dig optimizely.com TXT | jc --dig | jq -r '.[].answer[].data' # everybody has 'jq' installed, right?

Alternatively, you can just see it in this gist.

Remember that this is a Python module as well, and that means R folks can use it to process CLI output in a more straightforward way, such as:

library(reticulate)

jc <- import("jc")

system("dig optimizely.com TXT", intern = TRUE) |> 
  paste0(collapse = "\n") |> 
  jc$parse("dig", data = _) -> res

sapply(res[[1]]$answer, `[[`, "data") |> 
  writeLines()

which has the same output you'll find in the linked gist.

The Laws of Simplicity

We delve back into the world of design principles today with a look at The Laws of Simplicity by John Maeda.

The book was written at the time of the iPod, but the principles within it hold up to today, and the principles aren't just about product time. The book's subtitle is "Design, Technology, Business, Life", and the the essays withing are, indeed, applicable almost everywhere.

I'll let John describe how you can use this book:

The ten Laws outlined in the body of this book are generally independent of each other and can be used together or alone. There are three flavors of simplicity discussed here, where the successive set of three Laws (1 to 3, 4 to 6, and 7 to 9) correspond to increasingly complicated conditions of simplicity: basic, intermediate, and deep. Of the three clusters, basic simplicity (1 to 3) is immediately applicable to thinking about the design of a product or the layout of your living room. On the other hand, intermediate simplicity (4 to 6) is more subtle in meaning, and deep simplicity (7 to 9) ventures into thoughts that are still ripening on the vine. If you wish to save time (in accordance with the third Law of time), I suggest you start with basic simplicity (1 to 3) and then skip to the tenth Law of the one which sums up the entire set.
Each section is a collection of micro-essays that cluster around the main topic presented. Rarely do I have answers, but instead I have a lot of questions just like you. Every Law begins with an icon of my design that represents the basic concepts I present. The images are not a literal explanation of the con- tents, but may help you to better appreciate each of the Laws. There is also associated Web content at lawsofsimplicity.com where you can download the artwork as desktop patterns in case that will help to motivate you.

These are the ten "laws":

reduce: The simplest way to achieve simplicity is through thoughtful reduction. 2 organize Organization makes a system of many appear fewer.
time: Savings in time feel like simplicity.
learn: Knowledge makes everything simpler.
differences: Simplicity and complexity need each other.
context: What lies in the periphery of simplicity is definitely not peripheral.
emotion: More emotions are better than less.
trust: In simplicity we trust.
failure: Some things can never be made simple.
the one: Simplicity is about subtracting the obvious, and adding the meaningful.

I am especially fond of the essays, guidance, and examples in "Law 2" (organize) and check back in with it every so often to remind myself both how to engage in said process and the absolute need to do so if I'm going to remain sane.

It's a quick read, the website is a great companion, and the resources will, hopefully, be as useful to y'all as they have been for me.

The gif Gallery

This is just to give you a Friday distraction that doubles as a reminder of that the bad ol' days of "Geocities" pages were. You start off in one "room" and scan through the page of gifs to find the next one.

Sound up!; Don't be afraid to click on things; and, take a digital walk down animated memory lane.

Click the gif to get started!

FIN

Look for another newsletter drop later today! ☮