2022-06-23.01

Secrets; q; Hurl

Secrets

Photo by the blowup on Unsplash

Full-disclosure: When 1Password went all stupid lazy and moved from native OS apps to wrapping entire web browsers to present simple user-interfaces, I ditched them. In truth, I also ditched them due to how much they seem to hate macOS systems administrators, forcing them to do some tricks to get the software installed via automation. But many of you use 1P (and I have to now as well) so it makes sense to cover some new functionality.

Regardless of my own opines, 1Password does keep your secrets fairly safe (there are issues with all "normal user"-facing password managers) and just rolled out a new feature for users of Visual Studio Code that will help us keep our secrets, well, secret.

1Password (1P) for Visual Studio Code "provides you with a set of tools to integrate your development workflow with 1Password.". You need to have 1P v8 + the 1P CLI (v2.4.0+) installed, and enable biometric unlock for 1P CLI.

To save a new item in 1P from VS Code, just make 1+ selections, open the Command Palette, and choose "1Password: Save in 1Password". 1P will ask for some metadata, attempt to recognize the field type (i.e. "credit card") and will replace the thing is supposed to be secret with a "secret reference" in the document. Said references have the following structure:

Source: 1Password

This new 1P functionality came from their acquisition of SecretHub, so I trust it a bit more than I would if it were the standard 1P developers cranking a new feature out.

Integrating authenticated vault-backed secrets into your workflows is far better than relying on the safety of plaintext environment files, so I will definitely be giving this a go (my personal keeper of secrets is Bitwarden, but that's not what is used at $WORK). Stay tuned for a (hopefully) positive review.

q

Photo by Alexander Andrews on Unsplash

If two- and three-character arcanely named command-line utilities weren't confusing enough for you, meet q [GH]. For the sake of legibility and not having to mark each use of the letter "q" with inline code markup, i'll refer to it as "Q" from now on.

Q's purpose is "to bring SQL expressive power to the Linux command line by providing easy access to text as actual data, and allowing direct access to multi-file sqlite3 databases."

I sort of let the cat out of the bag (that poor feline) on Twitter yesterday that I'd be covering this today, so feel free to skip this section if you've already played with the tool.

As the description says, feed Q a CSV and do SQL queries on it, straight from the command-line. Have a bunch of sqlite3 databases with similar structure, query them all in one go with Q.

One brilliant feature is the (optional) use of caching for CSV files. If enabled, the dataset will be cached in custom .qsql files and the speed of future queries will be blazingly fast (there's a great table showing that in the GH repo).

I also like that one can do something like:

ps -ef |q -H \
  "SELECT UID, COUNT(*) cnt FROM - GROUP BY UID ORDER BY cnt DESC LIMIT 3"

and get the top three user IDs based on the number of processes they've launched. Said overall querying of command-line tool output idiom will work over most well-behaved programs.

As JD notes, it's a far cry from my beloved awk days:

JD Long @CMastication

@hrbrmstr I hadn’t seen q, but reading that page I was like “this feel so much less painful than sed/awk for some uses” … turns out that’s the point :)

Check out the massive update Q received last year and get ready to replace the q key on your keyboard next year from all the use it's about to get.

Hurl

Photo by Austrian National Library on Unsplash

Hurl [GH] is "a [Rust-backed] command line tool that runs HTTP requests defined in a simple plain text format."

With it, you can perform requests, capture values and evaluate queries on headers and body response both as one-off data/page grabs or for entire HTTP sessions.

The GitHub repo and project domain are ridiculously well-documented, but I'll provide a snapshot of some ways to use it here.

The ideal use-case for hurl is for doing what it says on the tin: testing HTTP endpoints. Ideally, you'd use hurl to test expected API responses given a set of parameters, or that the bit of HTML you expected to be on a page after the results of a UX tune-up sprint is, in fact, present. You can chain all these together and get a slick HTML report of all the test steps (including the actual test scripts).

Or, you can abuse hurl to your own evil wishes.

Say you wanted to know when CISA's KEV changed. You could use hurl with a test configuration like so:

GET https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json

# If we get a 200 response code

HTTP/* 200

[Asserts]
jsonpath "$.dateReleased" != "{{stored_date}}" # detect a new release

and, run it like so:

hurl --variable stored_date="2022-06-14T16:43:02.2817Z" cisa-kev.hurl

and (assuming you're on a decent operating system) test $? for a non-zero exit status. If so, then go get the KEV JSON and start processing a new release. I manually put the date in the variable parameter, but you can store parameters in files, so just update that parameter file with the new date each run.

Another abuse of the tool could be to capture the headers and HTML content of a bunch of pages in a HAR-like JSON format. So, if you have a "test" configuration like this:

GET https://www.r-project.org
HTTP/* 200
[Captures]
my_body: body

GET https://cran.r-project.org/mirrors.html
HTTP/* 200
[Captures]
my_body: body

GET https://www.r-project.org/about.html
HTTP/* 200
[Captures]
my_body: body

You'll get some JSON back with all the header responses and the entire HTML of all the pages that's super easy to process (i.e. build a low-rent content scraper).

Hurl is fast, well-designed, and dead simple to use, and I suspect it might be a great tool to add to your arsenal.

FIN

Make sure to check in on someone you haven't heard from in a while today. We live in some pretty tough times. ☮

#q #hurl #1password