Drop #268 (2023-05-24): Multi-threaded Edition v0.6.0

How To Start A Go Project In 2023; Nonlinear Dot Plots; Dropbox HTTP/3 Deep Dive

Programming note: Your humble hrbrmstr is actually taking a break from Friday-to-Monday, so no WPE and no weekend Bonus Drop. #2.1 is actually coming to pampa's abode! Also: I'll be scheduling a M-F drops for the first week of June since we'll all be traipsing across Acadia National Park on our annual fam retreat Down East. Hit me up if you'll be in the area!

"Multi-threaded 'splainer" for folks new to the Drop.

How To Start A Go Project In 2023

Photo by Maxime Horlaville on Unsplash

This recent post title: “How to start a Go project in 2023” on Ben E. C. Boyter's blog kind of says it all. It's a comprehensive guide to kick-starting a modern Golang project, but also includes tons of other useful tidbits.

Here's the layout:

• Install / Setup

• Starting a Project

• Learning Go

• Searching

• Building / Installing

• Linting / Static Analysis / Security Scanning

• Profiling

• Unit Testing

• Integration Testing

• Community

• Multiple Main Entry Points

• OS-Specific Code

• Docker

• Useful Tools/Packages

In the last section, I learned about a few new tools and packages that I'm “borrowing” to drop right in here (please hit up Ben's post so he gets some 👀 too).

As per my stated policy of telling you when I use ChatGPT (et al.) for anything ever since our AI overlords took over this year, I used the following prompt to get the links from the post into markdown for use here:

Please scrape https://boyter.org/posts/how-to-start-go-project-2023/#useful-toolspackages and convert the unordered lists in the "Tools" and "Packages" section to markdown, ensuring you also make the HTML links markdown links, and put the whole result into a code block.

• gow - A watch mode command. Run it with your arguments, and it will hot recompile under the hood for you. Very useful for HTTP development. For example gow -e=go,html,css run . will watch for file changes to any Go, HTML or CSS file, and if found rerun the go run . command, giving you a hot reload.

• hyperfine - A command line benchmarking tool. Think of it as a replacement for running time multiple times and averaging the results.

• dcd - Duplicate code detector. My own project (so I hesitate to add it) but you can run it to find examples of duplicated code in a project. Especially useful when looking to refactor.

• gotestsum - Alternate test runner. Gives different test outputs with format options that you might prefer. Can produce junit output format to work with CI/CD systems.

• JSON to Go generator - Goland can do this for you too, but this tool works pretty well for pasting in JSON and getting back a struct that can hold it.

• gofumpt - A stricter formatter than gofmt. I, personally, have not used this, but had it suggested to me.

• Static type checker and lint enforcer – Apply this from day one of your project, and it will save you a lot of cleanup. The suggestions it provides are always good, and it helps when asked the usual sort of audit questions. Hook it into your CI/CD pipeline as a deployment gate for best results.

• gitleaks - SAST tool to find and identify checked in secrets, passwords and such. Again works well to help pass audit questions.

• BFG Repo-Cleaner - Easiest way to remove large binaries or checked in secrets from a git repository. Very useful for fixing issues gitleaks finds.

• gjson - A quick way to get JSON values from a JSON document. Rather than deserialize into a struct, you can just get the value you want. Especially useful for integration tests when running against your own endpoints. Since importing the struct you used to marshal out won’t catch regressions, using this is a decent way to write non brittle tests. g := gjson.Get(resp.Body, "response.someValue").

• zerolog - Structured JSON logs. A fast way to get structured logs that make sense. My preference is to use this with a unique code date | md5 | cut -c 1-8 allowing you to track down errors to the exact line log.Error().Str(common.UniqueCode, "9822f401").Err(err).Msg(""). Add in context information to get the invoke details too, giving you some level of observability through your logs.

• mux - A replacement for the standard Go router, which is a bit janky.

If you are at all Golang-curious, or want to enhance any nascent Golang l33t $k1llz, I highly suggest bookmarking the URL and referring to poring over it.

P.S. ChatGPT's new web surfing powers are pretty darned cool/handy.

Nonlinear Dot Plots

A new R CRAN package caught my eye the other day, and I'm taking the "multi-threaded” opportunity to drop it on y'all.

Back in 2017, a pretty cool paper introduced a novel visualization technique known as “nonlinear dot plots”. This technique enables effective visualization of larger datasets with high dynamic ranges, all thanks to the unique aspect of scaling dot sizes according to the number of data points each column possesses.

Unlike the traditional dot plots, which tend to limit their usage to smaller datasets due to their constant dot sizes, nonlinear dot plots intelligently scale dot sizes. This means columns with higher data points get smaller dots, and thus, they're adept at illustrating larger datasets.

The paper's authors proposed an innovative algorithm — “two sweep passes” — to determine dot positions and sizes. This is a modified version of Wilkinson's original dot plot algo:

These plots are adept at showcasing outliers, as the dots in sparse columns remain considerably large. What's more, they make the counting of data points an easier task compared to histograms.

The R package uses base plot, but the algorithms are open, so aspiring {ggplot2} Geom/Stat wranglers can go to town on porting the code; I'll also likely try my hand at an Observable version.

Dropbox HTTP/3 Deep Dive

It turns out Dropbox is alive and (I guess) well. I make said assertion based on a recent blog post by Dropbox tech researchers: “Investigating the impact of HTTP/3 on network latency for search”.

These clever folks investigated using HTTP/3 to reduce network latency for their search feature. Currently, network latency accounts for about half of the total latency for search results. They conducted an experiment where they compared HTTP/3 and HTTP/2 by making parallel requests to a test API endpoint. They found that while HTTP/3 only reduced latency by around 5-15ms for most users, it decreased latency by up to 146ms for the slowest 5% of users, especially those in Asia. Given these sizable improvements for high-latency users, they plan to implement HTTP/3 in production to benefit search and other features that require fast network performance.

It’s well-worth the read to get more familiar with HTTP/3 and for how to perform a solid IT experiment.

Aside: I made the unamusing dig about Dropbox relevancy primarily out of nostalgia. I remember when it was “cool tech” and part of my own daily drivers. I don't use it at all anymore, but I'm glad for their second profitable year and ~18 million users!

FIN

How on earth is it Wednesday already? ☮