ShellCheck ✅; Dagger 🗡; CI/CD Design Principles
The drive back from seeing #2.1 was long, and Portland (ME) airport has one runway down, so @mrshrbrmstr's plane back from Reno to see #1 was also super delayed, thus there was no newsletter yesterday. So…you get two today! (i.e., there will be a 2022-04-22.02 in the afternoon.)
I have a confession to make: up until a decade ago, I was a linter slacker.
What's a "linter", you ask? Simply put, a linter is a tool that that helps you improve your code.
The concept of a linter was defined and implemented [direct PDF] by Stephen Johnson back in 1978 at Bell Labs. The linked paper describes a command-line utility
lint which was written to process C code, but there are many linters out there for virtually every programming language. Here's how Johnson defined a linter:
"Lint is a command which examines C source programs, detecting a number of bugs and obscurities. It enforces the type rules of C more strictly than the C compilers. It may also be used to enforce a number of portability restrictions involved in moving programs between different machines and/or operating systems. Another option detects a number of wasteful, or error prone, constructions which nevertheless are, strictly speaking, legal."
Three things have made me both a fan and user of linters. Two of those things are the RStudio and Xcode integrated development environments (IDEs). These GUI environments (IMO) a great job providing real-time lint notices and suggestions. Sure, my favorite editor,
vi (feel free to take a shot at that in the comments, emacs-folk) has ways of providing the same functionality, but I wasn't in the habit of running linters prior to Xcode/RStudio, so I've never really dug in to linting in
The third is having to create and share code/projects in teams. Linters help enforce coding standards, which is incredibly important when working on any sized project with one or more coding partners. It is fine to have a personal style when coding, but a team really needs to agree upon a set of rules to each member can expect to see source files in a certain format and using certain idioms, so everyone can Get Stuff Done™ as quickly and error-free as possible.
One area where I am still somewhat neglectful (as far as linters are concerned) is using them on shell scripts, specifically
zsh scripts. Since I made a commitment ("resolutions" are so pre-pandemic) to improving in many areas this year, I figured it was time to level up my shell scripts and start linting ones that go into (personal) production. The shell script linter I've found most useful is
ShellCheck (online version).
ShellCheck is a linter for shell scripts (which you likely figured out from the previous paragraph). It is written in Haskell (just like the incredibly awesome
pandoc!), in numerous operating system package repositories, works in almost every programming IDE (or plain editor), and is tailor made for CI/CD pipelines. (We'll also be discussing CI/CD in the next section).
The goals of
ShellCheck are (in the team's own words) to point out:
typical beginner's syntax issues that cause a shell to give cryptic error messages.
typical intermediate level semantic problems that cause a shell to behave strangely and counter-intuitively.
subtle caveats, corner cases and pitfalls that may cause an advanced user's otherwise working script to fail under future circumstances.
ShellCheck has absolutely made my
zsh scripts cleaner and safer. Truth be told, I may have made a majority of the mistakes in the
ShellCheck gallery of bad code,
and the tool has caught my forgetfulness about which shell I'm coding for (which is "a thing" mostly due to Apple).
To go all "full disclosure", thanks to
ShellCheck I'm also learning quite a bit about those two shell languages that I did not know before. When a tool can do its job and also make you smarter, you know it has to be good.
I heartily encourage even the most casual of shell scripters to give
ShellCheck a go.
As Michael Corleone said, "Keep your friends close, and your CI/CD pipelines closer."
The CI/CD link in the first section was meant to give a high level overview. This CI/CD 'splainer by GitHub does a much better job filling in the details. Here's how they define CI and CD²:
Continuous integration (CI) automatically builds, tests, and integrates code changes within a shared repository; then
Continuous delivery (CD) automatically delivers code changes to production-ready environments for approval; or
Continuous deployment (CD) automatically deploys code changes to customers directly.
And, here's an example workflow from the same article:
While GitHub has a vested interest in CI/CD due to their GitHub Actions service.
It has been a painful couple of years for such services.Both Travis CI and Codecov have suffered breaches (Travis has had more than one), and it is far too easy to misconfigure most of them (just like to glory days of Amazon S3 buckets). These incidents can shake the trust of organizations and individuals who use them with their private (and, for some, public) repos.
Setting safety and resilience aside, I'm not sure it's wise to rely on "the cloud" for everything, especially with the storage capacity and computing power available even on an individual's workstation. With containers and virtual environments, one can have a robust, cloud-like setup that will work even when your internet connection does not, and can help secrets (such as API keys), secret.
Dagger is a portable "devkit" for CI/CD pipelines developed by the creators of Docker. You can use any language you like for the CI/CD actions, and the entire workflow is defined in the CUE language, to avoid the problems and shortcomings of using a terrible format such YAML (which seems to be the "it" format for most cloud-based CI/CD services).
While I've just espoused the virtues of local CI/CD, Dagger also plays nicely in all the cloud CI/CD environments, so you can rest assured your codebase (and organization) can still be breached if you choose to stay in the 🌩.
I've only recently started tinkering with Dagger, and will blog about the home server setup in a few weeks after kicking the tyres a bit more.
CI/CD Design Principles
If you are new to CI/CD (or are a veteran who wants to see if your CI/CD mental model needs a tune up), these CI/CD design principles:
First Principle of CI/CD: Segregation for Stakeholder Responsibility
Second Principle of CI/CD: Risk Reduction
Third Principle of CI/CD: Short Feedback Loop
and two CI/CD context references:
Environment Implementations in CI/CD
Tools for CI/CD
are succinct, accessible, and should provide both food for thought (for veterans) and foundations beginners can use to get up to speed quickly.
Remember to be on the lookout for 2022-04-22.02 this afternoon, and be kind in the comments (especially if you
emacs folk are going to be sassing me) if you choose to engage. ☮