Drop #212 (2023-03-03): Weekend Project Edition

An Apt Endeavor

Programming note: The Drop is one away from 500 subscribers! Many thanks to all readers for providing the inspiration to keep this daily endeavour cranking!

While I've never done a survey of you dashing Drop readers, I suspect — since you read, sub, and support this newsletter — that the majority of you are very hands-on tech folks. It's also very likely, then, that you fairly regularly find yourselves at a Linux command prompt — probably a Debian/Ubuntu box or in a Debian/Ubuntu VM/container. If those assumptions are accurate, then you most certainly have installed a Linux package or two.

The arcane incantations of:

sudo apt update
sudo apt install $PACKAGE_NAME

cause libraries and binaries to magically appear on your system, and can even manifest entire new services out of spare electrons.

So, while you've used packages, have you ever created one?

OK. I see some of you nodding your heads in affirmation (you should really consider covering up that tiny camera when not in active use).

I suspect the affirmed percentage to be in the small double-digits, though. If I'm correct, then it's time to pull back the curtain on what these "packages" are, how to make them yourself (and why you should consider doing so), and also how to host your own, mini-package repository.

While I usually wait til the end to give you your mission, it should come as no surprise that said mission will be to perform these tasks, but I'll provide some specifics before the end.

I'm including a bunch of primary source information from my experiences building and maintaining (local) Debian packages, but will also be adding in many links. Normally, you might be able to get away with not checking out every single link I drop, but I'd encourage you to visit and bookmark each of the ones I do include, since I'm hoping to avoid writing an entire book (and, you could use a break from my blatherings).

So, What Are These "Packages", Anyway?

Photo by Mildlee on Unsplash

We're focused on Debian/Ubuntu (b/c if you're using something else I would bet a considerable amount of someone else's money that you have done all the package dances), so the package format we're focusing on is — unsurprisingly — Debian.

A Debian package is a binary package format used by the repeatedly aforementioned Linux distributions to distribute and install software. These packages contain all the files and metadata necessary to install and manage a particular piece of software (which could mean "library") on a Debian-based system. One common way to install and manage these packages is via the "Advanced Packaging Tool" (APT) or the dpkg CLI.

Running apt install … (successfully) causes a bunch of things to happen:

• The apt package manager checks the local system package cache to see if the package is already installed. If the package is not already installed or is past its prime, apt will download the package information from the configured package sources.

• Then, apt will retrieve package metadata from one or more of those package sources, typically from a Debian package repository that either came pre-configured with the system, or one you added. This metadata includes package names, versions, dependencies, and other information needed to install the package.

• Next, apt will figure out which version of the package to install based on the which version you've selected, and the dependencies specified in the package metadata for that particular version. If there are dependencies, apt will do the same dance for those packages as well.

• All packages identified for installation will be downloaded and stored in the local apt cache located in the /var/cache/apt/archives/ directory.

• Once downloaded, the dependencies and the selected package will be installed and configured. "Installation" just means extracting files from the package archive and copying them to their respective locations on the file system.

Using dpkg does something similar, but you're usually running that command on a package you've manually downloaded.

So, What's Inside The Box?

Photo by Christopher Bill on Unsplash

A Debian package consists of several components:

• Control files: These are just bits of metadata about the package (e.g., package name, version, dependencies, maintainer, description, etc.). They are stored in a directory called "DEBIAN" at the root of the package.

• Data files: These files contain the actual components — libraries, binaries, configuration files, data — that the package provides. They are stored in various directories depending on their purpose and are generally compressed using gzip, but can be shrunk by a different compression algorithm.

• Scripts: These are optional scripts that can be included in the package to perform pre-installation, post-installation, pre-removal, or post-removal tasks.

Let's say you want to make an awesome package that lets you say 👋🏼 to the 🌍. The package directory structure would look like this:

awesome-package
├── DEBIAN
│   └── control
└── usr
    └── bin
        └── hello-world

The control file has a ton of options, and the options are different for binary and source packages. This Drop is focused on binary ones (since they are simpler, and make more sense — IMO — for personal package repos). There are rules about these sorts of things you should be familiar with, and source packages are different beasts.

The one in this example is pretty minimal:

Package: awesome-package
Version: 0.1.0
Section: custom
Priority: optional
Architecture: all
Essential: no
Maintainer: person@example.com
Description: Hello, world!

hello-world is just an executable shell script that prints what you think it does. By the obvious tree structure, it's going to be in /usr/bin. In fact, everything not involved with package installation metadata and functions you provide within awesome-package will be mapped out to directories on the system.

If you're dropping an actual binary you've made, and it has dependencies, you can get help figuring those out and adding them

¹

. You can also declare them manually, which may be necessary if you're just dropping shell scripts that use other binaries that may not be installed on the system.

Making The Actual Package

Photo by Taisiia Shestopal on Unsplash

Make the directory tree in the previous section (perhaps not be as lame as I was and have the script do something more useful than print "hello world"). Please don't hesitate to use real/tweak info in control.

Now, move the directory containing awesome-package and do:

dpkg-deb --build awesome-package

It'll make awesome-package.deb, which you can inspect, since it's just a Debian ar archive:

$ ar t awesome-package.deb
debian-binary
control.tar.zst
data.tar.zst

You should be able to sudo dpkg -i awesome-package.deb and see all your files in the proper target places in your filesystem. If you replicated this toy package, you should also be able to just run hello-world.

Are You Being Served?

Photo by Sasha • Stories on Unsplash

You could end everything there, but wouldn't it be cool to be able to apt install kit you build?

Let's assume you are not a monster and use Nginx as your web server of choice. We'll further assume you've kept all the defaults in place (if you haven't then you'll know how to modify the configs anyway).

Create a new configuration file for your spiffy Debian package repository. This file should be placed in the /etc/nginx/sites-available/ directory. For example, create a file called spiffyrepo with this in it:

server {
    listen 80;
    server_name spiffyrepo.com;

    root /var/www/spiffyrepo;
    autoindex on;

    location / {
        try_files $uri $uri/ =404;
    }
}

You can hang it off of a directory in an existing nginx setup if you like.

The spiffy repo needs a predictable directory structure.

• create a directory for your repository and add a "dists" subdirectory.

• inside the "dists" directory, create subdirectories for each distribution that your repository will support (e.g., "stable," "testing," "unstable").

• inside each distribution directory, create subdirectories for each architecture that your repository will support (e.g., "amd64," "i386").

• finally, inside each architecture directory, create a "main" directory.

Copy your Debian package files to the appropriate "main" directories in the repository structure.

The most crucial step is to generate the package metadata for your spiffy repo. This needs to be done every time you add or update a package in the spiffy repo. The dpkg-scanpackages utility is what you'll use for that.

In each "main" directory, run:

sudo dpkg-scanpackages . /dev/null | gzip -9c > Packages.gz

Provided you've wired up nginx to use the new configuration, you should be able to add:

deb [trusted=yes] http://spiffyrepo.com distribution main

as a package source.

In theory, this should be an HTTPS endpoint, and you should use PGP if you intend to provide a package to others, but those steps are a bit out of scope for a weekend project (IMO).

And, technically, you don't need a web server if you'd rather skip that step. Your source can be a local directory:

deb [trusted=yes] file:/some/local/dir ./

Why Bother?

This is work. So, why do it?

First, you get to see how the sausage is made — which is always a good thing. But, let me see if I can convince you further.

By creating your own packages, you can tailor software installations to your specific needs. You can configure software to use certain options, install specific versions of dependencies, or apply patches to fix bugs or security vulnerabilities.

In the event you'd like to share things with a team or just other folks, creating packages can make distribution easier. Anyone can, then, simply add your package repository to their system's sources list, and they will be able to install and update your software just like any other package.

If you do a bit of customization and tweaking after installing things, you will ensure consistency across different systems by using packages. And, you can totally create packages that work across multiple distributions, architectures, and OS versions. Plus, doing so will fully ensure that all necessary dependencies are included.

By hosting your own, spiffy repo, you can have greater control over the security of the software you distribute. This is especially true if you hit that above link and sign your packages with a GPG key to verify their authenticity, and you can use HTTPS to encrypt communication with your repository.

Finally, you're not limited to hosting just stuff you build. A local package repo can be more efficient than relying on external repositories (and you don't send telemetry to your ISP and the upstream repository).

Your Mission

Photo by Clark Tibbs on Unsplash

Given that tons of folks were in harm's way this week and many more are about to be (thanks, "weather"), I'd posit that just reading this might suffice. It's not fun having to recover from terrible weather events.

If you're somewhere with less atmospheric troubles, then I'd say you can ✅ if you just work through the example in this edition.

Extra points for adding all your "usual suspects" Debian packages you generally have to install in a container or a VM/real system and making a local repo for them, so you may cease ceding telemetry upstream.

But, if there are some scripts and configurations you install regularly on new systems — for me that's a handful of local IoT-ish, weather, personal security scripts, and some data science odds and ends — consider wrapping them in Debian packages and at least keep them on a drive you can attach and install via file:// repos. I think you might be surprised how useful it turns out to be.

FIN

Premium subscribers 🙏: keep a watch out for the weekend Bonus Drop! ☮

1. Some more info on dpks-shlibdeps.