Caching Quality Of Life Improvements

This edition is primarily about one recent project of mine, but there are a few other resource links after the jump in case you care not about writing an API caching proxy in Go.

NIST maintains the National Vulnerability Database, which is a repository of standards-based vulnerability management data represented using the Security Content Automation Protocol (SCAP).

They’ve been threatening to shut down the giant vulnerability feed files for a while now, shunting folks to use their enhanced API endpoints. It’s a fancy and robust API, but — as a data scientist — those large file dumps are preferred, since we data folks tend to really dislike having to make individual API calls for single records.

Another bummer with NVD’s new approach is the enforcement of a ~1-second delay (longer if not auth’d) between subsequent, authenticated requests. I grok why they do this. Skeezy security vendors and clueless researchers would just end up using NVD as a janky “database” if such restrictions were not in place.

This additional API friction was just annoying enough for my personal projects that I ended up building a tiny, caching REST API “proxy” in Go for the https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=:cveid: that I host on my personal tailnet.

Today’s Bonus Drop provides a link to the project and includes a light walkthrough of the project. I went with the Knowledge Drop for this Bonus Drop as this tiny project has significantly improved my quality of life (when it comes to personal vulnerability info hacking). As such, it may be a useful framework for others to implement something similar, since I know we cybersecurity folk are not alone when it comes to having to deal with APIs vs “just give me the whole thing”.

Here’s the GitLab project for my nvdprox project.

For those not interested in today’s topic:

Here’s an:

• EPIC deep dive on “CSV Sniffing” by the folks @ DuckDB

• oddly useful article from Time on the best inventions of 2023

• useful 2023 FOSS pick list from Infoworld.

Now, onto today’s topic dive.

Go Cache Young Person [Woman. Man. Camera. TV.]

I picked Go for this project since:

• I wanted a single binary I could run anywhere

• Go has a robust ecosystem of HTTP web request “router” modules (I’m using Gin for this.)

• The go-sqlite3 module is ace (I’m using SQLite for caching.)

• There’s an excellent nvdapi module which means I don’t have to write the NVD interaction code from scratch.

• We primarily use Go at work, and context switching has been significantly more challenging in this continuing bout of long covid. I do miss Rust, though.

The aforebulleted Gin is a high-performance Go web framework written. It emphasizes composability and flexibility through middleware chaining. This middleware lets us intercept requests and responses and perform tasks like logging, reading cookies, authenticating, etc. The module is designed to be fast, efficient, and minimalistic while providing essential features to develop robust web applications and APIs. Because of it, this helpful caching server is just 67 lines of code:

$ tokei main.go
===========================================
 Language Files Lines Code Comments Blanks
===========================================
 Go           1   107   67       13     27
===========================================
 Total        1   107   67       13     27
===========================================

Gin is so handy I can toss code even in Substack’s code-hostile code blocks for a super minimal example of how easy it is to use it:

package main

import (
  "github.com/gin-gonic/gin"
)

func main() {

  // returns an Engine instance with the 
  // Logger and Recovery middleware 
  // already attached
  r := gin.Default() 

  // handler for requests to http://…/param/ANYTHING
  r.GET("/param/:value", func(c *gin.Context) {

    // retrieve the inpout
    value := c.Param("value")

    // spit it back as JSON
    c.JSON(200, gin.H{
      "value": value,
    })

  })

  // listen and serve on 0.0.0.0:8080
  // UNSAFE don't do this IRL
  r.Run() 
}

Lots of other programming languages have built-in or package-provided API router code that looks similar to this.

If you put the above in a main.go, you can quickly test it out without even bothering to set up a fancy project:

$ GO111MODULE=off go get github.com/gin-gonic/gin
$ GO111MODULE=off go build main.go
$ ./main & # shunts it to the background; there will also be some debug messages
$ curl -s http://localhost:8080/param/hello-world
{"value":"hello-world"}
$ fg
$ CTRL-C

The SQLite database is equally minimal. I don’t need Go to do surgery on the JSON returned from the API, so this is the entire table (this is the exact same SQL statement used in the project’s main.go):

CREATE TABLE IF NOT EXISTS cve_cache (
  cveid TEXT PRIMARY KEY, 
  data TEXT
)

Removal of potentially stale entries is as simple as a single DELETE FROM… (which is in the project’s Justfile).

I kept it this simple since it causes this microservice to return the same JSON responses the NVD API does, and because I have other projects that just dump the data column out to an ndjson/jsonlines file so DuckDB can turn it into lovely, purpose-built Parquet files so I can go back to having giant files of vulnerabilities in a proper data science context.

The server uses four environment variables for configuration:

• NVD_API_KEY: using an NVD API KEY

• NVD_CACHE: full path (with filename) to the SQLite cache database

• PORT: port you want to run the microservice on (just the port; it has localhost hard-coded)

• GIN_MODE: use release for the best performance

A sample run to get CVE-2018-1207 cached and uncached shows the performance improvements over having to wait for NVD’s timeout period:

1.214626666s | not cached
   274.375µs | cached
   256.958µs | cached

FIN

I’m leaning on the diminutive quality of the server code to be explanatory enough to not bombard you with blathering in this edition. Having said that, please do not hesitate to drop issues (or hit me up anywhere) with q’s.

PRs gladly accepted after review. Perhaps add a route to another NVD endpoint and submit it as a bonus WPE!

Credit would be 👍🏽 if you’re a cybersecurity vendor who does use this, but tis not required.

Be safe out there and 🙏🏽 for your support! ☮