It’s Go Time

While some schools in the U.S. have started back up, it’s still technically vacation season, and we’re going to continue the easygoing WPE theme for a few more weeks.

I’m unusually excited about the Go 1.21 release, so we’ll give y’all a starter Golang project to riff on this weekend.

Since this is code-heavy-ish, a more readable version is at https://rud.is/wpe/2023-08-11-drop-wpe.html.

RSS ALL THE THINGS

I occasionally muse about CISA’s Known Exploited Vulnerabilities (KEV) Catalog. We’re going to use it in the example, today, but please don’t let that stop you from following along (you can and should sub-out another data source that has meaning for you).

For a scant few weeks, CISA had an RSS feed for that information, but switched it to an email alert service (#NotHelping). I have a fairly robust setup doing all sorts of unnatural things to that data, including creating an RSS feed. JSON is great, but even its RSS-ish form (JSON Feed), it’s still not consumable by most RSS readers. RSS is super handy, and I have Inoreader set up to notify me whenever there is a new CISA KEV entry.

So, today, we’re going to see how to turn a regularly updated JSON data file into an RSS feed with Go.

Why Go?

Well, for starters, it’s what I have to use in production at work (so y’all should feel some of my pain, too 🙃).

Seriously, though, Golang may not be all “fancy” like some of its modern cousins, but it is pretty straightforward to read and write, and has far fewer “hieroglyphics” (odd/alien language tokens) than, say, Rust. It also lets us build a self-contained binary for any OS that we can also more easily cram into a lightweight Docker container or “serverless” function.

The Plan

We will write a Go program to:

• fetch CISA KEV JSON

• turn ^^ into something we can use in Go

• use the handy Gorilla feeds package to help us cobble together an RSS feed from the JSON entries

We’ll also use an external tool to validate the feed output, and Just.

You’ll need Go 1.21 installed (get it here), and I suggest keeping that URL in a tab if you’re not too familiar with Go. Please make sure you can execute go version at a command line and get a result that shows you’re running Go 1.21.

For those just itching to get to the source, it’s on GitLab.

I’ve broken the project down into three separate exercises:

1. just fetch and display the JSON

2. fetch the JSON, then build and print out the feed from it

3. fetch the JSON, build the feed, and save it to a file

As noted, in the Justfile, there’s a fourth step that validates the output.

NOTE: to keep this concise, I’m omitting error “handling” from the examples in situ. The project files have error handling baked in. I will make some comments about “logging”, though, so keep an eye out for that.

Marshal, Marshal, Marshal

CISA KEV JSON looks like this:

{
  "title": "CISA Catalog of Known Exploited Vulnerabilities",
  "catalogVersion": "2023.08.09",
  "dateReleased": "2023-08-09T09:49:53.0210Z",
  "count": 983,
  "vulnerabilities":
  [
    {
      "cveID": "CVE-2021-27104",
      "vendorProject": "Accellion",
      "product": "FTA",
      "vulnerabilityName": "Accellion FTA OS Command Injection Vulnerability",
      "dateAdded": "2021-11-03",
      "shortDescription": "Accellion FTA contains an OS command injection vulnerability exploited via a crafted POST request to various admin endpoints.",
      "requiredAction": "Apply updates per vendor instructions.",
      "dueDate": "2021-11-17",
      "notes": ""
    }
  ]
}

While that is some fine JSON, it doesn’t do us much good in Go without it being turned into native Go constructs. That process is called unmarshaling, which means we need to map the JSON to Go structures. Thankfully, we don’t have to go it alone.

Quicktype (GH) is both a web app, local utility, and IDE plugin to help with this type of JSON translation. Just feed in JSON and get useful language constructs back:

As you can see, you don’t need to paste in the entire JSON file. All you need is a representative sample — like the one above — to get what we need:

type KEV struct {
	Title           string          `json:"title"`
	CatalogVersion  string          `json:"catalogVersion"`
	DateReleased    string          `json:"dateReleased"`
	Count           int64           `json:"count"`
	Vulnerabilities []Vulnerability `json:"vulnerabilities"`
}

type Vulnerability struct {
	CveID             string `json:"cveID"`
	VendorProject     string `json:"vendorProject"`
	Product           string `json:"product"`
	VulnerabilityName string `json:"vulnerabilityName"`
	DateAdded         string `json:"dateAdded"`
	ShortDescription  string `json:"shortDescription"`
	RequiredAction    string `json:"requiredAction"`
	DueDate           string `json:"dueDate"`
	Notes             string `json:"notes"`
}

With that part covered, we can dig into the rest of this (short) exercise.

Part 1: Getting the JSON

Reference: https://gitlab.com/hrbrmstr/go-feed/-/blob/batman/01-getkev.go

Go has tons of batteries included. These three standard library packages:

• encoding/json

• io/ioutil

• net/http

make quick work out of retrieving data from a URL and getting it into the aforementioned structs:

// get the data
url := "https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json"
resp, err := http.Get(url)

defer resp.Body.Close() // see below

// read in the entire body of the response
body, err := ioutil.ReadAll(resp.Body)

// turn it into Go structs
var kevData KEV
err = json.Unmarshal(body, &kevData)

// display it
fmt.Printf("KEV data: %+v\n", kevData)

In Go, defer resp.Body.Close() is used to ensure that the response body of an HTTP request is closed after it has been processed. The defer keyword in Go allows you to schedule a function call to be executed just before the current function returns. In this case, it ensures that the Close() method is called on the response body, which is important for releasing resources and preventing resource leaks.

When you make an HTTP request, the response body is a stream of data that needs to be closed after it has been read to prevent resource leaks and ensure proper connection handling. If the response body is not closed, it can lead to increased memory and resource consumption, as well as issues with the underlying connection.

Using defer to close the response body is a good practice because it ensures that the Close() method is called regardless of whether the function returns early due to an error or completes successfully.

Keen eyes will see that we also relied on the “fmt” package from the standard library.

You should be able to run:

$ go run 01-getkev.go

or

$ just s01

to see the feed as formatted Golang data.

Part 2: More Alchemic Transformations

Reference: https://gitlab.com/hrbrmstr/go-feed/-/blob/batman/02-kev2feed.go

The Gorilla feeds package provides a handy set of high-level functions to help us craft and emit RSS XML. There’s an overall feed structure, and a similar one for the feed items. These are generic, too, since the framework can also output Atom and JSON feeds (we’re not doing that but feel encourage to play/experiment!).

Our set of imports has grown:

• fmt (printing)

• log (better output messaging)

• strings (we need to transform some strings)

• time (we need to convert some strings to timey wimey bits)

• encoding/json (unmarshaling)

• io/ioutil (reading the response body)

• net/http (retrieving the JSON)

• github.com/gorilla/feeds (working with Gorilla feeds)

Before we get to the Gorilla feed conversion, please compare both files to see how we’ve moved from “printing and returning” to “logging”. Using the Golang “log” package over fmt.Println has several advantages, particularly when it comes to providing more context and structure to your log messages. For enhanced observability, “log” package can automatically add useful information like timestamps or file names to your log messages, making it easier to trace and debug issues. And, we can implement structured logging using key/value attributes, which can be more easily parsed and analyzed programmatically. This is especially useful for monitoring and analyzing log data in larger systems.

However, In Go, using log.Fatal over fmt.Println + return (even in such a small program) is often preferred when you want to log an error message and terminate the program immediately. The main differences between the two approaches are related to error handling and program termination.

log.Fatal logs the error message and then calls os.Exit(1) to terminate the program with a non-zero exit status, indicating that an error has occurred. This is useful when you want to signal to the operating system or other programs that the program has encountered a fatal error and cannot continue execution.

On the other hand, using fmt.Println + return only prints the error message and returns from the current function. This approach does not necessarily terminate the program (it does in this small example, though), and the program will continue executing the remaining code. This can be useful in situations where you want to handle the error gracefully and allow the program to continue running.

Using log.Fatal has some drawbacks. When log.Fatal is called, deferred functions in the current goroutine (in this case, the main program) will not be executed, which can lead to resource leaks or improper cleanup. Our switch to using log.Fatal() also has us manually closing the response body.

Making the feed and printing it out is pretty straightforward. First, we build the Feed object from the CISA KEV metadata:

feed := &feeds.Feed{
	Title:       kevData.Title,
	Link:        &feeds.Link{Href: url},
	Description: "CISA's Known Exploited Vulnerability (KEV) catalog is an authoritative source of vulnerabilities that have been exploited in the wild, helping organizations prioritize remediation efforts to reduce cyber risks",
	Author:      &feeds.Author{Name: "CISA", Email: "Central@cisa.dhs.gov"},
	Created:     time.Now(),
}

Next, we iterate over the vulnerabilities and add each Item to the feed:

for _, vulnerability := range kevData.Vulnerabilities {

	kevDate, _ := time.Parse("2006-01-02", vulnerability.DateAdded) // See: https://pkg.go.dev/time#Parse

	item := &feeds.Item{
		Title:       vulnerability.VulnerabilityName,
		Link:        &feeds.Link{Href: fmt.Sprintf("https://nvd.nist.gov/vuln/detail/%s", strings.ToLower(vulnerability.CveID))},
		Description: vulnerability.ShortDescription,
		Id:          fmt.Sprintf("https://nvd.nist.gov/vuln/detail/%s", strings.ToLower(vulnerability.CveID)),
		Created:     kevDate,
	}

	feed.Items = append(feed.Items, item)

}

And, we then print it out:

rss, err := feed.ToRss()

fmt.Println(rss)

You should be able to run:

$ go run 02-kev2feed.go

or

$ just s02

To see the RSS feed.

TODO (For You)

• Add the dateAdded and cveID to the item titles

• Since we use it twice, store the Sprintf‘d string in a temporary variable

• CISA has requiredAction and notes fields; perhaps use them to add a formatted Content field?

Part 3: Write Out The Feed

Reference: https://gitlab.com/hrbrmstr/go-feed/-/blob/batman/03-kev2feedfile.go

Despite all the pointy braces, RSS XML is pretty human readable. As such, I prefer feeds that have the most recent entries first in the file. One reason we’re using the latest Golang version is that the “slices” package has a handy Reverse function to let us do that. So, we’ll need that and the “os” package (which helps us write out the contents).

Stick:

slices.Reverse(kevData.Vulnerabilities) // https://pkg.go.dev/slices@master#Reverse

right before the for loop, and

file, err := os.Create("kev-rss.xml")
_, err = file.WriteString(rss)
file.Close()

at the end of the file.

You should be able to run:

$ go run 03-kev2feedfile.go

or

$ just s03

to see the RSS feed.

TODO (For You):

• add more observability for what the program is doing in each step (use different logging levels)

• poke at the new log/slog package

• dig into the Justfile and run the tests

• use your own data source!

FIN

Remember, the full source is on GitLab.

You now have a standalone binary you can run anywhere to generate an RSS feed from data! Let me know how it went if you decided to take up this WPE challenge! ☮