Drop #221 (2023-03-16): Scan Me If You Can

Cloudflare Radar Beta Scanner; urlscan.io;

I was going to try to shove three resources into today's Drop, but I think you'll be busy enough between both the ones I've included that you'll quickly forget there are usually three.

Scanning The Entire Internet

https://github.com/hrbrmstr/rust-ipv4heatmap

(See, I jimmied in a third section anyway to set up the other two!)

My previous gig had me working in many areas of cybersecurity research, but the primary two centered around scanning the internet and listening to the internet. When my younger kids used to ask me what I did at my job, I would say “count IP addresses to make sure none are missing”. In some ways, that was more true than one might expect.

I still do both those activities in my current position, but the “scanning” part is quite different, and something I will likely be able to talk about in the latter half of the year.

I, now, have (possibly) the most comprehensive view of benign, malicious, and unknown (in terms of intent) activity on the internet. Yet, I do kind of miss knowing just how many sensitive endpoints still had Telnet enabled on them

¹

, right at my fingertips.

Scanning the internet can be done in different ways, and the two services I'm introducing today operate much differently than what I used to do. To put it in terms that are hopefully accessible to the largest majority of folks: the scans I used to perform were the equivalent of visiting all the buildings across the country, and quickly (+ softly) asking what the owner did for a living. We tried to be very lightweight and non-intrusive, but do just enough of an ask to be relatively certain we obtained a legit answer.

The services featured in this Drop pretty much drive up to your house in an armored S.W.A.T. vehicle. Then, take detailed photographs of the building's exterior, try all the doors and windows to see if they're open, and even rifle through anything that's within arm's reach. They each provide a comprehensive set of details that can help you determine whether a given site is safe to visit, get an idea of what's running under the hood, and provide all sorts of other metadata.

I'll provide some details of what they each do, but the first one has a comprehensive introductory blog post (it just became “a thing” this week) and the other one will be self-explanatory after you check out the first one.

So, fear not, my blathering is almost at an end.

Cloudflare Radar Beta Scanner

NOTE: I've done my best to refrain from personal commentary about Cloudflare in this section. I think I've managed to do an OK job, apart from what you can infer from this introductory sentence, and some things I really felt I needed to make clear.

Cloudflare Radar had been a hub that “only” showcases global Internet traffic, attacks, and technology trends and insights. It is powered by data from Cloudflare’s global network, as well as aggregated and anonymized data from Cloudflare’s 1.1.1.1 public DNS resolver, which you should never use.

Using Radar’s API you could access Cloudflare’s data on global Internet traffic. This API is free (reg required), enabling academics, data sleuths and other web enthusiasts to investigate Internet usage across the globe. All data available via Radar API endpoints is made available under the CC BY-NC 4.0

This week, they introduced a new scan component of this Radar service. The header image shows the scan I made of my primary domain, and you can check out the whole report if you are so inclined. I did not read enough to know how ephemeral those links are, so I might preserve it at the Wayback machine, and will update the link if I do.

The blog post is long, and you can infer quite a bit from the screencap. But, essentially, with Radar Scan:

• Users can provide a URL and receive a report containing information such as phishing scans, SSL certificate data, HTTP request and response data, page performance data, DNS records, cookie security settings, and technology stack.

• Reports are publicly accessible and divided into categories: Security, Cookies, Network, Technology, DOM, and Performance.

• There's a “Security” tab which helps determine if a page is safe to visit by checking phishing and SSL certificate status.

And, a “Cookies” tab which reveals privacy friendliness by displaying cookies set and their attribute values, including Secure and HttpOnly flags.

• “Technology” does what it says on the tin and enumerates technologies, frameworks, and libraries used on the page, providing insight into the technology stack.

• For the IP jockeys amongst us, the “Network” tab displays HTTP transactions and DNS records, offering information on content loading and fundamental aspects of the page.

• You can poke at the contents of the page they scan via the “DOM” tab, which collates hyperlinks, global JavaScript variables, and raw HTML for further analysis.

• One also gets access to a “Performance” tab which presents load times and Performance Navigation Timing metrics for evaluating user experience.

(Phew)

Honestly, it's easier just to submit a bunch of URLs or check out ones that have been scanned (all the data is public) than it is to read the blog post or suffer through that awkward bullet list.

You won't get in trouble for scanning anything, and you may find some issues with your own internet-facing bits that you can add to the #fixit #todo list.

urlscan.io

TL;DR it does almost the same thing, but is run by a fine chap vs. the…(bites tongue)…

Anyway…

Urlscan.io is a free service for scanning and analyzing websites, helping users identify potentially malicious activity. When a URL is submitted, the platform browses the site like a regular user, recording various details such as domains, IPs, resources, and more. Urlscan.io also captures a screenshot, DOM content, JavaScript variables, and cookies. If the site targets users of over 900 monitored brands, it's flagged as potentially harmful. While the core service is free, commercial options are available for heavy users and organizations. Urlscan.io aims to make it easy for anyone to confidently analyze unknown or suspicious websites, providing valuable context to help users understand potential risks. Oh, it also has an API, and if you hunt around on my GitHub, you'll find an API wrapper for it in R.

They are both super similar, but often provide noticeably different results. A topic we'll cover in some future (likely Bonus) edition.

FIN

Whether you're just learning about this type of remote site introspection, or have been a longtime user of urlscan.io, I'm re-encouraging folks to give both a go in your spare time. They are great diagnostic tools, and do reveal the headers and other metadata that you do not normally see for a site. ☮

1. I’ll bet you can’t say you got the entire government of Belgium to hate you ~8 years ago, can you?

#cloudflare #urlscan #internetscanning #cybersecurity