DEV Community: schmudde

Turning URLs Into Meaningful Names Using Clojure

schmudde — Wed, 01 Dec 2021 13:13:25 +0000

URLs are wonderful to work with because they are structured, human meaningful, globally unique identifiers. But they do have a few thorny edge cases.

The following examples explore the domain name system using Clojure. The goal is to derive meaningful information about a resource owner's identity from any URL. For example, the URLs http://wikipedia.org/wiki/Peoria and https://en.wikipedia.org/wiki/Peoria look different but they both resolve to the same document and they are both managed by the Wikimedia Foundation. They do not pass the threshold of being meaningfully different.

There is some subjectivity in this analysis. The exploration in this article goes all the way back to 1987 in an effort to automatically determine the unique identity of a domain's owner. The work primarily relies on two libraries, the Apache Commons Validator API and Java's Uniform Resource Identifier (URI) reference. The latter is our parser while the former is, unsurprisingly, our validator.

First, create the Clojure namespace with the requisite libraries. Java's URI library, java.net.URI, comes with Clojure. To use the Apache Commons Validator API, add commons-validator/commons-validator {:mvn/version "1.7"} to the project's deps.edn file.

(ns domain-names
  (:import [org.apache.commons.validator.routines UrlValidator DomainValidator]
           [java.net URI])
  (:require [clojure.string :as str]
            [clojure.edn :as edn]))

;; With that out of the way, it's time for some exploration.

Domain Names Explained

The URL

A Uniform Resource Locator (URL) is the addresse of a unique resource on the web. A URL can be validated using the .isValid method. It's easy to create a custom function in Clojure.

(defn url? [url]
   (-> (UrlValidator.)
       (.isValid url)))

(url? "http://schmud.de") ⇒ true

eMails won't pass. Neither will domain names.

(url? "email@example.com") ⇒ false

(url? "schmud.de") ⇒ false

What Is a Domain Name?

RFC 1034 laid the foundation for domain names in November 1987. The domain name indicates which server hosts a web resource. Domain names are owned by a person or an organization. Like all proper names, they are a form of identification.

This identification is what we're after. Who is the owner of this resource? It turns out answering that question is not so simple because subdomains can indicate meaningful distinctions. Consider these subdomains:

www.tumblr.com is not meaningfully different from tumblr.com. They are both used to indicate the entity known as Tumblr.
schmudde.tumblr.com is meaningfully unique from journalofanobody.tumblr.com. Within the domain of Tumblr, there is only one subdomain named schmudde, which points to a distinct person (myself).
https://azure.microsoft.com and https://www.microsoft.com/windows shows how Microsoft thinks about two different products. The www subdomain does not indicate a meaningful distinction while the azure prefix reflect's Microsoft's corporate structure; Azure makes up one of Microsoft's four distinct engineering divisions.

Depending on your purpose, subdomain distinctions may be significant. Some further understainding of domain name is necessary before moving on.

The Root Domain, the Top Level Domain, and the Subdomain

The Root Domain and Subdomains

Subdomains are defined relatively. RFC 1034 uses this example:

A.B.C.D is a subdomain of B.C.D, C.D, D, and " ".

Thus the root domain is the only absolute component of a URL. It is " ". A more concrete example from the RFC may help illustrate:

                               |
                               |
         +---------------------+------------------+
         |                     |                  |
        MIL                   EDU                ARPA
         |                     |                  |
         |                     |                  |
   +-----+-----+               |     +------+-----+-----+
   |     |     |               |     |      |           |
  BRL  NOSC  DARPA             |  IN-ADDR  SRI-NIC     ACC
                               |
   +--------+------------------+---------------+--------+
   |        |                  |               |        |
  UCI      MIT                 |              UDEL     YALE
            |                 ISI
            |                  |
        +---+---+              |
        |       |              |
       LCS  ACHILLES  +--+-----+-----+--------+
        |             |  |     |     |        |
        XX            A  C   VAXA  VENERA Mockapetris

The RFC explains:

The root domain has three immediate subdomains: MIL, EDU, and ARPA. The LCS.MIT.EDU domain has one immediate subdomain named XX.LCS.MIT.EDU. All of the leaves are also domains.

These days the root domain sometimes refers to the domain name registered with a domain name registrar. For example, I registered netart.today, not www.netart.today. The more precise definition is the domain which all other subdomains branch from.

The Top Level Domain

mil and edu eventually became known as generic Top Level Domains (gTLD) along with other familiar extentions like com and org. The Apache Commons Validator can help identify the Top Level Domains (TLDs) as currently defined and maintained by the Internet Assigned Numbers Authority (IANA).

The familiar gTLDs:
(.isValidGenericTld (DomainValidator/getInstance) ".edu") ⇒ true

Country codes as TLDs:
(.isValidCountryCodeTld (DomainValidator/getInstance) ".cn") ⇒ true

TLDs reserved for internet infrastructure:
(.isValidInfrastructureTld (DomainValidator/getInstance) ".arpa") ⇒ true

Now use the DomainValidator's general .isValid method to see if a gTLD + two subdomains is valid:
(.isValid (DomainValidator/getInstance) "www.schmud.de") ⇒ true

Add a protocol to the subdomains and the gTLD to form a URL:
(.isValid (DomainValidator/getInstance) "http://www.schmud.de") ⇒ false

A URL ≠ a domain name.

Getting the Domain Name

Parsing URLs

Getting the domain name from a URL means we need to parse the URL string. Parsing URLs in Clojure and Java can be done with java.net.URI. The URI object is parsed into these components: [scheme:][//authority][path][?query][#fragment].

The Authority

The Authority is the component that will hold the domain name (aka host name) in java.net.URI objects. It can also hold an authentication section (username and password) and an optional port number precededby a :. But I don't want those. To exclusively get the host, use .getHost.

Here's a URL with a port number, scheme, and path to demonstrate:

(-> (java.net.URI/create "https://schmud.de:443/books.html")
        .getHost)

⇒ "schmud.de"

The java.net.URI/create method expects strings that it can parse as a URI object. If you just give it a domain name, it will not fail but it also will not parse the host.

(-> (java.net.URI/create "schmud.de")
        .getHost)

⇒ nil

But it will fail if you give it something that it cannot parse.

(try
  (-> (java.net.URI/create "://schmud.de/books.html")
      .getHost)
  (catch Exception e e))

⇒ error

The best solution probably includes validation. The code won't even try to parse bad input.

(#(when-let [valid-url (when (url? %) %)]
    (-> valid-url
        java.net.URI/create
        .getHost)) "http://schmud.de/books.html")

⇒ "schmud.de"

(#(when-let [valid-url (when (url? %) %)]
    (-> valid-url
        java.net.URI/create
        .getHost)) "://schmud.de/books.html")

⇒ nil

The URL Validator also guards against TLDs that do not exist. In this case, the fictitious .ded TLD.

(#(when-let [valid-url (when (url? %) %)]
    (-> valid-url
        java.net.URI/create
        .getHost)) "http://schmud.ded")

⇒ nil

The .getHost method itself suffers one drawback: it can trip up when interpreting URL schemes that include alphabetic characters outside the Latin alphabet (essentially URL schemes after RFC 2396). Consider the difference:

(.getHost (java.net.URI/create "http://köpönyeg.hu")) ⇒ nil

(.getRawAuthority (java.net.URI/create "http://köpönyeg.hu")) ⇒ "köpönyeg.hu"

.getRawAuthority returns the value without interpreting any escaped octets. The strings returned by these methods may contain both escaped octets and other characters, and will not contain any illegal characters.

Parsing Subdomains

It's important to remember the goal when deciding how to deal with subdomains. I'm looking for any meaningfully unique identity. azure.microsoft.com and schmudde.tumblr.com are meaningfully unique in a way that en.wikipedia.org and www.schmud.de are not.

A more strict solution may consider all known public suffixes and only accept a top level domain + one subdomain.

My implementation takes the opposite approach. It removes common subdomains compiled by SecLists. Most of the 5000+ common subdomains on the list do not provide meaningful identity differentiation, such as www., en., m., shop., and www.news.

(def common-subdomains
  "Slurp and sort then drop the first 200 longest names because
   they seem especially esoteric."
  (->> (slurp "resources/subdomains-top1million-5000.txt")
                            clojure.edn/read-string
                            (sort-by count)
                            reverse
                            (drop 200)))

(defn remove-subdomain-prefix
  "In: Domain name (string)
   Out: Host without a prefix.

   has-more-than-one-subdomain? ensures that shop.google.com -> google.com and shop.com -> shop.com"
  [domain-name]
  (let [has-more-than-one-subdomain? (when (not= (str/index-of domain-name ".")
                                                 (str/last-index-of domain-name "."))
                                       true)
        matching-prefix (-> (partial str/starts-with? domain-name)
                            (filter common-subdomains)
                            first)]
    (if (and has-more-than-one-subdomain? matching-prefix)
      (str/replace-first domain-name matching-prefix "")
      domain-name)))

Common subdomains:

(remove-subdomain-prefix "www.company.com") ⇒ "company.com"

(remove-subdomain-prefix "news.company.com") ⇒ "company.com"

Common subdomain compounds are also caught. remove-subdomain-prefix sorts the list of common subdomain prefixes from longest to shortest. Therefore the function removes www.news. (longer) rather than just www. or news..

(remove-subdomain-prefix "www.news.company.com") ⇒ "company.com"

Common subdomain prefixes do not get confused with the primary subdomain:

(remove-subdomain-prefix "shop.google.com") ⇒ "google.com"

(remove-subdomain-prefix "shop.com") ⇒ "shop.com"

Significant subdomains are retained:

(remove-subdomain-prefix "www.news.chevrolet.gm.com") ⇒ "chevrolet.gm.com"

Getting a Meaningful Identifier

Put it all together to get meaningful domain names from any URL.

(defn get-host [x]
  (if (url? x)
    (-> (java.net.URI/create x)
        .getRawAuthority
        remove-subdomain-prefix)
   (str "`" x "` is not a valid url. Expecting [scheme:][//authority][path][?query][#fragment]")))

(get-host "http://m.google.com/community") ⇒ "google.com"

(get-host "ttp://m.google.com/community") ⇒ "ttp://m.google.com/community is not a valid url. Expecting [scheme:][//authority][path][?query][#fragment]"

(get-host "http://schmudde.tumblr.com/") ⇒ "schmudde.tumblr.com"

I believe that the get-host function makes the right trade offs when considering the stated goals. The next step could be to turn these domain name rules into reusable Clojure specs. Check out Conan Cook's A spec for URLs in Clojure and associated generators for a good introduction.

Most of my work deals with identity and time in computing systems. See my posts Storing Time - Part 1 and Storing Time - Part 2 for a deep dive into computational memory and recording time in Clojure and Java. Thanks for reading!

Getting Started With the Node Solid Server

schmudde — Fri, 10 Sep 2021 08:20:41 +0000

A dive into Solid, including a complete how-to for starting your own Solid server using Node.js.

Solid is a specification that lets people store their data in federated data stores called Pods. The information on each Pod is linked through a person's online identity. Identity is addressed with a WebID, which is just an HTTP URL that points to a profile.

As depicted in the image below, the personal data pod can include things like pictures, calendar information, and personal contacts. The left side of the image depicts how this information is stored in walled gardens on today's internet platforms. The right shows how the data is stored in a personal data pod. Applications can simply query a Pod to get the information they need to provide their service.

Figure: Applications can query a pod to get the data they need. CC-by-4.0 Ruben Verborgh

Individuals can therefore manage their personal information in whatever place they choose. The Solid standard makes it easy to grant and revoke access to personal information to specific individuals, organizations, and applications. This is a win for personal privacy.

Pods make the web more secure because data is not pooled in a single place. Personal information is spread across a federation of servers. While Solid does not prevent individual targeting, its adoption would curb the massive data breaches that have become commonplace.

Individual Pods can move between servers with very little friction. If a person does not like a server's security practices, terms of service, or notices other forms of abuse, they can simply move their Pod to another server or even host their Pod on their own server.

That last sentence is what this article is about. Running the code in this article will stand up a Solid server that can be found on the public internet. I will take you through each step along the way, including how to access the server from an outside Solid application. Let's get started.

Ngrok

The Solid server runs locally; Ngrok will provide a public URL for the local Solid server.

I have configured Ngrok to make the Solid server available at https://btf.ngrok.io. The full ngrok.yml file can be found in the Appendix. See the Ngrok Configuration section.

A reserved subdomain (btf in this case) is only available through the paid plan. If you don't have a paid plan, you'll have to provide Ngrok's generated subdomain to the Solid server before you start it up.

I'll add my authentication token to the ngrok.yml configuration file. The token is held in an environmental variable.

echo 'authtoken:' $ngrok | cat - ngrok.yml > /tmp/out && mv /tmp/out ngrok.yml

Now it's time to start the ngrok service in the background using nohup.

nohup ./ngrok start --all --config="/ngrok.yml" --log="/tmp/ngrok.log" &> /tmp/ngrok-run.log & sleep 1

The Ngrok tunnel is now live at https://btf.ngrok.io/. The next step is standing up a Solid server at https://localhost:8443.

Solid

There are two Solid server implementations that run on Node.js. The Community Solid Server (documentation) is newer, but this article uses the Node Solid Server (documentation). Both are open-source Node.js implementations of the same Solid standard, independently maintained.

The Solid server provides the necessary tools to store and manage access to data held in Pods on the sever.

The server provides the tools to assert your identity (you are who you say you are, Authentication)
The server provides the tools to use your identity (sharing personal information, having conversations on the internet, etc..., Authorization)

Authentication

Once you prove you are in control of your Pod, the authentication is done. This authentication step is not application-specific, it's Pod-specific. Therefore a person does **not* need a unique name and password for each application they use, they only need a single name and password for the Pod.*

Authorization

A Solid application will ask you to authorize certain access permissions. The ultimate authority on who has access to your information does not live within the application (which is generally true today), it lives within a person's Pod.

Install the server

Now that some of the Solid server's functions are clear, it's time to install the server using NPM.

npm install -g solid-server

Configure the Server

The server's configuration is held in config.json below. A few specific notes.

The serverUri must be set to the reverse proxy's URI, https://btf.ngrok.io.
The port is set to serve information over SSL at 8443. This number matches the port setting of Ngrok's addr (address), https://localhost:8443. See ngrok.yml in the Appendix for more context.

Setting webid to true enables authentication and access control by establishing a relation between the WebID URI and the person's public keys

{ "root": "/solid/data",
  "port": "8443",
  "serverUri": "https://btf.ngrok.io",
  "webid": true,
  "mount": "/",
  "configPath": "./config",
  "configFile": "./config.json",
  "dbPath": "./.db",
  "sslKey": "/keys/privkey.pem",
  "sslCert": "/keys/fullchain.pem",
  "multiuser": false,
  "server": {
    "name": "The Beyond the Frame Solid Server",
    "description": "This is a Solid server experiment.",
    "logo": ""
  }
}

The sslKey and sslCert still need to be generated and added to the environment. This can be done using the openssl req command. OpenSSL is a general purpose cryptographic SSL/TLS (Secure Sockets Layer/Transport Layer Security) toolkit, req is the specific command to fulfill a X.509 Certificate Signing Request.

Here is the configuration file that provides the necessary cryptographic information and fulfills the Distinguished Name requirements.

[ req ]
 default_bits       = 2048
 default_md         = sha256
 prompt             = no
 encrypt_key        = no # do not encrypt the keypair, same as -nodes
 distinguished_name = btf_server
[ btf_server ]
 countryName            = "US"
 stateOrProvinceName    = "Illinois"
 localityName           = "Chicago"
 organizationName       = "Beyond the Frame"
 organizationalUnitName = "None"
 commonName             = "example.com"
 emailAddress           = "example@example.com"

Run the req command to generate a 2048-bit (default_bits) private key (-keyout /keys/privkey.pem) and then fulfill a certificate request (-out /keys/fullchain.pem).

mkdir /keys
openssl req -outform PEM -keyform PEM -new -x509 -keyout /keys/privkey.pem -config openssl.cnf -days 365 -out /keys/fullchain.pem

Finally, the Solid server needs a folder to serve. The folder is set by root in the configuration file above. Make the corresponding directory:

mkdir -p /solid/data

Run the Server

solid start

Open another window and load https//btf.ngrok.io. This will be the result:

Some explanation on the magic happening behind the scenes is worthwhile.

Solid requires valid SSL certificates to function properly. They normally cannot be self-signed. The command solid-test start can be used to accept self-signed certificates. The command is found in solid-server/bin under the opt/nodejs folder.

However, this article gets away with self-signed certificates because Ngrok's servers use the registered ngrok.com certificates. Ngrok assumes that the server is running privately, so it doesn't need to check our self-signed certificates. If this was production instance, we would want to use our own certificates rather than Ngrok's.

Working With Pods

WebID

Click the blue Register button at https//btf.ngrok.io to create a WebID. The profile associated with the WebID will be at this URL: https://btf.ngrok.io/profile/card#me. Use this identifier to interact with Solid pods and apps.

Note: if multiuser was set to true in the config.yml, this WebID could be customized with a user's name to differentiate it from other Pods on this Solid server.

Schema

Relationships are the stuff of which information is made. Just about everything in the information system looks like a relationship.

~ Data and Reality (87)

Now that the personal datastore is addressable at https://btf.ngrok.io/profile/card#me, web applications can request personal information using a standard schema.

The schema does more than simply address the data, it can describe relationships between bits of information.

For example, you may host a photograph on your Pod that you share through an application. If I sign into the same application using my WebID, I might be able to leave a comment on the photograph. That comment could exist on my Pod because I am the original author, but you could still view it through the application interface. The relationship between the photo and the comment can be depicted by a schema. It may look something like:

<https://mypod.solid/comments/36756>
    <http://www.w3.org/ns/oa#hasTarget>
        <https://yourpod.solid/photos/beach>

As Ruben Verborgh illustrates in Solid: Personal Data Management Through Linked Data, the comment on my Pod is linked to the picture on your Pod through a type defined in a schema. We know that my comment is a response to your picture instead of the other way around.

Like the markup standards that ensure a website is displayed correctly, schema standards for the world wide web promised to create semantic relationships between pieces of information on the web. This effort is at least 20 years old and their website at schema.org just turned 10.

Wikidata is one of the most prominent sites built on structured data. They use an open schema called RDF. RDF graphs are sets of subject-predicate-object triples (e.g. <Bob> subject <is a> predicate <human> object). The schema also provides a data-modelling vocabulary for RDF data (e.g. <human> is a subClassOf <mammal>).

These structures allow programmers to create vocabularies - which are essentially rules about what information can and cannot be asserted about a domain. For example, a programmer can build a social graph using the Friend of a Friend vocabulary.

Solid is built to use RDF. Common vocabularies can parallel the power of social networks, shared calendars, document collaboration, music libraries, etc.... These vocabularies describe what interactions are possible between Pods. Since vocabularies are built on open standards, independent data sources and applications don't have to build their own custom API. Interoperability is baked in, centered on web identity.

Apps

Solid supports a rich ecosystem of applications that compete on the merits of their service. The best-performing applications will naturally be complimentary with other services that process the data on your pod. As Jacob O'Bryant opines on Solid:

There's an interesting analogy to functional programming: it's better to have 100 apps operate on 1 database than 10 apps operate on 10 databases.

Any one of Solid's official list of apps can be used to read and write data to this new Pod. For example, Notepod is a simple Solid-compatible note-taking app. Visit the app, click Connect, and add this Pod's URL.

When asked to authenticate, use the username and password previously created on the Solid server.

Grant the requested permissions and the app is connected. Anything created in the app is stored in the Pod. It's worth re-emphasizing that the app doesn't have its own username and password. Solid not only solves the problem of siloed data, it solves the proliferation of usernames and passwords that plague today's internet users.

Conclusion

I hope this article helped convey the power of Solid and how easy it is to get started with your own Solid server. The technology will become more powerful and useful as the federated network of servers grow. When working with the configuration files, make sure to configure Ngrok to use your own subdomain (paid Ngrok plan) or dynamically add the generated subdomain to your Solid server configuration (free Ngrok plan).

Find me at my blog where I discuss information science, art, and narrative: Beyond the Frame. I can also be reached at Mastodon and Twitter.

Appendix

Federated Servers

If you don't like the server that hosts your Pod, simply move it to another server. Here is a list of current Solid servers:

Inrupt Pod Spaces by Inrupt, Inc. hosted at Amazon in the USA. I have a Pod and public profile on this server.
inru pt.net by Inrupt, Inc. hosted at Amazon in the USA
solidcommunity.net by Solid Project hosted at Digital Ocean in the UK. I have a Pod and public profile on this server.
solidweb.org by Solid Grassroots hosted at Hosteurope in Germany
trinpod.us by Graphmetrix, Inc. hosted at Amazon in the USA
When this code is running, there is also a Solid server at https://btf.ngrok.io

You can choose a server based for a variety of ethical or technical reasons. Perhaps you disagree with a change in the terms of service, want more storage capacity, or high-speed transfer. This is illustrated in the data market section below:

Figure: Data and app markets. CC-by-4.0 Ruben Verborgh

Ngrok Configuration

If you use the code from this article, you can use the same Ngrok service, but your subdomain will be random, rather than btf. All other configurations are the same.

When you change the subdomin, remember that the full domain, subdomain, domain (.ngrok), and TLD (.io) must match in three places:

This Ngrok configuration file at subdomain.
The serverUri found in Solid's config.json above.
The URL in your web browser's address bar.

Here is the Ngrok configuration used in this article:

region: us
console_ui: true
tunnels:
  btf:
    proto: http
    addr: https://localhost:8443
    subdomain: btf
    bind_tls: true

Setting the https prefix of https://localhost:8443 forces Ngrok to speak HTTPS to the local server. The 8443 port suffix directs it to the Solid server's port.

proto: http - the protocol. The http protocol covers both HTTP and HTTPS
addr:https://localhost:8443 use the full URL to force HTTPS (equivalent to ngrok http https://localhost:8443 on the command line)
bind_tls - without this setting both HTTP and HTTPS will be available from ngrok
- bind_tls: true - ngrok only listens on an HTTPS tunnel endpoint
- bind_tls: false - ngrok only listens on an HTTP tunnel endpoint
host_header: localhost - unused. But some APIs will want to see the host header of your app rather than 'ngrok'

References

Kent, William, and Steve Hoberman. 2012. Data And Reality. 3rd ed. Westfield, N.J.: Technics Publications.
‘Linked Data – Design Issues’. Accessed 19 August 2021. https://www.w3.org/DesignIssues/LinkedData.html.
Verborgh, Ruben. ‘Solid: Personal Data Management Through Linked Data’, 2018. https://doi.org/10.25815/50W4-HK79.

This work is licensed under a Creative Commons Attribution 4.0 International License.

Simple "Have I Been Pwned" API Calls With Clojure

schmudde — Fri, 13 Nov 2020 09:45:28 +0000

';--have i been pwned? is the gold standard for seeing if a user's account has been compromised in a data breach. This is usually done using an eMail address, which is what I'll be demonstrating here.

I will be using the Have I Been Pwned (HIBP) API in this notebook. The API requires a key for a nominal charge of $3.50 a month.¹ Obviously, my key is not available to the public.

Here's a full blog post on why ';--have i been pwned? charges for this service. The API is pretty simple, so let's get started.

Grab the Data

Use a curl command to grab the data using the API.

curl -H "hibp-api-key:<your-secret>" -H "user-agent: Beyond the Frame" -sS https://haveibeenpwned.com/api/v3/breachedaccount/d%40schmud.com?truncateResponse=false -o "/pwned-accounts.json"

A breakdown of the switches I used:

🔑 -H "hibp-api-key:<your-secret>": An HIBP subscription key is required to make an authorized call and can be obtained on the API key page. The key is then passed in a hibp-api-key header. Replace <your-secret> with your own key.
-H "user-agent: Beyond the Frame": Each request to the API must be accompanied by a user agent request header. Typically this should be the name of the app consuming the service.
-o "/pwned-accounts.json": Output the returned JSON data.

The URL has two unique features:

d%40schmud.com: My eMail, encoded for a URL.
?truncateResponse=false: return the complete breach data for all data breaches

Examine the data² returned by Have I Been Pwned:

(require '[clojure.data.json :as json])

(def accounts 
    (json/read-str (slurp "/pwned-accounts.json") 
                   :key-fn keyword))

(pprint accounts)
accounts

output ⇒

[{"~:Domain":"8tracks.com","~:DataClasses":["Email addresses","Passwords"],"~:IsFabricated":false,"~:IsVerified":true,"~:BreachDate":"2017-06-27","~:PwnCount":17979961,"~:Title":"8tracks","~:IsSensitive":false,"~:IsSpamList":false,"~:LogoPath":"https://haveibeenpwned.com/Content/Images/PwnedLogos/8tracks.png","~:ModifiedDate":"2019-08-25T08:52:21Z","~:Name":"8tracks","~:Description":"In June 2017, the online playlists service known as <a href=\"https://blog.8tracks.com/2017/06/27/password-security-alert/\" target=\"_blank\" rel=\"noopener\">8Tracks suffered a data breach</a> which impacted 18 million accounts. In their disclosure, 8Tracks advised that &quot;the vector for the attack was an employee’s GitHub account, which was not secured using two-factor authentication&quot;. Salted SHA-1 password hashes for users who <em>didn't</em> sign up with either Google or Facebook authentication were also included. The data was provided to HIBP by whitehat security researcher and data analyst Adam Davies and contained almost 8 million unique email addresses. The complete set of 18M records was later provided by JimScott.Sec@protonmail.com and updated in HIBP accordingly.","~:IsRetired":false,"~:AddedDate":"2018-02-16T07:09:30Z"},{"~:Domain":"eatstreet.com","~:DataClasses":["Dates of birth","Email addresses","Genders","Names","Partial credit card data","Passwords","Phone numbers","Physical addresses","Social media profiles"],"~:IsFabricated":false,"~:IsVerified":true,"~:BreachDate":"2019-05-03","~:PwnCount":6353564,"~:Title":"EatStreet","~:IsSensitive":false,"~:IsSpamList":false,"~:LogoPath":"https://haveibeenpwned.com/Content/Images/PwnedLogos/EatStreet.png","~:ModifiedDate":"2019-07-19T11:29:35Z","~:Name":"EatStreet","~:Description":"In May 2019, the online food ordering service <a href=\"https://www.zdnet.com/article/eatstreet-food-ordering-service-discloses-security-breach/\" target=\"_blank\" rel=\"noopener\">EatStreet suffered a data breach affecting 6.4 million customers</a>. An extensive amount of personal data was obtained including names, phone numbers, addresses, partial credit card data and passwords stored as bcrypt hashes. The data was provided to HIBP by a source who requested it be attributed to &quot;JimScott.Sec@protonmail.com&quot;.","~:IsRetired":false,"~:AddedDate":"2019-07-19T11:29:35Z"},{"~:Domain":"ticketfly.com","~:DataClasses":["Email addresses","Names","Phone numbers","Physical addresses"],"~:IsFabricated":false,"~:IsVerified":true,"~:BreachDate":"2018-05-31","~:PwnCount":26151608,"~:Title":"Ticketfly","~:IsSensitive":false,"~:IsSpamList":false,"~:LogoPath":"https://haveibeenpwned.com/Content/Images/PwnedLogos/Ticketfly.png","~:ModifiedDate":"2018-07-14T06:06:15Z","~:Name":"Ticketfly","~:Description":"In May 2018, the website for the ticket distribution service <a href=\"https://motherboard.vice.com/en_us/article/mbk3nx/ticketfly-website-database-hacked-data-breach\" target=\"_blank\" rel=\"noopener\">Ticketfly was defaced by an attacker and was subsequently taken offline</a>. The attacker allegedly requested a ransom to share details of the vulnerability with Ticketfly but did not receive a reply and subsequently posted the breached data online to a publicly accessible location. The data included over 26 million unique email addresses along with names, physical addresses and phone numbers. Whilst there were no passwords in the publicly leaked data, <a href=https://support.ticketfly.com/customer/en/portal/articles/2941983-ticketfly-cyber-incident-update\" target=\"_blank\" rel=\"noopener\">Ticketfly later issued an incident update</a> and stated that &quot;It is possible, however, that hashed values of password credentials could have been accessed&quot;.","~:IsRetired":false,"~:AddedDate":"2018-06-03T06:14:14Z"}]

Parse the Data

Grab three parameters from every title returned and print as HTML.

Title: A descriptive title for the breach suitable for displaying to end users. It's unique across all breaches but individual values may change in the future (i.e. if another breach occurs against an organisation already in the system). If a stable value is required to reference the breach, refer to the "Name" attribute instead.
Domain: The domain of the primary website the breach occurred on.
BreachDate: The date (with no time) the breach originally occurred on in ISO 8601 format. This is not always accurate — frequently breaches are discovered and reported long after the original incident. Use this attribute as a guide only.

(require '[cljc.java-time.local-date :as ld])

(defn account->html [account]
  (let [title (:Title account)
        link (str "<a href=\"http://" (:Domain account) "\" target=\"_blank\">" title "</a>")
        date (ld/get-year (ld/parse (:BreachDate account)))]
    (str "<li>" link " in " date "</li>")))

(defn print-accounts [accounts]
  (str "<hr /><center><h2>Accounts Associated With This eMail</h2><br /><ul>"
     (apply str (map #(account->html %) accounts))
       "</ul></center><br /><br /><hr />"))

output ⇒

Accounts Associated With This eMail

8tracks in 2017
EatStreet in 2019
Ticketfly in 2018

That's it. Use your key and simply replace the eMail address with any other eMail address and you will get the pertinent results.

Appendix

¹ The language of HIBP's API is not very clear. If you cancel your subscription, your API key will remain functional until the subscription renewal is due after which it will cease to work. You can always reactivate it by returning to the API key page and purchasing another one.

² The deps.edn file:

{:deps
 {org.clojure/clojure {:mvn/version "1.10.1"}
  org.clojure/data.json {:mvn/version "1.0.0"}
  cljc.java-time {:mvn/version "0.1.11"}}}

This work is licensed under a Creative Commons Attribution 4.0 International License.

Have I Been Pwned is the source of the data on this page. Some of the writing is directly cut from their API v3 documentation, also under CC4.0.