Verification of Self-Signed Certificates

When interfacing to third-party web services, one often has to deal with self-signed SSL certificates that trigger verification errors. One workaround is to suppress those errors. (For instance, the Curl tool has the 'insecure' flag for this purpose.) However, at Smarking, we found ways to verify such certificates and to safeguard data communication from Man-in-the-Middle attacks.

Conventionally a web browser relies on a Public Key Infrastructure (PKI) to verify SSL certificates. Every certificate is signed by another (signing) certificate. That signing certificate must be signed by another, in a chain ending on a trusted certificate. This linkage allows a web server operator to switch to a new SSL certificate without requiring visitors to his website to update their web browsers. Alternatively, he could ask his users to trust his specific certificate so that the browser would not need to walk up the signature chain to verify it.

An SSL certificate carries inside it a public key of the webserver. On a conceptual level, the authenticity of that public key is the thing that allows us to establish an authenticated Diffie-Hellman key exchange between the browser and the server. Thus, if we could verify that it is the correct public key, we would have "verified" the certificate. However, instead of verifying a long public key, we could instead verify its checksum that is usually much shorter. A checksum of the entire certificate is called its fingerprint, and it's always formatted as a colon-separated list of hex codes. For instance, here is the SHA-256 fingerprint of the certificate served by https://google.com:

14:71:16:87:6D:F6:76:8E:98:E5:66:62:70:64:F1:0F:F8:0F:87:39:B8:55:4C:47:26:22:DF:FA:7D:1D:A5:FE

To retrieve the details of a website's certificate, click on the lock icon in your browser's URL bar and then inspecting SSL certificate details. Alternatively, you could use the following shell script:

#!/bin/bash
HOST=example.com
PORT=443
PROXY=1.2.3.4:8888

# If your environment does not require a HTTP proxy, delete the '-proxy $PROXY' parameter below
echo quit | openssl s_client -showcerts -servername $HOST -connect $HOST:$PORT -proxy $PROXY > result.txt

The output file result.txt includes the certificate in PEM format and metadata. The PEM format consists of binary data encoded using Base64 into ASCII, enveloped with "begin" and "end" lines like so:

-----BEGIN CERTIFICATE-----
<certificate encoded in base64 encoding>
-----END CERTIFICATE-----

You may feed the result.txt file into the following command to compute the fingerprint of the certificate. (The openssl tool would use the first certificate it finds in the input file and ignores everything else.)

$ openssl x509 -noout -fingerprint -sha256 -inform pem -in result.txt

The above command uses a -sha256 switch, which determines the length of the fingerprint to be 32 bytes. There are only a few widely-used variants, therefore the length of the fingerprint identifies the algorithm used to derive the fingerprint.

Here are three methods by which you can verify certificates by their fingerprints.

Method #1: Use Python

Verification of certificates by a fingerprint is supported out-of-the-box by the urllib3 library using the assert_fingerpint parameter,

import urllib3
from urllib.parse import urlparse

def http_get_request(url, fingerprint):
  parsed_url = urlparse(url)
  host = parsed_url.netloc
  path = parsed_url.path
  pool = urllib3.HTTPSConnectionPool(host, assert_fingerprint=fingerprint)
  response = pool.urlopen('GET', path)
  return response

response = http_get_request('https://example.com/a/b/c', '14:71:...')
print(response.data)

Notice that the fingerprint option configures an HTTPSConnectionPool object which could then be used to make a series of queries against a website, such that each of the queries would verify the fingerprint of the certificate.

The Python's requests library supports certificate fingerprint verification also because it builds upon the urllib3 library. It is based on adapter objects that return the HTTPSConnectionPool objects discussed above, and it provides a method Session::mount() which allows setting a custom adapter for a particular base URL. Putting this together, we have this code:

from urllib.parse import urlparse

def create_fingerprint_session(url, fingerprint):
  host = urlparse(url).netloc
  s = requests.Session()
  s.verify = False
  s.mount('https://{}/'.format(host), FingerprintAdapter(fingerprint))

session = create_fingerprint_session(('https://example.com/a/b/c', '14:71:...')
response = session.get(url)
print(response.text)

Note that the verify setting must be set to False, otherwise, the requests library would also try to verify the SSL certificate using the conventional way, by the signature chain and the domain name.

(Note that the verify parameter may also be set to a location of a certificate file that contains a concatenated list of trusted certificates in PEM format. However, a self-signed certificate is signed by a custom Certificate Authority (CA), but the certificate of the CA is usually unknown to us. Thus, we do not use this option but set verify to False.)

All that remains now is to implement the FingerprintAdapter. The quickest way is to subclass HTTPAdapter class and to modify the methods that create an HTTPPoolConnection object to include the assert_fingerprint option:

from requests.adapters import HTTPAdapter

class FingerprintAdapter(HTTPAdapter):
  """
  A TransportAdapter that allows to verify certificates by fingerprint
  """
  def __init__(self, fingerprint, *args, **kwargs):
    self._fingerprint = fingerprint
    HTTPAdapter.__init__(self, *args, **kwargs)

  def init_poolmanager(self, *args, **kwargs):
    kwargs['assert_fingerprint'] = self._fingerprint
    return super().init_poolmanager(*args, **kwargs)

  def proxy_manager_for(self, *args, **kwargs):
    kwargs['assert_fingerprint'] = self._fingerprint
    return super().proxy_manager_for(*args, **kwargs)

In summary, set verify=True when working with certificates signed by a trusted CA, otherwise set verify=False and mount a FingerprintAdapter when verifying self-signed certificates by fingerprint. Test that the verification is working by altering the fingerprint value and observing a security error.

Method #2: Site-wide

What if you wished to use other tools, besides Python, to query web sites signed with self-signed certificates? A site-wide solution is to add the self-signed certificate to a list of trusted certificates if it's there, no further signature checking will be made by the verifier.

However, the downside of this method is that a compromised trusted third party could now sign certificates for any domain which all programs on the machine would trust. This is a significant security risk for a long-lived server, but it may be tolerable if "site-wide" does not extend beyond a Docker container which runs a program that only connects to one endpoint.

The following instructions are for Ubuntu or Debian; for other distributions, make necessary adjustments.

Look in directory /usr/share/ca-certificates and you will see the directory mozilla, with many certificate files inside it. Make your own subdirectory on the same nesting level, for instance, /usr/share/ca-certficates/custom and put in it self-signed certificates of interest in PEM format, stored as separate files with extension .crt. Next, edit /var/ca-certificates.conf and list the custom certificates after the mozilla certificates. For instance,

...
mozilla/USERTrust_RSA_Certification_Authority.crt
custom/example-com-self-signed.crt
custom/another-example-com-self-signed.crt

Next, run update-ca-certificates command. Once that's done, symlinks to your certificates would appear in /etc/ssl/certs directory. At this point, the curl tool would work to accept the self-signed certificate from example.com.

However, the fingerprint method described previously, had the advantage that it worked even if there was a domain name mismatch. A mismatch would happen if you queried the target HTTPs server by an IP address (e.g.https://1.2.3.4/a/b/). If this is your situation, you can add an entry to /etc/hosts file to query the web server using the precise domain name that is listed inside the self-signed certificate.

The side-wide method works for all tools that rely on the libopenssl library, which includes curl. However, it is not sufficient for Python's requests library since it does some of its own checking of certificates.

Method #3: Strip SSL

Another way to allow a variety of tools to access HTTPs websites signed by self-signed certificates is to access them through a trusted proxy server that would strip the SSL after verifying the legitimacy of the self-signed certificates using fingerprints. We could implement such a proxy server in Python using the techniques above. Alternatively, we could use a utility program called stunnel that stands for the "Universal SSL Tunnel."

First, prepare a connection.conf configuration file like this one:

pid = /var/run/stunnel1.pid
CApath = /etc/ssl/certs
foreground=yes

[connection1]
verifyChain=no
verifyPeer=yes
client=yes
accept=8081
connect=1.2.3.4:443
sni=example.com

Then, run stunnel with the configuration file as the argument to have https://example.com proxied as http://localhost:8081,

$ stunnel connection.conf >& output.log &
$ curl 'http://localhost:8081/a/b/c'

An important thing to notice in the example configuration file is the verifyChain and verifyPeer options. They combine to verify the certificate by a fingerprint only and would ignore an incomplete signature chain. These options were added to stunnel in July 2016, in version 5.34. Another thing to notice that the domain name doesn't matter. The sni parameter is used only to instruct the webserver which virtual host you are interested in, but it plays no role in validating the certificate.

To run stunnel site-wide make the following configuration changes: store the configuration as /etc/stunnel/stunnel.conf, remove the foreground=yes bit, set pid to /var/run/stunnel.pid and add additional connection sections as needed.

Summary

We have demonstrated three ways to work with self-signed certificates without compromising security. At Smarking, we are trusted by vendors of parking systems to protect their data, and we use such techniques to justify their trust.

To learn more about Smarking, visit www.smarking.com.