To build a multi-protocol Apache Kafka Clusters to allow for SSL Client Authentication with
PLAINTEXT for inter broker communication, I needed to generate both broker and client SSL certificates.
There were many interesting things I learned in this process and wanted to share them.
An upcoming article will be using these certificates for setting up my Secured Apache Kafka Cluster.
Disclaimer
I build certificates to explore the security available in various systems; in this particular scenario, it is
to explore a mixed protocol Kafka Cluster. Please seek advice from your security teams when creating certificates.
Introduction
If you only need to create certificates as part of an enterprise setup, consider actively supported
projects such as Confluent's Ansible Playbooks.
There are excellent articles on SSL Certificates, including Confluent's documentation
and an excellent tutorial.
The documentation and tutorial focuses on SASL client-side authentication (SASL_SSL protocol)
and I uncovered some unique challenges with client-side SSL authentication.
My experience in certificate generation is based on using OpenSSL. There are other means of generating certificates,
so if another means is used for certificate creation, some of these highlights may not apply to you.
My full scripts are in the repository Kafka SSL Cluster.
Naming
Your certificate subject must start with /, and should contain at minimum a CN element, as in /CN=${hostname}.
For broker certificates, have the CN matches the hostname, and then leverage Subject Alternate Names for
ensuring the certificate can be used with and without domain name successfully.
There are no wildcard options with the standard Apache Kafka Authorizer, kafka.security.authorizer.AclAuthorizer.
So if you a complete subject of common name (CN), organization unit (OU), organization (OU), locality (L), state (S),
and country (C) the full name is what will be used as the User in ACLs.
This: User:CN=broker-1,OU=KAFKA,O=COMPANY,L=CITY,ST=MN,C=US
Not This: User:CN=broker-1
JKS key and keystore passwords should be the same
When creating Java key-stores (.jks files) for your JVMs, make sure the keystore password is the same as the key password.
This is due to limitations of the SunX509 KeyManagerFactory. Java Secure Socket Extension (JSSE) Reference Guide.
All keys in the KeyStore must be protected by the same password
I have found a few other documented cases of this with other systems, the most recent is with BitBucket.
Extensions
Be sure the certificates have the proper x509 extensions.
Basic Contraints extension for CA certificate
basicConstraints=CA:TRUE,pathlen:0
If you setup SSL certificates for client authentication, the CA certificate needs the following x509 extension indicating it is
indeed a CA certificate. Without this extension, it will not authenticate client certificates.
For additional information, see x509v3 config.
Subject Alternate Name Extension for Brokers
subjectAltName=DNS:${i}, DNS:${i}.${DOMAIN}
When it comes to SSL Certificates for encryption, it is important that it can have extensions to handle being used with and without
domain names.
Extended Key Usage Extension (for all)
extendedKeyUsage=serverAuth,clientAuth
The public key a certificate had to be noted that it can be used for server and client authentication. For my cluster,
I configured all certificates so they can be used for both server and client authentication. I added both to all
certificates.
Extensions are not Copied from Request to Certificate
By default, extensions added to a request is not propagated to the certificate. In order for extensions to be copied,
you would need copy_extensions = copy added to your openssl.cfg. I was unsuccessful in getting this to work in
MacOS so I updated my scripts to explicitly add extensions when certificate requests are signed.
It is a good reminder if you create certificate requests for another group to sign, be sure to indicate extensions you
need in your certificates.
Check Status after every call to OpenSSL
If you write scripts, make sure you check return status codes after each command and fail on error. I have spent many hours
troubleshooting script errors only to find out that something failed much earlier in the process.
After each command, add the following. Scripts are finicky, especially with loops. So double-check the error handling
works as well. This is the type of check I do after every openssl and every other command.
[ $? -eq 1 ] && echo "failure" && exit
Intermediate Certificates
Most enterprises do not sign machine or applications certificates with their top-level CA certificate; they use an
intermediate certificate. Using an intermediate certificate requires a certificate chain when creating the trust-store.
Personally, I use intermediate certificates, so it is a reminder on how to properly chain them when creating the pkcs12 file.
The openSSL configuration file
The configuration file of OpenSSL, openssl.cnf can vary greatly between Unix systems and even between Linux distributions.
For every command of openssl that uses openssl.cfg provide a configuration file; otherwise, a default file will be used
and it varies greatly between OS distributions. To avoid confusing on how OpenSSL executes your request, I explicitly
provide the openssl.cfg to use. I keep this file with my scripts. Furthermore, if you are scripting your certificate
process, leverage inline files to provide a custom configuration file to each specific command execution; it is then
self-documenting too.
An example of inlining a file concatenation.
-config <(cat ./openssl.cnf <(printf "\n[ext]\nbasicConstraints=CA:TRUE,pathlen:0"))
Check your OpenSSL documentation
OpenSSL varies between Unix variants and even between Linux distributions; please read the manual pages for your specific
version and release of openssl. Also, there could be a bug or limitation with the instance of openssl you are using.
For example, I was not successful leveraging copy_extensions on MacOS.
Conclusion
What I have documented here were the major pieces I had to figure out when building certificates for client authentication
for my Kafka Cluster. It wouldn't surprise me if there are other things I could uncover, especially limiting extensions
only between the broker certificates and client certificates.
If you want a glimpse to the scripts that I will use in part 2 of this blog where I configure my Secured (Mix-Protocol)
Kafka Cluster, you find them all at Kafka SSL Cluster.
Top comments (2)
Neil,
To implement the above, is clientAuth absolutely necessary when generating the certificates? Will kafka fail to establish an SSLHandshake with the client even if the credentials are all valid?
Best,
Marc
for SSL encryption, the setting wasn't needed. It was when I added client authentication did I see the need for it. However, I keep learning on the various nuances of certificates so I should attempt to remove as many extensions as possible. I created the broker and client certificates the same, and I am expecting that is not needed, so I should re-evaluate with the idea that the extensions between them are different.