Poor StartCom. Since 2009—ever since I read Glenn Fleishman’s Ars piece on how to get free SSL/TLS certificates—StartCom has been my go-to for certs. Most welcome has been StartCom’s pricing on wildcard certs (that is, certificates for *.yourdomain.com, allowing you to use a single certificate for everything, if you desire) While other certificate authorities charge rigoddamndiculous prices for individual wildcard certs, StartCom gives you unlimited wildcard certs for the price of a single $60 identity validation (yes, you read that correctly). It has always been and remains an unbeatable deal. Literally. No one else comes close. Buying wildcard certs w/StartCom felt like I’d discovered an Internet cheat code.
Then a Chinese company called WoSign bought StartCom and started doing shady stuff—including, most egregiously, issuing and backdating certificates to try to circumvent the upcoming mass SHA-1 deprecation. The response has been swift and devastating: WoSign and StartCom’s certs will become untrusted by major web browsers (details on exactly what that means can be read here).
Which means, after eight years, I needed to find a new CA. And with how heavily I’ve leaned on wildcard certs, it wasn’t looking good.
Out with the old, in with the new
The obvious answer was Let’s Encrypt, a free and automated CA run by the Internet Security Research Group. LE allows server admins to automatically request SSL/TLS certificates, without having to manually interface with a CA or deal with CSRs. Certificates are issued pretty much instantaneously, and are valid for 90 days. The short duration isn’t a problem, because with some scripting your server should take care of renewing the certificate before it expires.
I’d looked at LE several times over the past year with interest, but getting it to work with my overly-complicated stack seemed problematic, primarily because LE’s system for issuing and renewing certificates appeared to require access to port 80—the same port I’ve got HAProxy listening on. My brief investigations always ended with a shrug. Implementing it looked like a lot of work for not a lot of benefit. Why bother?
Now that StartCom has gone full-blown evil, though, it was time to make the switch. And, fortunately, I waited long enough that other folks solved the technical issues—proving once again that if you put off an important task long enough, someone else will probably take care of it for you!
A quick refresher: the stack
Just to set the stage, I’m working with a three-application stack here at BigDinosaur.org: HAProxy for SSL/TLS termination, Varnish for caching, and Nginx for the actual pages. It looks something like this:
All pages are served via HTTPS, with HAProxy handling the SSL/TLS part. HAProxy passes unencrypted data to Varnish (via the PROXY protocol). Varnish does its cache magic, and reverse-proxies unencrypted data (via plain ol’ HTTP) to Nginx. Nginx either serves up static assets and web sites, or reverse-proxies in turn to something else (php-fpm, or a web app inside a Docker container, or whatever).
Let’s Encrypt this stuff
I primarily followed this lovely blog post by Digital Ocean’s Mitchell Anicas, which gets us almost where we need to be (Mitchell has us set up LE before HAProxy, whereas in this post we’re going to assume HAProxy is already up and running). I got some additional info and inspiration from the BrixIT Blog’s similar post. As it turns out, the amount of work needed isn’t nearly as terrifying as I’d originally thought when I first looked at everything.
In broad strokes, here’s the process we’re going to follow:
- Download some Let’s Encrypt tools
- Configure HAProxy to watch for Let’s Encrypt traffic and shunt that traffic to a new Let’s Encrypt backend
- Set up a script that will allow us to both request new certs and renew expiring ones
- Use the script and get some certs
- Set up a cron job to auto-renew the certificates before they expire
Step one: Download CertBot
For simplicity’s sake, we’re going to use a tool called CertBot to handle the process of acquiring and renewing Let’s Encrypt certificates. There are other options (security researcher Scott Helme prefers Daniel Roesler’s acme-tiny script, for example), but CertBot is easy to deal with and there’s plenty of documentation available for it if something goes wrong.
To get started, we need to download CertBot, which we’ll do by cloning its GitHub repo to our server:
$ sudo apt install git #In case you don't already have git installed
$ sudo mkdir /opt/letsencrypt
$ sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencryptStep two: Configure HAProxy
This is the magic bit, and the problem I couldn’t see past until Mitchell and others demonstrated that it wasn’t actually that big of a problem at all.
Certificate Authorities are allowed to issue certificates because they (ostensibly) verify who you are and vouch for your identify—this is the core assumption that underlies the entire global encrypted World Wide Web (and, hell, lots of other things, too—e-mail servers, IM servers, voice chat systems, and tons of other applications use SSL/TLS). In the pre-LE days, a traditional CA would always do some kind of identity validation before issuing you a certificate. Depending on the type of cert you’re requesting, a CA might simply check to make sure you can receive e-mail at your domain’s webmaster address; for an advanced EV cert (the kind that turns your address bar green) they might do an extremely thorough vetting job and ask to see copies of government-issued IDs and tax records. The point is that the CA is vouching for your identity and they have to check who you are.
Let’s Encrypt takes a different tack: when you ask for a certificate for something.yourdomain.com, the LE service checks yourdomain.com’s DNS entries for an entry for “something.yourdomain.com,” and then tries to contact the server over the web via the ACME protocol. Your local LE tool (in our case, CertBot) is supposed to be up and running on your server, listening on port 80 for the LE service’s ACME call. LE does a cryptographic handshake, verifies that the server requesting the certificate for “something.yourdomain.com” is indeed the server that the DNS entry for “something.yourdomain.com” points at, and then quickly signs your server’s CSR and issues you a certificate.
The problem, obviously, is that HAProxy is already listening on port 80, so you can’t run CertBot—it’ll panic and fail. This is always where I’d stopped before.
Ah, but! HAProxy is a powerful application, and it’s very good at, well, proxying. All we need to do is run CertBot on an alternate TCP port, then tell HAProxy to listen for the specific type of connection the LE service will try to make and forward that connection to CertBot. And, thankfully, the ACME connection from the LE service tries to hit a specific, easy-to-identify path.
Therefore, configuring HAProxy is a two-step process of first setting up a new back-end to point to our CertBot instance, and then telling HAProxy to forward ACME requests to that back-end instead of our normal backend. Easy!
Open HAProxy’s config file (usually /etc/haproxy/haproxy.cfg) for editing, and make the following additions in your frontend section, beneath your bind statements:
acl letsencryptrequest path_beg -i /.well-known/acme-challenge/
use_backend letsencrypt if letsencryptrequestThen, add a new backend to handle the requests:
backend letsencrypt
mode http
server letsencrypt 127.0.0.1:54321
# If you're running HAProxy on a different server than
# CertBot, use that server's address instead of localhost.Save the file and restart HAProxy.
Step three: get your script
Now we need a script to take care of our certificate issuance and renewal. Mitchell has one that works great with lots of logic in it, but I had much better luck adapting the following script from BrixIT instead. Copy and save it as /usr/local/bin/le-renew.sh (make sure to chmod it as executable):
#!/bin/bash
# Path to the letsencrypt-auto tool
LE_TOOL=/opt/letsencrypt/letsencrypt-auto
# Directory where the acme client puts the generated certs
LE_OUTPUT=/etc/letsencrypt/live
# Concat the requested domains
DOMAINS=""
for DOM in "$@"
do
DOMAINS+=" -d $DOM"
done
# Create or renew certificate for the domain(s) supplied for this tool
$LE_TOOL --agree-tos --force-renewal --standalone --standalone-supported-challenges http-01 --http-01-port 54321 certonly $DOMAINS
# Cat the certificate chain and the private key together for haproxy
cat $LE_OUTPUT/$1/{fullchain.pem,privkey.pem} > /etc/haproxy/ssl/${1}.pem
# Reload the haproxy daemon to activate the cert
systemctl reload haproxyThis script takes as input the domain for which we want to issue or renew a certificate, then handles the request, puts the certificate and key in the /etc/haproxy/ssl directory, and restarts HAProxy.
Step four: get a cert
Now it’s time to use all these fancy applications and scripts and actually request a certificate. Make sure you’ve restarted HAProxy so that the front- and backend changes you made are active, and then run the script, supplying as an argument the domain you want a certificate for. Make sure the name you’re requesting has an entry in your domain’s zone file (either an A-record or a CNAME—LE has some lingering issues with IPv6), and you should be good to go:
$ sudo /usr/local/bin/le-renew.sh testbox.bigdinosaur.orgThe process will run and then report back when it completes:
To verify that the request worked, take a gander at your /etc/haproxy/ssl directory. You should see a shiny new concatenated servername.yourdomain.whatever.pem file there. (You might also want to chmod 400 the file to keep it locked down, since it contains the certificate’s unencrypted private key, but it’s not as necessary as it would be with a standard multi-year certificate—this one expires in 90 days!)
If you’ve got multiple certificates to request, now’s the time. Get ’em all generated, and then move on to the next step.
Step five: automate with cron
The last step is to set an entry in your system crontab to call le-renew.sh once a month and handle the renewals. Edit /etc/crontab (or, if you feel like doing things the proper way, create a new crontab file in /etc/cron.d) and add one entry per certificate:
37 0 1 * * root /usr/local/bin/le-renew.sh yourserver.yourdomain.whatever
37 1 1 * * root /usr/local/bin/le-renew.sh yourserver2.yourdomain.whatever
37 2 1 * * root /usr/local/bin/le-renew.sh yourserver3.yourdomain.whateverOn the first of each month, starting at thirty-seven minutes after midnight, cron will kick off your renewals. I’ve got them spaced one hour apart; you can set them however you’d like.
Alternately, you can run the script without any arguments to trigger a renewal of all of your certificates. You could also take a more aggressive stance on renewal and run the script weekly instead of monthly. However, you don’t want to use an interval shorter than weekly because you risk running into the LE rate limits.
Step six: one last bit of HAProxy configuration
Okay, I lied—there’s one more step. We need to go back in and make sure HAProxy is actually using the certificates we generated rather than whatever old certs it was configured to use before. Open /etc/haproxy/haproxy.cfg back up for editing and modify your frontend configuration, substituting in your new certificates:
frontend yourfrontendname
bind *:80
bind *:443 ssl crt /etc/haproxy/ssl/firstcert.pem crt /etc/haproxy/ssl/secondcert.pem ...…and so on, adding each of your certificates into the list. Then restart HAProxy, and you’re all set. You shouldn’t need to change your HAProxy configuration in the future, since the renewal certs created by le-renew.sh will always have the same names.
Optional step seven: smarter renewals
As Scott Helme points out, the renewal process we’ve got here is pretty basic and has no recourse in the event of renewal failure. LE certs expire in 90 days, and by running the renewal script once per month you’ve essentially got three chances to renew before your certificate expires.
Things do go wrong sometimes (stupid computers!), and so Scott has an alternate “smart renewal” process that should be much more reliable in noticing when a renewal is needed and in following through with that renewal. I haven’t implemented it yet, but it’s probably a good idea.
Congrats, you’re done
If your setup is like mine, then no further configuration is needed—you don’t have to screw with Varnish or Nginx, since HAProxy is the only part of the stack that cares about SSL/TLS. Let’s Encrypt is still less convenient than a single wildcard certificate for everything, but it’s also arguably less risky, since a compromised key doesn’t give an attacker carte blanche to put up any site they want with your domain name.
Many thanks to Mitchell Anicas, BrixIT, and Scott Helme—this blog post is essentially a synthesis of their posts. Any praise should go entirely to them, and any mistakes are mine alone.