How Datto is Issuing Let’s Encrypt Certificates for 65,000 Servers

By Philipp Heckel

October 4, 2018

By: Philipp Heckel

Let’s Encrypt is a revolutionary new certificate authority that provides free certificates in a completely automated process. These certificates are issued via the ACME protocol. Over the past two years or so, the Internet has widely adopted Let’s Encrypt — over 50 percent of the web’s SSL/TLS certificates are now issued by Let’s Encrypt.

But while there are many tools to automatically renew certificates for publicly available web servers it’s hard to find any useful information on how to issue certificates for internal non Internet facing servers with Let’s Encrypt. In this blog, we’re filling the gap by describing a way to issue certificates to internal servers using Let’s Encrypt. We used this exact mechanism to deploy a Let’s Encrypt certificate to each of our 65,000+ BCDR appliances.

How Does it Work?

To issue a certificate through Let’s Encrypt, you must prove that you either own the website you want to issue the certificate for or that you own the domain it runs on. Typically, automated tools like certbot use the HTTP challenge to prove site ownership using the well-known directory. While this works beautifully if the site is Internet-facing (and Let’s Encrypt can verify the HTTP challenge files via a simple HTTP request), it doesn’t work if your server runs on 10.1.1.4 or any other internal address.

The DNS challenge solves this problem by letting you prove domain ownership through the DNS TXT record _acme-challenge.example.com. Let’s Encrypt will verify that the record matches what it expects and issue your certificate if it all adds up.

So really, the only things you need to issue certificates for servers behind a NAT are:

  • A dedicated DNS zone for all your internal devices, e.g. xi8qz.example.com, and a dynamic DNS server to manage this zone
  • An ACME client capable of using the Let’s Encrypt’s DNS challenge to prove domain ownership

The general approach is simple: The appliance regularly reaches out to our control server to ensure that it can be reached via its own subdomain. If its local IP address changes, it triggers an update of its own subdomain. In addition, it checks regularly if the certificate is still valid, and requests a renewal if it’s outdated.

To learn more about the process, head over to my personal blog for a full write up including diagrams and screenshots