Let’s Encrypt Doomsday Scenario Narrowly Avoided
1 in 3 Android Devices Were Nearly Blocked from Millions of Websites
For months, Let’s Encrypt has been warning site owners that a change in their root was coming due to the expiration of the IdenTrust root they had been using to cross-sign their certificates. Alarming posts discussed how certain Android devices would not trust Let’s Encrypt certificates and users visiting websites would see errors.
Looking back at 2020, November and December were long months online for most — we lived day-to-day trying to avoid the Covid-19 virus, became experts at social distancing and building family memories via Zoom and other virtualized solutions like FaceTime.
For many website owners, this was also a time to prepare for a massive distrust event that was to start unfolding on January 11, 2021. Let’s Encrypt had tried to delay the unavoidable, but the time had come to start issuing certificates from their own root, ISRG Root X1. The only issue was that one third of Android devices were still using versions older than 7.1.1 and would not trust the certificates issued under the Let’s Encrypt root. 225 million websites would block one third of Android users. Site owners scrambled to find new certificate authorities (CAs) to replace Let’s Encrypt with certificates issued from widely trusted roots.
Root Change: Why it Was a Looming Threat to Millions of Businesses
In 2015 when Let’s Encrypt was started by the Internet Security Research Group (ISRG) and partners, they started by cross-signing to an existing certificate authority’s trusted root, IdenTrust’s DST Root X3 certificate. This enabled Let’s Encrypt to instantly issue certificates that were trusted across the internet, effectively biding them time while their root was propagated across browsers and operating systems.
Why? In order for a TLS/SSL certificate to be trusted by your browser, it must chain up to a trusted root certificate that’s in your browser’s root certificate trust store — but it can take years for a root certificate to be trusted by all the major browsers and operating systems.
However, the IdenTrust DST Root X3 is set to expire on September 30, 2021, so Let’s Encrypt was going to start transitioning to their root on January 11, 2021 to give site owners time to adjust to the change ahead of the hard date of September 30, 2021. In a post by Josh Aas, ISRG Executive Director, it’s clear that this was not an ideal solution since the Let’s Encrypt root is still not trusted by one-third of Android devices.
This was a doomsday scenario. But now there’s new news and it’s time to reflect.
Fast Forward: What You Can Learn from the Near Miss
It’s 2021 now, the sun is shining, the birds are not yet out, but it is a new day for Let’s Encrypt and millions of website owners can sigh a collective sigh of relief. It seems that Let’s Encrypt has been hard at work looking for a solution.
On December 21, 2020, Josh Aas and Aaron Gable posted an article titled, Extending Android Device Compatibility for Let’s Encrypt Certificates, that states the following:
“IdenTrust has agreed to issue a 3-year cross-sign for our ISRG Root X1 from their DST Root CA X3. The new cross-sign will be somewhat novel because it extends beyond the expiration of DST Root CA X3. This solution works because Android intentionally does not enforce the expiration dates of certificates used as trust anchors. ISRG and IdenTrust reached out to our auditors and root programs to review this plan and ensure there weren’t any compliance concerns.
As such, we will be able to provide subscribers with a chain that contains both ISRG Root X1 and DST Root CA X3, ensuring uninterrupted service to all users and avoiding the potential breakage we have been concerned about.
We will not be performing our previously-planned chain switch on January 11th, 2021. Instead, we will be switching to provide this new chain by default in late January or early February. The transition should have no impact on Let’s Encrypt subscribers, much like our switch to our R3 intermediate earlier this month.”
In a nutshell, this extended expiration prevents:
- one-third of Android device users being blocked from 225 million websites,
- you having to use a workaround until September 30, 2021; and,
- you having to shop for new certificate authority.
Hold that last thought…what you should be doing now
As a site owner, you might be thinking that you no longer need to consider having a plan B. Your gut may tell you that the stars will always align and you can get certificates for free forever, without having to worry. Or your gut may say that this was a close call and that having a plan B in place would be a smart idea to ensure your websites are not impacted in the future. In this virtual world, your website is your front door. Without it, your business can be severely disrupted, costing you thousands or millions of dollars of lost revenue.
If there is a lesson we can learn from PKI history, it’s that things change, the future is unpredictable and having a certificate management solution in place is always a good idea.
To learn about what can go wrong with PKI, Researchers Serrano, Hadan and Jean Camp from Indiana University Bloomington analyzed 379 reported instances of failures in certificate issuance to pinpoint the most common causes as well as systemic issues that contribute to these happening.
At a broad-brush level, one can also look at The Feisty Duck’s recently released TLS/SSL certificate PKI history timeline. It is quite illuminating and covers major PKI events from 1994 to the present.
The sophistication of PKI has evolved greatly over time (remember SHA-1?), distrust events are common, and techniques used by bad actors are increasingly complex. Reduced compatibility events like the one that nearly occurred for Let’s Encrypt are quite common and can result from different causes such as a CA compromise, browser distrust, or in this case a lack of compatibility for the root CA on older Android devices. The last major distrust event occurred in 2017 when Google announced its plans to distrust Symantec certificates due to mis-issuing thousands of transport layer security (SSL/TLS) certificates.
Could this happen again? You bet.
According to the research from Indiana University Bloomington, Google was very clear that they could decide to distrust other root certificates, “Google Chrome reserves the right to distrust root certificates present in the operating system’s root certificate list. At the core of trust in the PKI system is the fact that the operation of Root CAs is beyond reproach.”
It’s prudent to have a hedging strategy when it comes to using a CA, this is what is called CA agility. Having the ability to change CAs at any time using a single switch can be a powerful protective measure so you aren’t left scrambling. It’s just like when you invest in the stock market, you want to diversify your portfolio by not putting all your eggs into one basket and use a centralized platform to trade your stocks.
What to Do: Planning for an Uncertain Future
Although the Let’s Encrypt change was averted (for now), organizations using Let’s Encrypt should take heed and put in place a proper plan for managing their SSL/TLS certificates, also known as machine identities. When looking for a certificate management solution that delivers on the promise of maximum flexibility and control, here are five questions you should ask:
1. How quickly can I migrate to a different CA?
Organizations need to be able to respond quickly if a CA implementation is compromised through unauthorized access or corrupted due to lost keys — or due to a distrust event. The ability to switch CAs easily offers a critical security advantage and support business continuity.
2. How quickly can I respond to a CA compromise or serious error?
If large groups of certificates are disabled or distrusted, it leaves many CA customers scrambling to find another CA, perform the necessary validation steps with the CA to be able to request publicly trusted certificates from the CA, and replace existing certificates with new ones issued by the new CA. If you need to switch CAs, any automated processes, integrations with security infrastructure, and policies that were developed using the compromised CAs must be rewritten. By establishing a relationship with another CA and investing in solutions that allow you to automate switching between CAs at any time, CA compromise or distrust can become a virtual non-event for your organization.
3. What automated certificate renewal options does it offer?
Let’s Encrypt is an easy-to-use, automated solution that’s free but it’s not the only option. Most certificate management solutions offer ACME for automating renewals from multiple CAs with the same level of convenience — in addition, they can automate renewals on private endpoints — such as web and application servers — to save human resource time and stop outages due to expired certificates, something that Let’s Encrypt cannot do.
4. Will I be able to organize my certificates by application to assign ownership?
Most multi-tier applications have certificates installed on everything from the load balancer and web server to the application server. To assign ownership to certificates and manage certificates at a holistic level by application, it’s important to be able to organize certificates in a logical fashion. This is also helpful to ensure that critical business applications don’t experience downtime due to an expired certificate.
5. Can I get certificate intelligence from more than one CA?
For organizations that use multiple CAs, it’s generally not possible to manage TLS/SSL certificates across multiple CAs using a single CA management console. This means a holistic view of all your organizations’ keys and certificates will require ongoing manual effort that is error-prone and time-consuming.
6. Will I be able to identify and react to a certificate-related outage or breach?
If you rely on multiple CAs and don’t have enterprise-wide awareness of your key and certificate security posture, you may not be able to quickly detect and respond to a misused certificate, an unplanned outage, or a vulnerability. And when you’re experiencing an outage or a breach, time is critical. The longer the outage or breach continues, the greater the potential damage to your organization.
To maximize business agility, look for certificate management solutions that allow you to actively manage all your certificates from a single console as well as integrate with multiple CAs for obtaining public/privately trusted certificates. You won’t tie your organization’s security posture to any single CA vendor. Plus, you’ll be equipped to implement consistent security policies across all TLS/SSL certificate machine identities and deliver audit-ready reports and documentation about your company-wide key and certificate management program.
Free Resources
If you want to get a handle on your TLS/SSL certificates, check out Venafi Cloud, it’s a free certificate management solution to help you:
- Find every certificate and where it is used
- Know where your wildcard certificates are installed
- Use ACME to auto-renew certificates from leading CAs
- Get alerted of expiring certificates (stop outages)
- Request certificates from multiple CAs (flip the switch)
- Administer ownership by application (organize)
- Integrate certificates into CI/CD pipelines by application to assign ownership?
