Earlier in the week, you may have noticed some… issues when connecting to our sites. Last Friday, I had updated our SSL certificates to new SHA2 certificates from Gandi. I went through the normal process to get a replacement certificate: specifying SHA2 in the CSR, grabbing the newly generated certificate and intermediate certificate, replacing them in the ansible scripts, and deploying them to our servers. Everything looked ok in Firefox, so I pushed to production. A little bit later, we started receiving reports that some of the sites wouldn’t load only in Chrome for Android. Oops.

Of course, issues with one specific mobile browser isn’t necessarily an emergency; it signifies something to research and look into, though. Fast forward a couple days, and we started experiencing issues loading the sites almost at random: some people could load some of the sites in Chrome, others’ couldn’t load any, Safari was refusing to load anything, and Firefox would present OCSP errors only when connecting to the servers running Apache until Apache was restarted. Then a few minutes later the OCSP errors would return. Major oops. This is now an emergency. At this point, if you aren’t familiar with how PKI (Public Key Infrastructure) works, I suggest reading this article. Don’t worry, it’s fun.

Investigating the issue

Looking at the certificate information in Firefox showed everything looking normal. The chain went up to the root CA and there were no errors. Qualys’ SSL Labs test came back with a A+ rating and no glaring errors in the certificate chain. However, SSLShopper’s SSL Checker showed this:

Um. Ok. The trust chain is broken. But on a site that was still using the old SHA1 Gandi cert it was showing this:

The setup for the two sites was identical. The only things that had changed were swapping out the server certificate and the intermediate certificate. And yet, the SSL Checker tool was implicitly trusting the old Gandi cert.

The Gandi certs are in the Firefox certificate store, though.

What happened?

So it looks like the cause of all this is that some certificate stores, like desktop Firefox, have the new SHA2 Gandi CA2 cert in their trust stores while others do not yet. Referring back to the metaphor used in the article linked above, our old SHA1 certificate for our server was the credentials for Ensign Tony, which were signed by Lt. La Forge (Gandi). Our browser knows Lt. La Forge (his signature is in the trust store) and already knows the trust chain goes up to Picard. When we changed to the SHA2 certificate, we replaced Ensign Tony with Ensign Lopez. Ensign Lopez reports instead to Lt. Worf, who some browsers already know and other do not. We therefore need to build the trust chain to Picard for those browsers who do not know Lt. Worf.

There’s a slight wrinkle in this as the Gandi certificate is not signed by the root certificate (AddTrust External CA Root), but by another intermediate certificate (USERTrust RSA CA). So we have to add another link in our trust chain, the officer who signs Worf’s credentials: Commander Riker.

All browsers should trust Commander Riker’s credentials because they were signed directly by Captain Picard himself, who all browsers trust implicitly on reputation. Therefore, once presented with the proper chain (Ensign Lopez -> Lt. Worf -> Commander Riker) there’s no reason not to trust Ensign Lopez is who he says he is even if we haven’t met Lt. Worf; the Captain is essentially vouching for him!

OK, so how do I fix it?

In order to get every browser to accept the certificates, we have to present three certificates: our server certificate, the Gandi Standard SSL CA2 intermediate certificate, and the USERTrust RSA Certification Authority intermediate certificate. Apache and Nginx ended up handling the chains differently, so I’ll go over the configurations I came up with for each.

Apache

First, download all of three of the needed certificates. For this example I’ll call them star-domain-com.crt, GandiSSLCA2.pem, and USERTrustRSA.pem. Since Apache has both the SSLCertificateFile and SSLCertificateChainFile directives, it needs two certificate files: a file with the server certificate and a file with the full chain of certificates. To create the full chain file, star-domain-com-fullchain.crt:

cat star-domain-com.crt GandiSSLCA2.pem USERTRustRSA.pem >> star-domain-com-fullchain.crt

Now, add the server certificate and fullchain certificates to the following directives in the Apache vhost config:

SSLCertificateFile /path/to/star-domain-com.crt
SSLCertificateChainFile /path/to/star-domain-com-fullchain.crt

Doing so creates the following chain in the SSL Checker tool:

Nginx

Unlike Apache, Nginx does not have separate server certificate and certificate chain file directives. It only has the ssl_certificate directive which takes the full chain directly. Just like for Apache, create a fullchain certificate file like so:

cat star-domain-com.crt GandiSSLCA2.pem USERTRustRSA.pem >> star-domain-com-fullchain.crt

And then use the following directive in the vhost config:

ssl_certificate /path/to/star-domain-com-fullchain.crt

The Nginx site will now show the full chain in the SSL Checker tool. Every browser (that supports SHA2 certificates) should now be able to connect without error!

Banner Photo: Catena a Boboli via photopin (license)