Rendered at 19:33:17 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
jaas 2 days ago [-]
Let's Encrypt has been working normally for most of the day. There was a ~90 minute period during which some of our users would have received a higher error rate due to upstream networking issues, but the majority of requests were successful even during that period.
It seems our status.io notes are being misinterpreted as much more severe than they were intended to reflect.
Edit: Note that this was written in response to a previous submission title implying that Let's Encrypt was entirely down most of the day.
widdakay 2 days ago [-]
I'm not sure if your higher error rate is sticky per user or something, but I've tried 10+ times throughout the day and have had 0 successes. They all come back as internal server error. That's why I eventually posted.
jaas 2 days ago [-]
It would not have been sticky for the entire day. If it was sticky at all, it would have been only during the 90 minute period I referenced. It's most likely that there is some other issue with how you're requesting the cert. Folks can help debug at: https://community.letsencrypt.org/
widdakay 2 days ago [-]
I ran the exact same command now and it's working, so it is possible I was unlucky and was hitting all the worst possible cases.
sgt 2 days ago [-]
Could it be that he was simply throttled while retrying? That seems plausible, and it would make it seem like a long outage.
widdakay 2 days ago [-]
I updated the post title to say (Fixed) now.
jaas 2 days ago [-]
Since Let's Encrypt wasn't down most of the day if would be helpful if you could update the title to reflect that.
widdakay 2 days ago [-]
I updated the title. Let me know if you think it's more accurate. It did appear as down for me though.
jaas 2 days ago [-]
Yeah, thanks
widdakay 2 days ago [-]
I did not intend this to hit the top of the front page lol. I just posted it and then came back 15 minutes later to it having exploded.
jaas 2 days ago [-]
No worries
taspeotis 2 days ago [-]
Thanks for securing the web
sam_lowry_ 2 days ago [-]
Thank them for making the web depend on a single US-based shady org, as if DNS was not enough.
cpach 1 days ago [-]
Feel free to launch your own CA.
sam_lowry_ 1 days ago [-]
No-no, I would rather go back to the good old HTTP/1.1.
P.S. JS injection into TCP packets and other meddling with passthrough data should be banned legally, not technically via encryption.
soco 1 days ago [-]
I wish you good luck in court trying to get compensation for the damage you've got through a JS injection attack. Because people prefer to lock their valuables instead of constantly having to identify and sue thieves.
sam_lowry_ 1 days ago [-]
Not court, regulation. Wanna be a carrier? Then don't meddle with traffic. Otherwise, you are liable for all the child porn and drug trade that happen to cross your boundary.
Natfan 1 days ago [-]
am i a carrier when i host my own wifi network?
soco 1 days ago [-]
Right, we actually agree on this. And where is regulation enforced? In courts. Who says the provider is liable? A court. This was my point: not locking your house means a lot of processing to get back to the place where you have been before - if it can ever happen. And I'd very much not go through a whole trial...
sam_lowry_ 1 days ago [-]
Regulation is enforced in courts only in the US, heh.
teekert 2 days ago [-]
Why are you trying? Doesn’t Caddy (or something) just takes care of this well in advance and should have no issues with one or several days of my service at all at any time?
Edit: my bad. I’ve tried as well recently, when you’re rushing to get your new domain up of course…
dlcarrier 2 days ago [-]
That explains why one of my IoT vendors is using an expired certificate.
I wish Firefox would just give a mild warning for a recently expired certificate, instead of treating it the same as a true man-in-the-middle attach. It's not like someone who couldn't factor the private key in 200 days could in 201 days or even 300 days.
I'm convinced that we'd have better security, if we didn't have so much security theater. You'd think TLS is useless, from the warning my phone gives if I connected to a public Wi-Fi AP, but then again there's nothing in TLS (or WPA) that prevents it from being used in a way that is completely useless: https://www.youtube.com/watch?v=M1si1y5lvkk
jaas 2 days ago [-]
> That explains why one of my IoT vendors is using an expired certificate.
I don't think so. There was a dip in success rates for 90 minutes today, but nobody should be renewing their certificate within 90 minutes of expiration. If you're at that point, something went wrong weeks ago.
mannyv 2 days ago [-]
"nobody should be renewing their certificate within 90 minutes of expiration"
You obviously haven't worked with hardware guys.
"I mean, what's the point of those last 30 days if you need to renew it 30 days before expiration? Why not just renew it before it expires? If I'm required to renew it 30 days before the expiration date then the expiration date is a lie, isn't it?"
ozim 2 days ago [-]
If they make 7 days grace period then expiration date will be a lie and of course every one will use grace period like it would be normal thing ;)
NewJazz 2 days ago [-]
Roulette grace period, keep them on their toes.
selcuka 2 days ago [-]
> If I'm required to renew it 30 days before the expiration date then the expiration date is a lie, isn't it?
Many countries won't let you enter if your passport expires less than 6 months after your planned departure date. Basically the effective validity of a passport is 0.5 years less than the period you pay for.
LtWorf 2 days ago [-]
> weeks ago
How long do you think a certificate lives?
jaas 2 days ago [-]
Mostly 90 days, and we recommend renewing at 60 days for 90 day certs. That gives more than four weeks of leeway.
If you're one of the few early adopters of short-lived (6-day) certs you should renew at 3 days, giving you 3 days for a successful renewal. A 90 minute outage, even if it was a full outage, would not interfere with a successful renewal.
selcuka 2 days ago [-]
> If you're one of the few early adopters of short-lived (6-day) certs you should renew at 3 days
Apparently certificates are becoming OCSP-only with a TTL.
nottorp 2 days ago [-]
How's the push for 48 hour certificates going?
bebop 2 days ago [-]
90 days moving to 45 but you can and should renew earlier than that. Automating this process means that you should be request a new certificates roughly 60 days (or 30 soon) after the issuance of the previous certificate. That way you would have plenty of time to deal with renewal issues. The process for renewal should have back off and retries built in. This prevents a situation where a down time for the issuer means that your production environments are non-functional.
Biganon 2 days ago [-]
They work at letsencrypt, I'm pretty sure they know.
dingaling 2 days ago [-]
> I wish Firefox would just give a mild warning for a recently expired certificate
Nope, if the SSL industry continues to insist on increasingly short cert lifetimes then I want Firefox to give no quarter when a cert expires.
Play by their rules and fall by their rules too.
mannyv 2 days ago [-]
Certificate expiry is less severe than an untrusted issuer or a host mismatch.
The former is most likely an administrative error (ie: someone forgot to renew, or the auto-renew is failing). The latter is more likely to be an MTM attack.
I'm not sure how you would use an expired cert as an attack vector. By loading in an old cert into an expired domain so you could spoof older content?
mcpherrinm 2 days ago [-]
If a key is breached, the certificate can be revoked, but that revocation goes away once the certificate is expired.
Expiry is a pretty fundamental part of the security model of certificates.
tgsovlerkhgsel 2 days ago [-]
Revocation information may not be available for expired certificates. Not that it matters much because the last time I checked revocation didn't really work for non-expired certificates either, but I think that (+ the risk of people treating expired certificates as worthless and thus increasing the risk of exposure) is the main reason.
Also of course domains changing owners, but again... I don't think we have good monitoring for that during the current long lifetime, so maybe a grace period where a warning is shown but it's easier to click through would be a good idea. Perhaps combined with a requirement to keep revocation information (and keep revoking expired certificates) X days past expiry.
How does that help? Seems like mostly the end user suffers.
hannob 2 days ago [-]
There are reasons browsers do things the way they do.
Experience and user studies have shown that users have a hard time decoding what error messages mean. "This certificate is expired, but only for a little while" isn't meaningful for people who don't have a mental model of what a certificate is.
Furthermore, "downgrading" warnings increases the incentive to ignore issues, potentially causing more problems down the line.
bruce511 2 days ago [-]
But it's only the extreme warning that alerts the website (usually via a customer complaining) that the cert hasn't been renewed. Having the lesser warning just kicks the can down the road.
The IoT should have updated the certs weeks in advance. If they haven't done it by day 0 then their process is broken and delaying the scary warning to say day +5 won't solve anything.
lambdaone 1 days ago [-]
What might be better is to, in addition to failing hard when the certificate expires, web browsers were to give a 'soft' click-through user warning if the certificate on the site - while still within its validity period - has less than say 7 days to go before expiry.
That's probably long enough for most companies to be alerted to the problem in time and to get their act together to fix the problem.
tgsovlerkhgsel 2 days ago [-]
A warning with a clear clickthrough button would work for alerting - the default TLS warnings are designed to be somewhat hard to bypass to make people think twice.
2 days ago [-]
bluesign 1 days ago [-]
What you want is warning when certificate expiry in next 7 days, then everyone would update before the warning.
fragmede 2 days ago [-]
omg new tom7!
saagarjha 2 days ago [-]
Seems not ideal for an entity who seems to be pushing for shorter expiration periods all the time
Dylan16807 2 days ago [-]
If it goes past 24 hours, that becomes a real worry.
If anyone is renewing certificates with less than a day remaining, that's an issue on their end far more than anything else.
xp84 2 days ago [-]
I think it’s mostly Apple and maybe Google who have the hard-ons for the shortest expiries possible.
fragmede 2 days ago [-]
To be fair, if someone managed to steal a set of keys to Gmail.com and icloud.com, I would want them to expire as short a time as possible too.
spragl 2 days ago [-]
That is right, but one thing is not like the other. You have always been free to set expiry low on your own certificates, but that is not the same as enforcing it on everyones ceritificate.
notrealyme123 2 days ago [-]
I think revoking them would be better in such a case.
flakes 2 days ago [-]
One is not really better, you want both. Certificate revocation lists are loaded out of band and depending on the client can be poorly enforced.
Questions come up: do you block a request if you fail to download the latest CRL? How often do you refresh it?
When the cert expires, it can be removed from the CRL, so shorter lived certs will allow CRLs to be smaller and faster to transfer.
naturalmovement 1 days ago [-]
> Questions come up: do you block a request if you fail to download the latest CRL? How often do you refresh it?
In the before times we left settings like this up to competent system administrators to decide based on risk and not hardcoded by a handful of people at Google.
dijit 1 days ago [-]
> competent system administrators
Sorry, we don't hire those anymore.
Best I can do is a YAML monkey who knows how to glue cloud services together..
icedchai 1 days ago [-]
So true. The last time I worked with a person with an actual "system administrator" title was 2009!
Stale news. Mozilla introduced a new solution for certificate revocation that solves nearly all the problems with old methods. While it hasn't really taken off outside of Firefox, that's mostly because Google and Apple haven't embraced it because they are too busy trying to shorten certificate life unnecessarily.
> While it hasn't really taken off outside of Firefox
Thus doesn’t really work. Sadly.
zx8080 1 days ago [-]
What is the reason that they are shortening it?
naturalmovement 1 days ago [-]
Revocation doesn't work because a cabal of arrogant Googlenos and friends decided it's too hard to fix so we won't do it at all.
The last browser where revocation worked properly is Internet Fucking Explorer.
tonyhart7 2 days ago [-]
isn't this the other way around ??? because shorter expiration time resulting on more issuing cert and therefore make it more prone to downtime
RetroTechie 1 days ago [-]
And perhaps more opportunities to insert bad certificate somehow.
Kesseki 2 days ago [-]
To be clear, “Degraded Performance” means just that, not “down.” Let’s Encrypt’s issuance is mostly working fine.
saagarjha 2 days ago [-]
I see you are unfamiliar with status page-ese. “Degraded performance” is a term which means some form of “the entire datacenter is probably on fire”.
Kesseki 2 days ago [-]
Although I only post here personally, I work for Let’s Encrypt.
number6 2 days ago [-]
Thanks you for your work!
ofrzeta 2 days ago [-]
It would be better to say this upfront. I am not blaming you in any way but this would prevent responses such as the parent's (hopefully).
dlcarrier 2 days ago [-]
Let them know that they're having an outage. If their monitors aren't telling them so, they might need to host them off-site.
Kesseki 2 days ago [-]
Let's Encrypt is operating normally. If you're having trouble, please post the details on the community forum so that folks can help you out. There is external monitoring in place.
AceJohnny2 2 days ago [-]
A common confusion; this interpretation only applies to OVH.
That would a Microsoft'ese, "Some regions are encountering issues" => "The entire world is down, but our status page is working"
zelphirkalt 1 days ago [-]
I also see that fitting into the corporate language of Gaslightese.
AceJohnny2 2 days ago [-]
I thought it meant "electricity has ceased to be a physical phenomenon in the general vicinity of our servers"
widdakay 2 days ago [-]
I have tried many times to renew my certs and have had 0 successes throughout today. It seems to be 100% degraded to me.
Kesseki 2 days ago [-]
That’s unexpected. Please post details on the “Help” topic of the Let’s Encrypt community forum so that folks can take a look.
2 days ago [-]
greatgib 1 days ago [-]
They claim "Degraded Performance", but 400 and 500 error responses is a non fully working service and not a performance that is just "less good".
> Some clients may encounter 400 and 500 error responses.
gib444 2 days ago [-]
What % of requests succeeded vs failed? How many certificates were issued during the outage vs the average? That might actually clear things up
pibaker 2 days ago [-]
What are the viable alternatives to LE? And in case none exists, what does it take to build one?
Requirements: free, available to everyone, automation friendly, issues certificates that are actually considered trustworthy by other parties.
treesknees 2 days ago [-]
ZeroSSL – free 90-day certs via ACME, also has a web UI for cert management
Google Trust Services – free ACME certs, requires a Google account for registration
SSL.com Free DV SSL – offers free 90-day certs through ACME
polpo 2 days ago [-]
I use acme.sh for certs on my personal server and was a little surprised when it started using ZeroSSL by default. Despite being more "corporate" I decided to roll with it and it's worked just fine.
It's a bit mathy, but if you can make it through that, I highly recommend watching the whole video, especially if you like dad jokes.
evbogue 2 days ago [-]
Like peers could sign sites?
ksimukka 2 days ago [-]
[dead]
otabdeveloper4 2 days ago [-]
> What are the viable alternatives to LE?
None. Big tech intentionally made Let's Encrypt a single point of giant failure.
> And in case none exists, what does it take to build one?
A new Internet and Web standards stack. The whole problem is self-imposed -- we could have published self-signed Ed25519 keys on the DNS instead, and the result would be more secure than whatever it is we have now.
icedchai 1 days ago [-]
Do you remember the early days of SSL certificates? It took an act of god just to get a certificate: verification rituals like faxing corporate paper work, phone calls, manually reissuing certs because someone forgot the "www", forgotten renewals...
Let's Encrypt is incredible.
ardeaver 2 days ago [-]
I realize this is very much not the point, but the fact that the "Active Incident" banner is green is upsetting.
Kesseki 2 days ago [-]
The banner's colour is based on the "Incident Status;" it's green because services are currently operational. It would be yellow or red if the impact were more severe.
dxdm 2 days ago [-]
Using only color to communicate the status is confusing. If you want to communicate something, it's often best to just say it. The color can be a visual reinforcement of that. Then your explanation would not be needed.
Kesseki 2 days ago [-]
We do say it. That's what the "Incident Status" field is there for.
dxdm 1 days ago [-]
But that's not were the confusion is created. I don't even see the status field on mobile without scrolling. You don't have a missing status field, you have too much confusion, because the field and/or the color have a placement mismatch.
dlcarrier 2 days ago [-]
Their monitors don't seem to be detecting the outage. Sometimes they run directly on the server, and aren't able to detect routing or DNS problems.
NewJazz 2 days ago [-]
We're operating normally, but with reduced redundancy. We continue to work with our upstream ISP to identify and resolve the issue.
nubinetwork 2 days ago [-]
It's a good thing that acme clients try to renew early, rather than leaving it to the last minute...
po1nt 2 days ago [-]
Let's encrypt is a single point of failure for a large percentage of the internet.
gsliepen 1 days ago [-]
No, it's not. You can always switch to a different SSL provider. There are other free ones (as mentioned in other comments).
However, thinking about how to make your own setup more robust without having to manually change configuration when one SSL provider stops working is a good exercise. I wonder if you can just get your server's private key signed by multiple SSL providers, and serve multiple certificates to clients, and whether all browsers handle that correctly.
po1nt 21 hours ago [-]
If you couldn't switch, that would be a monopoly. But single point of failure is when you put all your fruit in one basket. Airplanes have redundant systems, even though you can always buy new components. But it's much harder to change them mid-flight.
gsliepen 10 hours ago [-]
Ok, but that would just be your own website having a single point of failure, not that Let's Encrypt is a single point of failure. Otherwise you could call every certificate authority a single point of failure.
doublerabbit 1 days ago [-]
Nothing is a point of failure if you can switch but that's not really true unless you have fail-over.
If LE was to go nope right now, how fast could you move your stack from LE?
You can't use multiple SSL certificates as redundancy. You could probably create something bespoke with a Load Balancer and SSL offloading but that's just more overhead for really nothing.
po1nt 21 hours ago [-]
Just picture the massive load spikes on other SSL providers in that moment. And the fact that even those might not work, as their backends might rely on LE SSL 3rd party services for ID checking or something.
anal_reactor 1 days ago [-]
Hot take, but in general single points of failure are less of an issue than it seems because usually outages simply aren't that common. Meanwhile maintaining whole infrastructure to avoid single point of failure is often very expensive.
po1nt 21 hours ago [-]
In theory this sounds great, but you only realize how much do you rely on a single point of failure, once it fails. Just see github outages or even electricity outages at your home.
anal_reactor 13 hours ago [-]
> electricity outages at your home
I haven't had one in 20 years, which kinda proves my point.
drsalt 2 days ago [-]
thats too bad
ta3029382 1 days ago [-]
[dead]
tomalbrc 2 days ago [-]
[flagged]
bruce511 2 days ago [-]
You are getting down-voted for this, which I think is a bit unfair. (I expect I'll get the same.)
Although you don't expand your thesis, as a general feeling, I agree. But, to be fair, it has always been thus, and it has been this way in every forum ever.
I'm old enough to remember the irony in "I read about it on the internet so it must be true" statements, which have existed since the internet was News (NNTP) not web.
In truth, any time you get a random group of people together, of different ages and backgrounds, all of whom self-describe as "smart" you're going to get a lot of chaff mixed in with the wheat.
To some extent you need to simply ignore the nonsense. There's plenty of it and "correcting people who are wrong" is seldom received well.
It seems our status.io notes are being misinterpreted as much more severe than they were intended to reflect.
Edit: Note that this was written in response to a previous submission title implying that Let's Encrypt was entirely down most of the day.
P.S. JS injection into TCP packets and other meddling with passthrough data should be banned legally, not technically via encryption.
Edit: my bad. I’ve tried as well recently, when you’re rushing to get your new domain up of course…
I wish Firefox would just give a mild warning for a recently expired certificate, instead of treating it the same as a true man-in-the-middle attach. It's not like someone who couldn't factor the private key in 200 days could in 201 days or even 300 days.
I'm convinced that we'd have better security, if we didn't have so much security theater. You'd think TLS is useless, from the warning my phone gives if I connected to a public Wi-Fi AP, but then again there's nothing in TLS (or WPA) that prevents it from being used in a way that is completely useless: https://www.youtube.com/watch?v=M1si1y5lvkk
I don't think so. There was a dip in success rates for 90 minutes today, but nobody should be renewing their certificate within 90 minutes of expiration. If you're at that point, something went wrong weeks ago.
You obviously haven't worked with hardware guys.
"I mean, what's the point of those last 30 days if you need to renew it 30 days before expiration? Why not just renew it before it expires? If I'm required to renew it 30 days before the expiration date then the expiration date is a lie, isn't it?"
Many countries won't let you enter if your passport expires less than 6 months after your planned departure date. Basically the effective validity of a passport is 0.5 years less than the period you pay for.
How long do you think a certificate lives?
If you're one of the few early adopters of short-lived (6-day) certs you should renew at 3 days, giving you 3 days for a successful renewal. A 90 minute outage, even if it was a full outage, would not interfere with a successful renewal.
Apparently certificates are becoming OCSP-only with a TTL.
Nope, if the SSL industry continues to insist on increasingly short cert lifetimes then I want Firefox to give no quarter when a cert expires.
Play by their rules and fall by their rules too.
The former is most likely an administrative error (ie: someone forgot to renew, or the auto-renew is failing). The latter is more likely to be an MTM attack.
I'm not sure how you would use an expired cert as an attack vector. By loading in an old cert into an expired domain so you could spoof older content?
Expiry is a pretty fundamental part of the security model of certificates.
Also of course domains changing owners, but again... I don't think we have good monitoring for that during the current long lifetime, so maybe a grace period where a warning is shown but it's easier to click through would be a good idea. Perhaps combined with a requirement to keep revocation information (and keep revoking expired certificates) X days past expiry.
Experience and user studies have shown that users have a hard time decoding what error messages mean. "This certificate is expired, but only for a little while" isn't meaningful for people who don't have a mental model of what a certificate is.
Furthermore, "downgrading" warnings increases the incentive to ignore issues, potentially causing more problems down the line.
The IoT should have updated the certs weeks in advance. If they haven't done it by day 0 then their process is broken and delaying the scary warning to say day +5 won't solve anything.
That's probably long enough for most companies to be alerted to the problem in time and to get their act together to fix the problem.
If anyone is renewing certificates with less than a day remaining, that's an issue on their end far more than anything else.
Questions come up: do you block a request if you fail to download the latest CRL? How often do you refresh it?
When the cert expires, it can be removed from the CRL, so shorter lived certs will allow CRLs to be smaller and faster to transfer.
In the before times we left settings like this up to competent system administrators to decide based on risk and not hardcoded by a handful of people at Google.
Sorry, we don't hire those anymore.
Best I can do is a YAML monkey who knows how to glue cloud services together..
https://garantir.io/certificate-revocation-challenges-and-be...
https://hacks.mozilla.org/2025/08/crlite-fast-private-and-co...
Thus doesn’t really work. Sadly.
The last browser where revocation worked properly is Internet Fucking Explorer.
ref: https://www.reuters.com/article/world/millions-of-websites-o...
> Some clients may encounter 400 and 500 error responses.
Requirements: free, available to everyone, automation friendly, issues certificates that are actually considered trustworthy by other parties.
Google Trust Services – free ACME certs, requires a Google account for registration
SSL.com Free DV SSL – offers free 90-day certs through ACME
It seems a bit silly that a service that could be forced by EO to revoke foreign certificates is the backbone of so much of the internet.
It's a bit mathy, but if you can make it through that, I highly recommend watching the whole video, especially if you like dad jokes.
None. Big tech intentionally made Let's Encrypt a single point of giant failure.
> And in case none exists, what does it take to build one?
A new Internet and Web standards stack. The whole problem is self-imposed -- we could have published self-signed Ed25519 keys on the DNS instead, and the result would be more secure than whatever it is we have now.
Let's Encrypt is incredible.
However, thinking about how to make your own setup more robust without having to manually change configuration when one SSL provider stops working is a good exercise. I wonder if you can just get your server's private key signed by multiple SSL providers, and serve multiple certificates to clients, and whether all browsers handle that correctly.
If LE was to go nope right now, how fast could you move your stack from LE?
You can't use multiple SSL certificates as redundancy. You could probably create something bespoke with a Load Balancer and SSL offloading but that's just more overhead for really nothing.
I haven't had one in 20 years, which kinda proves my point.
Although you don't expand your thesis, as a general feeling, I agree. But, to be fair, it has always been thus, and it has been this way in every forum ever.
I'm old enough to remember the irony in "I read about it on the internet so it must be true" statements, which have existed since the internet was News (NNTP) not web.
In truth, any time you get a random group of people together, of different ages and backgrounds, all of whom self-describe as "smart" you're going to get a lot of chaff mixed in with the wheat.
To some extent you need to simply ignore the nonsense. There's plenty of it and "correcting people who are wrong" is seldom received well.