CDN Issues
Incident Report for Netlify
Postmortem

We’ve written up the timeline, details, mitigations, and future plans in this blog post: https://www.netlify.com/blog/2021/04/02/load-balancer-service-degradation-march-25-2021/

Posted Apr 02, 2021 - 14:09 PDT

Resolved
Service by our old Load Balancer has been restored and all sites should be working whether or not you have updated your DNS as recommended in earlier updates. If you still have any trouble, please follow up in the helpdesk if you have a Pro+ plan, or in https://answers.netlify.com/t/followups-from-25-mar-2021-service-degradation/34913 if you do not.

We will post a root cause analysis about this incident and how we'll prevent recurrence going forward on our blog, and link it from this status page incident as well.
Posted Mar 25, 2021 - 11:59 PDT
Update
Our upstream provider is working to resolve the issue which is impacting bare domains which use the now-old 104.198.14.52 IP. The provider advises us the estimated time to fix is still several hours, so we advise you to update the A record value for your site(s) bare domain to 75.2.60.5, in order to mitigate against this outage. This link – https://docs.netlify.com/domains-https/custom-domains/configure-external-dns/#configure-an-apex-domain – contains instructions to perform this action.
Posted Mar 25, 2021 - 10:42 PDT
Update
Our team have created a new load balancer instance which is not associated with the upstream provider who is currently experiencing issues. Please update A record values for your site(s) bare domain to 75.2.60.5 to mitigate against this outage. This link – https://docs.netlify.com/domains-https/custom-domains/configure-external-dns/#configure-an-apex-domain – contains instructions to perform this action.
Posted Mar 25, 2021 - 09:46 PDT
Update
Our team are actively working to mitigate the issue attributed to an upstream provider. Please check https://answers.netlify.com/t/support-guide-minimizing-impact-of-load-balancer-not-working/34888 for potential mitigation steps.
Posted Mar 25, 2021 - 09:10 PDT
Update
We are continuing to work on a fix for this issue. Please check https://answers.netlify.com/t/support-guide-minimizing-impact-of-load-balancer-not-working/34888 for potential mitigation steps.
Posted Mar 25, 2021 - 08:44 PDT
Update
We are continuing to work on a fix for this issue.
Posted Mar 25, 2021 - 08:28 PDT
Identified
We have identified the issue and it is attributed to an upstream provider. We will continue to provide updates as we receive them.
For clarification, only sites which use our special load balancer IP as an A record value with their DNS not hosted on Netlify are affected (learn more about this config at https://docs.netlify.com/domains-https/custom-domains/configure-external-dns/#configure-an-apex-domain).
Posted Mar 25, 2021 - 08:06 PDT
Investigating
We are currently experiencing issues in serving traffic via our CDN where the apex domain is the primary domain in the Netlify UI and Netlify DNS is not used.
Posted Mar 25, 2021 - 07:49 PDT
This incident affected: Enterprise Content Distribution Network and Content Distribution Network.