Most of our service automatically fails over to a secondary storage provider when there is an outage in the primary, and this happened tonight in all cases except for saving build caches. Our updates to build cache saving have proven stable and effective.
Our team will continue to monitor the situation.
Note that our initial message about CDN degradation was incorrect; both CDN's continued to function at full capacity.
Posted Mar 12, 2019 - 20:55 PDT
We are continuing to work on a fix for this issue.
Posted Mar 12, 2019 - 20:34 PDT
We have a fix in place to allow builds to succeed despite storage errors, and we are continuing to monitor.
Posted Mar 12, 2019 - 20:30 PDT
We are seeing elevated error rates from a storage service that is affecting service of uncached assets from our CDN's and causing some build errors as well. Our team is investigating.
Posted Mar 12, 2019 - 19:47 PDT
This incident affected: Enterprise Content Distribution Network, Content Distribution Network, and Build Pipeline.