All customer-facing impact has been resolved.
Most of our service automatically fails over to a secondary storage provider when there is an outage in the primary, and this happened tonight in all cases except for saving build caches. Our updates to build cache saving have proven stable and effective.
Our team will continue to monitor the situation.
Note that our initial message about CDN degradation was incorrect; both CDN's continued to function at full capacity.
Mar 12, 20:55 PDT
We are continuing to work on a fix for this issue.
Mar 12, 20:34 PDT
We have a fix in place to allow builds to succeed despite storage errors, and we are continuing to monitor.
Mar 12, 20:30 PDT
We are seeing elevated error rates from a storage service that is affecting service of uncached assets from our CDN's and causing some build errors as well. Our team is investigating.
Mar 12, 19:47 PDT