We looked into an issue causing some pages to be cached as 404s on at least one CDN node. We identified and reconfigured a bad DNS resolver, and we refreshed all active sites.
Follow-up: This was a side effect of a larger Google DNS outage that ended up affecting DNS lookups between our origin servers and the object store that holds uploaded files, causing the 404s some users were seeing. Normally we would have gotten spikes in our exception logging service, but their service was affected by the same Google DNS issue, and that meant they never received our exception reports. We’ve now setup alerting for any spike in these kinds of exceptions on our internal log processing software and will be alerted if something like this happens again. We’re also looked into and repaired the behavior of our (some of) our origin servers not correctly failing over to a secondary DNS service once Google started giving errors.