On June 8, a number of popular websites went down simultaneously, including giants like Amazon, Spotify, eBay, Reddit, and Pinterest. The underlying cause was an outage of the systems run by Fastly, one of the more prominent names in the cloud industry.
The service in question was Fastly’s CDN—a solution used to deliver content quickly worldwide and ease the load on primary servers hosting an application. It is relied upon by numerous major websites, many of which had their pages replaced with 503 errors when the incident happened.
It was first assumed that the CDN went down due to a cybersecurity attack. Indeed, it is not uncommon for cybercriminals to target such services given the opportunity. Fastly clarified the incident quickly, stating that a software bug caused 85% of its network to go down.
Fortunately, network monitoring caught the downtime within a minute. The issue was isolated and mostly resolved in less than 50 minutes.
This story illustrates the importance of catching issues early and having a solid disaster recovery plan. Regular users might do with a good cloud backup solution can but infrastructure providers need a serious contingency plan.
Single User Causes Outage and Drives Stock Prices Up
Funnily enough, the episode was caused by a single user unknowingly configuring their CDN settings in a specific way. In a perfect storm, the configuration triggered a bug that has been present since May and took down the network.
The outage was not all bad for Fastly. The company’s stock prices fell initially but ultimately rose about $5 per share. The cause for the increase is a subject of debate.
However, it is most likely explained by the speed with which Fastly handled the outage, as well as it regularly keeping users updated.
The outage was much more costly for companies using Fastly. Only Amazon is projected to have lost $32 million due to its site being down.
That’s more than 15% of Fastly’s annual revenue!
Some good news is that the outage is not likely to recur, according to Fastly’s report.