Facebook is coming back online, but new details from today's massive worldwide outage are emerging, including the fact that employees couldn’t fix server hardware because their ID badges wouldn’t even let them enter company buildings.
Facebook appears to be coming back online for some users. Yet when it crashed this morning, the company produced an absolutely laughable understatement that “We’re aware that some people are having trouble accessing our apps and products.” And by “some people” they meant the entire planet, anyone on any Facebook-owned app or product like Instagram, WhatsApp, Messenger, or Oculus.
No rush. Really.— Kathy Griffin (@kathygriffin) October 4, 2021
As the outage lasted nearly the entire business day, details emerged from inside the Facebook fortress. Or rather, from people trying to get into the Facebook fortress. Multiple journalists on the Facebook beat have reported that the outage was so widespread, that employees could not get into their email accounts, or any of their internal company tools, or for that matter, even their buildings. Several employees' ID badges are reportedly were not even working, which seems like trouble if you had to fix a server.
Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors.— Sheera Frenkel (@sheeraf) October 4, 2021
To be fair, there are some posts on message boards claiming to be from Facebook employees who say their badges are working. And most of what we know about the outage is also from people claiming to be Facebook employees on message boards. A Reddit user named “ramenporn,” who has since deleted their account and is sure to be fired, summed it up as such: “There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified.
“Part of this is also due to lower staffing in data centers due to pandemic measures.”
Don't yet know exactly what's behind the DNS issue that's knocked Facebook/Instagram/WhatsApp offline, but it's really bad. Pretty much everything that runs through those three companies are inaccessible. Employees can't even enter conference rooms because they're IoT!— Kevin Collier (@kevincollier) October 4, 2021
The explanation contains highly technical language and confusing three-letter acronyms, but Krebs on Security offers a handy explainer. “Sometime this morning Facebook took away the map telling the world’s computers how to find its various online properties,” the site says. “As a result, when one types Facebook.com into a web browser, the browser has no idea where to find Facebook.com, and so returns an error page.”
woah that was quick pic.twitter.com/xt9IWCip5j— Joe Groff (@jckarter) October 4, 2021
It appears Facebook is coming back online for some. According to the New York Times, “A Facebook spokesman confirmed that the services were slowly coming back online, but cautioned that it would take some time for the services to stabilize.”
Of course, booking websites now taking bets on when Facebook will come back online. Currently at 2.5 hours and counting…— Sheera Frenkel (@sheeraf) October 4, 2021
This outage came after an already terrible couple of weeks for the company — following a series of pieces in the Wall Street Journal dubbed "The Facebook Files" based on a trove of leaked documents from whistleblower and former Facebook data scientist Frances Haugen. Haugen's leaks have prompted a congressional inquiry, and she went on 60 Minutes on Sunday to fully solidify the seriousness with which she wants people to take her whistleblowing.
Facebook has weathered countless scandals, yet never seems to take a real hit, because its stock price never takes a hit. Not so today, as the stock dropped 5% in just one day, and Zuckerberg lost $6 billion in net worth.
When Facebook loses the investor class, that’s when it has real trouble. Facebook-watcher Kevin Roose’s op-ed "Facebook Is Weaker Than We Knew," published barely an hour before Facebook went down, concludes that “Godzilla eventually died, and as the The Facebook Files make clear, so will Facebook.”
Related: Facebook Whistleblower Who Shared Docs With Wall Street Journal Goes on '60 Minutes' to Spill More Dirt [SFist]
WASHINGTON, DC - APRIL 11: Facebook co-founder, Chairman and CEO Mark Zuckerberg testifies before the House Energy and Commerce Committee in the Rayburn House Office Building on Capitol Hill April 11, 2018 in Washington, DC. This is the second day of testimony before Congress by Zuckerberg, 33, after it was reported that 87 million Facebook users had their personal information harvested by Cambridge Analytica, a British political consulting firm linked to the Trump campaign. (Photo by Chip Somodevilla/Getty Images)