Update (2200ET): Facebook’s services are now back online and the company is actively working to fully return them to regular operations.
“Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication,” the company said in a post on its blog. “This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt”
“We want to make clear at this time we believe the root cause of this outage was a faulty configuration change. We also have no evidence that user data was compromised as a result of this downtime”
The company also apologized "to all the people and businesses around the world who depend on us." Come to think of it, everyone feels sorry for people who depend on Facebook.
Below is the full Facebook blog post:
To all the people and businesses around the world who depend on us, we are sorry for the inconvenience caused by today’s outage across our platforms. We’ve been working as hard as we can to restore access, and our systems are now back up and running. The underlying cause of this outage also impacted many of the internal tools and systems we use in our day-to-day operations, complicating our attempts to quickly diagnose and resolve the problem.
Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt.
Our services are now back online and we’re actively working to fully return them to regular operations. We want to make clear at this time we believe the root cause of this outage was a faulty configuration change. We also have no evidence that user data was compromised as a result of this downtime.
People and businesses around the world rely on us everyday to stay connected. We understand the impact outages like these have on people’s lives, and our responsibility to keep people informed about disruptions to our services. We apologize to all those affected, and we’re working to understand more about what happened today so we can continue to make our infrastructure more resilient.
* * *
Update (1905ET): Today's Facebook whistleblower debacle and global outage cost CEO Mark Zuckerberg $6 billion of his net worth. He's still worth $121 billion but well off the highs of $140+ billion from late summer.
* * *
Update (1829 ET): After six hours offline, the estimated global economic impact of the internet disruption for Facebook, WhatsApp, and Facebook Messenger comes out to nearly $1 billion.
The NetBlocks Cost of Shutdown Tool (COST) estimated the outage had a total economic impact of a whopping $968 million on the global economy.
* * *
Update (1750 ET): Facebook is currently experiencing one of the largest global outages ever that has also taken out Instagram, WhatsApp, and Facebook Messenger, according to Downdetector.
After a Domain Name System (DNS) issue that knocked Facebook and other related products off the internet for more than six hours, there's a sign of life coming from Facebook's DNS servers which are broadcasting again, according to DNS Checker.
Senior Editor at The Verge tweeted, "Facebook seems to be slowly coming back to life. I'm seeing BGP announcements and DNS is back."
Facebook seems to be slowly coming back to life. I'm seeing BGP announcements and DNS is back pic.twitter.com/LNO76M5PhR— Tom Warren (@tomwarren) October 4, 2021
Even though Facebook's DNS is back up, this doesn't mean everything is fixed; the social media platform is still inaccessible to some users.
Just as a reminder, this circus show comes one day before a Facebook whistleblower is set to testify before Congress.
* * *
Update (1507ET): Facebook appears to have more than DNS issues, as NYT's Sheera Frenkel reports:
"Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren't working to access doors."
Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors.— Sheera Frenkel (@sheeraf) October 4, 2021
This may suggest Facebook's internal systems are down. DNS issues that affect the website don't hinder employees from entering buildings.
* * *
Update (1448ET): It's been several hours, and Facebook, Instagram, Whatsapp, and Facebook Messenger, are still down. According to internet researchers, Facebook appears to be dealing with a Domain Name System (DNS) issue.
Internet security researcher Brian Krebs tweeted:
Confirmed: The DNS records that tell systems how to find Facebook.com or Instagram.com got withdrawn this morning from the global routing tables. Can you imagine working at FB right now, when your email no longer works & all your internal FB-based tools fail?
To be more precise (and Geek Factor 5) the BGP routes serving Facebook's authoritative DNS were withdrawn, rendering all Facebook domains inaccessible. That's per @DougMadory , who knows a few things about BGP/DNS.
Searching for Facebook's DNS records comes up empty-handed.
NBC News' Kevin Collier called the DNS issues at Facebook "really bad."
Don't yet know exactly what's behind the DNS issue that's knocked Facebook/Instagram/WhatsApp offline, but it's really bad. Pretty much everything that runs through those three companies are inaccessible. Employees can't even enter conference rooms because they're IoT!— Kevin Collier (@kevincollier) October 4, 2021
There's even a ripple effect across the internet.
JUST IN: Facebook's DNS problem is actually affecting the internet. DNS providers are being hammered with queries pic.twitter.com/GrzsJ5SHWm— Insider Paper (@TheInsiderPaper) October 4, 2021
Meanwhile, everyone on Twitter...
* * *
Downdector reports that Facebook, Instagram, Whatsapp, and Facebook Messenger are experiencing issues and /or outages worldwide. The outage comes just before a Facebook whistleblower is set to testify before Congress on Tuesday.
Users began to report issues with the social media platform around 1120 ET.
Facebook users report the website won't load with an error message that reads:
"Sorry, something went wrong. We're working on it and we'll get it fixed as soon as we can."
Facebook spokesperson Andy Stone acknowledged on Twitter that "some people are having trouble accessing our apps and products."
We’re aware that some people are having trouble accessing our apps and products. We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience.— Andy Stone (@andymstone) October 4, 2021
The timing of the widespread outages comes as a Facebook whistleblower will be testifying before Congress on Tuesday. The whistleblower appeared on "60 Minutes" Sunday, detailing the tech company's mission to put profit before doing "what was good for the public," including clamping down on hate speech.
Shares of the tech company slid 6% by early afternoon, tumbling not just on the whistleblower but also the global outage.
Facebook users took to Twitter to complain about the outages:
Facebook, Instagram and WhatsApp experiencing worldwide outage!!!!!! pic.twitter.com/EIMKqrtUFK— Nate (@natedoggie1134) October 4, 2021
The Facebook DNS outage will quickly expose the hidden dependencies some of us have on Facebook (company), even if we barely use the social network.— Danny Kopping (@dannykopping) October 4, 2021
For me, it's WhatsApp.
Reminds me of the story of when Google took down their "Chubby" service:
... and there's this:
Does anyone else think this Facebook outage is timed perfectly if they want to clean house in the face of the whistleblower’s interview so that even more dirt isn’t found?— Tim Palomo (@timpalomo1) October 4, 2021
Twitter user Chad Loder tweeted: "someone nuked the DNS A and AAA records for Facebook, Instagram, and WhatsApp."
Twitter CEO Jack Dorsey responds to Loder's tweet and asks "how much?"
how much? https://t.co/fH0zXw7rV9— jack⚡️ (@jack) October 4, 2021
Let the conspiracy theorists run wild with today's events:
"Facebook is down because it was trending for a whistleblower testifying before congress today. Now they are trending for being down. Clever PR stunt," said one Twitter user.
On the bright side, with a broad part of social media platforms down today, maybe people will get back to work and boost global productivity since the outage is broad.