NATIONAL NEWS

Facebook blames outage on error during routine maintenance

Oct 6, 2021, 9:32 AM | Updated: 9:55 am
In this photo illustration, the Facebook and Instagram apps are seen on the screen of an iPhone on ...
In this photo illustration, the Facebook and Instagram apps are seen on the screen of an iPhone on October 04, 2021 in San Anselmo, California. (Photo Illustration by Justin Sullivan/Getty Images)
(Photo Illustration by Justin Sullivan/Getty Images)

LONDON (AP) — The global outage that knocked Facebook and its other platforms offline for hours was caused by an error during routine maintenance, the company said.

Santosh Janardhan, Facebook’s vice president of infrastructure, said in a blog post that Facebook, Instagram and WhatsApp going dark was “caused not by malicious activity, but an error of our own making.”

The problem occurred as engineers were carrying out day-to-day work on Facebook’s global backbone network; the computers, routers and software in its data centers around the world along with the fiber-optic cables connecting them.

Facebook, WhatsApp, Instagram suffer worldwide outage

“During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally,” Janardhan said Tuesday.

Facebook’s systems are designed to catch such mistakes but in this case a bug in the audit tool prevented it from properly stopping the command, Janardhan said.

That change also triggered a second problem that made things worse by making it impossible to reach Facebook’s servers even though they were operational.

Engineers scrambled to fix the problem on site, but this took time because of the extra layers of security, Janardhan said. The data centers are “hard to get into, and once you’re inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them.”

Once connectivity was restored, services were brought back gradually to avoid traffic surges that could cause more crashes.

It was an “unforeseen anomaly” for a faulty maintenance update to take down Facebook’s backbone network, but the company probably could have avoided a scenario in which its servers were completely taken offline, making it impossible to access the tools needed to fix it, said Angelique Medina, of Cisco Systems’ ThousandEyes, a firm that monitors internet outages.

“The big question is why so many internal tools and systems could have a single source of failure,” Medina said. “Facebook would still have been down because of the network outage, but they could have resolved the outage sooner if they had internal access.”

KSL 5 TV Live

Top Stories

National News

R&B singer R. Kelly (L) arrives at the Cook County courthouse where jury selection is scheduled to ...
Claire Savage and Michael Tarm, Associated Press

Chicago prosecutor dropping R. Kelly sex-abuse charges

A Chicago prosecutor said Monday that she's dropping sex-abuse charges against singer R. Kelly following federal convictions in two courts that should guarantee the disgraced R&B star will be locked up for decades.
1 day ago
The booking photo for Jami Farmer. (Lincoln County Sheriff’ Office)...
Michael Houck

Nevada attempted murder suspect arrested in Washington County

An attempted murder suspect from Nevada is in police custody after being found at a Utah gas station Monday.
1 day ago
Amy Robach and TJ Holmes attend the 2022 ABC Disney Upfront at Basketball City - Pier 36 - South St...
Associated Press

TV anchors T.J. Holmes, Amy Robach leave ABC amid romance

T.J. Holmes and Amy Robach, anchors at the afternoon extension of ABC’s “Good Morning America,” are leaving the network.
1 day ago
DALLAS, TX - JUNE 07:  A general view of the skyline of downtown Dallas, Texas as the American Airl...
KELSY MITTAUER

Arrest made in sex trafficking of teen girl who disappeared at Mavs game

Nine months after a Texas girl was sex trafficked after disappearing from a Mavericks game, an arrest has been made.
1 day ago
Lisa Loring...
Toyin Owoseje and Amanda Watts, CNN

Lisa Loring, original Wednesday Addams actress, dead at 64

Lisa Loring, best known as the first actress to play Wednesday Addams in the original "The Addams Family" sitcom, has died at the age of 64.
1 day ago
FILE - Brevard County Sheriff Wayne Ivey gets ready to spin his popular "Wheel of Fugitive" in July...
Associated Press

Florida sheriff sued for ‘Wheel of Fugitive’ defamation

A man has filed a defamation lawsuit against a Florida sheriff who posts weekly “Wheel of Fugitive” videos on social media.
1 day ago

Sponsored Articles

Fiber Optical cables connected to an optic ports and Network cables connected to ethernet ports...
Brian Huston, CE and Anthony Perkins, BICSI

Why Every Business Needs a Structured Cabling System

A structured cabling system benefits businesses by giving you faster processing speeds and making your network more efficient and reliable.
notebook with password notes highlighted...
PC Laptops

How to Create Strong Passwords You Can Actually Remember

Learn how you can create strong passwords that are actually easy to remember! In a short time you can create new ones in seconds.
house with for rent sign posted...
Chase Harrington, president and COO of Entrata

Top 5 Reasons You May Want to Consider Apartment Life Over Owning a Home

There are many benefits of renting that can be overshadowed by the allure of buying a home. Here are five reasons why renting might be right for you.
Festive kitchen in Christmas decorations. Christmas dining room....
Lighting Design

6 Holiday Decor Trends to Try in 2022

We've rounded out the top 6 holiday decor trends for 2022 so you can be ahead of the game before you start shopping. 
Happy diverse college or university students are having fun on their graduation day...
BYU MBA at the Marriott School of Business

How to Choose What MBA Program is Right for You: Take this Quiz Before You Apply!

Wondering what MBA program is right for you? Take this quiz before you apply to see if it will help you meet your goals.
Diverse Group of Energetic Professionals Team Meeting in Modern Office: Brainstorming IT Programmer...
Les Olson

Don’t Let a Ransomware Attack Get You Down | Protect Your Workplace Today with Cyber Insurance

Business owners and operators should be on guard to protect their workplace. Cyber insurance can protect you from online attacks.
Facebook blames outage on error during routine maintenance