NATIONAL NEWS

Facebook blames outage on error during routine maintenance

Oct 6, 2021, 9:32 AM | Updated: 9:55 am
In this photo illustration, the Facebook and Instagram apps are seen on the screen of an iPhone on ...
In this photo illustration, the Facebook and Instagram apps are seen on the screen of an iPhone on October 04, 2021 in San Anselmo, California. (Photo Illustration by Justin Sullivan/Getty Images)
(Photo Illustration by Justin Sullivan/Getty Images)

LONDON (AP) — The global outage that knocked Facebook and its other platforms offline for hours was caused by an error during routine maintenance, the company said.

Santosh Janardhan, Facebook’s vice president of infrastructure, said in a blog post that Facebook, Instagram and WhatsApp going dark was “caused not by malicious activity, but an error of our own making.”

The problem occurred as engineers were carrying out day-to-day work on Facebook’s global backbone network; the computers, routers and software in its data centers around the world along with the fiber-optic cables connecting them.

Facebook, WhatsApp, Instagram suffer worldwide outage

“During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally,” Janardhan said Tuesday.

Facebook’s systems are designed to catch such mistakes but in this case a bug in the audit tool prevented it from properly stopping the command, Janardhan said.

That change also triggered a second problem that made things worse by making it impossible to reach Facebook’s servers even though they were operational.

Engineers scrambled to fix the problem on site, but this took time because of the extra layers of security, Janardhan said. The data centers are “hard to get into, and once you’re inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them.”

Once connectivity was restored, services were brought back gradually to avoid traffic surges that could cause more crashes.

It was an “unforeseen anomaly” for a faulty maintenance update to take down Facebook’s backbone network, but the company probably could have avoided a scenario in which its servers were completely taken offline, making it impossible to access the tools needed to fix it, said Angelique Medina, of Cisco Systems’ ThousandEyes, a firm that monitors internet outages.

“The big question is why so many internal tools and systems could have a single source of failure,” Medina said. “Facebook would still have been down because of the network outage, but they could have resolved the outage sooner if they had internal access.”

KSL 5 TV Live

Top Stories

National News

A package of Plan B contraceptive is displayed at Jack's Pharmacy on April 5, 2013 in San Anselmo, ...
HALELUYA HADERO, ASSOCIATED PRESS

Amazon, Rite Aid cap purchase of emergency contraceptives

Amazon is limiting how many emergency contraceptives consumers can buy, joining other retailers who put in place similar caps following the Supreme Court decision overruling Roe v. Wade.
1 day ago
Guns of many varieties are organized by type and caliber for use in training, and to fix malfunctio...
DON THOMPSON, Associated Press

Personal info on California gun owners wrongly made public

The California Department of Justice says it wrongly made public the personal information of hundreds of thousands of gun owners in their state-operated databases.
1 day ago
The American Heart Association added sleep duration to its cardiovascular health checklist. It's a ...
Megan Marples, CNN

Sleep duration matters for heart health, according to new recommendations

If you needed another reason to get enough sleep, here it is: It may help your heart health.
1 day ago
FILE: Former President Donald Trump speaks at a rally on September 25, 2021 in Perry, Georgia. Repu...
MICHAEL R. SISAK, Associated Press

Judge ends Trump contempt order after lengthy legal fight

A New York judge has ruled that former President Donald Trump is no longer in contempt of court.
1 day ago
FILE (Getty Images)...
Associated Press

Police: Virginia toddler left in car dies, father kills self

A toddler accidentally left in a vehicle for hours died Tuesday and police said his father was found dead in an apparent suicide at their Virginia home.
1 day ago
Cameron Diaz is making a new movie with Jamie Foxx. (Weiss Eubanks/NBCUniversal/Getty Images via CN...
Leah Asmelash, CNN

Cameron Diaz is coming out of retirement, thanks to Jamie Foxx

After an eight-year hiatus, Cameron Diaz is making movies again.
1 day ago

Sponsored Articles

hand holding 3d rendering mobile connect with security camera for security solutions...
Les Olson

Wondering what security solutions are right for you? Find out more about how to protect your surroundings

Physical security helps everyone. Keep your employees, clients, and customers safe with security solutions that protect your workplace.
Many rattan pendant lights, hay hang from the ceiling.Traditional and simple lighting....
Lighting Design

The Best Ways to Style Rattan Pendant Lighting in Your Home

Rattan pendant lights create a rustic and breezy feel, and are an easy way to incorporate this hot trend into your home decor.
Earth day 2022...
1-800-GOT-JUNK?

How Are You Celebrating Earth Day 2022? | 4 Simple Ways to Celebrate Earth Day and Protect the Environment

Earth Day is a great time to reflect on how we can be more environmentally conscious. Here are some tips for celebrating Earth Day.
Get Money Online...

More Ways to Get Money Online Right Now in Your Spare Time

Here are 4 easy ways that you can get more money online if you have some free time and want to make a little extra on the side.
Lighting trends 2022...

Lighting Trends 2022 | 5 Beautiful Home Lighting Trends You Can Expect to See this Year and Beyond

This is where you can see the latest lighting trends for 2022 straight from the Lightovation Show at the Dallas World Trade Center.
What Can't You Throw Away in the Trash...

What Can’t You Throw Away in the Trash? | 5 Things You Shouldn’t Throw in to Your Trash Can

What can't you throw away in the trash? Believe it or not, there are actually many items that shouldn't be thrown straight into the trash.
Facebook blames outage on error during routine maintenance