Nationwide CenturyLink outage affecting 911 calls in Western Washington

a reply to: The Great Day

I notice that these kinds of things tend to happen in waves.

Keep an eye out for more infrastructure breaches.

Before the 16' elections we had the Emergency Alert System hacked in Utica NY displaying text on TVs in the warning bar from a Dr Seuss book "would you could you on a train". The next day a train crashed in NJ.

Then there was the tornado siren hack in Dallas, and a small town in Colorado.

Also several fishy widespread internet/phone outages around the same time period.

I'm not pointing fingers at the who, but obviously several angles are to be had for these kinds of systems going down. Whether you're a geopolitical belligerent or a profiteer.

we use centurylink at work. this outage has been a real, real pain in my ass.

CenturyLink is having some major issues.

There were significant Internet outages yesterday across the western US.

The timing is curious. Telcos across the nation are struggling right now with migrating off of a legacy technology known as SONET. Technologies like MOE (Metro Ethernet) are the likely successors, but the migration path is not straight forward at all and painful at the very least. Most of the legacy SONET transport gear is obsolete, no longer manufactured and sourcing of spares is becoming increasingly difficult if not impossible.
MOE is in wide use around the country, but this is only at the 'edge'. Much of the core high-speed distribution level architecture / infrastructure still relies on SONET for all the major links. We use SONET technology for our distribution also, and we're facing the same issues.

I suspect the CenturyLink issues are related to this (but I doubt they'd ever admit it). They're likely trying to migrate segments of traffic and having issues.

The migration to MOE across the board is pretty simple for straight digital ethernet services, but it becomes WAY more complicated when you start getting into specialized circuits like T-1's, DS-0's and 911 services (and countless others). And then, when you couple this with all the CPE (customer provided) infrastructure out there which relies on these services to function correctly the problem of the migration becomes exponentially more complicated (by orders of magnitude).

For the non-technical reader, here's how this translates into plain English; here are two examples...

1. Imagine a business who has a data line which comes onto their premises as what is known as a "T-1" line (or a fractional T-1). There are literally tens of millions of these customers in the US. All of a sudden the service provider can no longer deliver a T-1, but need to deliver a MOE circuit instead. Now the customer suddenly has to change equipment at their premises for their system to continue to work. When you scale this up to the millions of customers who use this technology the problem becomes apparent. Unfortunately, it gets worse though...

2. Even Mom & Pop businesses are affected. Let's say they have a gift shop or convenience store. They don't need a great big data "pipe" like a T-1, nor can they afford the $5,000/month Telco bill either, right? Wrong. The mechanism they use to take credit cars is called a POS terminal, or "Point of Sale" terminal. This terminal requires what essentially amounts to a FAX line to function properly. It's not VOIP, but rather requires an analog telephone line (which, anymore, is actually a 'special order'). That analog line is delivered on what is known as a DS-O type service, but MOE can't provide DS-0 services, only VOIP. So when the DS-0's go away...so does Mom & Pop's POS terminal. Now they need different technology in order to continue to accept credit cards. Because over 85% of all transactions today are done via plastic (debit/credit cards) this means that Mom & Pop suddenly lose 85% of their revenue until they make the switch.

These are just two simple examples of about 100 that I can cite. They are some of the easier of the examples to understand. Some are infinitely more complicated, especially when you get into banking and trading.

So what these companies are doing is trying to stand up parallel technologies to "load shed" some of the easier traffic of onto the new technology, but they don't have solutions for the millions who require new equipment on the customer end...and there's no way (no possible way) the Telcos can afford to foot the bill for all this equipment.

Bottom line...there will be more outages (a lot more) before this storm passes!

P.S. - You heard it here first, folks!

a reply to: The Great Day

Just got this message and I live in Massachusetts. What is up with this?

a reply to: The Great Day

The 1st floor tenant living below my neighbor who warned me last night, got their car broken into last night...

But honestly it's not super uncommon around here.

About to go check the damage report for our own vehicle. Fingers crossed...



UPDATE- We didnt get hit by the druggies last night. We are moving soon but I don't really know yet if it's gonna be any better in the new neighborhood. Gonna have to wait and see.


Yeah I used to be a druggy but I never broke into a car or house or shoplifted to support my addiction. When times got tough, I panhandled. Let someone Decide if they wanna give money to a stranger.

originally posted by: Lagomorphe
Why not just try calming down and just chill out America?

Take a break with those you love... if you have no-one to love then just love yourself.

Go for it ....

People are upset because it's not just homes out..... Its affected people's businesses and livelihoods which in turn affects those families and loved ones you're referencing

originally posted by: Daranini
a reply to: The Great Day

Just got this message and I live in Massachusetts. What is up with this?

Really!? That's somewhat concerning this is still ongoing and spreading. Thanks for reporting in, let us know if you find out anything more about it.

a reply to: Flyingclaydisk

Solid intel! Sounds stormy indeed. Thanks for putting that in layman's terms...though it still skimmed just over my head

a reply to: The Great Day

Translated another way (in very simplistic terms); say you had a business where you needed your TV in your house to work in order to conduct your business. But the cable provider for your TV was operating on old equipment which was obsolete. The only solution the cable provider has is to change to a different technology, but that requires you to get a new, much more expensive, TV and a whole bunch of other equipment to make the TV work. The cable company knows this, so they try to hang on as long as they possibly can with the old equipment (way past it's lifespan). The cable company's equipment still works, so they take a..."if it's not broken, don't fix it"...approach. They hope that by waiting some new solution will come along which won't require you to buy all this stuff (and scream bloody murder as a result). However, no solution comes, and the old equipment just gets older and older, then starts failing. Then there's no choice.

In the meantime all the new customers have been set up with new technology, but the vast majority of customers rely on this old technology because they can take a "do-nothing" approach. So this is the "load shedding" I was talking about, but it's not enough, and the old gear starts failing (i.e. outages). If it's just local stuff, it's no big deal, but when the big transport gear (the core equipment) starts failing you have nationwide issues.

I used TV as an example because there is a parallel many will understand, and this was the transition to "digital TV", or "HDTV" which happened in the past 5 years. Traditional analog TV was basically phased out, and people had to buy new equipment to get HDTV. The transition to digital TV was pretty painless in comparison to what the Telco's are facing, but it serves as a good illustration.

For the techies...

On a historical note, some of the international Telco's tried to migrate away from SONET back in the 90's with technology like ATM. ATM had all sorts of growing pains and was ultimately eclipsed by IP Ethernet technology. Unfortunately, the timing was all wrong (literally and metaphorically) for ATM, because SONET was still widely supported then. So, there was no compelling need to migrate away from SONET as a transport protocol. Now things are much different, and it looks like IP based protocols are the only solution left.

So, from a technology perspective we start getting into things like Coarse Wavelength Division Multiplexing (CWDM) and Dense Wavelength Division Multiplexing (DWDM) to support metropolitan level sized Ethernet environments in order to capitalize on existing fiber optic infrastructure.

There's a lot of really cool technology out there right now, but getting from where we are presently to that level has lots of challenges, many of which people don't know or understand. The next 5-10 years are going to be interesting.

ETA - In many ways what we see coming is a 'perfect storm' of the digital age. There are legions of technical people (engineers and technicians) who understand the legacy technology, but they're all nearing retirement, and they don't have any real interest in learning the new technology. At the same time there are legions of young people who understand what we call "the edge technology". So there are lots of smart guys on things like Cisco and routing in the LAN environment, but that's all just local. Much like bricks and mortar construction trades, there are very few of the next generation youth who are interested in the 'big iron' that it takes to move big data and telephony services for long haul. The learning curves are steep, and even though the technology is there, there really isn't enough technical support to support it.

And no, contrary to common belief, "The Cloud" won't fix all this! (they'd love you to think it will though!) That's all just smoke-n-mirrors in a vacuum of cold hard facts.

originally posted by: Daranini
a reply to: The Great Day

Just got this message and I live in Massachusetts. What is up with this?

Verizon no longer has the E911 system it’s been updated to Centurylink out of La, If you have a landline 911 will still work on that.

a reply to: Flyingclaydisk

This is interesting - looks like it could be a solar-related 'space weather' event from an incoming coronal hole stream from the sun creating a mild 'geomagnetic storm' event, with some power line and communications problems resulting:

www.youtube.com...

It seems the more reliant we become on technology, the more vulnerable we also become. Ouch!

a reply to: Flyingclaydisk

Solar related vid to the above.
www.youtube.com...

Also think CL grew toooo big tooo fast especially after acquiring Level 3.

Weird, my Dad's business in Rome, NY had a power glitch a few nights ago and next morning found one of his servers was hacked, badly. Then yesterday his internet service went down, followed by his hosting provider tech support for one of his servers could not be reached, then his business landline went down for several hours. Back up today but service has been sporadic.
My Dad is old school but smart enough that he has all his 'cloud' data backed up to SSD's kept in a fireproof safe each night.

twitter.com...

a reply to: EndtheMadnessNow

Solar related vid to the above.

It would be good to be skeptical about Suspicious0bservers.

Earth Catastrophe Cycle | SOLAR MICRONOVA

Riigggghhht.

a reply to: The Great Day

Have you lot tried turning the Internet off and back on again?

See if Trump would just leave Twitter alone for 5 minutes i imagine the problem would sort itself out.

Great posts by Flyingclaydisk.

This is not the first time CenturyLink has had a major 9-11 trouble. The last time was in 2014 and resulted in fines from the FCC and from the state of Washington. Total of fines was over $20 million. Search on 'centurylink' 'fcc' and 'fine'. If you read the FCC findings it is clear that CenturyLink attempted to pass the blame on to sub-contractors. However, as the prime contractor, CenturyLink could not legally divest itself of the responsibility. CenturyLink's unscrupulous behavior led to the rather large fines.

Edit to add a link the FCC consent decree which CTL signed and paid the fines:
www.fcc.gov...

The outage of December 2018 was orders of magnitude more severe for CenturyLink and the country. Ajit Pai has ordered the FCC to investigate. There will be very little publicity until that investigation is completed. Meanwhile, the story is *DEAD* in the news. CenturyLink went on record blaming a piece of hardware in Denver, Colorado.

Sure, guys. We believe you (not).

CenturyLink is now a gangster's paradise. They are a Frankenstein monster of former telcos. The mergers & acquisitions only serve to clean the slate of previous wrongdoings, scams and gang activity. Departments are rearranged, executives are shuffled, managers are given new directives, the names of operating companies are changed but *the gangsters are still in business.*

The telco gangsters even take turns running the FCC (Ajit Pai, hyuck, hyuck, VZN, hyuck, hyuck)! How far will this FCC investigation go? Only as far as it needs and no farther. In my opinion, they are right now concocting the cover up plan under the guise of a federal investigation.