KPUD County wide outage 12/9/2021 2PM till 6PM
Ouch. Pass the Advil please! Around 2PM today the primary 100gb link from Bremerton to Seattle died. This link was subject to scheduled maintenance from 1am till 5am this morning. Hmmm.
KPUD has a redundant 100gb path out of Kitsap through Olympia. It has worked in the past, like when the bozo with a pile driver couldn’t read the 8x16 foot billboard saying “no anchoring, underwater cable” and drilled through the fiber. But today it didn’t fail over correctly.
KPUD has a third backup route, but it is only 1gbit. The county traffic trying to get out the back door was well in excess of 4gbits.
Upstream of KPUD is a statewide network operator called NoaNet. They had transport problems as well. You may have heard that 911 service was down in the western counties around Puget sound. NoaNet carries the 911 traffic for WA state now.
Once services restored around 6:15PM, we ran into another issue: many homeowners tried plugging various things in to their internet port. As a result all our global IPv4 addresses were exhausted as we hand out 3 day leases to minimize DHCP traffic and maximize the chance that if we had a DHCP issue, you wouldn’t notice it.
Here is the communication from KPUD about the outage:
“Yesterday at 2:00 pm our Upstream Internet Provider had an equipment configuration change that took down the statewide network. This affected our Internet feed from Seattle and Portland, and was still affecting traffic this morning. A fix that went into place today, caused a quick total outage, and now we are back up.”
When you have an outage try sending an email to service@net253.net and check our website if possible first. Today I fielded 400 phone calls in 2.5 hours. averaging one call per 12 seconds non stop. I literally couldn’t post an update because I couldn’t stop my phone from ringing long enough to load my squarespace app on the mobile phone. When your internet goes out, please try basic power cycles and do not start shuffling cords/computers around!
Note : This outage affected all commercial circuits including the transport that KPUD offers to the residential service providers. We were all hammered by this event.