Routine Fiber Maintaince in Port Orchard this morning cost me $100K

This morning, a routine maintenance event in the McCormick Trails/Village area, a high-density housing hub with significant KPUD fiber infrastructure, took an unexpected turn. KPUD was working to improve a fiber case to accommodate future expansion. While the planned outage was scheduled to last from 1:00 AM to 5:00 AM, our phones began lighting up by 5:45 AM. As KPUD struggled to restore service, an unrelated power and generator failure at the Bremerton airport upstream of the neighborhood created a catastrophic triple fault.

When Net253 first launched in 2012, we relied on KPUD as our primary internet provider with a 1Gb uplink. As our bandwidth requirements increased, we added a NoaNet 5Gb link. Eventually we upgraded our NoaNet link to full 10Gb. Last year we upgraded to a 100Gb Tier-1 Lumen (AS3356) feed as our primary connection. However, because our secondary NoaNet link uses KPUD fiber from Bremerton to Bainbridge, we remain vulnerable to a cascaded failure. To mitigate this risk, we have been finalizing negotiations with Cogent (AS174) for a 10Gb backup circuit delivered via physically isolated Comcast Enterprise fiber. Similar to the DSL line we maintain at our office as a final bootstrap. This new connection provides 100% path diversity for our primary and secondary out of county transit operations. While 10Gb is the maximum capacity Comcast Enterprise can currently deliver as a tail circuit in Kitsap County, it represents the most robust solution available to ensure our service’s availability. No catastrophic event on KPUDs backbone can disconnect Net253 from the world.

The events of this morning solidified our decision. We have signed a three-year, $100,000 contract for this triple redundant service. While that investment could have easily replaced the 2005 Dodge minivan you might see us driving to service calls, our priority remains providing the most reliable internet possible rather than upgrading our fleet. Rest assured, the minivan is here to stay, and no, it is not for sale.