Service blip today 5/21/2024

At 10:38 AM PSE lost 2/3 of their Murden Cove Substation on Bainbridge Island. Our facility went dark. No big deal… Ten seconds later, our generator had not spooled up. What the heck? Everything was running on battery backup here, but that doesn’t last forever. We went outside and found that the service technicians who recently did an annual maintenance on the 50KW generator (208V 3 phase 125 amp service Cummins RS50) left the unit locked off. After a few stressful minutes I manned up, manually started the generator and manually flipped the 600AMP auto transfer switch. Think Frankenstein knife blades with big warning signs of lethal energy present. If you are inclined, search for OTEC600 600A 3ph-120/208V. Its a beast and we installed it for your service protection. It has worked many times over the last six years.

Root Cause of today’s outage - Our new redundant routes switch rebooted during this event. We are investigating what went wrong. I will update this blog post once I have root cause. We take disruption seriously; we all depend on reliable service.

Update on Resolution: - We added a tertiary, independent, 24VDC battery backed UPS to a secondary DC input on the redundancy switch. We replaced the batteries on the dedicated primary AC UPS for the redundancy switch. These batteries were scheduled to be replaced January 2025. We replace batteries at four years in our vault which maintains 70F+/-4F. We now have three UPS plus a generator plus an auto transfer switch protecting the power to the core switch unit. We added labels to the generator stating “MUST LEAVE IN AUTO START” “MUST HAVE OWNER WITNESS BEFORE YOU LEAVE” That is to address the human element of todays chain of events that led to an outage. It really should have not happened, but worst case should have been 90 seconds. The twelve minute outage was unacceptable to us.