Blip on the phones this morning from 7:50AM to 8:43 AM

We connect to the “old fashioned phone system” through a major nationwide wholesaler called Vitelity, part of a bigger company called Voyant, absorbed into Inteliquent, soon to be acquired by Sinch. Inteliquent powers Zoom, Ooma, 8x8. We have our oars in the main telecom river, not some discount backwater.

A human error at Inteliquent caused the domain named “outbound.vitelity.com” to no longer have an IP address registered in DNS. Once the name to IP address expired on our servers, our PBX systems went offline. At 7:50 this morning every PBX system, every phone line we serve, went offline. We have backups, but we don’t have backups for the backups.

Fortunately, there is a process where we can toss money at the trouble ticket system and immediately go to the top of the queue. If support agrees your issue is truly an emergency, they don’t charge the “line cutter” fee. In this case, they resolved the problem and did not charge us the fee. I can say this is the first “complete blackout” we have experienced using this provider in over 10 years. As frustrated as I am that it happened, I am satisfied with the speed of the resolution.

Please, do not call 911 unless you are having a true emergency. PBX records show a few people tried dialing 911 when other local calls failed.