KGMN550 Outage 12/10/2024

The KGMN550 machine has suffered a catastrophic outage. Neither of the controllers will transmit audio, and we have completely lost internet connectivity and any remote access to the machine. We are working with our site manager to coordinate a trip to the tower site now, and we will keep you updated in this post as we know more information. 

Update 1: [Dec 10, 2024 5:32PM] Wizard (WQXC747) has a repeater on 462.650 that is open to licensed GMRS users. If you would like information on how to set it up in your radio, I will be happy to assist. Feel free to use the contact form on the website, or send a DM to Cody (WRVY789). Another option is 462.725Mhz Simplex, the local Kingman Krowd simplex channel.

Update 2: [Dec 11, 2024, 10:05 AM] The repeater was confirmed to be transmitting audio again. WRVY789 disabled the damaged controller and swapped to the backup controller. Internet connection is still unavailable, so we will still be making a trip to the site to investigate the issue. For now the repeater is online and operational. NOTE: You may hear an operator on the air asking to clear traffic, please do so until you hear the all clear to resume normal traffic.

Update 3: [Dec 12, 2024, 7:16 AM] We have scheduled a trip to the tower site for diagnostics and repair on Sunday, December 15th. We do not have a planned maintenance time, but an announcement will be made before we shutdown the hardware over the air and on the website top banner. We appreciate your patience and understanding. 

Update 4: [Dec 15, 2024, 12:06 PM] We are heading to the repeater site for maintenance, we will make an announcement before power down. Estimated maintenance window is 3-5 hours. We appreciate your patience and understanding.

Update 5: [Dec 16, 2024, 3:24 PM] All systems are now operational. We are still digging through the logs to get a better understanding of the issue that occurred. However, we believe this was caused by a power fault. Not all systems were fully offlined, giving us logs to compare between the states of the devices.

It appears the power fault occurred at 258am on December 10th, knocking our live stream node offline completely, and causing memory issues with our primary repeater controller. We are unsure if a second incident occurred, or if the modem suffered the same issue as the primary controller at a much later time. The modem logs showed the device moved into a "connecting to network" state from December 10th @ 913AM until it was rebooted on the 15th.

The controller was able to fully recover on its own 24H later but was disabled in the event the same error occurred. The memory issue caused all audio into the controller to exit out of port 2 only, and since the live stream server, and remote access server were offline, we were unable to determine the issue at the time of the outage. We have added a reboot routine into the controller to ensure that it can recover on its own after an error.