Storage network switch update
Scheduled Maintenance Report for Wisconsin Public Radio
Postmortem

The updates to the network switches were successful, however this resulted in some uncommon, but not unexpected failure modes, this caused an interruption in service on air.

This type of update is very uncommon, and our usual protocol for emergency updates did not cover a situation like this. Furthermore, we were unaware of how our new automation system would behave in a situation like this. We have now documented the results, and in the coming weeks we will create a more comprehensive plan to handle updates like this and the corresponding outages that may be caused by these updates.

Overall this upgrade will improve the system reliability and prevent further emergency outages that would cause unknown or unplanned downtime.

We would like to apologize for any inconvenience this upgrade has caused, and thank you for your patience while we improve our broadcasting systems.

Posted Jul 21, 2021 - 21:14 CDT

Completed
The scheduled maintenance has been completed.
Posted Jul 21, 2021 - 21:00 CDT
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Jul 21, 2021 - 20:00 CDT
Scheduled
During this time we will be updating our core storage switches to resolve a bug in the switch OS. This should not impact any services, however we will be ready to respond in the event of any outages.
Posted Jul 21, 2021 - 14:24 CDT
This scheduled maintenance affected: Storage Infrastructure (MSN Primary Array, MSN Replication Array, Archive Shared Drive, Audio Common Shared Drive).