Superior Network Outage
Update - The network has been up, however we have been dropping audio commands. We are continuing to monitor the situation.
Oct 26, 10:39 CDT
Monitoring - Ping to devices has returned. we will be monitoring packet loss for a while before resolving.
Oct 23, 22:32 CDT
Investigating - Our monitoring has identified that we have lost connectivity to the internal networks in Superior. The cause is unknown at this time.

All local services (local AV playback, Livewire Audio, Recording) will continue to function as planned, however network breaks may be missed while the network is down.
Oct 23, 18:49 CDT
Wausau Switch Outage
Identified - The network switch failed during a firmware upgrade, a replacement switch has been ordered, when it arrives we will schedule a time for installation.
Oct 23, 18:51 CDT
Investigating - The switch that feeds office network connectivity did not come back after a routine software upgrade. This will cause an outage for the 3 offices outside of the studio area in the Wausau bureau.

I will try having a local tech reboot the switch in the AM, if that does not resolve the problem I will head on site with a replacement switch.

This network switch outage will not cause issues with AV or studio networking, only office networking.
Oct 15, 19:51 CDT
Uptime over the past 90 days. View historical uptime.
Core Networking Operational
90 days ago
100.0 % uptime
Today
WPR Core Firewall ? Operational
90 days ago
100.0 % uptime
Today
WPR Engineering Business Network ? Operational
90 days ago
100.0 % uptime
Today
Sysnet Primary ? Operational
Sysnet Backup ? Operational
Internal VoIP Phones ? Operational
DR Site Firewall ? Operational
90 days ago
100.0 % uptime
Today
Madison Audio Networking Operational
90 days ago
100.0 % uptime
Today
CSSC Axia Primary ? Operational
90 days ago
100.0 % uptime
Today
CSSC Axia Secondary ? Operational
90 days ago
100.0 % uptime
Today
Vilas Audio Networking ? Operational
90 days ago
100.0 % uptime
Today
Primary Virtual Infrastructure Operational
90 days ago
100.0 % uptime
Today
WPR Primary Virtual Cluster ? Operational
90 days ago
100.0 % uptime
Today
WPR Primary Virtual Desktop Infrastructure ? Operational
90 days ago
100.0 % uptime
Today
VCenter Server 1 ? Operational
90 days ago
100.0 % uptime
Today
VCenter Server 2 ? Operational
90 days ago
100.0 % uptime
Today
VM Host 1 ? Operational
VM Host 2 ? Operational
VM Host 3 ? Operational
VM Host 4 ? Operational
VM Host 5 ? Operational
VM Host 6 ? Operational
VM Host 7 ? Operational
VM Host 8 ? Operational
VDI Host 1 ? Operational
VDI Host 2 ? Operational
VDI Host 3 ? Operational
VDI Host 4 ? Operational
Storage Infrastructure Operational
90 days ago
100.0 % uptime
Today
MSN Primary Array ? Operational
90 days ago
100.0 % uptime
Today
MSN Replication Array ? Operational
90 days ago
100.0 % uptime
Today
Archive Shared Drive Operational
90 days ago
100.0 % uptime
Today
Audio Common Shared Drive Operational
90 days ago
100.0 % uptime
Today
Headend Audio Operational
Ideas Network Audio ? Operational
Music Network Audio Operational
HD2 (All Classical) Network Audio Operational
Web Services Operational
Monitoring ? Operational
Ticketing System ? Operational
Docs / Wiki ? Operational
Wausau Partial Outage
Wausau Networking ? Partial Outage
Wausau AudioVault Operational
Wausau Virtualization Operational
Eau Claire Operational
Eau Claire Networking ? Operational
Superior Degraded Performance
Superior Axia ? Operational
Superior Networking ? Degraded Performance
Superior Virtualization Operational
Superior AudioVault Operational
External Services Operational
Slack Login/SSO Operational
Slack Messaging Operational
Slack Notifications Operational
Slack Apps/Integrations Operational
Slack Workspace/Org Administration Operational
Opsgenie Voice Notification Delivery Operational
Opsgenie Email Notification Delivery Operational
Opsgenie Alert Flow Operational
Opsgenie Incident Flow Operational
Opsgenie Incoming Email Service Operational
Opsgenie Incoming Integration Flow Operational
Opsgenie Incoming Call Routing Operational
FortiClient VPN ? Operational
Newsboss Operational
90 days ago
100.0 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.

No incidents or maintenance related to this downtime.

Past Incidents
Oct 28, 2020

No incidents reported today.

Oct 27, 2020

No incidents reported.

Oct 26, 2020
Jira NetID login Outage
Resolved - Jira Logins are 100% operational. Please clear your browser cache if you are having issues logging in.
Oct 26, 19:34 CDT
Monitoring - The certificate has been updated on the NetID system and logins are now working. You may need to clear your browser cache to be able to login without errors

We will continue to monitor this for a little while.
Oct 26, 10:52 CDT
Identified - The certificate for NetID Logins on Jira expired, and is preventing anyone from authenticating to jira, we have sent the information over to DoIT to resolve this issue.

Emailing support tickets in will still create new tickets, anyone in the system already should not log out. If admin access is required contact eric.bartos@wpr.org
Oct 26, 10:38 CDT
Oct 25, 2020

No incidents reported.

Oct 24, 2020

No incidents reported.

Oct 23, 2020

Unresolved incident: Wausau Switch Outage.

Oct 22, 2020

No incidents reported.

Oct 21, 2020

No incidents reported.

Oct 20, 2020

No incidents reported.

Oct 19, 2020
Docs Outage
Resolved - The application is back online
Oct 19, 09:12 CDT
Identified - The issue has been identified as full VM disk. We are working to resolve this issue now.
Oct 19, 09:01 CDT
Investigating - Confluence is currently down, we are investigating the issue.
Oct 19, 08:57 CDT
Oct 18, 2020

No incidents reported.

Oct 17, 2020

No incidents reported.

Oct 16, 2020

No incidents reported.

Oct 15, 2020
Emergency Router Firmware Upgrades
Completed - All routers upgraded successfully.

One switch in Wausau is still not coming online. Creating a separate incident for the offline switch.
Oct 15, 19:48 CDT
Verifying - All upgrades complete one fortiswitch in Wausau (managed by fortigate) did not come back online after upgrades. currently investigating the root cause of this.
Oct 15, 19:15 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Oct 15, 18:01 CDT
Scheduled - During this maintenance window we will be upgrading all FortiGate routers in the network to a new version of code to resolve bugs and several security issues.

Internet connectivity in the Eau Claire studios/offices and Wausau studios/offices will be interrupted during the upgrade. There should be no interruptions in connectivity in Madison.

Each router will take about 30 minutes to upgrade, I will be posting here after each successful upgrade. I will start with the routers in the field and finish with Madison.

Please email wpr-eng@wpr.org if there are any issues after the upgrades are complete.
Oct 14, 17:32 CDT
Oct 14, 2020

No incidents reported.

← Incident History Powered by Statuspage