Prism - Game Panel Load Balancing Inactive (still functional) – Incident details

# Postmortem: Panel Migration & Stability Improvements

### Incident Summary

On January 28, 2025, we initiated preparations for migrating our panel infrastructure to a more stable and redundant setup. This transition involved moving to a new load-balanced system with multiple replicas in Ashburn. The process caused temporary sluggishness, brief outages, and some asymmetric syncing issues that were later addressed.

### Timeline of Events (MST)

- 1/28/2025 - 8:42 PM: Preparations began for migrating the panel, leading to some sluggish performance.

- 1/29/2025 - 6:16 PM: The panel was temporarily taken offline to complete the transfer to the new system.

- 1/29/2025 - 6:34 PM: The cutover was completed, and the panel was restored on the new infrastructure.

- 1/29/2025 - 8:40 PM: Stability was fully regained after minor intermittent outages.

### Outcome & Changes Implemented

- The panel is now operating on two replicas in Ashburn, ensuring redundancy and improved availability.

- PANEL-ASH-2 has been set as the default route due to PANEL-ASH-1 experiencing slight performance degradation from asymmetric syncing, which will be addressed soon.

- Users can check which instance they are on by looking at the bottom of their panel page.

### Next Steps

- Address the asymmetric syncing issue on PANEL-ASH-1 to bring its performance in line with PANEL-ASH-2.

- Monitor performance closely and make further optimizations as necessary.

- Continue ensuring smooth operation with minimal disruptions for users.

We appreciate everyone's patience during this transition and are confident that this upgrade will provide a more resilient and responsive panel experience moving forward. If you encounter any lingering issues, please open a support ticket.

— The Prism Team