Clubs PSMT - AS400 slow performance and AWS tunnels down on 2024-10-10 – Incident details

The incident has been successfully resolved and is now closed.

Brief description of the problem:

There was a service disruption affecting certain resources within the VNet, leading to connectivity issues and failures in communication between services hosted in Azure.

Actual state:

• The affected services are now operational, and network connectivity between the impacted resources has been restored.

• All related VNet configurations have been reverted to their previous, working state to resolve the immediate issue.

Root Cause:

The problem was triggered by a policy change applied to the Azure Virtual Networks (VNets). Specifically, the new policy inadvertently modified network security settings, causing unexpected connectivity restrictions. This resulted in some services being unable to communicate across subnets or with external resources. The change also affected route tables and security group rules, leading to disruptions in the internal traffic flow within the VNet.

The issue was resolved by identifying and rolling back the policy change, restoring the original VNet configurations. The Azure team is investigating the need for additional safeguards to prevent such policy impacts in the future.

Our systems are back to normal operation. Thank you for your understanding and support throughout this process.

The system is stable but the situation is being actively monitored. We are closely watching for any potential impact and taking necessary measures. Further updates to follow. Thank you for your patience and understanding.

The issue has been identified, and our team is still working on resolving it. We appreciate your patience as we continue working towards a resolution. Further updates will follow. Thank you for your understanding.

The issue has been identified and our team is working on resolving it. We appreciate your patience as we work towards a resolution. Further updates to follow. Thank you for your understanding.