Clubs PSMT - Major outage - POS - 2024-09-23 – Incident details

The incident has been successfully resolved and is now closed.

Brief description of the problem:

An outage was experienced due to AWS tunnels going down, leading to loss of connectivity between our on-premise infrastructure and the cloud resources hosted on AWS. This impacted the availability and functionality of applications dependent on this connection.

Actual state:

The AWS tunnels were restored, and connectivity has been re-established.

Root Cause:

The incident was caused by a disruption in AWS Site-to-Site VPN tunnels, likely due to either network misconfigurations, a temporary issue with AWS networking services, or a failure in VPN endpoints. Further analysis revealed that the automatic failover mechanism did not engage as expected, prolonging the downtime.

Our systems are back to normal operation. Thank you for your understanding and support throughout this process.

The system is stable but the situation is being actively monitored. We are closely watching for any potential impact and taking necessary measures. Further updates to follow. Thank you for your patience and understanding.

The issue has been identified and our team is working on resolving it. We appreciate your patience as we work towards a resolution. Further updates to follow. Thank you for your understanding.

The incident has been successfully resolved and is now closed.

Brief description of the problem:

An outage was experienced due to AWS tunnels going down, leading to loss of connectivity between our on-premise infrastructure and the cloud resources hosted on AWS. This impacted the availability and functionality of applications dependent on this connection.

Actual state:

The AWS tunnels were restored, and connectivity has been re-established.

Root Cause:

The incident was caused by a disruption in AWS Site-to-Site VPN tunnels, likely due to either network misconfigurations, a temporary issue with AWS networking services, or a failure in VPN endpoints. Further analysis revealed that the automatic failover mechanism did not engage as expected, prolonging the downtime.

Our systems are back to normal operation. Thank you for your understanding and support throughout this process.

The system is stable but the situation is being actively monitored. We are closely watching for any potential impact and taking necessary measures. Further updates to follow. Thank you for your patience and understanding.

The issue has been identified and our team is working on resolving it. We appreciate your patience as we work towards a resolution. Further updates to follow. Thank you for your understanding.