First, the impact on internal monitoring limited our ability to understand the problem. This progressed slowly for several reasons. Operators continued working on a set of remediation actions to reduce congestion on the internal network including identifying the top sources of traffic to isolate to dedicated network devices, disabling some heavy network traffic services, and bringing additional networking capacity online. Importantly, monitoring data was still not visible to our operations team so they had to continue resolving the issue with reduced system visibility. This change improved the availability of several impacted services by reducing load on the impacted networking devices, but did not fully resolve the AWS service impact or eliminate the congestion.

At 9:28 AM PST, the team completed this work and DNS resolution errors fully recovered. Because internal DNS is foundational for all services and this traffic was believed to be contributing to the congestion, the teams focused on moving the internal DNS traffic away from the congested network paths. Operators instead relied on logs to understand what was happening and initially identified elevated internal DNS errors. This congestion immediately impacted the availability of real-time monitoring data for our internal operations teams, which impaired their ability to find the source of congestion and resolve it. This led to persistent congestion and performance issues on the devices connecting the two networks. These delays increased latency and errors for services communicating between these networks, resulting in even more connection attempts and retries. This resulted in a large surge of connection activity that overwhelmed the networking devices between the internal network and the main AWS network, resulting in delays for communication between these networks. At 7:30 AM PST, an automated activity to scale capacity of one of the AWS services hosted in the main AWS network triggered an unexpected behavior from a large number of clients inside the internal network. These networking devices provide additional routing and network address translation that allow AWS services to communicate between the internal network and the main AWS network. Because of the importance of these services in this internal network, we connect this network with multiple geographically isolated networking devices and scale the capacity of this network significantly to ensure high availability of this network connection. While the majority of AWS services and all customer applications run within the main AWS network, AWS makes use of an internal network to host foundational services including monitoring, internal DNS, authorization services, and parts of the EC2 control plane. To explain this event, we need to share a little about the internals of the AWS network. We want to provide you with some additional information about the service disruption that occurred in the Northern Virginia (US-EAST-1) Region on December 7th, 2021.

Awa outage