Certain Amazon web and Elastic Compute Cloud services are down, and sites are unreachable, as we post this on Thursday, July 28, 2022. According to a notice from Amazon, the outage started at 10:11 AM PDT, when they issued a notice saying that “We are investigating network connectivity issues for some instances and increased error rates and latencies for the EC2 APIs within the US-EAST-2 Region.”
According to Amazon this is an “Operational issue” affecting “Amazon Elastic Compute Cloud (Ohio)”. Amazon calls the “Severity” level of this outage “Informational”. It may be informational for them, but it’s markedly more severe for the countless Amazon cloud services customers who rely on them for email, website, and other hosting services.
This is a developing story and we will stay on top of it and update it as information becomes available.
UPDATE from Amazon: “Instance Impairments 10:25 AM PDT We can confirm that some instances within a single Availability Zone (USE2-AZ1) in the US-EAST-2 Region have experienced a loss of power. The loss of power is affecting part of a single data center within the affected Availability Zone. Power has been restored to the affected facility and at this stage the majority of the affected EC2 instances have recovered. We expect to recover the vast majority of EC2 instances within the next hour. For customers that need immediate recovery, we recommend failing away from the affected Availability Zone as other Availability Zones are not affected by this issue. The following AWS services have been affected by this issue: Amazon Elastic Load Balancing and Amazon Relational Database Service”
UPDATE from The Internet Patrol: At around 10:40 AM PDT the “Severity” level changed from “Informational” to “Degradation”.
The Internet Patrol is completely free, and reader-supported. Your tips via CashApp, Venmo, or Paypal are appreciated! Receipts will come from ISIPP.
CashApp us
Venmo us
Paypal us
UPDATE from Amazon: “10:49 AM PDT We continue to see recovery of EC2 instances that were affected by the loss of power in a single Availability Zone in the US-EAST-2 Region. At this stage, the vast majority of affected EC2 instances and EBS volumes have returned to a healthy state and we continue to work on the remaining EC2 instances and EBS volumes. Elastic Load Balancing has shifted traffic away from the affected Availability Zone. Single-AZ RDS databases were also affected and will recover as the underlying EC2 instance recovers. Multi-AZ RDS databases would have mitigated impact by failing away from the affected Availability Zone. While the vast majority of Lambda functions continue operating normally, some functions are experiencing invocation failures and latencies, but we expect this to improve over the next 30 minutes. Power has been restored to all affected resources and remains stable. We expect the recovery of EC2 instances and EBS volumes to continue to improve over the next 45 minutes. For customers that need immediate recovery, we recommend failing away from the affected Availability Zone as other Availability Zones are not affected by this issue.”
UPDATE from Amazon: “11:25 AM PDT We continue to make progress in recovering the remaining EC2 instances and EBS volumes affected by the loss of power in a single Availability Zone in the US-EAST-2 Region. The vast majority of EC2 instances are now healthy, but we continue to work on recovering the remaining EBS volumes affected by the issue. EC2 API error rates and latencies have returned to normal levels. Elastic Load Balancing remains weighted away from the affected Availability Zone. Error rates and latencies for Lambda function invocations have now returned to normal levels. Power has been restored to all affected resources and remains stable. We expect the recovery of EC2 instances and EBS volumes to continue to improve over the next 30 minutes. For customers that need immediate recovery, we recommend failing away from the affected Availability Zone as other Availability Zones are not affected by this issue.”
UPDATE from The Internet Patrol: At about 12:45 PM PDT we noticed that our own view into this showed that most systems were back up.
LAST UPDATE from Amazon: “12:45 PM PDT Starting at 9:57 AM PDT some EC2 instances and EBS volumes experienced a loss of power within a single Availability Zone in the US-EAST-2 Region. Power was restored at 10:19 AM PDT and EC2 instances and EBS volumes began to recover. By 10:23 AM PDT, the vast majority of EC2 instances and EBS volumes has fully recovered and by 11:37 AM PDT, all but a very small number of EC2 instances and EBS volumes had recovered. Elastic Load Balancing shifted traffic away from the affected Availability Zone, which has now been shifted back. RDS impact for single-AZ databases, which have also been recovered. Other services (tagged below) saw impact during the event, but most have fully recovered. Those that are still seeing impact, will provide updates via the Personal Health Dashboard as they work towards full recovery. The issue has been resolved and the service is operating normally.”
The Internet Patrol is completely free, and reader-supported. Your tips via CashApp, Venmo, or Paypal are appreciated! Receipts will come from ISIPP.
CashApp us
Venmo us
Paypal us
