maborak.com~/blog/cloud-outages-2026-failure-patternssoftware & infrastructure engineering
MaborakTechnologies
Contact
Back to all articles

Twelve 2026 cloud outages, investigated one by one

Overview

Twelve provider postmortems from February to October 2026, each investigated in its own section: trigger, the safeguard that missed it, who was exposed, a local reproduction where one is possible, the fix and how to verify it.

By the Maborak engineering team · · 41 min read

Twelve outages in 2026 have an official cause statement from the provider: Cloudflare, Microsoft Azure, AWS, Railway, Google Cloud and GitHub. Eleven have a full root-cause analysis; AWS published only a status-dashboard entry. Each incident below has its own section: the official RCA documents, what happened, how it unfolded step by step, who is exposed to the same failure, a local reproduction where the mechanism is software, the controls that stop it, and how to verify them.

Checked 30 October 2026. Provider records retrieved 19:05 to 19:25 UTC and re-read in full 19:55 to 20:05 UTC; proofs run 19:24 to 19:31 UTC. Provider statements are [vendor], local runs are [reproduced], and analysis and recommendations are [inferred]. The proofs reproduce mechanisms, not the providers’ systems.

Background

These are every 2026 outage, up to 30 October, for which the provider published a cause that this research could retrieve. The “Official RCA” line at the top of each section lists the provider’s documents, including later addenda and follow-ups. Cloudflare’s post-mortem tag (opens in a new tab) has no outage write-up after February, and AWS’s post-event summary index (opens in a new tab) has no 2026 entry. Oracle, Fastly, Akamai, Slack, Datadog and Microsoft 365 were not searched.

# Date (UTC) Incident Pattern Local proof
1 20 Feb Cloudflare BYOIP Scope Yes
2 24 Apr Azure East US control plane Health Yes
3 7 May AWS us-east-1 (use1-az4) Facility Inspection
4 19 May Railway Shared Yes
5 14 Jul Google Cloud VMware Engine Health Yes
6 15 Jul Google Cloud europe-west4-a Facility Inspection
7 17 Aug GitHub Shared Uses #4
8 20 Aug Google Cloud us-west1 Health Inspection
9 1 Sep Google Cloud us-central1 Scope Uses #1
10 29 Sep Azure OpenAI, Sweden Central Health Configuration check
11 30 Sep Azure networking gateways Scope Uses #1 and #4
12 20 Oct Google Cloud europe-west2-b Facility Inspection

The patterns are: Scope, automation or a procedure acting on more targets than intended; Health, a health check or failover that watched the wrong signal or could not finish; Shared, independent systems with one dependency underneath; Facility, power or cooling. Several incidents show more than one; the table gives the dominant one.

These twelve are not a sample of all outages. IncidentHub (opens in a new tab) counted 30,246 status-page outage posts from 1,082 providers between January and June, a count it says is “not a measure of customer impact”. Uptime Institute’s Annual outage analysis 2026 (opens in a new tab) reports that “power remains the leading cause of impactful outages”.

The local proofs ran on macOS 27.0.1 (arm64) with Go 1.27.3, Python 3.15.2 and Docker 29.5.2 (linux/arm64). Each is complete: save the code block under the file name shown and run the command. None contacts another host, apart from apk fetching two packages inside a container in incident 5.

1. Cloudflare BYOIP, 20 February

Official RCA: Cloudflare outage on February 20, 2026 (opens in a new tab), postmortem by D. Tuber and D. Trumic, published 21 February (page modified 22 July, change not stated). Status incident kwy3dt82bwbt (opens in a new tab). Cloudflare’s 1 May post Code Orange: Fail Small is complete (opens in a new tab) does not mention this incident, so it is not evidence that these fixes shipped.

What happened

Bring Your Own IP customers lost their BGP routes. About 1,100 of 4,306 BYOIP prefixes, 25%, were withdrawn between 17:56 and 18:46 UTC. CDN, Spectrum, Dedicated Egress and Magic Transit were affected; the 1.1.1.1 website returned 403 errors while the resolver kept working. The postmortem’s opening gives 17:48 UTC and a duration of “6 hours and 7 minutes”; its own timeline gives 17:56 to 23:03 UTC. [vendor]

How it unfolded

  1. Code merged on 5 February and released at 17:46 UTC on 20 February automated a customer request that had been manual: removing BYOIP prefixes. It ran as a “regularly running sub-task”.
  2. At 17:56 the sub-task called /v1/prefixes?pending_delete with no value. The server filtered only when Query().Get("pending_delete") was non-empty, so “the API server interprets this as a request for all BYOIP prefixes.”
  3. The sub-task deleted prefixes “and all of their related dependent objects including service bindings”. Address changes propagate to the edge immediately; only a quarter of customers were hit because the change “was applied iteratively”.
  4. Nothing stopped it: test mock data “did not cover a scenario where the task-runner service would independently execute changes to user data without explicit input”, and snapshot-based rollback “is not in Production today”.
  5. Detection came from a symptom: engineers engaged at 18:13 on failures of the 1.1.1.1 website, and declared an incident at 18:18.
  6. At 18:46 the sub-task was “terminated by an engineer and regular execution disabled”.
  7. Recovery depended on how much had been deleted. Customers could re-advertise withdrawn-only prefixes from the dashboard (guidance at 19:19); a release restored about 800 prefixes around 20:20; the ~300 whose service bindings were gone needed a global configuration rollout to every edge machine, from 21:08 to 23:03 UTC.

[vendor]

Impact

Exposed: any job that selects its targets through an API filter and then acts on the result, such as cleanup tasks, reconcilers and bulk API clients, when the filter can fall through to “all”. The damage scales with the job’s permissions; this one could delete service bindings, not only prefixes. [inferred]

Probe

The handler below copies the branch Cloudflare published. Eight prefixes, two marked for deletion. Save as emptyflag.go:

// Empty query flag: a cleanup job asks for "pending deletion" prefixes and
// receives every prefix.
// Run: go run emptyflag.go [-breaker] [-strict] [-query=pending_delete=true]
package main

import (
	"encoding/json"
	"flag"
	"fmt"
	"net/http"
	"net/http/httptest"
	"strconv"
	"sync"
)

type prefix struct {
	CIDR          string `json:"cidr"`
	PendingDelete bool   `json:"pending_delete"`
}

var (
	mu    sync.Mutex
	store = []prefix{
		{"192.0.2.0/24", true}, {"198.51.100.0/24", false}, {"203.0.113.0/24", false},
		{"100.64.0.0/24", false}, {"100.64.1.0/24", false}, {"100.64.2.0/24", true},
		{"100.64.3.0/24", false}, {"100.64.4.0/24", false},
	}
)

// list mirrors the branch Cloudflare published: the filter runs only for a
// non-empty value, so "?pending_delete" falls through to "return everything".
func list(w http.ResponseWriter, r *http.Request) {
	mu.Lock()
	defer mu.Unlock()
	out := store
	if v := r.URL.Query().Get("pending_delete"); v != "" {
		out = nil
		for _, p := range store {
			if p.PendingDelete {
				out = append(out, p)
			}
		}
	}
	json.NewEncoder(w).Encode(out)
}

// strictList is the fixed handler: the flag must parse as a boolean.
func strictList(w http.ResponseWriter, r *http.Request) {
	q := r.URL.Query()
	if q.Has("pending_delete") {
		if _, err := strconv.ParseBool(q.Get("pending_delete")); err != nil {
			http.Error(w, "pending_delete must be true or false", http.StatusBadRequest)
			return
		}
	}
	list(w, r)
}

func remove(cidr string) {
	mu.Lock()
	defer mu.Unlock()
	for i, p := range store {
		if p.CIDR == cidr {
			store = append(store[:i], store[i+1:]...)
			return
		}
	}
}

func main() {
	breaker := flag.Bool("breaker", false, "stop batches larger than 25% of the inventory")
	strict := flag.Bool("strict", false, "reject a pending_delete flag that is not a boolean")
	query := flag.String("query", "pending_delete", "query string the cleanup job sends")
	flag.Parse()
	handler := http.HandlerFunc(list)
	if *strict {
		handler = strictList
	}
	srv := httptest.NewServer(handler)
	defer srv.Close()

	total := len(store)
	resp, err := http.Get(srv.URL + "/v1/prefixes?" + *query) // default: flag present, value empty
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()
	fmt.Printf("GET ?%s -> HTTP %d\n", *query, resp.StatusCode)
	if resp.StatusCode != http.StatusOK {
		fmt.Printf("job aborted; %d of %d prefixes remain\n", len(store), total)
		return
	}
	var batch []prefix
	json.NewDecoder(resp.Body).Decode(&batch)
	fmt.Printf("prefixes returned for deletion: %d of %d\n", len(batch), total)

	// Circuit breaker: refuse a batch larger than 25% of the inventory.
	if *breaker && len(batch)*4 > total {
		fmt.Println("circuit breaker: batch exceeds 25% of inventory, job stopped")
		return
	}
	for _, p := range batch {
		remove(p.CIDR)
	}
	fmt.Printf("prefixes remaining: %d of %d\n", len(store), total)
}
$ go run emptyflag.go
GET ?pending_delete -> HTTP 200
prefixes returned for deletion: 8 of 8
prefixes remaining: 0 of 8
$ go run emptyflag.go -query=pending_delete=true
GET ?pending_delete=true -> HTTP 200
prefixes returned for deletion: 2 of 8
prefixes remaining: 6 of 8

[reproduced] The only difference between the runs is =true. The proof shows the selection bug, not Cloudflare’s deletion rate or restore path.

Mitigation

  • Parse control flags strictly and reject what does not parse (-strict above). Cloudflare says it will adopt a stricter API schema, because “the pending_delete flag was interpreted as a string.” [vendor]
  • Refuse batches larger than a fixed share of the inventory (-breaker above, 25%). Cloudflare says it will add a circuit breaker for changes that move “too fast or too broadly”, and roll out configuration snapshots behind health metrics. No later Cloudflare post confirms these shipped. [vendor]
  • Rollback: both controls are flags; removing them restores the original behaviour.

Verification

$ go run emptyflag.go -strict
GET ?pending_delete -> HTTP 400
job aborted; 8 of 8 prefixes remain
$ go run emptyflag.go -breaker
GET ?pending_delete -> HTTP 200
prefixes returned for deletion: 8 of 8
circuit breaker: batch exceeds 25% of inventory, job stopped

[reproduced] On a running API, send the malformed request in staging and expect 400, not 200: curl -s -o /dev/null -w '%{http_code}\n' 'https://api.staging.example/v1/prefixes?pending_delete'. [inferred]

2. Azure East US control plane, 24 April

Official RCA: Post Incident Review 5GP8-W0G (opens in a new tab), “Multiple services – Control plane issues in East US”, final version (no preliminary remains on the page). Azure Incident Retrospective recording, YouTube (opens in a new tab), uploaded 19 May 2026, about 29 minutes.

What happened

From 11:30 to 23:22 UTC, 11 hours 52 minutes, customers in East US saw failures or delays when they tried to “provision, scale, or update resources”, and some newly provisioned workloads had connectivity problems. [vendor]

How it unfolded

  1. A newly deployed PubSub version, PubSub being “a key component of the networking control plane”, carried “a latent regression” that slowed replica builds in large zones. It “had not surfaced during earlier testing, or in any other regions which had already received the update.”
  2. At 11:30 one partition in physical zone AZ-01 hit lock contention and its operations timed out; this was detected at 11:38.
  3. The automatic failover “did not complete successfully”, and a manual failover from 11:38 to 13:40 also failed: the regression “contributed to the failure of the failover process”.
  4. Azure’s automated messaging sent the first targeted notices at 11:59; the first public status update came at 15:36.
  5. At 13:40 the update was identified as the likely trigger. AZ-03 was affected from 13:50 and rolled back from 14:07; AZ-01 was rolled back from 15:04 and was above 99% at 19:02.
  6. Rollback was slow by design: it runs “in stages… using update domains”, sometimes rebuilding “full replicas on new nodes”. During it, quorum was lost twice, and “as load redistributed” AZ-02 was affected at 19:05.
  7. At 21:02 AZ-03 relapsed, because the rollback “had not yet completed across all update domains”. It finished at 22:39, and all impact was mitigated at 23:22.

[vendor]

Impact

Exposed: any replicated service whose failover must rebuild a replica within a deadline, when that deadline was measured on data smaller than the largest production partition. The failure appears only where the data is largest. [inferred]

Probe

A replica rebuild copies 1 MiB chunks under a lock and must finish within 1.0 s. The “new release” holds the lock 8 ms per chunk instead of 2 ms. Save as failover.py:

"""Failover that cannot finish: a replica rebuild copies a partition chunk by
chunk under a lock, and must finish before the failover deadline. A new
release holds the lock longer per chunk. Only the large zone misses.

Run: python3 failover.py [hold_ms_new]
"""
import sys
import threading
import time

DEADLINE_S = 1.0
LOCK = threading.Lock()


def rebuild(chunks: int, hold_ms: float) -> float:
    """Copy `chunks` 1 MiB chunks, holding the partition lock `hold_ms` each."""
    payload = bytes(1 << 20)
    start = time.perf_counter()
    for _ in range(chunks):
        with LOCK:
            _ = bytearray(payload)        # the copy itself
            time.sleep(hold_ms / 1000)    # lock held for bookkeeping
    return time.perf_counter() - start


if __name__ == "__main__":
    hold_new = float(sys.argv[1]) if len(sys.argv) > 1 else 8.0
    for zone, chunks in (("small", 40), ("large", 200)):
        for release, hold_ms in (("old", 2.0), ("new", hold_new)):
            elapsed = rebuild(chunks, hold_ms)
            result = "completed" if elapsed <= DEADLINE_S else "TIMED OUT"
            print(f"{zone:<6} {chunks:>4} MiB  {release:<4} {hold_ms:>4.1f} ms/chunk  "
                  f"rebuild {elapsed:5.2f} s  deadline {DEADLINE_S:.1f} s  {result}")
$ python3 failover.py
small    40 MiB  old   2.0 ms/chunk  rebuild  0.10 s  deadline 1.0 s  completed
small    40 MiB  new   8.0 ms/chunk  rebuild  0.38 s  deadline 1.0 s  completed
large   200 MiB  old   2.0 ms/chunk  rebuild  0.50 s  deadline 1.0 s  completed
large   200 MiB  new   8.0 ms/chunk  rebuild  1.90 s  deadline 1.0 s  TIMED OUT

[reproduced] A test at 40 MiB passes on both releases; only the largest size exposes the regression, the property Microsoft describes. This is a timing model, not PubSub: the numbers are chosen, and only the shape carries over.

Mitigation

  • Set failover deadlines from measurements at the largest partition, and re-measure after every release of the replication component. [inferred]
  • Microsoft rolled the update back in other regions with large zones (completed). It lists wider test coverage and smaller PubSub partitions for June 2026, automatic customer notification for August, and protecting recovery operations from resource limits for September; the PIR still shows these as estimated. [vendor]
  • Rollback: the release itself. Microsoft tried two failovers first; the rollback began at 14:07 and took until 22:39 to finish everywhere.

Verification

$ python3 failover.py 3
large   200 MiB  new   3.0 ms/chunk  rebuild  0.74 s  deadline 1.0 s  completed

[reproduced] Output trimmed to the changed line: with a 3 ms hold the large rebuild fits the deadline. On a real system, time a staging failover at the size of the largest production partition after each release, and record the margin to the deadline. [inferred]

3. AWS us-east-1, 7 May

Official RCA: none. AWS has published no post-event summary (opens in a new tab); the index still ends in October 2025, although AWS’s own criteria for publishing one include total power failure. The only official record is the Health Dashboard event (opens in a new tab), kept for about 12 months.

What happened

Impact began at 23:20 UTC in “a single facility (data center) within a single Availability Zone (use1-az4)”. EC2 and EBS were affected, and the event lists nine further services as impacted, from NAT Gateway to SageMaker. AWS marked the event resolved at 03:04 UTC on 9 May with a “small number” of resources still impaired, and gave no full-recovery time. [vendor]

How it unfolded

  1. At 23:20 UTC on 7 May, temperatures rose in one data centre. AWS’s only cause statement is “a thermal event resulting in a loss of power”; it does not say what failed in the cooling system.
  2. The loss of power was protective: “servers automatically shut down when the temperatures exceeded the operating thresholds in order to protect the hardware.”
  3. “Engineers were automatically engaged within minutes.” At 00:06 UTC on 8 May AWS shifted traffic away from the zone, 19 minutes before its first public post at 00:25.
  4. Dependent services degraded separately: Redshift errors came from “an upstream dependency” and were fixed at 16:30, and MSK partition timeouts began at 18:12.
  5. Power returned to part of the hardware at 04:12; racks came back in phases as cooling capacity was added.
  6. At 20:50 UTC cooling was back to “pre-event levels” and most instances and volumes were restored.

[vendor]

Impact

Exposed: workloads whose instances or volumes sit only in use1-az4. That ID is not the zone name an account sees: AWS maps names such as us-east-1a to physical zones “randomly” per account, while an AZ ID like use1-az4 “represents the same physical location in every AWS account.” [vendor] (AWS AZ IDs (opens in a new tab))

Probe

Facility failures cannot be reproduced locally. The inspection that applies is finding which of your zone names is use1-az4 and what runs there, read-only:

az=$(aws ec2 describe-availability-zones --region us-east-1 \
  --query "AvailabilityZones[?ZoneId=='use1-az4'].ZoneName" --output text)
echo "use1-az4 is $az in this account"
aws ec2 describe-instances --region us-east-1 \
  --filters "Name=availability-zone,Values=$az" "Name=instance-state-name,Values=running" \
  --query 'length(Reservations[].Instances[])'

Not run here: it needs an AWS account. The first command is the zone-mapping query AWS documents; the second counts running instances in that zone.

Mitigation

  • Spread instances and volumes for each service across at least two AZ IDs, and know which zone name maps to which ID in each account. [inferred]
  • AWS published no follow-ups for this event.

Verification

Group running instances by zone and confirm no service has all of them in one: aws ec2 describe-instances --region us-east-1 --query 'Reservations[].Instances[].Placement.AvailabilityZone' --output text | tr '\t' '\n' | sort | uniq -c. [inferred]

4. Railway, 19 May

Official RCA: Incident Report: May 19, 2026 – GCP Account Suspension (opens in a new tab), dated 20 May, edited later without a revision note (see below). Status incident I23M92U0 (opens in a new tab); staff FAQ (opens in a new tab), 20 May. Follow-ups: Going multi-cloud (opens in a new tab), 1 June, and a later routing incident report (opens in a new tab), 30 September. Google has published no statement, report or post about the suspension.

What happened

From 22:20 UTC on 19 May, Railway’s “API, control plane and databases” went offline. Workloads on Railway’s own hardware and on AWS kept running but became unreachable, returning 404 errors, “even though the workloads themselves remained online”. The API was back at 04:00 UTC on 20 May, monitoring began at 06:14, and the status page marked the incident resolved at 07:57. [vendor]

How it unfolded

  1. At 22:20 Google Cloud suspended Railway’s production project. The report’s “What Happened?” section says “incorrectly, as part of an automated action”. Its opening paragraph has changed: archived copies from 20 May to 26 June say Google “incorrectly” placed the account in suspension; by 29 June it cited “detected activity flagged as violation of Terms of Service”; by 13 July it added that Google had sent “previous notices of violations” but “did not provide us a notice prior to this suspension”.
  2. Monitoring caught API health-check failures at 22:10, and the cause was identified at 22:19; the report does not reconcile 22:10 with the 22:20 start. A P0 ticket went to Google at 22:22.
  3. The suspension removed the API, the Cloud SQL database and the Google-hosted compute at once.
  4. Account access came back at 22:29, but “All compute instances remained stopped and persistent disks inaccessible”; disks, compute and networking each needed separate recovery. Railway is still “awaiting confirmation” whether that delay was on Google’s side.
  5. At 22:35 cached routes began expiring. The edge proxies “maintain a cache of routing tables from the network control plane, which is hosted within Google Cloud”, so once the cache expired, workloads on Metal and AWS returned 404s. Railway was “in the middle of fully distributing the routing table” when it happened.
  6. Disks recovered from 23:09 to 23:54, compute and edge from 01:30 to 01:38, builds at 01:57, and the API at 04:00 UTC.

[vendor]

Impact

Exposed: any multi-cloud or multi-site design whose routing, configuration or account lives with one provider. Backends that never failed become unreachable when the router can no longer refresh its routes. [inferred]

Probe

Two backends, labelled metal and aws, sit behind a router that learns routes from one control plane and caches them for 2 s. Save as shareddep.py:

"""Hidden shared dependency: two backends on "independent" infrastructure sit
behind a router that learns its routes from one control plane and caches them
for TTL seconds. Stop the control plane and both backends disappear once the
cache expires, although neither backend failed.

Run: python3 shareddep.py            # router drops expired routes
     python3 shareddep.py --stale-ok # router keeps the last known good table
"""
import json
import sys
import threading
import time
import urllib.error
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

TTL_S = 2.0


def serve(handler_cls):
    srv = ThreadingHTTPServer(("127.0.0.1", 0), handler_cls)
    threading.Thread(target=srv.serve_forever, daemon=True).start()
    return srv


def backend(name):
    class H(BaseHTTPRequestHandler):
        def do_GET(self):
            self.send_response(200)
            self.end_headers()
            self.wfile.write(f"hello from {name}".encode())

        def log_message(self, *a):
            pass
    return serve(H)


metal, aws = backend("metal"), backend("aws")
ROUTES = {"app-metal": metal.server_port, "app-aws": aws.server_port}


class ControlPlane(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.end_headers()
        self.wfile.write(json.dumps(ROUTES).encode())

    def log_message(self, *a):
        pass


control = serve(ControlPlane)
CONTROL_URL = f"http://127.0.0.1:{control.server_port}/routes"


class Router:
    def __init__(self, stale_ok):
        self.stale_ok, self.table, self.fetched = stale_ok, {}, 0.0

    def routes(self):
        if time.monotonic() - self.fetched > TTL_S:
            try:
                with urllib.request.urlopen(CONTROL_URL, timeout=0.3) as r:
                    self.table, self.fetched = json.load(r), time.monotonic()
            except (urllib.error.URLError, OSError):
                if not self.stale_ok:
                    self.table = {}  # expired and cannot refresh: forget routes
        return self.table

    def get(self, host):
        port = self.routes().get(host)
        if port is None:
            return "404 no route"
        with urllib.request.urlopen(f"http://127.0.0.1:{port}/", timeout=0.3) as r:
            return f"200 {r.read().decode()}"


router = Router(stale_ok="--stale-ok" in sys.argv)
start = time.monotonic()
for step in range(7):
    if step == 1:
        control.shutdown()
        control.server_close()
        print(f"t={time.monotonic() - start:4.1f}s control plane stopped (backends still up)")
    t = time.monotonic() - start
    print(f"t={t:4.1f}s  app-metal: {router.get('app-metal'):<22} app-aws: {router.get('app-aws')}")
    time.sleep(0.5)
$ python3 shareddep.py
t= 0.0s  app-metal: 200 hello from metal   app-aws: 200 hello from aws
t= 1.0s control plane stopped (backends still up)
t= 1.0s  app-metal: 200 hello from metal   app-aws: 200 hello from aws
t= 1.5s  app-metal: 200 hello from metal   app-aws: 200 hello from aws
t= 2.0s  app-metal: 404 no route           app-aws: 404 no route
t= 2.5s  app-metal: 404 no route           app-aws: 404 no route

[reproduced] Output trimmed to six lines. The stop time varies between 0.5 s and 1.0 s from run to run, because the server’s shutdown waits for its 0.5 s poll interval; the 404s always start once the 2 s cache expires. Neither backend failed.

Mitigation

  • Keep the last known good routing table when the control plane cannot be reached, instead of dropping routes when the cache expires (--stale-ok above). [inferred]
  • Railway said it would remove the routing dependency on the Google-hosted control plane, extend its high-availability database shards across AWS and its own hardware, and keep Google Cloud “only for secondary/failover”. On 1 June it reported the first version of distributed routing “now live for all customers”. On 30 September that new routing control plane returned 404s for every domain for about five minutes, because “the origin proxies couldn’t fall back to the last route it had seen for that domain”: the same gap this probe shows. [vendor]
  • Rollback: the flag; without it, the router returns to dropping expired routes.

Verification

$ python3 shareddep.py --stale-ok
t= 0.0s  app-metal: 200 hello from metal   app-aws: 200 hello from aws
t= 1.0s control plane stopped (backends still up)
t= 2.0s  app-metal: 200 hello from metal   app-aws: 200 hello from aws
t= 3.5s  app-metal: 200 hello from metal   app-aws: 200 hello from aws

[reproduced] Trimmed to four lines; every sample after the stop returned 200. On a real system, block the control plane’s address from one router in staging, wait past the route TTL and confirm traffic still returns 200. [inferred]

5. Google Cloud VMware Engine stretched clusters, 14 July

Official RCA: Google Cloud incident T8gmtofFSTGT5tbhyciF (opens in a new tab): nine status updates (14–15 July), a Preliminary Incident Report (20 July) and the final Incident Report (24 July), all on the same page. No blog post.

What happened

For 10 hours 40 minutes, from 17:00 UTC on 14 July to 03:40 UTC on 15 July, VMware Engine stretched clusters in australia-southeast2 and europe-west3 lost communication between their zones. VMware HA restarted or moved VMs “as designed” for “some customers, depending on their architecture”. [vendor]

How it unfolded

  1. At 17:00 a configuration update “deployed to the foundational network control plane” was “structurally valid” but exposed “an implementation gap within the control plane’s routing logic”. It “passed initial payload validations.”
  2. Hosts misprogrammed their routing tables, and “traffic destined for the private IP address space used by GCVE Stretched Clusters was dropped.”
  3. Stretched clusters carry their inter-zone traffic on that range, so the two sites lost each other and their witness appliance, the third site that decides which side keeps running. HA events followed; VMs not replicated with vSAN are pinned to one zone. An interim update warned that VMs on isolated sites “may be left without writable data.”
  4. Fabric BGP and BFD “remained fully functional because their control plane sessions run on separate, unaffected address spaces”, so standard failover “failed to detect” the loss. One interim update did mention “BGP session flapping”.
  5. The first public update came at 20:24 UTC, 3 hours 24 minutes after the start. Google named the configuration change as the likely cause at 23:05 and began a remediation rollout at 00:13.
  6. The fix was a rollback “to its last-known good value” plus rerouting over working paths; service was restored at 03:40 UTC.

[vendor]

Impact

Exposed: any failover decided by a check that uses a different address or path from the workload’s traffic. The check stays green while users fail. [inferred]

Probe

Two network namespaces are joined by a veth pair. The health check pings 10.0.0.2; the workload uses 10.200.0.2. The faulty update is one blackhole route for the workload range. Save as datapath.sh:

#!/bin/sh
# Healthy control plane, dead data path. Two network namespaces joined by a
# veth pair: the health check uses 10.0.0.2, the workload uses 10.200.0.2.
# A bad routing entry drops only the workload range. Runs inside a container.
set -eu

ip netns add site-a
ip netns add site-b
ip link add veth-a type veth peer name veth-b
ip link set veth-a netns site-a
ip link set veth-b netns site-b
ip -n site-a addr add 10.0.0.1/24 dev veth-a
ip -n site-b addr add 10.0.0.2/24 dev veth-b
ip -n site-b addr add 10.200.0.2/32 dev lo
for ns in site-a site-b; do ip -n "$ns" link set lo up; done
ip -n site-a link set veth-a up
ip -n site-b link set veth-b up
ip -n site-a route add 10.200.0.0/24 via 10.0.0.2

probe() { # name address
  if ip netns exec site-a ping -c 1 -W 1 "$2" >/dev/null 2>&1; then
    echo "  $1 ($2): healthy"; return 0
  fi
  echo "  $1 ($2): FAILED"; return 1
}

decide() {
  echo "$1"
  control=0; data=0
  probe "control-plane check" 10.0.0.2 || control=1
  probe "data-path probe    " 10.200.0.2 || data=1
  [ "$control" -eq 0 ] && echo "  failover on control-plane check: no" || echo "  failover on control-plane check: yes"
  [ "$data" -eq 0 ] && echo "  failover on data-path probe:     no" || echo "  failover on data-path probe:     yes"
}

decide "before the change:"
# The "structurally valid" update: a route that drops the workload range.
ip -n site-a route replace blackhole 10.200.0.0/24
decide "after the routing change:"
ip -n site-a route show | sed 's/^/  route: /'

Creating namespaces needs a privileged container; it is removed on exit and touches no host interface.

$ docker run --rm -i --privileged alpine:3.24 \
    sh -c 'apk add -q --no-cache iproute2 iputils >/dev/null && sh -s' < datapath.sh
before the change:
  control-plane check (10.0.0.2): healthy
  data-path probe     (10.200.0.2): healthy
  failover on control-plane check: no
  failover on data-path probe:     no
after the routing change:
  control-plane check (10.0.0.2): healthy
  data-path probe     (10.200.0.2): FAILED
  failover on control-plane check: no
  failover on data-path probe:     yes
  route: 10.0.0.0/24 dev veth-a proto kernel scope link src 10.0.0.1
  route: blackhole 10.200.0.0/24

[reproduced] Image alpine:3.24 (digest sha256:be2656061b53…), iproute2 7.0.0. A ping stands in for BGP and BFD; what matters is which address the check uses.

Mitigation

  • Drive failover from a probe inside the workload’s own address range, not only from the routing session. [inferred]
  • Google’s follow-ups include data-path failover that probes “the specific data-tunneling traffic space”, detailed GCVE network setups in testing, faster inter-zone alerts, tuned HA and storage settings, and outreach to customers whose VMs are pinned to one zone. No dates are given. [vendor]

Verification

The failover on data-path probe: yes line above is the control working. On a real system, read the health-check target from the deployed configuration and confirm it is an address inside the data range, not the session address. [inferred]

6. Google Cloud europe-west4-a, 15 July

Official RCA: Google Cloud incident 3BvH3LVGcupoYqV6F4Nw (opens in a new tab): ten status updates (16 July), a Preliminary Incident Report (18 July, 00:22 UTC) and the final Incident Report (25 July) with separate facility, VMware Engine, Bare Metal Solution and NetApp timelines. No blog post.

What happened

At 23:24 UTC on 15 July, 01:24 on 16 July in the Netherlands, “a 3ms voltage drop in the power feed from the utility provider” affected both utility feeds of the data centre. VMware Engine, Bare Metal Solution and NetApp Volumes were interrupted for a combined 14 hours 55 minutes; the per-service windows in the preliminary report were 9 h 24 min, 12 h 57 min and 8 h 31 min. Impact covered 24 VMware Engine private clouds for 20 customers, 9 Bare Metal Solution customers and all 6 NetApp Volumes clusters. [vendor]

How it unfolded

  1. At 23:24 the voltage drop tripped the utility breakers on both feeds. Side B’s diesel rotary UPS took the load; side A’s “failed to take over the facility load due to electrical component failures”.
  2. At 23:37 rows 1 and 2 transferred to feed B. Row 3 failed to transfer when an “overload protection breaker trip[ped]”, caused by a “load deployment discrepancy” still under investigation, and lost all power.
  3. Also at 23:24, “the chiller controller dropped offline during the voltage transient event, failing to signal the chilled water distribution pumps to restart.” Chiller A stopped. The redundant chiller “was not available due to known ongoing construction work at the facility.”
  4. Temperatures climbed: network switches passed 60 °C at 00:05, and Google was alerted to rising temperatures at 00:44. The NetApp clusters shut themselves down at 00:32.
  5. The hall reached 44 °C at 01:29. Bare Metal Solution storage nodes overheated at 01:37, and Google shut down reachable devices deliberately from 02:00 to 02:55.
  6. Staff restored circulation by switching the pumps from automatic to manual; cooling was back at 04:05. A portable UPS for the chiller controllers and a spare diesel rotary UPS were brought in as interim fixes.
  7. NetApp Volumes recovered by 06:55, VMware Engine by 09:36 after an on-site console-server recovery, and the last Bare Metal Solution server was rebooted at 13:32 UTC on 16 July.

[vendor]

Impact

Exposed: the 20 VMware Engine and 9 Bare Metal Solution customers Google counts, and in general any workload whose only copy runs in one zone. [inferred]

Probe

Not reproducible locally. The inspection is listing what runs in the zone, read-only, for example gcloud compute instances list --filter="zone:europe-west4-a" --format="value(name)", plus the GCVE private clouds and Bare Metal Solution servers registered there. Not run here: it needs a Google Cloud project.

Mitigation

  • A recovery plan for single-zone products that does not assume the zone comes back, tested at least once. [inferred]
  • Google listed seven follow-ups due August to October 2026, including the cause of the failed backup transfer, chiller pump control redundancy, early cooling alerts and faster automated shutdown during thermal runaway. The record has no update after 25 July. [vendor]

Verification

A recorded restore of one single-zone workload into another zone, with the time it took. [inferred]

7. GitHub front door and authentication, 17 August

Official RCA: status incident zkxwbgr0cnmx (opens in a new tab), whose resolution post is the most detailed account; the CTO’s post The August 17 outage, and the work ahead (opens in a new tab), 20 August; and the August 2026 availability report (opens in a new tab), 9 September.

What happened

From 13:28 to 21:15 UTC, 7 hours 47 minutes by the status incident and the CTO’s post, issues, pull requests, the REST and GraphQL APIs, Actions, Copilot, sign-in and webhooks returned errors and slow responses. The availability report gives 7 hours 35 minutes in its header and about 6 hours 44 minutes in its body. At the peak, 56.07% of front-door requests to the affected services “failed or ran slow”; the hard error rate on web and API traffic was about 20% early on. About 29,000 organisations saw at least one failed or slow request, about 4.8 million failed or slow requests in total. “No data was lost.” [vendor]

How it unfolded

  1. At 13:28 a new traffic peak reached the Central US data centre. An Istio sidecar hit its concurrency limit and did not autoscale, “because of a misconfigured policy that watched host service but not sidecar limits”.
  2. The backlog spread until “four HAProxy nodes exhausted their flow limits, degrading the gateway auth path”. Every service routed through that data centre depended on that path, and the gateway’s “optimistic retry logic” added load to the internal load balancers.
  3. Automated monitoring opened the incident at 13:40.
  4. Engineers moved some traffic to Northern Virginia and cut gateway retries. “Pausing HAProxy on those nodes simultaneously produced immediate broad recovery”: most services were back by 16:36, and Actions by about 18:03.
  5. During recovery, “a latent client retry bug” amplified traffic to the Copilot Token Service about tenfold, from 7,000–9,000 to 70,000–100,000 requests per second. GitHub blocked those token requests at the load balancers and ramped traffic back up site by site.
  6. Copilot recovered at 21:02, and the incident was resolved at 21:15 UTC.

[vendor]

Impact

Exposed: services that share one authentication or gateway path, and clients that retry without a budget. A partial slowdown becomes a full outage when every failed request is sent again. [inferred]

Probe

The shared-path half is the same mechanism as incident 4: replace the control plane in shareddep.py with an authentication gateway, and every service behind it fails together. The retry half needs no simulation: GitHub measured it. A client bug turned 7,000–9,000 requests per second into 70,000–100,000 against one endpoint during recovery, and gateway retries had already added load during the initial overload. [vendor]

Mitigation

  • Retry budgets and exponential backoff with jitter on clients, and a cap on retries at gateways. [inferred]
  • GitHub’s follow-ups: autoscaling that accounts for sidecar concurrency, an audit of mesh limits, retry and backoff limits across gateways and clients, a fix for the retry bug, and better load-balancer capacity monitoring and failover safeguards. [vendor]

Verification

Read the retry settings from each client’s deployed configuration and confirm a maximum attempt count and backoff are set. In a load test, make the authentication dependency return errors and confirm request volume to it does not rise above the incoming rate. [inferred]

8. Google Cloud us-west1, 20 August

Official RCA: Google Cloud incident utF3FMFdQfwBzJcGG6vf (opens in a new tab): status updates on 20 August, a preliminary report (24 August) and the final Incident Report (27 August), all on the same page. No blog post.

What happened

From 15:00 to 17:22 UTC, 2 hours 22 minutes, capacity between data-centre campuses in the us-west1 region, in the Dalles, Oregon metro area, dropped. The report lists 26 affected services, among them Compute Engine, Cloud Run, Bigtable, Cloud Storage, Cloud IAM, Persistent Disk and Google Kubernetes Engine; some backlogs and VM provisioning in us-west1-c recovered after 17:22. [vendor]

How it unfolded

  1. At 15:00 “scheduled fiber optic maintenance… unexpectedly compromised network capacity between data centers within the us-west1 region.”
  2. “Automated rerouting mechanisms failed to properly redistribute traffic to alternate capacity.” Google does not say why.
  3. Traffic volumes exceeded the remaining bandwidth, causing “severe packet loss, request throttling, and increased latency across inter-campus dependencies”.
  4. Spanner’s Paxos consensus and the Unified Metadata Server slowed down, and the delay “cascaded into timeouts” for the products built on them, in both control-plane and data-plane requests. “Increased retry volume” added load.
  5. Internal monitoring raised the alert; the first public update came at 16:44 UTC. Engineers drained traffic away from the affected links, and Google advised customers to fail over “to other regions where feasible”.
  6. At 17:22 fiber capacity was restored, and traffic was reintroduced “through controlled validation stages”.

[vendor]

Impact

Exposed: workloads in us-west1 that depend on regional storage, IAM or GKE control planes, including workloads spread across all three zones, because the failure was between data centres in the region. [inferred]

Probe

Not reproducible locally: it needs real capacity between sites. Google does not explain why rerouting failed; treating it as a sibling of the health-check gap reproduced in incident 5 is our reading, not Google’s. [inferred] The inspection for a tenant is listing which regional services a workload calls synchronously.

Mitigation

  • For workloads that cannot tolerate a regional degradation, a second region that does not call the first on the request path. [inferred]
  • Google lists “key focus areas” still “being finalized and assigned to owning teams”: “stricter safety checks and circuit redundancy requirements prior to routine maintenance”, and refined automated regional traffic failover. No dates are given. [vendor]

Verification

A test that blocks the workload’s calls to one region and confirms requests are served from the other. [inferred]

9. Google Cloud us-central1, 1 September

Official RCA: Google Cloud incident J5ia5t9p3g9Q5Wi7r8Ev (opens in a new tab): status updates on 1 September, a preliminary report (3 September), the final Incident Report (10 September) and an addendum (29 September) that widened the described scope from us-central1-b to -b and -f and added per-product detail. No blog post.

What happened

From 14:41 to 18:52 UTC, 4 hours 11 minutes, “only two clusters in a datacenter in us-central1-b and us-central1-f” lost network connectivity. Workloads in those clusters were affected, along with Spanner and Firestore in the nam5 multi-region, Private Service Connect clients “in specific circumstances”, Cloud Router, VPN and Interconnect. [vendor]

How it unfolded

  1. The plan was to replace optical transceivers one router at a time “over several days”, with traffic diversion and verification between routers. The redundant routers sit in separate rooms of one building.
  2. The new higher-density transceivers were “not compatible with the transceivers still in place in the data center fabric, causing a loss of connectivity for each fiber”. Google does not say why.
  3. “A procedural error in the manually orchestrated upgrade process caused the complete list of transceiver replacements across all routers to be issued to the technician without instructions to sequence the work one router at a time.” The workflow also “did not include the expected human and software verification steps”.
  4. From 14:41 the technician “unplugged all fiber paths across the affected devices within 13 minutes”, and “a standing procedure to halt if light is detected on any fiber-optic cable after unplug was not followed.” Live light on an unplugged fiber means the link was still carrying traffic.
  5. Monitoring detected the loss immediately. Traffic was diverted away at 14:45 and 15:00; regional products were affected for about 19 minutes until then.
  6. The original transceivers were reinserted: most links were back at 15:50 and diversions were removed at 16:19.
  7. The diverted traffic had raised errors in other clusters, and after the restore Cloud Run and App Engine hit a “‘thundering herd’ of concurrent instance initializations” that “bypassed warm caches and overwhelmed internal file-serving tiers” until emergency capacity was added. Full recovery came at 18:52.

[vendor]

Impact

Exposed: any change procedure, manual or automated, that receives the whole target list at once and has no stop condition between targets. [inferred]

Probe

This is the shape reproduced in incident 1 with a work order instead of an API: every target released at once. The -breaker run there is the equivalent of a work order that stops after a set share of devices.

Mitigation

  • Issue work orders one device at a time, with a check of the previous device before the next is released. [inferred]
  • Google’s follow-ups: retraining on the live-light check, work-stop alerts for live-fiber disconnects, and finishing the move of network upgrades to “a fully automatically sequenced orchestration system”: regional workflows were automated in 2025, zonal ones such as this are “in progress”. Automated draining of faulty zones is planned to cut the 19-minute step to about 5. Google also halted maintenance in the region pending audits. [vendor]

Verification

Review the change tooling: a fleet-wide change should be impossible to submit as one unsequenced batch, and the stop condition should be enforced by the tool, not only written in a procedure. [inferred]

10. Azure OpenAI and Foundry, Sweden Central, 29 September

Official RCA: Post Incident Review 7QL5-Z50 (opens in a new tab). Azure Incident Retrospective livestreams on 22 and 23 October (session 1 (opens in a new tab), session 2 (opens in a new tab)); no recording had appeared in the retrospectives playlist (opens in a new tab) by 30 October.

What happened

From 10:03 to 15:58 UTC, about 5 hours 55 minutes, customers of Azure OpenAI, Foundry Agent Service, Foundry Models and Azure AI services in Sweden Central saw “intermittent request failures, increased latency, and/or HTTP 500 and 503 errors”. [vendor]

How it unfolded

  1. At 09:38 a planned deployment of an internal service that calls the backend cleared that service’s own caches, at peak traffic.
  2. “Two other defects in how that service read from its caches” sent most lookups straight to the backend; by 10:00 backend traffic was “approximately 20 times its normal level”. The PIR does not describe the defects further.
  3. The backend’s cache layer reached full utilisation and latency rose; connections stayed open longer and retries added requests, until the “network connection capacity on the infrastructure node instances was exhausted”. Customer impact began at 10:03.
  4. “Automated health checks for each instance depended on successfully connecting to the database layer.” With no outbound connections left, instances failed the check and were “automatically restarted by design”, leaving fewer to serve. The database itself was healthy.
  5. An alert fired at 10:17, node timeouts were identified at 11:30, and Microsoft notes its metrics “did not effectively indicate the connection exhaustion”.
  6. Scaling attempts from 13:55, and more compute and higher limits at 15:00, did not fix it.
  7. At 15:40 Microsoft “reconfigured the automated health check to no longer depend on remote connectivity, which stopped the restart loop”; recovery was complete at 15:58. The traffic source was traced to the deployment only at 16:30.

[vendor]

Impact

Exposed: services whose restart-triggering health check calls a remote dependency. When the dependency is slow, the check restarts healthy instances and removes capacity exactly when it is needed. [inferred]

Probe

In Kubernetes terms, this is a liveness probe that checks the database. The Kubernetes documentation separates the two: a failed liveness probe means “the kubelet kills the container and restarts it”, while a readiness probe only stops traffic, and is the place for an application that may “depend on external services after startup”. [vendor] (Kubernetes probes (opens in a new tab)) The inspection is reading which endpoints each probe calls:

kubectl get deploy -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.spec.template.spec.containers[*].livenessProbe.httpGet.path}{"\n"}{end}'

Not run here: it needs a cluster. Any liveness path that queries a database or another service is this incident’s pattern.

Mitigation

  • Keep restart-triggering checks local to the process; put dependency checks in readiness. [inferred]
  • Microsoft’s follow-ups: the health-check change, cache-layer monitoring and updated runbooks are completed; throttling, more cache capacity and better diagnostic metrics were due October 2026; partitioning the cache layer is due December 2026. [vendor]

Verification

In staging, make the database unreachable and confirm the pods become not-ready but their restart count does not increase: kubectl get pods -o wide before and after. [inferred]

11. Azure networking gateways, 30 September

Official RCA: Post Incident Review 7Q30-010 (opens in a new tab). Azure Incident Retrospective livestreams on 22 and 23 October (session 1 (opens in a new tab), session 2 (opens in a new tab)); no recording had appeared in the retrospectives playlist (opens in a new tab) by 30 October.

What happened

From 20:30 UTC on 30 September to 02:15 UTC on 1 October, about 5 hours 45 minutes, customers in 33 regions saw “intermittent failures when performing service management operations” on ExpressRoute, Application Gateway, Firewall, Bastion, Virtual WAN and VPN Gateway. “A subset of upgraded Azure ExpressRoute gateways” lost connectivity, and Azure VMware Solution customers “may also have” lost it. [vendor]

How it unfolded

  1. “A previously deployed performance change” had made configuration operations on “a shared regional networking control plane service” slower. The PIR does not say what the change was.
  2. “A temporary deployment restriction had paused operating system updates for a group of networking gateways. When this restriction expired, a large number of gateway updates resumed within a short period”: a surge of restarts and configuration requests.
  3. From 20:30 the control plane was overwhelmed. “Existing request-throttling and load-protection mechanisms did not sufficiently protect critical processing”, and adding capacity ran into “platform issues that prevented the control plane service from scaling out”.
  4. Restarting ExpressRoute instances “were marked healthy before their configuration and routing state had been fully restored”, so the update moved on to the redundant instance, and “neither instance could retrieve the required routing configuration.”
  5. Monitoring alerted at 20:43 and 20:58, and teams engaged at 21:12. Updates were paused at 21:45; shifting traffic gave only “temporary relief”.
  6. At 22:36, “after approximately 30 minutes of unsuccessful configuration-retrieval attempts”, the ExpressRoute gateways applied their “locally stored last known good (LKG) configurations” and their connectivity returned.
  7. The performance change was identified at 23:55. Its rollback began in one region at 01:15, was validated, and finished at 02:15, which ended the incident.

[vendor]

Impact

Exposed: deployment systems where a pause holds back work that is all released when the pause ends, and instances that report healthy before they can serve. [inferred]

Probe

Three mechanisms meet here, each shown in another section. The burst is the unbounded batch of incident 1; its -breaker run is the equivalent of a staged release. The “healthy before configured” flag is the health-check gap of incident 5. The ExpressRoute recovery, last known good configuration, is the --stale-ok run of incident 4; the incident as a whole ended with the rollback.

Mitigation

  • Resume paused deployments in stages with a rate limit, never all at once when the pause expires. [inferred]
  • Report an instance healthy only after its configuration and routes are restored. [inferred]
  • Microsoft’s follow-ups: the regressing change is rolled back; staged resumption of paused updates was due October 2026; stronger ExpressRoute health checks and control-plane scale-out fixes are due March 2027; throttling and load shedding, June 2027. [vendor]

Verification

In the deployment system, check that resuming a pause enqueues work at the configured rate, not as one batch. For each gateway type, check that the readiness signal includes a configuration-loaded condition. [inferred]

12. Google Cloud europe-west2-b, 20 October

Official RCA: Google Cloud incident Xs2XTHzAVC9YnafnPyzj (opens in a new tab): status updates on 20 October, a Preliminary Incident Report (22 October, giving 6 h 10 min) and the final Incident Report (27 October, 6 h 8 min). No blog post.

What happened

Zone europe-west2-b lost power at 06:00 UTC on 20 October. Power returned at 06:54 and the data halls were re-energised at 07:20, but service impact lasted until 12:08 UTC, 6 hours 8 minutes, mostly spent recovering storage. Some services and customers in zones europe-west2-a and -c were degraded too: “not a full regional control-plane outage, but certainly a degraded state”. [vendor]

How it unfolded

  1. The facility was running on one feed: feeder B “was physically opened and key-interlocked” for repairs after “issues identified during a quarterly generator load test on 14 Oct 2026”. The report does not say what the test found.
  2. At 05:53 “the primary utility provider experienced an outage on primary utility feed A, caused by an external underground cable fault.”
  3. The generator controllers, a redundant pair, had their “undervoltage sensing inputs… wired upstream (line side) of the isolated utility feed B breaker”. Feed B was still live on that side of the open breaker, so “both PLCs sensed live upstream voltage”, reported “a healthy utility source”, and “did not issue a start command to the generator plant.” The UPS batteries carried the load.
  4. A remote load transfer through the controller was tried, but “the system inhibited the action”; the report does not say why. Staff turned to the standby utility instead.
  5. At 06:00 the batteries, rated for “5 minutes of autonomous runtime”, ran out about seven minutes after feed A was lost, and the zone went dark. Monitoring detected it at 06:04, and failover began at 06:09.
  6. In zones a and c, “cross-zone remote procedure calls from regional service control planes” timed out, and the failover surge “saturated available compute slots”. Some failovers failed because of “capacity constraints and difficulty in replicating the underlying storage mechanism”. Affected products included Cloud SQL, AlloyDB, BigQuery, Dataflow, Pub/Sub, GKE, Regional Persistent Disk and Vertex AI Prediction.
  7. Power came back from the standby utility at 06:54. Storage devices with “unclean shutdowns began recovery” at 07:50 and were ready at 10:13; traffic was readmitted from 10:45, new VM creation resumed at 11:46, and 99.5% of VMs were healthy at 12:08 UTC.

[vendor]

Impact

Exposed: workloads in europe-west2-b, and workloads in -a and -c whose regional services made control-plane calls into -b or needed failover capacity at the same moment. Zone spread alone did not contain this outage. [inferred]

Probe

The power sequence cannot be reproduced locally. The cross-zone part is the shared-dependency mechanism of incident 4: a component in one zone that the other zones call on their request path. The generator controller is a health check reading the wrong signal, the gap reproduced in incident 5.

Mitigation

  • Know which of a workload’s regional dependencies keep working when one zone is gone, and test it. [inferred]
  • Google’s follow-ups: the controller is reconfigured and validated (completed 26 October); a code review of the controller was due 31 October; checks of similar controllers at “all global sites with similar topologies” are due 15 November; a freeze on planned maintenance at similar sites was due 30 October. [vendor]

Verification

In a zone-loss exercise, take one zone out of a staging deployment and confirm that creating, updating and deleting regional resources in the remaining zones still works within normal API latency. [inferred]

What the twelve have in common

In eight incidents, a safeguard existed and did not hold: Azure’s failover could not finish (2), BGP and BFD watched the wrong addresses (5), a backup UPS failed with the spare chiller already out of service (6), automated rerouting failed (8), the live-light rule was not followed (9), a health check depended on the database (10), half-configured gateways were marked healthy (11) and the generator controller sensed the wrong side of a breaker (12). In five, a change the provider made started it (1, 2, 5, 10, 11), and two more began in planned maintenance (8, 9). Twelve incidents cannot measure frequency, but the shape repeats: the trigger is ordinary, and the outage is the safeguard’s failure. [inferred]

Sources

Topics: sre, infrastructure

Back to all articles

Technologies we work with

  • Argo CD
  • Kubernetes
  • Docker
  • Podman
  • Ansible
  • Jenkins
  • Chef
  • Proxmox VE
  • Google Cloud
  • Linux
  • Red Hat
  • CentOS
  • NGINX
  • Apache
  • Traefik
  • Prometheus
  • Grafana
  • PostgreSQL
  • MySQL
  • MariaDB
  • Redis
  • etcd
  • Qdrant
  • Elasticsearch
  • Logstash
  • Kibana
  • PagerDuty
  • Go
  • Python
  • FastAPI
  • PHP
  • TypeScript
  • React