What does it cost to declare an incident?
Overview
Google's SRE book puts an incident at 6 hours of engineering time; incident.io's data at 3 to 11 person-hours. The dollar figures come from downtime surveys.
An incident takes about a working day of engineering time on average. Google’s SRE book (opens in a new tab) puts it at 6 hours, counting root-cause analysis, remediation, the postmortem and fixing bugs. On incident.io’s platform (opens in a new tab), with over 100,000 incidents, a minor incident takes a mean of 3.1 to 5.0 person-hours and a critical one 5.0 to 10.8, depending on company size. Much of that is the fault itself, which needs fixing whether or not anyone declares it. What declaring adds is the process around the fix: more people paged, a lead and status updates, the review, the follow-up tickets and, at GitLab.com, a deploy freeze at the two highest severities.
The figures that circulate measure the outage instead. PagerDuty’s 2024 survey (opens in a new tab) arrives at $793,957 per customer-facing incident by multiplying the $4,537 a minute that 500 IT leaders estimated by a 175-minute average resolution.
Checked 11 November 2026.
Declared incidents cost hours, pages and follow-up work
The sources count different things, so the rows sit side by side and are not added up. Our own counts carry their n and a 95% interval.
| Part | Number | Who counted it, and how |
|---|---|---|
| Engineering time | 6 hours on average, including remediation, the postmortem and fixing bugs | Google SRE book (opens in a new tab), chapter 11 (2016), from Google’s experience; no sample size given |
| Mean 3.1–5.0 person-hours for a minor incident, 4.6–7.5 for a major one, 5.0–10.8 for a critical one, by company size | incident.io (opens in a new tab), over 100,000 incidents on its platform, report of February 2025 | |
| Median 2.6 responder-hours (95% CI 2.1–6.7, n = 94); a rough product that misses work after the impact ends and counts responders who left early; 3 reports give zero | Our count of Wikimedia’s incident reports (opens in a new tab), 2022–2026: responders × the window of user impact | |
| Time open | Median 1.1 hours for a critical incident, 2.3–4.0 hours for the other levels; the means run 3 to 9 days | Rootly (opens in a new tab), 314,862 incidents resolved on its platform from October 2025 to September 2026, declaration to resolution |
| User impact ended within an hour in 53 of 107 incidents (40–59%); median 1.05 hours (0.61–2.23), but 0.45 hours in 2022 (n = 49) against 2.06 hours in 2023–2026 (n = 58) | Our count of Wikimedia’s reports, from the start and end of user impact | |
| People pulled in | 93% of organisations bring in at least three engineers for a major incident | NeuBird survey (opens in a new tab), 1,000+ practitioners, 2026 |
| The fastest incidents had six responders; incidents with one more had an 18% higher mean time to resolve, a correlation | FireHydrant (opens in a new tab), 50,000 incidents on its platform, 2019–2022 | |
| Median 1 person paged (0–2, n = 109), or 2 if six entries reading “batphone” or “sre” count as five or more; 46 of the 109 paged nobody (33–52%); where five or more were paged (n = 30), a median of 19 were paged and 5 responded, and 7 of those 30 paged exactly 26 | Our count of Wikimedia’s reports | |
| Paging load | Pages per declared incident on the median team: 8.2 below $100M revenue, 24.5 up to $1B, 60.5 above, while the best quarter of teams at every size stays under 4; a page is an alert routed to on-call, counted once however many people it reaches | Rootly platform data |
| Nights and weekends | The median customer gets 23.2% of its pages in sleeping hours and does 10–16% of its incident work outside working hours | incident.io platform data |
| The median responder had 2 off-hours interruptions a month in 2021; the most interrupted tenth had 19 a month over January 2019 to April 2021 | PagerDuty (opens in a new tab) platform data, 2022 report and 2021 release (opens in a new tab) | |
| Of the 44 reports that paged anyone and answer the business-hours question, 17 paged outside business hours (39%, 26–53%) | Our count of Wikimedia’s scorecards | |
| The review | The median customer sends 29%, 64% and 82% of major and critical incidents into a postmortem at small, mid-sized and large companies; finishing and sharing one takes about 4 days of elapsed time on average | incident.io platform data and its 2024 analysis (opens in a new tab) of about 13,000 incidents |
| Retros after about 29% of low-severity and 42% of high-severity incidents | FireHydrant (opens in a new tab), quoting its 50,000-incident benchmark | |
| About 35% of recent high-severity incidents had no complete postmortem report | Microsoft Teams study (opens in a new tab), 152 incidents, May 2021 to May 2022 (SoCC 2022) | |
| Follow-up work | 391 of 1,085 public incidents (36%, 33–39%) got at least one corrective action, from 32% in late 2025 to 41% in May to August 2026; half the actions were closed within 31 days, counting the 21% still open | Our count, GitLab.com: incidents from 15 September 2025 to 31 August 2026, and 844 corrective actions opened to 1 November 2026 |
| The median customer completes a follow-up in 10.2 to 12.3 days; across 14,000 follow-ups in 2024 the median was 7 days | incident.io report (2025) and blog (2024) | |
| Routing | 5% to 68% of incidents first went to the wrong team, 27% on average; the misrouted ones took an average of 11 discussion items to reach the right team | Microsoft (opens in a new tab), eight online services (ASE 2019) |
| Incidents closed without a fix | False alarms, transient faults, intended behaviour, customer errors, won’t-fix and unreproducible incidents: an average of “up to 50.32%” of incidents (12–71% by service), taking an average of “up to 55.05%” of the time incidents stayed open (11–77%) | Microsoft (opens in a new tab), 18 online services, mostly monitor-raised tickets (ASE 2020) |
| Duplicate declarations | 97 of 1,395 public incidents (7.0%, 5.7–8.4%) were merged into another as duplicates: 3.9% from September 2025 to January 2026, 7.8% from February 2026 (Fisher’s exact test, p = 0.016) | Our count, GitLab.com’s public incident tracker |
| Deploy freeze | S1 and S2 incidents freeze production deploys by rule; GitLab’s weekly reports charge them 155 of the 1,230.5 hours charged to issues in GitLab’s production tracker in 59 weeks, about 2.6 hours a week, and 46 of the 83 S1 and S2 incidents had any | Our count from GitLab’s weekly deployment-blocker reports (opens in a new tab); 216.5 of the hours sit on incidents missing from the public tracker |
Google’s 6 hours and incident.io’s person-hours are means, and long incidents pull them up. Rootly’s mean time to resolve a critical incident is 9.1 days against a median of 1.1 hours, and incident.io’s report says major and critical incidents are “often spanning multiple workdays worth of workload”. Google uses its 6 hours to cap on-call at two incidents per 12-hour shift.
Wikimedia incident reports dated 2022 to 2026 (n = 165): duration of user impact, people paged, responders and the scorecard's business-hours answer
Impact is the scorecard’s start to end, read as UTC when written as a date and time; other forms show as –. Paged and responders show the number we parsed (a leading number; “6 + 3” as 9), or the word written when there is none; free text is shown as [text]. The business-hours answer is the scorecard’s answer to “Were pages routed to online (business hours) engineers?”; the after-hours count uses only reports with at least one person paged. 78 of the 165 reports date from 2022 and 3 from 2026.
GitLab.com public incidents, duplicates and corrective actions by month, 15 September 2025 to 1 November 2026 (1,395 incidents, 844 corrective actions)
| Month | S1 or S2 | S3 | S4 | Merged as duplicates | With a linked corrective action | Corrective actions opened |
|---|---|---|---|---|---|---|
| 2025-09 | 4 | 19 | 16 | 3 | 16 of 39 | 25 |
| 2025-10 | 5 | 36 | 31 | 2 | 17 of 72 | 42 |
| 2025-11 | 3 | 23 | 31 | 2 | 22 of 57 | 26 |
| 2025-12 | 4 | 28 | 28 | 3 | 19 of 60 | 25 |
| 2026-01 | 3 | 32 | 34 | 2 | 27 of 69 | 41 |
| 2026-02 | 4 | 48 | 78 | 15 | 36 of 130 | 56 |
| 2026-03 | 12 | 53 | 55 | 11 | 46 of 120 | 80 |
| 2026-04 | 7 | 50 | 67 | 5 | 38 of 124 | 76 |
| 2026-05 | 10 | 50 | 77 | 10 | 53 of 137 | 82 |
| 2026-06 | 7 | 41 | 34 | 2 | 44 of 82 | 81 |
| 2026-07 | 3 | 34 | 59 | 9 | 33 of 96 | 52 |
| 2026-08 | 9 | 54 | 36 | 17 | 40 of 99 | 91 |
| 2026-09 | 5 | 53 | 44 | 7 | – | 69 |
| 2026-10 | 7 | 59 | 44 | 9 | – | 97 |
| 2026-11 | 0 | 1 | 0 | 0 | – | 1 |
| Total | 83 | 581 | 634 | 97 | 391 of 1085 | 844 |
September 2025 starts on the 15th and November 2026 holds only the 1st. Severity is the label each incident ended with. Corrective actions are issues labelled corrective action in GitLab’s production-engineering project; an incident counts as linked when a corrective action’s description carries its incident.io ID, which public incidents carry until August 2026. Every issue in the tracker was created by GitLab’s incident.io integration account.
Production deploy hours charged to issues in GitLab's weekly Deployment Blockers reports, weeks of 15 September 2025 to 26 October 2026, by final severity, and the 46 S1 and S2 incidents
| Group | Issues with hours | Hours |
|---|---|---|
| S1 or S2 | 46 | 155 |
| S3 | 144 | 716.5 |
| S4 | 36 | 93 |
| Merged as duplicates | 3 | 49.5 |
| Not in the public tracker | 47 | 216.5 |
| Total | 276 | 1230.5 |
| Incident | Opened | Hours | Week’s report | Title |
|---|---|---|---|---|
| #21778 (opens in a new tab) | 2026-04-14 | 9 | 2026-04-13 (opens in a new tab) | Customers getting “pull access denied” from registry.gitlab.com/gitlab-org/ images |
| #22777 (opens in a new tab) | 2026-08-24 | 8 | 2026-08-24 (opens in a new tab) | KAS deployments failing |
| #22785 (opens in a new tab) | 2026-08-25 | 8 | 2026-08-24 (opens in a new tab) | 500s in ops.gitlab.net |
| #22016 (opens in a new tab) | 2026-05-07 | 6 | 2026-05-04 (opens in a new tab) | Jobs duration is incorrect |
| #22714 (opens in a new tab) | 2026-08-17 | 6 | 2026-08-17 (opens in a new tab) | Project/Group deletions are failing on .com |
| #22750 (opens in a new tab) | 2026-08-20 | 6 | 2026-08-17 (opens in a new tab) | Gitaly concurrency saturation |
| #21190 (opens in a new tab) | 2026-02-03 | 5 | 2026-02-02 (opens in a new tab) | Sidekiq queueing SLI below SLO on low-urgency-cpu-bound shard (apdex 79.43%) |
| #21192 (opens in a new tab) | 2026-02-03 | 5 | 2026-02-02 (opens in a new tab) | Redis deployment error causes delayed CI processing and GitLab Duo outage |
| #21498 (opens in a new tab) | 2026-03-10 | 5 | 2026-03-09 (opens in a new tab) | Sidekiq queueing apdex SLO violation on low-urgency-cpu-bound shard |
| #21573 (opens in a new tab) | 2026-03-17 | 5 | 2026-03-16 (opens in a new tab) | 500 error when creating MR and API request |
| #22031 (opens in a new tab) | 2026-05-08 | 5 | 2026-05-04 (opens in a new tab) | Upgrading DUO_CLI_VERSION to “8.93.0” causes all flows to fail on installation of duo CL |
| #22163 (opens in a new tab) | 2026-05-19 | 5 | 2026-05-18 (opens in a new tab) | Redis primary CPU saturation on redis-sidekiq nodes approaching capacity limit |
| #23240 (opens in a new tab) | 2026-10-15 | 5 | 2026-10-12 (opens in a new tab) | Blackbox probe availability for customers.gitlab.com/-/liveness/database,migrations,cache |
| #20607 (opens in a new tab) | 2025-09-24 | 4 | 2025-09-22 (opens in a new tab) | Image pull failures on SaaS Linux runners exceeding SLO |
| #22331 (opens in a new tab) | 2026-06-17 | 4 | 2026-06-15 (opens in a new tab) | Broken master |
| #22729 (opens in a new tab) | 2026-08-18 | 4 | 2026-08-17 (opens in a new tab) | Loadbalancer error rate violating SLO |
| #23280 (opens in a new tab) | 2026-10-19 | 4 | 2026-10-19 (opens in a new tab) | ai-assisted rails_request error rate at 10.99% |
| #20914 (opens in a new tab) | 2025-12-01 | 3 | 2025-12-01 (opens in a new tab) | Merged tab on the MR homepage times out |
| #21484 (opens in a new tab) | 2026-03-09 | 3 | 2026-03-09 (opens in a new tab) | Sidekiq queueing SLO violation on catchall-b shard with low apdex |
| #22008 (opens in a new tab) | 2026-05-06 | 3 | 2026-05-04 (opens in a new tab) | Loadbalancer error rate for api service in cny stage exceeds SLO (3.7%) |
| #22057 (opens in a new tab) | 2026-05-12 | 3 | 2026-05-11 (opens in a new tab) | Sidekiq queueing apdex SLO violation on shard catchall-b (apdex at 70.19%) |
| #22103 (opens in a new tab) | 2026-05-12 | 3 | 2026-05-11 (opens in a new tab) | Error rates violating SLO |
| #22256 (opens in a new tab) | 2026-06-05 | 3 | 2026-06-01 (opens in a new tab) | The loadbalancer SLI of the ci-runners service (main stage) has an error rate violating |
| #22412 (opens in a new tab) | 2026-06-30 | 3 | 2026-06-29 (opens in a new tab) | CI/CD pipelines not being triggered on merge requests. |
| #22418 (opens in a new tab) | 2026-06-30 | 3 | 2026-06-29 (opens in a new tab) | Gitaly goserver apdex SLO violation in cny |
| #22424 (opens in a new tab) | 2026-06-30 | 3 | 2026-06-29 (opens in a new tab) | server_code_completions error rate 31.87% in europe-west2 |
| #22661 (opens in a new tab) | 2026-08-06 | 3 | 2026-08-03 (opens in a new tab) | sidekiq_queueing apdex 70.08% on low-urgency-cpu-bound shard |
| #23061 (opens in a new tab) | 2026-09-30 | 3 | 2026-09-28 (opens in a new tab) | GitLab Email delivery to Yahoo and related providers delayed/disrupted |
| #23131 (opens in a new tab) | 2026-10-08 | 3 | 2026-10-05 (opens in a new tab) | CI job finalization & webhook delays — Sidekiq concurrency-limiter deferral latch |
| #23158 (opens in a new tab) | 2026-10-12 | 3 | 2026-10-12 (opens in a new tab) | tool_use error rate at 30.25% in duo-workflow-svc |
| #21046 (opens in a new tab) | 2026-01-06 | 2 | 2026-01-05 (opens in a new tab) | Sidekiq queueing SLO violation on urgent-cpu-bound shard (apdex 73.88%) |
| #21462 (opens in a new tab) | 2026-03-05 | 2 | 2026-03-02 (opens in a new tab) | CI/CD Pipelines stuck in running status |
| #21487 (opens in a new tab) | 2026-03-09 | 2 | 2026-03-09 (opens in a new tab) | Gitaly goserver apdex SLO violation in cny stage |
| #21936 (opens in a new tab) | 2026-04-30 | 2 | 2026-04-27 (opens in a new tab) | The ci_runner_jobs SLI of the ci-runners service on shard saas-linux-small-amd64 has a |
| #22241 (opens in a new tab) | 2026-06-03 | 2 | 2026-06-01 (opens in a new tab) | Duo Agent Platform workflows fail to start — Websocket connection closes abnormally (1006) |
| #22364 (opens in a new tab) | 2026-06-23 | 2 | 2026-06-22 (opens in a new tab) | git pushes are failing via ssh |
| #22897 (opens in a new tab) | 2026-09-09 | 2 | 2026-09-07 (opens in a new tab) | Redis memory utilization near max on redis-cluster-shared-state |
| #22904 (opens in a new tab) | 2026-09-10 | 2 | 2026-09-07 (opens in a new tab) | Sidekiq jobs slow to process and being deferred |
| #20680 (opens in a new tab) | 2025-10-07 | 1 | 2025-10-06 (opens in a new tab) | The gitlab_sshd git service unavailable (apdex violating SLO) |
| #20686 (opens in a new tab) | 2025-10-07 | 1 | 2025-10-06 (opens in a new tab) | Gitlab unavailable due to load |
| #21448 (opens in a new tab) | 2026-03-04 | 1 | 2026-03-02 (opens in a new tab) | Duo agentic chat losing messages with some models |
| #22754 (opens in a new tab) | 2026-08-20 | 1 | 2026-08-17 (opens in a new tab) | Gitaly IO saturation |
| #21528 (opens in a new tab) | 2026-03-12 | 0.5 | 2026-03-09 (opens in a new tab) | Sidekiq jobs queueing duration SLO violation on all urgent shards |
| #21864 (opens in a new tab) | 2026-04-23 | 0.5 | 2026-04-20 (opens in a new tab) | PGBouncer client connections per process nearing max_client_conn limit may cause refused |
| #21976 (opens in a new tab) | 2026-05-04 | 0.5 | 2026-05-04 (opens in a new tab) | GITLAB_DEPENCY_TOKEN fails to authenticate across multiple projects |
| #22875 (opens in a new tab) | 2026-09-07 | 0.5 | 2026-09-07 (opens in a new tab) | 68.9% of chat workflows are stopped |
Hours are the gprd column of each week’s report; a row repeated within one report counts once, and where a week has two report issues with the same table, the later one is used. GitLab’s incident labels today show far fewer hours than these reports (19.5 hours on S1 and S2 incidents), so the reports are the record used here.
The dollar figures are survey estimates of downtime
Each one counts something different:
| Source | Figure | What was counted | Who answered |
|---|---|---|---|
| PagerDuty (opens in a new tab), June 2024 | $4,537 a minute; $793,957 per customer-facing incident | “the true cost of downtime”, multiplied by a 175-minute average resolution | 500 IT leaders at companies of 1,000+ employees in the US, UK and Australia (Censuswide) |
| New Relic (opens in a new tab), September 2025 | Median $2 million an hour | “high-impact outages”, not defined on the public pages | 1,700 IT and engineering staff in 23 countries (ETR), April and May 2025 |
| Splunk with Oxford Economics (opens in a new tab), June 2024 | $400 billion a year across the Global 2000, 9% of profits | Downtime including slowness, calculated by Oxford Economics from the answers | 2,000 executives at Global 2000 companies |
| ITIC (opens in a new tab), September 2024 | Over $300,000 an hour for over 90% of mid-size and large enterprises | Respondents’ estimate of one hour of downtime, without litigation or fines | Over 1,000 firms, November 2023 to March 2024 |
New Relic reported a median of $2 million an hour in 2025 and a mean of $1.85 million (opens in a new tab) in 2026, so the two years cannot be read as a fall. Google’s SRE book (opens in a new tab) makes the case for declaring early: “It is better to declare an incident early and then find a simple fix and close out the incident than to have to spin up the incident management framework hours into a burgeoning problem.”
None of ten incident tools charges per declared incident
We read ten incident-management pricing pages on 11 November 2026. Eight bill per user, seat, responder or agent, from Splunk On-Call (opens in a new tab) at “$5 per user, per month” for up to 10 seats to Better Stack (opens in a new tab) at $34 a month per responder licence. AWS Incident Manager (opens in a new tab) bills “$7 per response plan per month” and no longer takes new customers, and Statuspage (opens in a new tab) sells flat tiers by audience size.
incident.io’s (opens in a new tab) free Viewer seat can “Declare incidents”, Datadog’s (opens in a new tab) billing page says “a user does not need a seat to create incidents”, and FireHydrant (opens in a new tab) lets unlicensed users declare if an administrator enables it. Grafana Cloud IRM (opens in a new tab) counts anyone who “Creates, edits, or updates an incident” as a billable user for that month. The only cap on incident count we found was on Opsgenie (opens in a new tab) Essentials, “100 incidents per month (account-wide)”, a plan Atlassian stopped selling on 4 June 2025. incident.io’s buyer’s guide (opens in a new tab) warns that under per-incident pricing “teams may avoid declaring incidents to save budget”, without naming a vendor that charges that way.
Billing units on ten incident-management pricing pages, read 11 November 2026
| Product | Billing unit, in the vendor’s words | Incidents |
|---|---|---|
| PagerDuty (opens in a new tab) | Professional “$25 / $21 … per user / month” (monthly / annual) | No limit stated |
| incident.io (opens in a new tab) | Team “$15 Per user / month” (annual); “Our pricing is based on the number of Responder and On-call seats you use” | Viewers declare free |
| Rootly (opens in a new tab) | “Incident Response … $20 per user / month” | No limit stated |
| FireHydrant (opens in a new tab) | Pro “$25 /responder/month, billed annually” | Unlicensed users can declare, if enabled |
| Jira Service Management (opens in a new tab) | Standard “$21.50 per agent / month” | No limit stated |
| Grafana Cloud IRM (opens in a new tab) | “Starts at $20/ active IRM user” | Creating an incident makes a user active for the month |
| Splunk On-Call (opens in a new tab) | “Starting at just $5 per user, per month, billed annually for up to 10 seats” | No limit stated |
| Better Stack (opens in a new tab) | “Responder (Incidents + Telemetry)”: $34 a month, $29 billed yearly, per licence | No limit stated |
| AWS Incident Manager (opens in a new tab) | “$7 per response plan per month”; closed to new customers | No incident charge |
| Statuspage (opens in a new tab) | Flat monthly plans by subscribers, employees or users | Incidents in every plan |
EU deadlines start when a firm becomes aware of a serious incident
Under DORA, a financial entity sends its initial notification “within four hours from the classification of the ICT-related incident as a major ICT-related incident and no later than 24 hours from the moment the financial entity has become aware” of it, an intermediate report within 72 hours of that and a final report within a month (Delegated Regulation (EU) 2025/301 (opens in a new tab), Article 5). The final report has to state the “direct and indirect costs and losses”. NIS2 asks for an early warning “within 24 hours of becoming aware of the significant incident”, a notification within 72 hours and a final report within a month (Directive (EU) 2022/2555 (opens in a new tab), Article 23). For breaking those rules, member states must be able to fine “essential entities” up to at least €10 million or 2% of worldwide turnover, whichever is higher (Article 34). A cloud provider’s incident becomes reportable when the service is “completely unavailable for more than 30 minutes” (Implementing Regulation (EU) 2024/2690 (opens in a new tab), Article 7).
Financial entities reported 3,383 major ICT incidents under DORA in 2025, 0.18 per entity, and the European supervisory authorities (opens in a new tab) found that “two thirds of major incidents resulted in no or minor disruption to clients and transactions”.
What to count in your own incidents
Most of these numbers come from someone else’s platform, and a team’s own incident tool holds the same records. Five counts cover the parts that declaring adds: pages routed to on-call per declared incident, the share of declarations merged as duplicates, the share first assigned to the wrong team, person-hours per incident by severity, and the time to close corrective actions with the open ones counted. Rootly publishes the first for its median team, GitLab’s public tracker yields the second, Microsoft’s papers report the third and incident.io the fourth.
Sources
- Google, Site Reliability Engineering, chapter 11, “Being On-Call” (Andrea Spadaccini), O’Reilly 2016, online edition at sre.google (opens in a new tab), read 11 November 2026. Six hours per incident; two incidents per 12-hour shift.
- Google, Site Reliability Engineering, chapter 14, “Managing Incidents” (Andrew Stribblehill), sre.google (opens in a new tab), read 11 November 2026. The advice to declare early.
- incident.io, “What does good incident management look like?”, web report (opens in a new tab) and PDF (opens in a new tab) (PDF created 10 February 2025). Over 100,000 incidents: person-hours per incident, pages in sleeping hours, workload outside working hours, postmortem entry rates.
- incident.io, “Where does the time go after you resolve an incident?” (Eryn Carman), blog (opens in a new tab), 29 July 2024. Postmortem completion in about 4 days; follow-ups in a median of 7 days.
- incident.io, “What to look for in incident response software pricing”, blog (opens in a new tab), 6 July 2026. The warning on per-incident pricing.
- Rootly, “How to achieve great incident management”, web report (opens in a new tab), 8 October 2026. 314,862 resolved incidents, October 2025 to September 2026: time from declaration to resolution, pages per declared incident.
- FireHydrant, “Two data-backed ways to resolve incidents faster” (Chris Kelly), blog (opens in a new tab), 18 April 2023, and “Beyond Downtime”, ebook (opens in a new tab), 2023. 50,000 incidents, 2019 to 2022: six responders and 18%; retros after 29% and 42% of incidents.
- PagerDuty, “State of Digital Operations 2022”, report page (opens in a new tab), 6 June 2022, and the 2021 press release (opens in a new tab), 28 July 2021. Off-hours interruptions per responder, from platform data.
- PagerDuty, “The Cost of Incidents” survey, whitepaper (opens in a new tab) (26 June 2024) and press release (opens in a new tab) (27 June 2024). $4,537 a minute, 175 minutes, $793,957.
- NeuBird, “State of Production Reliability”, blog (opens in a new tab), 20 April 2026. 93% bring in at least three engineers.
- New Relic, Observability Forecast press releases of 17 September 2025 (opens in a new tab) and 22 September 2026 (opens in a new tab). Median $2 million an hour (2025), mean $1.85 million (2026).
- Splunk with Oxford Economics, “The Hidden Costs of Downtime”, press release (opens in a new tab), 11 June 2024. $400 billion a year.
- ITIC, “2024 Hourly Cost of Downtime Report”, part 1 (opens in a new tab), 3 September 2024, and part 2 (opens in a new tab), 10 September 2024. Over $300,000 an hour for over 90% of mid-size and large enterprises.
- J. Chen et al., “How Incidental are the Incidents?”, ASE 2020, author copy (opens in a new tab). 18 Microsoft services: incidental incidents and their share of the time to resolve.
- J. Chen et al., “Continuous Incident Triage for Large-Scale Online Service Systems”, ASE 2019, author copy (opens in a new tab). Eight Microsoft services: wrong first assignment, discussion items before correct triage.
- S. Ghosh, M. Shetty, C. Bansal and S. Nath, “How to Fight Production Incidents? An Empirical Study on a Large-scale Cloud Service”, SoCC 2022, PDF (opens in a new tab). 152 high-severity Microsoft Teams incidents, May 2021 to May 2022; about 35% of recent high-severity incidents lacked a complete postmortem report.
- Commission Delegated Regulation (EU) 2025/301, EUR-Lex (opens in a new tab), Articles 4 and 5. DORA reporting deadlines and the cost field.
- Directive (EU) 2022/2555 (NIS2), EUR-Lex (opens in a new tab), Articles 23 and 34. Reporting deadlines and fines.
- Commission Implementing Regulation (EU) 2024/2690, EUR-Lex (opens in a new tab), Article 7. When a cloud incident is reportable.
- ESAs Joint Committee, “2025 Report on major ICT-related incidents” (JC 2026 16), PDF (opens in a new tab), 3 June 2026. 3,383 major incidents in 2025.
- Incident-management pricing pages listed in the table above, read 11 November 2026.
- Wikimedia Foundation, incident documentation (opens in a new tab) on wikitech, 165 reports dated 2022 to 2026, read through the MediaWiki API on 11 November 2026. Our counts of duration, people paged, responders and after-hours pages.
- GitLab, production incident tracker (opens in a new tab), weekly Deployment Blockers reports in gitlab-org/release/tasks (opens in a new tab), corrective actions in gitlab-com/gl-infra/production-engineering, read through the public API on 11 November 2026; incident management handbook (opens in a new tab) (commit b81a25c9, 18 August 2026) and deployments handbook (opens in a new tab) (commit 488c11b2, 22 October 2026). Our counts of incidents, duplicates, corrective actions and deploy-freeze hours.
