Engineering notes
Security, runtime behavior and operations.
Technical analysis with primary sources, configuration details and verification procedures.
- What does it cost to declare an incident?Google's SRE book puts an incident at 6 hours of engineering time; incident.io's data at 3 to 11 person-hours. The dollar figures come from downtime surveys.
- Microsoft's root cause for EX1545193, checked against sender logsMicrosoft blames a spam-rule update for the Outlook.com rejections of November 2026 and dates them from 6 November. Sender logs start a day earlier and show a rule that refused domains with no DMARC record.
- Kubernetes Secrets Management: Which Approach Fits, With a Working Demo of EachNative Secrets, RBAC, encryption at rest with KMS v2, Sealed Secrets, SOPS with Flux and Argo CD, External Secrets Operator, the Vault Secrets Operator, the Secrets Store CSI driver, the Vault Agent sidecar and direct Vault access. What each changes, a demo with captured output, and the catch we hit.
- Argo CD 3.6 Upgrade Day: Synced Is Not Always SyncedOn Argo CD 3.6.1, a failed server-side dry run turns into a Synced badge, so OutOfSync alerts and automated sync never act. We reproduced it three ways and on a 50-app upgrade.
- Argo CD 3.6 in a Lab: The Release Post's 32 Claims, CheckedWe ran Argo CD 3.5.7 and 3.6.1 side by side, tested all 32 claims in the 3.6 announcement and screenshot every new feature. Most hold; the SSA diff change is missing.
- PagerDuty vs Jira Service Management: an SRE's field guide to every on-call conceptWhat each PagerDuty object becomes in Jira Service Management: events, dedup, escalation, phone paging, Slack, Terraform, rate limits and webhooks, checked against both APIs, both Terraform providers and Atlassian's bug tracker.
- Claude Code's new email guard, and two curl calls that get past itClaude Code 2.1.317 added a guard to stop the model putting your email in User-Agent headers. On 2.1.318 the address still reached our test server through git config and shell quoting.
- SSHole (CVE-2026-123158): a 257-character password opens locked accounts on Alpine's OpenSSHOn musl-based systems without PAM, sshd accepts any 257 to 1,024 byte password for an account whose hash is '*', root on stock Alpine included. Reproduced from source, with the second-attempt detail, detection commands and tested mitigations.
- Four AI judges re-scored our AI on-call lab: they agree on what happened, not on whether the reports were trueThe same 260 blinded runs from our 13-model Kubernetes on-call lab, scored by Claude Opus 5.5 twice and by GPT-6.1 Sol and GPT-6 Astra. The judges agree on causes and fixes; they split on whether incident reports claimed things the logs do not show, and the rule's own wording sides with the stricter reading.
- CVE-2026-116472: one Ingress annotation drops all ingress-nginx traffic, and no upstream fix is comingAny user who can create an Ingress can point mirror-target at the controller's own configuration API and empty every backend. Reproduced on ingress-nginx v1.15.1, with the two mitigations tested and their side effects.
