LinkedIn Post Ideas for SREs

The health check we built to catch a failing service instead took it down. Let me explain the irony properly. We added an aggressive liveness probe — hit the service every 2 seconds, restart on 3 failures. Sounds reasonable. Under load, the probe itself became enough extra traffic to push response times past the failure threshold, which triggered restarts, which caused a thundering herd of reconnects, which slowed things down further. The system that existed to prevent an outage manufactured one, in about 90 seconds, entirely on its own. We now load-test our health checks under the same conditions as the service itself. If your reliability tooling has never been through a game day, it's still a hypothesis.

Five nines is 5.26 minutes of downtime a year. Three and a half nines is about 4.4 hours. I'd bet your users can't reliably tell the difference. We spent two quarters chasing an extra nine before I pulled the actual support ticket data. Zero tickets referenced brief availability blips. Plenty referenced slow page loads and confusing error messages during the incidents we did have. Perceived reliability is about how an outage feels, not how many seconds it lasted. A clear status page and a fast, honest incident update did more for user trust than the infrastructure spend that chased the extra nine. Pick your SLO from what users actually notice, not what looks good in a QBR slide.

We claim blameless postmortems. Most teams do. The mechanics of actually staying blameless under pressure are what nobody writes down. Rule one: build the timeline before anyone assigns cause. Facts only — 'deploy went out at 14:02,' not 'the engineer deployed without checking.' Language shapes blame faster than intent does. Rule two: the facilitator is never the person closest to the incident. Too much personal stake, even with good intentions. Rule three: every action item gets an owner and a date in the room, not 'someone should look into this.' Vague action items are where blameless culture quietly dies — they signal the process didn't matter enough to finish. We review our own postmortem quality every quarter now, the same way we review the incidents.

We measured toil for a full quarter, defined strictly: manual, repetitive, automatable work with no lasting value. The number was 43% of the team's engineering time. That's two out of every five hours going to ticket triage nobody automated, manual scaling nobody scripted, and access requests we kept meaning to self-serve. We picked the top three toil sources by hours logged and built automation for exactly those, nothing else. Six months later toil is down to 19%. The SRE book has preached this measurement for years. Almost nobody actually tracks it, because the tracking itself feels like more toil. It took us one spreadsheet and 15 minutes a week. That's the whole barrier most teams never cross.

A product team burned their entire monthly error budget in five days chasing a launch deadline. Then came the actual test of whether our policy meant anything. Per policy, that should freeze new feature releases until the budget resets. The team pushed back hard — VP-level pressure, a customer commitment, the works. We held the freeze. Not easily, and not without an uncomfortable meeting. But holding it was the only thing that made the policy real instead of decorative. Two things happened after: the team got noticeably more careful about risky changes near a launch, and leadership started asking about error budget status before committing to dates, not after. A policy that folds under the first real pressure test was never a policy.

Three alerts I deleted last year that made our system more reliable, not less. First: a disk usage warning at 70% that fired weekly and got dismissed every time, training the team to ignore all disk alerts, including the ones that mattered. Second: a latency alert on an endpoint nobody actually used anymore. It paged people for a service with zero real users. Third: a duplicate of our primary uptime check running on a different tool, adding noise with zero new signal. Every team defaults to adding monitoring. Almost nobody schedules time to remove it. Alert fatigue isn't caused by having too few good alerts — it's caused by never subtracting the bad ones.

We let AI draft an update to one of our runbooks. The on-call review caught two fabricated command flags that didn't exist on the tool in question. Where it helped: turning a rough set of Slack notes from an incident into a clean, structured first draft. That saved real time. Where it failed: it filled gaps in the actual commands with plausible-sounding syntax instead of flagging that it didn't know. Confidently wrong is worse than obviously incomplete in a document someone will run at 3am under pressure. Our rule now: AI drafts, a human who has actually run the procedure verifies every command line by line before it ships. No exceptions, no matter how good the draft looks.

My on-call week, logged honestly. Monday: 1 page, false alarm, 6am, back asleep in 20 minutes. Tuesday-Wednesday: quiet. Thursday: 2 pages, real issue, a slow memory leak I patched with a restart and ticketed for real fix. Friday: the one that mattered. 4am Friday, a page for elevated error rates. Half-asleep, I had a choice: roll back a deploy from 9 hours earlier, or dig deeper first. I rolled back. Turned out to be the right call — a bad config had shipped — but I made that decision on 3 hours of sleep with real ambiguity. That's the part on-call summaries never show: the decision made in a fog, that happened to be correct. Slept 5.5 hours total that week outside normal hours. Logged it. Talked to my manager about it Monday.

Six questions I ask before any service goes to production, because they predict whether it'll page someone at night. 1. Is there a single point of failure we haven't load-tested around? 2. What happens on a retry storm — does the downstream fall over too? 3. What's the timeout on every external call, and is it actually set? 4. Can we roll back a bad deploy in under 5 minutes? 5. Does an on-call engineer have a runbook, or just a Slack channel to panic in? 6. Who besides the author understands this well enough to debug it asleep? Services that fail more than one of these get flagged in review, not blocked — just flagged, so whoever's on-call that week knows what they're inheriting.

SREs: what's the weirdest root cause you've ever chased down? I'll go first. A service failed reliably every February 29th, and only that day, for two release cycles before anyone connected the pattern. Root cause: a certificate rotation script that calculated 'one year from now' using a library that choked on leap days, silently generating a cert with a past expiry. Took us 11 hours and three engineers to find, because nobody thought to ask 'what's special about today.' This industry runs on absurd causality — a load balancer's stale DNS cache, a full disk from a log nobody rotated, a timezone bug that only fires during daylight saving transitions. Drop yours below. I read every one of these threads twice.