On-Call Difficulty Ratings: A Simple Metric That Transforms Shift Management

Every SRE team tracks MTTR. Nobody tracks how the shift felt.

Your dashboards show alert volume, mean time to resolve, error budgets, and SLO burn rates. All system metrics. All important. All incomplete. They tell you everything about the infrastructure and nothing about the person who spent eight hours keeping it alive.

Missing from every on-call dashboard is a signal from the human who actually worked the shift. How hard was it? Was it manageable or overwhelming? Is this person approaching burnout — or have they already crossed that line? This article introduces a metric that fills that gap: the On-Call Difficulty Rating.

The metrics everyone tracks

Before critiquing the status quo, it is worth acknowledging what teams already measure. The standard on-call metrics toolkit is well-established:

• MTTR (Mean Time to Resolve) — how quickly incidents get fixed once detected.
• MTTD (Mean Time to Detect) — how long before an issue is noticed, whether by monitoring or a human.
• Alert volume per shift — the raw count of pages, grouped by severity.
• After-hours interruption count — how many times an engineer is paged outside business hours.
• SLO burn rate and error budget — how fast you are consuming your reliability budget.
• Toil percentage— Google's SRE book recommends keeping toil below 50% of an engineer's time.

These are all valuable. They measure system health, operational efficiency, and service reliability. They are also, without exception, system metrics. They measure what happened to the infrastructure.

They do not measure what happened to the engineer.

The blind spot — no one measures the human experience

System metrics tell you the database was down for 47 minutes. They do not tell you:

• The engineer was dealing with three concurrent issues, not just the database.
• It was 3 AM and they had already been paged twice that night.
• The runbook was outdated and they had to improvise a fix from memory.
• They are now running on four hours of sleep with a full day of meetings ahead.

End-of-shift report templates from tools like Shiftbase, Smartsheet, and SafetyCulture capture what happened during a shift. None of them capture how it felt. They log incidents, tasks, and handover notes. They do not ask the engineer whether the shift was manageable or whether they are running on fumes.

Google's toil measurement framework uses quarterly surveys to gauge operational burden. Quarterly is too slow. Burnout does not build quarter by quarter — it builds shift by shift. By the time a quarterly survey detects a problem, the engineer is already updating their LinkedIn profile.

Introducing the On-Call Difficulty Rating (ODR)

The On-Call Difficulty Rating is a simple 1–5 scale captured at the end of every shift. One number. Five seconds to submit. A longitudinal signal that no other metric provides.

It takes five seconds to submit. It creates a dataset that, over weeks and months, reveals patterns invisible to every other metric in your SRE toolkit.

What the data enables

A single number per shift seems modest. But aggregated across engineers and weeks, it unlocks five capabilities that no combination of system metrics can provide.

Burnout early warning

If one engineer's average difficulty trends upward over four to six weeks, that is a leading indicator of burnout — before they hand in their notice. Alert volume alone does not catch this. An engineer might have low alert volume but consistently high difficulty because the alerts they receive are complex, the documentation is poor, or they are covering services they were not trained on.

A rolling four-week average of difficulty ratings per engineer is a better burnout predictor than any system metric. When that average crosses 3.5, it is time for a conversation — not a quarterly survey.

Fair burden distribution

Raw hours are a poor measure of on-call burden. Eight hours on a quiet Saturday is not the same as eight hours during a P1 on a Friday night. Multiply shift hours by difficulty rating and you get Difficulty-Weighted Hours (DWH) — a composite metric that captures both duration and intensity.

If one engineer's DWH is twice the team average, the schedule needs rebalancing — even if everyone worked the same number of hours. This is the foundation of fair on-call scheduling.

Staffing decisions

If average difficulty across the team trends from 2.5 to 3.5 over a quarter, that is a staffing signal — not an automation signal. It means your engineers are consistently dealing with harder shifts, and no amount of runbook automation will fix the fact that you need more people in the rotation.

Difficulty ratings give engineering managers concrete data for headcount requests. Instead of "the team feels stretched," you can say "average shift difficulty has increased 40% over the last quarter, and two engineers are consistently above 4.0."

Compensation justification

On-call compensation is one of the most contentious topics in SRE. Some companies pay flat stipends. Others pay per-page. Many pay nothing. Difficulty ratings provide a data-driven argument for on-call stipends or comp time: "Our engineers averaged difficulty 3.8 last quarter. This is not a quiet rotation — it is a demanding second job that disrupts sleep, weekends, and personal time."

That argument is far harder to dismiss when it comes with three months of shift-level data.

New hire readiness

Track a new engineer's difficulty ratings during their shadow and reverse-shadow periods. If they consistently rate shifts 4–5 when peers covering the same shift rate 2–3, that is a training signal — not a performance signal. They need more context on the systems, better runbooks, or a longer shadow period. Not more pressure.

For more on structured on-call onboarding, see our on-call onboarding guide.

How to implement difficulty ratings

Implementation is straightforward. The hard part is not the technology — it is the discipline to collect the data consistently and the culture to act on it without weaponising it.

• Add a required 1–5 difficulty field to your end-of-shift sign-off. Make it mandatory — optional fields get skipped within a week.
• Make it frictionless: one click, not a survey. If it takes more than ten seconds, adoption will collapse. A row of five buttons is ideal.
• Store ratings alongside shift metadata — date, engineer, hours worked, incident count, and any handover notes. Context makes the data useful.
• Review monthly: team average, individual trends, and high-difficulty patterns. Look for shifts, time slots, or services that consistently drive high ratings.
• Act on the data: if average difficulty exceeds 3.5 for a month, something systemic needs to change — staffing, tooling, runbooks, or service ownership.
• Never use individual ratings punitively. This is the single most important rule. The moment engineers believe their ratings will be used against them, they will game the system and the data becomes worthless.

The analogy — lessons from healthcare shift scoring

This concept is not new. Nurses have used shift intensity scoring for decades. Patient acuity scores — numerical ratings of how sick each patient is — drive staffing ratios in every hospital. Higher acuity means more nurses per patient. The math is simple and the logic is obvious: harder patients require more staff.

IT has no equivalent. A two-person on-call team covers the same services whether it is a quiet Tuesday or Black Friday. The staffing model does not flex with demand because there is no signal telling managers that demand has changed. Alert volume is a partial signal at best — it measures quantity, not cognitive load.

Difficulty ratings are the IT equivalent of patient acuity. They turn staffing from gut feel into data. A hospital that staffed every shift identically regardless of patient acuity would be considered negligent. Yet this is exactly how most engineering organisations run their on-call rotations.

Frequently asked questions