It is patch night. An engineer's phone has buzzed forty times since dinner: services restarting, CPU spikes on machines rebooting, a monitor that has cried wolf every fifteen minutes since March. Buzz forty-one is a client's primary Hyper-V host going down for real, and it gets swiped away with the rest. Nobody was careless. That swipe is alert fatigue, and it is learned behaviour: every alert that needed no action taught the engineer a little more firmly that the next one will not either.
Hospitals measured this before IT did, because for them the failure mode is lethal. The Joint Commission's Sentinel Event Alert on medical device alarms estimates that "between 85 and 99 percent of alarm signals do not require clinical intervention", and its database recorded 98 alarm-related events over three and a half years, 80 of them deaths. The most common contributing factor listed is alarm fatigue itself. Clinicians did not stop caring; the noise trained them not to hear.
An MSP's RMM stack runs the same experiment at lower stakes: hundreds of checks per endpoint, multiplied across every client, firing into one queue. This post is a five-step method for reversing it: baseline the noise per client, kill the always-firing checks, deduplicate, suppress maintenance, then measure forever.
85 to 99%
Hospital alarm signals needing no clinical intervention
Joint Commission, Sentinel Event Alert 50
100+ / day
Alerts received by VA primary care clinicians
AHRQ PSNet, alert fatigue primer
>US$100k
Cost of more than half of significant outages
Uptime Institute, 2024 outage analysis
Alert fatigue is trained behaviour, not laziness
The research is consistent on how desensitisation works. The US Agency for Healthcare Research and Quality's primer on alert fatigue notes that fatigue increases with growing exposure: the more alerts a person receives, the more they override, including critical ones. In one of its examples, Veterans Affairs primary care clinicians were receiving more than 100 alerts per day. Past a certain volume, triage stops being a decision and becomes a reflex, and the reflex is dismissal.
Google's SRE book chapter on monitoring translates the same finding into operations language. Its rule is five words: "Every page should be actionable." When pages fire too often, the chapter observes, people begin to skim and second-guess them, and real pages get lost in the noise. Its authors are blunt about email alerts in particular, describing alert email as something that is rarely read or acted on. If your after-hours alerting runs through an inbox, it has already failed; the inbox is where alerts go to be ignored politely.
The reframe that makes the rest of this post work: every un-actionable alert you allow through is a training rep. You are not just tolerating noise, you are actively teaching your team to ignore the alert that matters. Cutting noise is not housekeeping; it is untraining a reflex before it costs you a client. The stakes scale with what your clients run: in Uptime Institute's 2024 outage analysis, more than half of significant outages cost over US$100,000.
Why MSP RMM stacks are noise machines
None of this is the RMM vendor's fault. Datto RMM, NinjaOne, N-able and their peers all ship capable threshold, dampening and suppression controls. The noise comes from how MSPs deploy them, and three habits do most of the damage:
- One monitoring policy, every client.The default policy gets cloned fleet-wide at onboarding, so a threshold tuned for nothing in particular is applied to a medical practice's database server and a landscaper's reception PC alike. Sensible defaults for one environment are noise generators in another.
- Set at onboarding, never revisited. Thresholds are configured on day one and then left. Disks fill, workloads grow, and a warning level that was rare in year one fires daily by year three. Nobody owns re-tuning because tuning was never scheduled.
- Noise is per client, tuning is global. The distinctive MSP problem. Alert volume concentrates: a handful of clients with old hardware or chronically full disks usually produce most of the queue. But because everything lands in one stream, nobody can see which contract the noise belongs to, so it gets treated as weather instead of as a fixable, attributable cost.
The routing side of this problem, getting the alerts that survive tuning to the engineer actually on call, is its own discipline; our hub guide to RMM alert management for MSPs covers it end to end. This post is about the filtering that should happen first, because routing a firehose accurately just soaks one engineer with precision.
The five-step method
Step 1: baseline noise per client before touching anything
Resist the urge to start deleting monitors. For two weeks, just count. Two numbers per client: total alerts per week, and the actionable rate, meaning alerts where a human actually did something divided by total alerts. A spreadsheet fed from your RMM's alert report is enough. Tag each of the top offenders with the check that produced it.
Two things happen every time an MSP runs this exercise. First, the volume concentrates: a small set of clients and an even smaller set of checks produce most of the noise, which means the fix is narrower than the queue makes it feel. Second, the actionable rate lands far lower than anyone guessed, which turns "the alerts are a bit much" from a feeling into a figure you can put in front of the team, or in front of the client whose ageing servers generate a third of your after-hours pages. Baselining first also gives you the before-and-after evidence that the next four steps worked, which matters when the work has to be justified against billable time.
Step 2: kill or fix the always-firing checks
Work the offender list from step 1 top down. For each check there are only three honest options: delete it because nobody has ever acted on it, raise its threshold because it fires on normal operation, or dampen it because it fires on brief spikes that resolve themselves.
The mechanics live in your RMM today. NinjaOne's own tuning guidance (updated August 2026) is a good model of what the platforms can do when configured deliberately. Separate entry and exit thresholds stop flapping: a warning that triggers above 85% utilisation but does not clear until usage holds below 80% cannot ping-pong on a boundary. Time-based dampening stops spike noise: requiring CPU to hold above 95% for three minutes before a critical alert exists means a virus scan no longer pages anyone. Static thresholds suit absolute limits like disk space; baselines suit volatile metrics like CPU. Every major RMM offers equivalents. The gap between a noisy stack and a quiet one is rarely missing features; it is that these settings were never revisited after onboarding.
Step 3: deduplicate, so one incident is one page
A failing service that re-alerts every fifteen minutes generates twenty-four pages by morning, and every one after the first is pure training-to-ignore. One incident should equal one page. Use your monitoring stack's cooldown or re-alert suppression so a firing check does not renotify while the first alert is still open, and collapse correlated symptoms where you can: a host going down should page as the host, not as the host plus every check that lived on it.
The exception worth engineering deliberately: a repeat should be allowed to change severity rather than repeat the notification. The fourth recurrence of a warning inside an hour is information; escalating that alert to critical is useful, while sending the same warning four times is noise. Deduplication is also what makes an escalation policy template workable at all: escalation ladders assume one alert per incident, and a duplicate storm turns a well-designed ladder into four parallel ladders paging the same two people.
Step 4: suppress your own maintenance
Patch night is self-inflicted paging. Your team scheduled the reboots, and the monitoring stack dutifully reports every one of them as an outage. Maintenance noise is the purest training rep there is, because the engineer knows with certainty that the alerts are false, and spends the evening practising the swipe. That is the state our opening engineer was in when the real alert arrived.
The fix is scoped suppression: windows defined per client and per device group, with a start and an end, never a global mute somebody has to remember to lift. Two rules keep windows honest. Heartbeat-style is-it-alive checks stay active through the window, because the failure you must still catch on patch night is the server that never came back. And windows are time-boxed by the change that justified them: a window that outlives its maintenance is just a blind spot with paperwork.
Step 5: measure again, forever
Rerun the step 1 numbers monthly: alerts per client per week, actionable rate, and time to acknowledge. Tuning decays, because fleets change underneath it. A client's noise trend bending upward is an early warning in its own right, and usually means something cheaper to fix than the tickets it will become: a policy cloned to new machines untuned, or hardware ageing into its failure years.
Per-client noise data also stops being purely an ops metric the moment you have it. The client whose environment produces triple the noise of any other contract is consuming your after-hours attention and your engineers' goodwill, and that belongs in the QBR and the renewal conversation, priced or remediated. The burden side of this, what noisy rotations do to the people carrying them, is measurable too; our research on the hidden cost of MSP shift changes covers what uneven load does to a team before anyone resigns over it.
What good looks like
No honest benchmark says what your alert volume should be; it depends on fleet size, client mix and contract shape, and any universal number would be invented. The bar worth holding is the SRE book's: every page actionable, every page requiring a human decision. Applied as a test to each alert that pages someone after hours: if the engineer it wakes cannot do something about it tonight, and it could not have waited for the morning queue, one of those two facts needs to change.
Where Shiftctl fits
Steps 2 and 4 partly live in your RMM, and should: threshold tuning belongs where the checks run. What Shiftctl Enterprise adds is the layer after the RMM decides to fire. Alerts from ConnectWise RMM and Automate, N-able, PRTG, Datto RMM or any generic webhook arrive in one place, get deduplicated with repeats able to escalate severity instead of renotifying, respect per-client maintenance windows, and route to whoever the roster says is on call right now, escalating over SMS, voice call with press-1 acknowledgement, Slack, email and push until a human acknowledges. Steps 1 and 5 come built: per-client noise, reliability and tech-burden reporting is the baseline-and-remeasure loop from this post running continuously, exportable as a client-facing QBR PDF. And anything still open at shift end carries into the next engineer's on-call handover brief with its acknowledgement history attached, so the alert that survived your filters cannot die at a shift boundary.
The honest caveats: alert ingestion, routing and the per-client reporting live on the Enterprise plan, not Team, and Shiftctl has no inbound number clients can dial. If a contract requires a phone line a person answers, run a dialer-based tool alongside Shiftctl for that piece. Setup details are in the docs, and per-seat pricing is on the pricing page.
Make the pager trustworthy again
Route your RMM alerts to the engineer actually on call, and carry anything still open into their handover. 14-day free trial, full access, no credit card.
Frequently asked questions
What is alert fatigue?
Alert fatigue is the learned desensitisation that sets in when most alerts a person receives require no action: they begin to skim, delay and dismiss, including the alerts that matter. It is best documented in healthcare, where the Joint Commission estimates 85 to 99 percent of device alarm signals need no clinical intervention and lists alarm fatigue as the most common contributing factor in alarm-related deaths. The same mechanism applies to an MSP's RMM queue: it is trained by exposure, one un-actionable alert at a time, and it is reversed by cutting the noise, not by asking people to concentrate harder.
How much alert noise is normal for an MSP?
There is no honest published benchmark, and any tool that quotes one is guessing. The measure that matters is your actionable rate: of the alerts that reached a human, how many led to a human doing something. Baseline it per client for two weeks, then judge yourself against your own trend line. Volume falling and actionable rate rising means the tuning is working, whatever the absolute numbers are.
Should I just turn off the noisy checks?
Sometimes, yes. If nobody has ever acted on a check's alerts, deleting it is the correct and under-used option. But always-firing checks usually deserve one of the gentler fixes first: raise the threshold if it fires on normal operation, or add time-based dampening if it fires on short-lived spikes. The one thing not to do is leave a check firing and rely on everyone ignoring it, because that ignoring is a habit, and habits do not check severity levels before they trigger.
Should I tune alerts in the RMM or in the alert routing layer?
Both, at different jobs. Threshold and sensor tuning belongs in the RMM, where the checks run and where vendors like NinjaOne and Datto provide the dampening and overlapping-threshold controls. Deduplication across sources, per-client maintenance windows, severity mapping and routing to the current on-call engineer belong in the layer after the RMM fires, because that layer sees alerts from every tool in one stream. Tuning only the RMM leaves duplicates and routing unsolved; tuning only the routing layer means filtering a firehose the RMM could have quieted at the source.
How long does it take to reduce RMM alert noise?
The baseline takes two weeks of passive counting. The first tuning pass over the top-ten offender list is typically a few focused hours, and because alert volume concentrates heavily in a few checks and clients, that first pass usually delivers most of the reduction. The real answer, though, is that it never finishes: fleets change, so a monthly remeasure and a small correction is the steady state. An hour a month is the price of a pager your engineers still believe.
Does reducing alert noise help with engineer burnout?
It is one of the highest-leverage changes available, because noise multiplies across every person carrying the pager: a hundred junk alerts a week is a hundred interruptions for whoever is on call, at home, at dinner, asleep. Healthcare research found fatigue increases with exposure, so the load compounds. Noise reduction shrinks the interruptions; measuring per-client burden and keeping rotations fair distributes what remains. Our guide to the hidden cost of MSP shift changes covers the measurement side.