Monitoring that stays quiet
I ran a full metrics stack for a single small server for about a year. It was fun to build and I looked at the dashboards maybe four times. The alerts, on the other hand, I saw constantly, and almost none of them needed me.
I replaced it with a script that checks five things: disk space, memory pressure, whether the web server answers, certificate expiry, and whether last night's backup finished. It prints nothing when all is well.
df --output=pcent / | tail -1 | tr -dc 0-9 # warn above 85 openssl s_client -connect localhost:443 </dev/null 2>/dev/null \ | openssl x509 -noout -checkend 1209600 # warn under 14 days
Cron mails me only when there is output. In eight months I have had three messages, and all three were worth reading.