Taming PagerDuty on iOS

I’ve switched recently to PagerDuty for oncall notifications and, being a new app for me, I left it on default settings for some days, to get a general feeling of it.

While on default settings it achieves its main purpose (to make you aware there’s an issue) it’s also horribly disturbing. Because it’s using “Critical Alerts” facility on iOS, it overwrites the normal sound volume and sound alert and it just makes a very loud… alerting noise, that will give you a heart attack. If it’s when you sleep, it will do the same thing for you AND your significant other. While “Critical Alerts” should be like this because they are… well… critical, I think in this case it’s abused by Pager Duty. OK, somewhere a piece of infrastructure has issues, it’s not an earthquake.

So that was the first thing I modified in settings. Exit Critical Alerts, enter Sad Trombone sound. A lot better.

Now, what to do during the nights? OK, it’s a Sad Trombone, but it will still wake you AND your significant other. And only one is getting paid here.

Enter any fitness smart band. I bought a Xiaomi Mi Smart Band 6, which is probably the cheapest decent thing you can buy (37€ on amazon.de), but you can also try any other cheap random band. The important thing is, to have an app that is able to forward app notifications / phone calls to your band so it would vibrate.

The first thing I had in mind was to just forward the app notifications / phone calls to my band, but unfortunately on iOS, even if you do that, the phone will still ring. Apps are not allowed to modify the sound volume on iOS devices.

Fine, next thing I tried was to activate this AND silent hours. But if you do this, then instead of getting the call forwarded, the call from PagerDuty is discarded by iphone. You get nothing. Still not OK.

Fine. I set the phone on silent (from the hardware button) and activated the app / phone forwarding. Now it works as expected, on alerts my fitness band vibrates, phone doesn’t make a sound, I am able to wake up and my wife won’t hear a thing. It sounds good, but what happens if there’s a problem with the band? If it crashes, or it’s falling from my hand, or bluetooth decides to whatever and not work: you would wake up in the morning with the entire infrastructure burning for several hours already.

I “fixed” this by adding an additional action on PagerDuty, to call me on my private phone. That is not silenced during the night, for phone calls, when I’m on call. So now, if I get an alert, I’ll get a short vibrate on my hand, that would wake me. If I’m in a deep sleep and that’s not enough, I will get a phone call on my work phone (or 2, or 3, you know you better) that would vibrate like hell on my hand and would wake me up. Phone would still be silent. And if somehow this doesn’t work, then the last action would be to get a phonecall on my private phone, which would wake me (and the rest of the house, but oh well). But this, until then, never happened. I found the band connection to be very stable.

So to make a summary:

  • remove “Critical Alerts” option and put a decent sound and a decent volume

  • pair your work phone with a fitness band

  • activate the app / calls forwarding to the fitness band (I use Mi Fit app, works like a charm)

  • add an extra action in PagerDuty to be called on your private phone (a catch all)

  • set your business phone on silent during the night

Your family will be very grateful for this setup. And your health as well.