I’m firmly in the camp that the person in the SRE team with on-call duty on a given week, should be the person handling the interrupt work. It’s a fight I’ve lost in the past when I couldn’t persuade other managers to see my point of view, but it’s the only thing I’ve seen that leads to a happy and efficient team. First of all, let me explain some key terms.
SRE
Site Reliability Engineers. We keep the lights on and balance the business’s need to push new products and features, while providing stability and uptime to the existing products that people are depending on.
On call
SREs usually operate an on-call rotation. That person is responsible for the week to deal with any alerts and incidents that occur. They can be paged out of hours for serious incidents and are generally expected to log in within a short time frame and work towards a resolution.
If the workplace operates a “follow the sun” model, they are usually only on call during their own working hours and weekends. People in the other geographic areas will pick up alerts while the designated on-call SRE is off shift. It makes for a better work-life balance and stops on-call engineers getting sick or burned out when done correctly.
Interrupts
I class interrupts as ad hoc requests that come in during the workday. They can be questions, minor job failures and people chasing up existing requests.
So why should on call handle interrupts
An on-call engineer who is still trying to make significant progress with their project work is setting themselves up to do a bad job at both. Unless on-call is quiet, you have very few alerts and none of them are potentially noise, then you have work to do. Your on call shift is going to be a combination of proactive and reactive responses to alerts and incidents. You will get some breathing space but in the meantime, requests and tickets will also likely be coming into your team.
These requests are often quick and easy to handle and if it’s your focus as the on-call engineer, you allow your team mates to have uninterrupted blocks of time where they can think, plan and develop solutions. Similarly when you are not on call, you will also reap the benefit of your colleague picking up these small tasks. Furthermore, it saves management effort allocating work if there is a recognised system in place, allowing them to work on being good people managers and good strategists.
Someone who has got into a flow in project work, will take time to reach that state again if they have been interrupted for minor tasks. It’s why I recommend that the on-call engineer handles as much of this as possible during their rotation. They are doing shorter tasks anyway and “shouldn’t” be expected to make much progress on project work.
Are interrupts toil?
They can be. Every interrupt should be looked at on its own merit. If the task can be easily automated then yes. Of course doing some of these tasks manually can be a good learning experience but once they have been done a few times, you aren’t learning anything each time you repeat the task. Paying close attention to the type and frequency of interrupts you get, could actually give you ideas on tasks which could be automated.
Interrupts for junior staff?
To an extent, yes. In the short term, it can lighten the load on others whilst giving newer team members the chance to learn. This should never be a permanent state of affairs as it leads to boredom and dissatisfaction, as well as stifling opportunities for junior people to learn more advanced skills and knowledge.
How to do the mechanics
Set up workflows in Slack and Teams. They should do two things. Auto answer time wasting questions and point people to documentation and process if they can help themselves.
- Why has my ticket not been done yet? - You are 4th in the queue and we expect to handle your request today. The ticket was submitted yesterday at 16:00 and is still within our 48 hour SLA.
- How do I do x? - Please read this SOP which has all of the steps outlined in plain English.
Have your workflow create JIRA tickets. Use a specific label or component to separate them from other requests. Publish the link which shows the ticket queue. This will hopefully stop people chasing, and tell your team what to work on next.
Why is this automation helpful?
- Automation reduces repetitive work needed by the SRE team
- It saves the requestors from having to wait for the answer
- Being up front on timings and where to check the information, helps people stop chasing and reduces future interrupts
- Learning the skills to automate is crucial to becoming an effective SRE
- Automated tasks do it the same way every time
Agree or disagree
I’ve disabled comment on my blog for now as I moved to static HTML to save costs. Let me know on LinkedIn or via email if you wish to continue the discussion. If you have a better way of organising your teams, I’d like to know.
