Introduction
Near the start of my tech career, I remember handling a ticket and thinking, why am I doing this again? It was the sort of request that came in from developers to DBAs, multiple times per week and easily took 30 to 60 minutes each time. That started a process where I suggested and implemented a self service system for devs, that allowed them to bypass DBAs for specific repeatable tasks where sometimes all that changed were some parameters. It eliminated dev waiting time on DBA availability, prevented us from doing monotonous repeatable tasks, and made the service consistent each time. I didn’t know it in 2014 but I was working on toil elimination.
Definition of toil
Toil as an SRE concept was introduced to the world by Google. It was defined as “the repetitive, predictable, constant stream of tasks related to maintaining a service”.
Toil adds no enduring value. I remember thinking, once I’ve done this a few times already, I’m not learning anything each time I do it. The agent jobs, metadata tables and SSIS packages had already been developed, I was simply pushing buttons, running scripts and waiting for one part to finish before going on to the next bit.
I’ll go on to talk about the impacts of toil on employee morale. This was the thing that struck me most. I was late starting in tech. I wanted to play catch up to colleagues of a similar age, and this toil was preventing me from learning the skills to work on interesting projects. So I made the automation of these tasks my project.
Why toil elimination matters
Reliability
At the heart of SRE is reliability. Reliable services endeavouring for high nines levels of uptime need to be self diagnosing and self healing as much as possible. Whilst you should have a playbook for when things go wrong, and standard operating procedures for manual processes, if you have to touch it to fix it then it’s likely toil. Self healing processes also reduce mean time to recovery resulting in higher availability, more satisfied customers and happier engineers (who weren’t called into a 3am Sev 1 incident).
Employee morale
Some people might get some comfort out of working through established SOPs, but everyone has a tipping point. Add to that, no one likes being paged out of hours, or even during office hours so it really is to the benefit of everyone that toil is eliminated wherever possible.
Furthermore, the work gets boring and most people generally like to learn new things and deliver value at work. Some don’t but they don’t tend to be high performing engineers, and possibly aren’t the best fit for a modern SRE organisation.
Scalability
Google talks about sublinear scaling. In essence, if you double your output without doubling support requirements, you have achieved this.
A traditional operations role where all tasks are done manually such as patching has to be automated in order to scale effectively. The first step in this is writing scripts to do the work. The logical evolution is for the scripts to belong to a software solution that handles scheduling, makes the changes and ensures smooth failovers.
What isn’t toil
Not all manual tasks are toil. If there is a derived benefit either to a person’s skill set or the service they are supporting, manual work can and does add value. It also stops skill atrophy which is a real effect of over-automation.
Toil isn’t just about saving FTE either. Of course it’s an added benefit to organisations that as they scale, their costs don’t need to increase linearly however I hate to see SRE as a means to cutting jobs. It’s just a better way of doing things and it leads to opportunities to increase personal skill level. That makes the engineers more employable and that’s never a bad thing.
How does toil elimination fit into SRE
I mentioned earlier that reliability is at the heart of SRE. The same is true of engineering. SREs should primarily be able to write code. SREs are like software engineers but with competing priorities to their colleagues that ship products and features. By writing effective solutions in code, they are quite literally engineering reliability into the service.
Toil is often a result of technical debt. A consequence of saying I know how to fix this so I’ll write an SOP, or I’ll put a debugging script on confluence. That isn’t SRE, it’s operations and it doesn’t scale.
How does an organisation drive this
There needs to be buy in from management. I was lucky early in my career that my manager encouraged my self service project and the benefits were clear. However not everyone will have that support. If you don’t have it, it’s an opportunity to learn how to influence and here’s how I would do it.
Measuring toil and your elimination efforts
Measurement is essential, with or without buy in but where you don’t have it, it adds validity to your efforts.
First of all, work out how much time is spent on specific tasks. Maybe start with one task as a proof of concept. Look for tasks that are needed to keep the lights on (ktlo) but just don’t contribute to personal or business growth. Even if your team doesn’t track time spent on tickets, search historic tickets and put an estimate on it. Remember to include time for context switching.
Secondly, measure the impact of your engineered solution once it is operating. How many hours of toil did you eliminate? How did it affect service uptime? Possibly harder to quantify but worth thinking about is, how did it improve team morale?
When you have the numbers, the benefit cannot be denied. Good luck with your toil elimination efforts.
