Category: Testing in Production

By Abby Bangser

I want to take a few minutes to share with youhow I have grown to once again realise how common so many of our challenges arein tech. This time the commonality is between the world of test automation andoperational alerting. Before I dive into these details though, let me quicklyexplain my career journey which led me here.

From the beginning of 2012 to the middle of2018 I worked as a test consultant in organisations ranging from small non-profitsthrough to large multinational financial institutions. During this time I sawthe scale, financing, and cultures change fairly drastically, however thechallenges were often quite similar when it came down to the team level ofdelivering software. Most notably, over these years I realised that being aspecialist in quality often meant my biggest contribution was in how we delivercode through Continuous Integration and Delivery processes rather than just how I test things. This led me to working with many DevOps and infrastructure relatedteams and activities and eventually on to joining MOO as a Test Engineer ontheir Platform Engineering team.

Introduction to our platformengineering team

At MOO the Platform Engineering team isresponsible for the infrastructure of both production and pre-productionenvironments as well as the running of internal tooling like source control andbacklog management tooling. As you may expect this includes being on call forthese systems 24 hours a day, 7 days a week. From what I can see around theindustry our team is not particularly unique in how we handle this.

We have a rota which makes sure we share theresponsibility and our team generates both warning and critical alerts to makesure we respond if issues occur. These alerts provide us huge opportunity forspeeding up our Mean Time To Recovery (MTTR) by decreasing our time todiscovery. However when I joined the team, they were also generating a lot ofstress which is why one of my first tasks was to review our alerts for valueand how to remove that stress.

Tackling our alerting challenges

Our alert stress was generated for a few majorreasons. First of all, some of our alerts were things that our MOO team memberscould do nothing about. For example, if our Google authentication is failingthere is an impact to our internal users which could trigger an alert to our oncall engineer, however there is nothing we can do but wait until Google sortsout their incident. Another issue with our alerts was critical alerts (meaningsomeone is woken up in the middle of the night) that resolved themselves withinminutes. Often these were issues with machine health in AWS which has its ownself-healing mechanisms and our thresholds were just too low. Finally, we hadsome alerts that made people wonder if they were worthwhile as they had nevertriggered once in over a year of tracking.

When I joined the team there was a ticket onthe backlog to “Improve team alert fatigue”. This was intentionally a prettyvague topic and was meant to be an analysis placeholder as there wasn’t justone solution to all these concerns with our alerts. Alert fatigue can bedescribed as “exposed to a large number of frequent alarms and consequentlybecomes desensitized to them. Desensitization can lead to longer response timesor to missing important alarms.” and our team was definitely showing the earlysymptoms. After spending some time with each team member we decided thatdefinition of done for this ticket would be to:

1. Identify the shape andrequirements of a “good” alert
2. Review our alerts against this definition
3. Set up future work to bring ourcurrent alerts in line with our desired state

Achieving a working solution byputting it to the test

To achieve these goals I went about puttingtogether a straw man definition of what a warning and a critical alert wouldneed to have. Then scheduled reviews of all of our alerts against thesedefinitions to let them be hardened and evolved by our real context. This was a(very) long process but by grouping alarms together, sharing the load acrossall team members in small groups and doing no more than 30 minute meetings wewere able to get through the review in less than a month and with only alimited amount of meeting fatigue!

In the end these became our team’s workingdefinitions:

All critical alerts should have…
…an immediate and quantifiable end user impact which must require a specific human interaction to rectify
…and should be paged on immediately

Whereas
All warning alerts should have…
…no short term action required, but do have proven critical impact to the system if left unmanaged
…and should be paged on after 72 hours in a persistent state

We of course dove into each of the detailslike how to define “short term” but seeing this high level framework canhopefully demonstrate how we also uncovered the need to discuss the informationprovided by an alert (if it requires human interaction, shouldn’t that actionbe clear from the get go?) and also how we handle different types of services(grafana which just displays data and is used most heavily in office hours maynot need to be paged on Sunday morning).

Outcomes and learnings from theprocess

By the end of the review we identified somealerts that could be removed, some alerts which should be retired throughretiring the systems they were attached to, and many alerts which could usesome sprucing up. In all these cases we created story cards with the reviewcontext and have been regularly feeding them into our workload since then.

As we are now about four months on, I thought it may be a good time to retrospect on the experience and the value of the work done. My biggest takeaway is that this experience closely mirrored my experience with test automation suite reviews.

In particular, stepping away from a singlestory card or a single feature helped our team realise the bigger picture ofalerts and come up with a framework to keep our alerts in order. Nothing wecame up with was novel to our team, it is just that we had not been able to sitdown and agree while simultaneously being able to apply or theories againstreal world scenarios. The alert review is what really helped cement agreementover our alert framework through discussion and debate about possible outliers.

Another similarity to test automation is howthe team cared deeply about quality alerts, but given the intermittent pain ofpoor alerts and lack of customer facing impact, it was hard to spend the timeto improve them. To be honest, this continues to be a challenge as we try andstrike a balance between interrupt driven work (incidents or alerts on a givenday), feature delivery, and long term maintenance. That being said, having thethoughtfully created and thorough documentation of fixes in places has meantwhen a “mildly annoying” alert goes off again we can often pluck that card from the backlog and prioritise it quickly.

Finally, the test community doesn’t seem tohave as well known a term for red build fatigue as the operations community hasfor alert fatigue, but trust me they are the same thing! Our team started torely on their instincts and experience to decide which warning alerts actually mattered and more often thannot just wanted to remove the culprit alerts because it was the only way theysaw any chance of improving the situation.

I am not suggesting we are in an ideal placenow, but by introducing some practices for bringing the pain forward likeescalating all warnings after 72 hours, it has increased our on call engineerand product owners focus on alert quality for the sake of team sanity.    

Abby Bangser is a software tester with a keen interest in working on products where fellow engineers are the users. Abby brings the techniques of analysing and testing customer facing products to tools like delivery pipelines and logging so as to generate clearer feedback and greater value. Currently Abby is a Test Engineer on the Platform Engineering team at MOO which supports the shared infrastructure and tooling needs of the organisation.