A place to share ideas, techniques, resources about all the testing activities needed to feel confident about delivering value to our customers frequently and sustainably.
I am going to share five handy tips for testers in today’s DevOps world from my experience in this post.
Learn Continuous Testing
The first and foremost thing to learn about is: what is
continuous testing
? How do DevOps processes impact the tester’s approach for transitioning from traditional waterfall to agile testing? Before starting with continuous testing, for each sprint, we were testing incrementally and then did the regression testing before deployed to production. Up to now, our sprint cycles for some projects have been weekly or bi-weekly. But this cycle becomes too long to roll out a feature or customer request in a timely way. So, we have adopted DevOps in our process.
Testing over the whole lifecycle by shifting left and shift right, i.e. testing continuously from beginning to end.
Learn about DevOps jargon.
I recommend taking a look at
Lisa Crispin’s course
in Test Automation U to learn basic concepts and terminology around continuous delivery and DevOps. That will help you communicate and collaborate with everyone involved in delivering your product, including operations and infrastructure specialists.
A Culture Shift from an Independent Team to Whole Team Approach
In a waterfall methodology, testing is confined to an independent verification and validation team. In agile development, it becomes a collaborative effort.
People come with baggage: preconceived notions, past experiences, prejudices. The primary responsibility is educating the team about DevOps culture.
In DevOps culture, the focus is on the product, not building fiefdoms and amassing political power. The DevOps slogan is, “Everyone involved with delivering software is responsible for delivering high-quality software.” In short, the delivery becomes a shared responsibility.
Build A Successful Test Strategy
Building the right test strategy is critical in continuous testing. When we have 500 automated tests, we can’t execute all the 500 tests for each change committed in the pipeline. It will take too much execution time. So, we have analyzed and prioritized the test cases by their core functionality as well as using input from the customer support and business teams. In the end, we have come up with 100 automated regression tests which are enough to give us confidence that the latest change hasn’t broken existing features.
Don’t Just Make The Tests Pass, Focus on Efficiency
When we started with DevOps, in our continuous pipeline, our automation tests failed in between or at the last step of the pipeline. We analyzed the test results and the error logs. Most of the errors were due to timeouts or because of the new feature or enhancement changes. We have updated our scripts periodically to take care of this. After deployment, when we cross verified the product/project using manual exploratory testing, we saw a few more failures. Then we did a brainstorming session with our entire product team to make our automated tests more efficient. For this, we have created/updated the automation scripts in the preproduction phase to save time and resources.
Choose Your Tests Wisely
This is related to the points discussed in the previous paragraph. When we started doing the deployment through the DevOps practices, as testers, we need to choose our tests wisely to add those to the pipeline. Along with the regression test suites, 24/7 monitoring should be added. This helps us to understand how the product is to be used by the customers. For choosing the tests, we use input from the user behaviors that we learn from our monitoring tools, as well as discussions with the business stakeholders.
I hope the information from this experience helps you to enhance your testing!
I hold a Certified Software Test Manager and Six Sigma Green Belt. I have a passion for testing and learning, as well as implementing and mentoring the latest automation tools. I spent the last 16+ years working in IT, with a focus on software quality assurance and software testing. I am Testing Guild 2018 conference speaker and a frequent blogger. I manage a fast-growing, dynamic team bringing you
https://www.qatouch.com
. Twitter: . Linkedin: . Blog :
www.bhavaniraj.com
We’re adding webinars to our Testing in DevOps community learning opportunities! Our first one is a panel discussion with awesome leading practitioners , , , and I will also join! Many thanks to
for hosting the webinar (they host this site as well).
So what will we talk about? As many of our guest posts here have pointed out, DevOps is about
the whole team
– including operations specialists – working together to deliver value to customers frequently, without drama or stress. It’s that continual loop: together we identify the most valuable capability to delivery next.
In the webinar, we’ll talk about how we build and test it, test and build it, together. We have created a pipeline to deploy it to production, together. We’ll probably get into topics such as monitoring and observability to learn how customers use the product and respond to problems, and get ideas for the next valuable feature slice to build. We will answer your questions!
For more details and to register, please go to my
(I am too lazy to copy it all over here!)
We welcome your questions in advance, just tweet them to @! Or, simply ask on the day in the Zoom chat. We look forward to an insightful and fun discussion!
The different pages of this website have links to so many helpful articles, podcasts, videos, webinars, books and more. Here are three that you might want to prioritize to read or listen to first, to build your understanding of devOps, continuous delivery, testing in production, and what makes software delivery teams successful.
Charity Majors on testing in production
Does the phrase “testing in production” scare you? Historically, it was an irresponsible practice – throw our changes into production and let the customers tell us what’s wrong. Today, it is a leading practice. In her recent article “
I test in prod
“, says it’s a superpower!
Most interesting man in the world
Charity points out that we test in production all the time. The key is to do it safely. It takes an investment in tooling and training. This is especially important with the complicated distributed apps we support today. We need to test more than the code – we need to test the whole system, including the environment, the infrastructure, and the customers.
There’s so much to learn in this one article, from Charity’s experiences and insights. She breaks down the three aspects needed to start testing in production in a useful and beneficial way: technical, cultural and managerial. Plus, she works in so many classic memes to hilarious effect! Give it a read and start overcoming the fear of what will happen with each new change released to production.
Theresa Neate: Stop the whisper campaign that devOps means no testers needed!
We’ve been fortunate to have a
guest post
here from , one of the leading testing practitioner who is also a leader in devOps practices. She gives us a preview of her
TestBash Australia
talk in this
Ministry of Testing podcast
with Michael Lang.
Theresa starts with one of the best explanations of devOps and continuous delivery that I have heard. If you are not yet familiar with these terms, listen to the first few minutes of this podcast.
One of the most important messages Theresa delivers here is that we need to stop the “whisper campaign” that devOps means “no testers needed” and “no testers welcome here”. We know we testers can add huge value by participating in devOps practices and helping our teams succeed with continuous delivery. it’s up to us to use our natural curiosity, learn important new skills, and show what we can do.
Photo from Christina Morillo on Pexels.com
This podcast is less than 30 minutes long, and it’s going to be the best half hour you’ve invested in your career in awhile.
I wish I could go to TestBash Australia and participate in Theresa’s workshop “Hands-on Intro to AWS and a Taste of DevOps, for Testers”!
Accelerate: State of DevOps 2019
This led by is building on 6 years of data from a wide group of respondents. The executive summary alone is a gold mine of information about what practices and capabilities correlate with team success.
From the
The report includes insights into testing practices. For example, though previous years found that test automation has a significant impact on continuous delivery (CD), this year they found that automated testing positively impacts continuous integration (CI).
The ability to reproduce and fix failures, gather feedback from tests, improve test quality and iterate test runs quickly also ties into automated testing.
The report also focuses on disaster recovery testing. One finding is that organizations who do disaster recovery testing and create and implement action items based on what they learn from it are more likely to be in the elite performing group.
The report delves into the “Four Key Metrics” of software development and delivery, one of which is culture – too often overlooked by organizations. To improve, organizations need to start with the foundations, and adopt a continuous improvement mindset.
Read the report and help your own team be one of the many teams who are continuing to improve their ability to deploy small changes to production frequently, while enjoying their work.
A continual problem that we must deal with when we run automated tests in our DevOps pipeline is maintenance. I could spend hours sharing posts and blogs about this matter, telling stories about how it affected our team’s capacity and time to market, and giving you several examples of what teams have done to sort this problem out. In fact, all the effort that we invest in maintenance is not visible to the business stakeholders’ eyes. Maintenance consumes our team capacity, but it does not add value to our clients directly.
Test automation challenges at scale
In one of my jobs wefaced this problem on a large scale. We had more an average of more than 1000automated tests running at each code check in, plus one big run at midnightthat we called full regression. Our suites where organized in those levels:
After each run, we noticed that 80% of UI and regression automated tests failed
due
fake failures. After reading my numbers, you can say if you automated crap, you will get crap results and that is correct — but please remember that we were building our automated test suites, with a continuous testing approach and DevOps mindset and pipelines, so we were trying to fail fast, learn and improve. First, we decided to analyze our test results and data, so we collected all the information that we got from our test runs and analyzed, we discovered that our tests failed
because
of those main reasons:
Changes inUI components.
Badcoding.
Exceptionsand errors handling
Time outseverywhere
Data
A new experiment
We had to make a decision. We had invested lot of efforts in testing automation, and we were the product team with more automated tests compare to the rest of product teams globally in the company. This brought us attention from lots of managers, who were keeping their eyes on our DevOps and test automation activities. In addition, we were participating in a DevOps competition which involved all the IT teams of the corporation. So, we decided to invest time to get the failed tests in good shape and have them passing in the pipeline. We called this mindset “Keep them green and running”. We communicated this effort to all the product teams. We asked the Product Owners to consider maintenance time as part of their team capacity, include tasks for this purpose, and keep it part of their definitions of “ready” and “done”.
Unfortunately, we did notachieve the goals we had hoped for. We still had the same test results, withmore than 80% of tests failing. In almost every case, they were fake failuresthat meant that the tests were stopped long before their last steps. To sum up,our rest results were not accurate at all and did not bring confidence to ourteams — which is something critical in testing automation.
Thinking outside the box
We needed to try something different. After some discussion, research, investigation and thinking out of the box, we decided to use gamification to keep our tests in a good shape, running and be green. We named the game “Game of Testing” and we established some critical points about it:
Maturitylevels: The gamers can be developers, testers or anybody who owned an automatedtest running in the pipeline, they were maturing in the game, from a beginnerlevel to a master level.
Visibility:We shared the game boards including ranking, test results and so, on using bigmonitors in the office, allowing everyone to be aware of the gamers andresults.
Weight:This is a very complicated and mathematical part, we needed to stablishformulas and weights for a point system for the game, so we could evaluate thetest results, data per gamer and teams, and gave points and make them get newlevels.
Recognition:We tried to recognize all the gamers taking part. For the best gamers, weshared the results with local and global senior managers, so their achievementscould be known and recognized. Thishelped encourage more team members to participate.
The game was quitesimple:
Gamers: All the team members who owned an automated test running in the pipeline.
Point system: A gamer won
point
s for test running in green and negative points if the test failed for fake results. Later we added negative points of breaking builds.
Rules: You needed to keep all your
test
and team tests running in the pipeline in a good shape for each run. So, you had to invest efforts on “maintenance” and continuous improvement if you want to rock in the game.
The Game of Testing
This approach helps usa lot to decrease false positive runs and keep the tests up and running in thepipeline. We decreased the number of 80% of failed tests to only 20% inapproximately three months and our numbers have continued decreasing. Furthermore, people had fun while playing thegame and they internalized that quality can be fun and not boring like usualmaintenance was.
While we got several criticisms about our failing tests at the beginning of our efforts to improve them, and people lost trust on our quality assurance process, we didn’t give up trying to fix the problems. In fact, it motivated our team to think out of the box and turn this problem into a useful and fun project. We spent several hours investigating continuous testing, scripting, and related topics. Besides that, we wanted to do something different, something motivates other team members to join.
Gamification can provide that motivation. You may not achieve desired results immediately, but if you prepare your environment and involve everyone in trying potential solutions, the game will give you results for sure. If you address the test issues you can get them pass or fail, but if you involve and motivate people to work on problems together, you always win and have fun. I recommend you try gamification to sort your problems out, you may be surprised at the results!
Jorge Luis Castro Toribio
Jorge is an
agilist
who feels passion for Testing Automation, Agile, DevOps and business agility. He enjoys research and learning about business agility and working with cutting-edge technologies. He has been learning continuously for more than 12 years and is still learning new ways to foster business agility and team greatness. He has worked in several roles (developer, tester, manager, agile coach) which helps him to see the big picture and operational and team members dynamics happening inside organizations. This experience lets him help teams design, build and implement digital, DevOps and agile transformation strategies. He encourages focus on: People, Productivity, Continuous improvement, Innovation and having fun to enjoy success in our agile journeys.
One fine day disruption was created inour organisation as part of digital transformation with few people fromoperations joining us – Developer, Tester, Product-owner.
…And we all moved into a NEW world ofDevOps. There were lot of uncertainties about what it would be like? Lot ofquestions whether there will be any role as tester? If there is what that rolewould involve?
As we all moved into DevOps world, allthese questions were getting clear. Our application demanded lots ofnon-functional testing along-with functional. In a DevOps world, it was allabout getting the feedback early and do frequent releases with a good quality.For a tester what it meant for me is to learn the process of continuousintegration and delivery and the testing for not just functional but also fornon-functional. Non-functional testing involved Security, Performance, Visual,Accessibility and reduce dependency on external teams to carry-out thesetesting to delivery faster. In short a Swiss Army Knife’s worth of skills (knowingdifferent types of testing). To top that it should be automated for quick-releases,so enough time is spent in doing exploratory testing and finding the unknowns.
To address that I came-up with the test-automation pyramid model, as illustrated by
to help team visualize & discuss test automation strategy. I will share how I modelled the test-automation pyramid to suit our DevOps world and overview all the testing involved in terms of functional & non-functional in an automated way to deliver faster without compromising on quality.
There are so many forms & interpretations of Test automation pyramid. All point out the same thing it should have more no. of unit tests, followed by service (consisting of API, Integration, Component) tests and finally UI tests.
There’s this another way of looking test pyramid as a Bug-Filter.
But have you noticed it doesn’t talk aboutnon-functional testing. Does that mean non-functional testing are not automatedor part of test automation? For me test automation stands for – tests which areautomated, run by machine/tool scripts and asserted pass/fail on ourexpectations.
At least it should say Functional Testpyramid. But I can see functional test automation is the most widelyimplemented form of automation and it has kind-of become synonymous with test automation.Also, Manual/Exploratory Testing bit is not captured properly. Exploratory Testingcan happen at any level – Unit, API, UI and for different types – Security,Performance, etc.
The test pyramid is an effective visualisationform outlining your testing approach which can be easily explained to others.Just make it visible on a board/wall and you have a very good indicator of howyou want to organise your testing efforts in one picture. It is a better visualreplacement for long test strategy documents.
As earlier mentioned, in our application we have lot of non-functional testing. So, I have taken these existing models and made changes to better suit our application. I will now explain how I view this test pyramid and apply it to testing in DevOps.
Test Automation Pyramid re-visited
So here is my interpretation.
Software testing is about identifyingRISKS and not finding bugs. Based on this, Imagine Test Pyramid to beupside-down as a RISK Filter, having all the different types of testing –Functional in UI, Service, Unit. Non-functional in Performance, Security and Others.
Now imagine the newly developedproduct or feature enters the pyramid for identifying risks, meaning carryingout functional tests in UI, Service, Unit-level and identifying functionalrisks.
Similarly, performance testing foridentifying performance risks, security testing for identifying security risks,and so on for other risks.
All these risks identified are byautomation meaning asserting our expectations or checking known behaviours. Whatabout Unknowns or new information. These will be identified by Exploratorytesting at every level – Unit, API, UI and for different types – Security,Performance, etc.
Do all the identified risks need to befixed? Absolutely NO. The team will prioritise it as high, medium, low etc. andfixed as per priority. Once all the prioritised risks are fixed then releasethe product/feature with known risks.
This is how test pyramid is applied in our case in testing in DevOps, making sure all the testings are done at each and every level from RISK perspective to achieve quality and deliver faster.
Functional Automation
Let’s talk about functional test automation. I will be just covering UI testing. There are so many functional test-automation tools, most popular ones being Selenium, Sahi, Katalon, Ranorex, NightwatchJS. And to implement best practices test there are frameworks such as Page-Object Models, BDD, TDD, Keyword & Data driven, Hybrid and so on. Depending upon your preference select the framework suiting your needs. There are no good or bad frameworks, it’s all about what best suits your application and your team. Same goes with the tools.
Non-Functional Testing
Quality of a software depends not onlyon Functional but Non-functional testing such as Security, Performance, VisualUI, Accessibility and so on.
Security Testing
Securitytesting is a process of identifying weakness in the security mechanisms ofan application that protects data and maintain functionality asintended. Purpose is to make the application immune to SQL injections,cross-site scripting, sensitive data exposure to protect Unauthorized andUnauthenticated users to enter in to your application. All these can bedriven through a tool and tested through test-automation.
Thereare various tools available where you can automate security tests and run aspart of CI/CD process. This automated activity necessarily doesn’t removemanual Penetration testing requirement. Through automation, it is beneficial tofind basic vulnerabilities in our application as much as possible so thatSecurity Specialists/Penetration testers can focus on identifying high-hangingbugs. This helps in delivering at a faster pace by finding basicvulnerabilities early in development phase which in turn saves Penetrationtester’s time so that they can concentrate where it adds value.
Performance Testing
Performance testing is about identifyingperformance issues such as slow response time, support for concurrent users,how much load the application can handle? Does the application crash? Also, it involvesnot only testing the server-side but the client-side as well. As can be seenfrom the chart, 80% of the end-user response time is spent on the front-end.Most of this time is tied up in downloading all the components of the page – images,stylesheets, scripts, Flash, etc.
To verify if an application is fast and efficient enough we perform client-side performance tests. This means checking the response time of a web application from the point of view of a single user. Both server and client side can be automated with pass/fail criterion if it doesn’t match the performance benchmarks. Usually performance tests are run once the functional tests are done and/or just before few weeks before the release. But executing performance tests early whenever new API is built (Server-side) and a new page is built (Client-side) helps to get early feedback and reduce any delays incurring from the issues identified later in the development phase.
Visual Testing
Visual UI testing consists about
Does your UI look the way it should, to allyour users?
Visual UI testing is about verifying that UI appears correctlyto your users. For detecting correct icon-size, padding, colour, layout, text,element position etc. It’s a quick way to check all these through visualautomation. It saves time and eye strain.
Angie Jones:
Even if the application passes functional testing, we still must check the look and feel of UI because the element might be not aligned properly or hidden behind some other element which the end user is not able to see. Visual UI testing is the best way to identify such issues and if integrated with functional tests gives a lot more confidence about the UI. This can be easily seen in the below image. Functional tests for this would pass but not Visual UI testing.
Accessibility Testing
Accessibility testing is a subsetof usability testing, and it is performed to ensure that the applicationbeing tested is usable by all the people and not only for people with disabilities.Accessibility is about ensuring inclusive applications are built for one andall, making an application equally accessible and usable for a normal user too.Purpose is to provide everyone with equal access to the application and meetingthe standards set in web content accessibility guidelines –
WCAG
.
People with disability will use navigation strategies like tabbing through the headings, links on webpages, images and videos with properly described text for accessing. So as a tester we need to test whether it follows the standard or meet the compliance having proper tab flow, recommended headings, alternate text for images, videos etc. Accessibility testing can also be automated to check whether the application has elements well structured, alternate texts and contents are described well or not.
Exploratory Testing
Exploratorytesting is the key to high quality software, it’s highly intellectual and time-consumingactivity. It is an investigative approach and main intent is to uncover newinformation or covering the unknowns in all areas of testing from functional,security, performance, visual and accessibility or any other types of testing. Oftenthis activity is pushed off to end-users.
The image below shows how testing from a risk perspective can be used to bridge the gap between automated testing (asserting expectations) and manual/exploratory testing (uncovering new information), so that lot of unknowns will be explored and converted into automated known behaviours to improve the quality. Exploratory testing helps in creating good automation tests to figure out all the things you want to go and check.
Continuous Testing
Continuous testing
right from requirement to production and taking feedback from production to requirement, this loop of continuous testing is key to achieve highest quality. Below image shows all the different types of testing we perform in various stages of DevOps.
Conclusion
As explained in my DevOps Testingpyramid, testing should always be done from a risk perspective and identifiedrisks should be prioritised and fixed accordingly for delivering qualityproduct faster.
In DevOps, automated testing is key tospeed-up the development cycle. By automation, I mean non-functional testing aswell. In a DevOps world, tester should possess Swiss Army Knife’s worth ofskills, which means as a tester it’s our responsibility to do all otherrelevant types of testing as much as possible apart from functional testing toget early feedback and not depend on external teams. Depending on externalteams might increase the risk for not getting the feedback early and delay therelease. Also, don’t forget to put all the tests including non-functional inCI/CD pipeline to get early and constant feedback on quality.
Continuous testing in all the stages iskey to achieve highest quality. It’s important to be tool aware and practiceidentifying good tools and use them whatever suits to your application and toyour team. Exploratory testing is the key to high quality software and pleasedon’t push it to end-users. Utilize as much time as possible in exploring theapplication for uncovering new information for improving quality.
Lastly, no matter what strategy andprocesses you have the team should have Quality mindset to go extra mile andadd that extra effort to deliver quality to the end user. This is accomplishedby Continuous Testing and doing all the relevant functional and non-functionaltesting at every stage so that
quality is cascaded
down to theend-user.
Testers have many diverse capabilities – just like a Swiss Army Knife!
I would like to end my blog – DevOpsis all about early feedback and frequent releases and for that
Testers should possess a Swiss Army Knife’sworth of skills, versatile and approaching the application from all the differentaspects of testing!!!
About the author
Amit Kulkarni
Amit Kulkarni (@amtoya on Twitter) is a test automation specialist and has over 10 years of experience delivering quality product for clients in Australia, Europe, Japan and USA. Amit is passionate about test automation for both functional and non-functional including security and performance testing. Amit has a diversified experience in Semiconductor technology, Embedded and Software industry in the field of Banking, Healthcare and Consumer Electronics working for companies such as Fujitsu, Mitsubishi, Qualcomm and Telstra. Amit is currently working as a Senior Quality Engineer in Bankwest and driving best quality practices in test automation and continuous testing to deliver quality product faster.
I was at
TestBash Brighton
recently, and a common theme kept coming up:
“You can be technical without knowing how to code!”
, TestBash organiser, even did a 99 second talk on the topic. He listed various skills – git, debugging, changing the microwave clock. If you have these skills as a tester, there are plenty of uses for these skills. If you have an agile or DevOps culture – where roles are less set in stone, and teams swarm on problems – you’ll get loads of opportunities to use them.
Well, apart from the microwave clock.
Learning the ecosystem
A yak dominates an ecosystem
It’s important to realise most developer skills aren’t coding. If you’ve tried to learn coding, you’ll know lots of other skills can get in the way. You start writing a simple application and lots of good practice pops up. Your developer friend says you should set up a git repository. Then instead of installing dependencies manually, it’s better to set up a dependency manager. Then you learn how to compile and run the code. The list goes on. This can be very daunting, and quickly turns into
yak-shaving
.
Learning doesn’t stop with developer skills
This seemingly never ending list of things to learn doesn’t stop with a developer’s skillset. Run a production-grade environment, and SRE/Ops skills will get in the way. Ops ensure that users can use your environments. They do all kinds of things. Backups. User and authentication control. Familiarity with the Linux CLI to debug issues. Log monitoring and analysis. Failover processes like blue/green. Containerisation like
Docker
, or container orchestration like
Kubernetes
. Your workplace may differ in the specifics, but production environments have lots of additional complex functionality. Complex functionality you can’t replicate by running the code on your laptop, or even sometimes in a test environment.
Broad skills are valuable too
Your broad skills are valuable
There are specialists in these subjects. Nobody is expecting testers to be experts with all of these skills. Much like testers don’t have to have all the coding skills of developers. That said, knowing about these things will help us understand our application. It will help understand facets of our application otherwise overlooked. If testing is about understanding the quality of our application, then we need to look beyond the code. As far as your application is concerned, code is the tip of the iceberg.
Let’s take an example: logging in to an application. Your users are not only expecting that they can login successfully. They’re expecting that the database stores their user credentials. They’re expecting that their data is backed up in the event of disaster. They’re expecting that they can still login if lots of other people are. They’re expecting their confidential data is securely stored and transported.
Much of that functionality is not done in code by developers. It is often the responsibility of ops, or sysadmins, or database admins. They configure databases and networks and servers that ensure all these things happen.
Ops opportunities for testers
Before DevOps, it was difficult for testers to have influence upon this. Much was siloed away from us. We test the code, then let them deal with that stuff. Out of sight, out of mind.
Cross-functional teams include Ops specialists
But now, Ops specialists are being moved into our teams, or working closely with our teams (hopefully not as a permanent ‘DevOps team’!). We see the work they do, and it needs testing too. That’s because it’s part of the application. It’s part of what users expect. We don’t just deliver code, we deliver a service.
Pair with an ops expert to assess risks
So how can testers start helping? To test code, we didn’t need to be developers. But learning related skills helped us empathise and co-operate. It helped us ask the right questions. So try pairing with an ops expert on your team and work out what risks might arise. Or doing some shaping of some infrastructure tickets, and thinking about who the users are (with pipelines, they’re often developers!). Or try writing some automation scripts to ensure that associated problems get caught before production. Or go to a devops meetup/conference. Learn about bad practice and security/infrastructure smells. Start treating ops like developers – to be assisted and questioned.
Because our application is more than just code. It’s all the processes, and the infrastructure too. That stuff needs testing too.
Matt Long is a QA & devops consultant for OpenCredo, an open source software consultancy based in London.
He has worked as a QA consultant for many years, and built test automation frameworks in half a dozen programming languages. After working on many cloud infrastructure projects, he’s become particularly interested in the intersection of modern cloud infrastructure and testing.
Similarities between SRE and testing: Alert/automation suite maintenance
By Abby Bangser
I want to take a few minutes to share with youhow I have grown to once again realise how common so many of our challenges arein tech. This time the commonality is between the world of test automation andoperational alerting. Before I dive into these details though, let me quicklyexplain my career journey which led me here.
From the beginning of 2012 to the middle of2018 I worked as a test consultant in organisations ranging from small non-profitsthrough to large multinational financial institutions. During this time I sawthe scale, financing, and cultures change fairly drastically, however thechallenges were often quite similar when it came down to the team level ofdelivering software. Most notably, over these years I realised that being aspecialist in quality often meant my biggest contribution was in how we delivercode through Continuous Integration and Delivery processes rather than just how
I
test things. This led me to working with many DevOps and infrastructure relatedteams and activities and eventually on to joining MOO as a Test Engineer ontheir Platform Engineering team.
Introduction to our platformengineering team
At MOO the Platform Engineering team isresponsible for the infrastructure of both production and pre-productionenvironments as well as the running of internal tooling like source control andbacklog management tooling. As you may expect this includes being on call forthese systems 24 hours a day, 7 days a week. From what I can see around theindustry our team is not particularly unique in how we handle this.
We have a rota which makes sure we share theresponsibility and our team generates both warning and critical alerts to makesure we respond if issues occur. These alerts provide us huge opportunity forspeeding up our Mean Time To Recovery (MTTR) by decreasing our time todiscovery. However when I joined the team, they were also generating a lot ofstress which is why one of my first tasks was to review our alerts for valueand how to remove that stress.
Tackling our alerting challenges
Our alert stress was generated for a few majorreasons. First of all, some of our alerts were things that our MOO team memberscould do nothing about. For example, if our Google authentication is failingthere is an impact to our internal users which could trigger an alert to our oncall engineer, however there is nothing we can do but wait until Google sortsout their incident. Another issue with our alerts was critical alerts (meaningsomeone is woken up in the middle of the night) that resolved themselves withinminutes. Often these were issues with machine health in AWS which has its ownself-healing mechanisms and our thresholds were just too low. Finally, we hadsome alerts that made people wonder if they were worthwhile as they had nevertriggered once in over a year of tracking.
When I joined the team there was a ticket onthe backlog to “Improve team alert fatigue”. This was intentionally a prettyvague topic and was meant to be an analysis placeholder as there wasn’t justone solution to all these concerns with our alerts. Alert fatigue can bedescribed as “exposed to a large number of frequent alarms and consequentlybecomes desensitized to them. Desensitization can lead to longer response timesor to missing important alarms.” and our team was definitely showing the earlysymptoms. After spending some time with each team member we decided thatdefinition of done for this ticket would be to:
Identify the shape andrequirements of a “good” alert
Review our alerts against this definition
Set up future work to bring ourcurrent alerts in line with our desired state
Achieving a working solution byputting it to the test
To achieve these goals I went about puttingtogether a straw man definition of what a warning and a critical alert wouldneed to have. Then scheduled reviews of all of our alerts against thesedefinitions to let them be hardened and evolved by our real context. This was a(very) long process but by grouping alarms together, sharing the load acrossall team members in small groups and doing no more than 30 minute meetings wewere able to get through the review in less than a month and with only alimited amount of meeting fatigue!
In the end these became our team’s workingdefinitions:
All critical alerts should have…
…an immediate and quantifiable end user impact which must require a specific human interaction to rectify
…and should be paged on immediately
Whereas
All warning alerts should have…
…no short term action required, but do have proven critical impact to the system if left unmanaged
…and should be paged on after 72 hours in a persistent state
We of course dove into each of the detailslike how to define “short term” but seeing this high level framework canhopefully demonstrate how we also uncovered the need to discuss the informationprovided by an alert (if it requires human interaction, shouldn’t that actionbe clear from the get go?) and also how we handle different types of services(grafana which just displays data and is used most heavily in office hours maynot need to be paged on Sunday morning).
Outcomes and learnings from theprocess
By the end of the review we identified somealerts that could be removed, some alerts which should be retired throughretiring the systems they were attached to, and many alerts which could usesome sprucing up. In all these cases we created story cards with the reviewcontext and have been regularly feeding them into our workload since then.
As we are now about four months on, I thought it may be a good time to retrospect on the experience and the value of the work done. My biggest takeaway is that this experience closely mirrored my experience with test automation suite reviews.
In particular, stepping away from a singlestory card or a single feature helped our team realise the bigger picture ofalerts and come up with a framework to keep our alerts in order. Nothing wecame up with was novel to our team, it is just that we had not been able to sitdown and agree while simultaneously being able to apply or theories againstreal world scenarios. The alert review is what really helped cement agreementover our alert framework through discussion and debate about possible outliers.
Another similarity to test automation is howthe team cared deeply about quality alerts, but given the intermittent pain ofpoor alerts and lack of customer facing impact, it was hard to spend the timeto improve them. To be honest, this continues to be a challenge as we try andstrike a balance between interrupt driven work (incidents or alerts on a givenday), feature delivery, and long term maintenance. That being said, having thethoughtfully created and thorough documentation of fixes in places has meantwhen a “mildly annoying” alert goes off
again
we can often pluck that card from the backlog and prioritise it quickly.
Finally, the test community doesn’t seem tohave as well known a term for red build fatigue as the operations community hasfor alert fatigue, but trust me they are the same thing! Our team started torely on their instincts and experience to decide which warning alerts
actually
mattered and more often thannot just wanted to remove the culprit alerts because it was the only way theysaw any chance of improving the situation.
I am not suggesting we are in an ideal placenow, but by introducing some practices for bringing the pain forward likeescalating all warnings after 72 hours, it has increased our on call engineerand product owners focus on alert quality for the sake of team sanity.
Abby Bangser is a software tester with a keen interest in working on products where fellow engineers are the users. Abby brings the techniques of analysing and testing customer facing products to tools like delivery pipelines and logging so as to generate clearer feedback and greater value. Currently Abby is a Test Engineer on the Platform Engineering team at MOO which supports the shared infrastructure and tooling needs of the organisation.
Open Sourcing Your Mindset
By Brendan Connolly
Asdevelopment has shifted from waterfall to agile and now DevOps, at each stageyou see developers taking on more responsibility and accountability.
Waterfallwas isolationist, building silos around different groups in softwaredevelopment. Developers owned the act of coding and in the stereotypicaldeveloper way, had little to no interest what happened before or after theycoded. Other roles on teams in many ways acted as insulators to keep externalforces away from developers.
Agilewas developers pulling that insulation away, shifting themselves left andtaking more ownership of the product creation process. Developers found theyrequired a real time feedback loop to build the right things and successfullyreact to change.
DevOpsis essentially the same process on the right side of the spectrum. Developmenttakes more ownership of deployment, monitoring and supporting the applicationshealth. It’s just too slow and cumbersome to react to the requirements ofmaintaining applications by filtering feedback through separate teams at scale.
Thebenefits of integration outweigh the cost of the additional responsibility.
Code Continuous
Tocope with the additional responsibilities, developers have turned to theirtrusty friend
code
.
Withagile we see rise of tooling developed for
ContinuousIntegration
to alleviate the pain and reduce the risk in iterativedevelopment.
InDevOps we see the rise of code and tools supporting
Continuous Deployment
. Even when many teams already had productinstallers, or automated deployments what was missing was the integration ofthat information.
Ifdevelopers are owning more of the product as well as release and deployments,then testing naturally gets swept up in the process. In fact we see thispattern emerging with the concept of
ContinuousTesting
. Code is being written, tools created to help solve the issue ofintegrating test execution and reporting into teams release pipeline. Thefeedback loop is extended to provide rapid information to make it easier andfaster for development to be responsive to change.
Continuous Testing
Automatedtesting is a loaded term. In developer circles it almost exclusively means unitand possibly integration tests. In QA it almost almost exclusively means end toend (e2e) tests, and is often synonymous with
Selenium
.
Onteams where both sets of tests exist, their worlds may seldom interact.Developers may be aware of the
e2e
tests but they probably aren’t writing them and even less likely running them.
ContinuousTesting has nothing to do with the testing that most QA team members are mostfamiliar with, exploratory testing. The purpose of the code and tooling thatcomprises a pipeline isn’t for there to be some manual intervention requiredpart of the way through it. Putting exploratory testing somewhere in the laterstages for the pipeline is akin to pointing a fire hose at someone and askingthem to direct the flow to the next stage.
Ifthe
speed of DevOps
wasn’t enough, the presence of
continuous testing
only puts morepressure on testers, simply because more attention is being drawn to testing.
Automationengineers and product engineers are being drawn closer together. The importanceand relevance of end to end automated tests is becoming more of a reality forproduct developers as they become run regularly as a post deploy verificationtests.
Whilecode is being thrown at testing with newfound passion, exploratory testers needto be prepared for their time in the spotlight. If automated tests are runningall the time what tests are you doing? What makes them special? Why are theyrequired before we can deploy?
What do you do different than Selenium?
Beforeyou dismiss that question, pause and consider what all selenium and the testsit drives truly provide.
Seleniumis cross browser, scalable, schedulable, repeatable, and can provide logs andscreenshots of its tests. Can you compete with that?
Flaky,slow, and wrong sometimes, you say? And you and all other humans aren’t?
Conversationsthat try to explain using people instead of machines for testing software canend up justifying them under the guise of
thereare some tests that aren’t good candidates for automation
.
Don’t settle for executing tests that don’t have enough ROIto automate.
Thistype of conversation relies on the belief that testing is comprised to twoelements:
test case creation
and
test execution
. Perhaps at its most baselevel testing is just that, the value proposition of the role is in the nuancethat a skilled tester can provide to teams through those elements.
Testersdo more than a fill a gap in automated test coverage. The problem is manytesters express this nuance solely through actions.
Developersproduce code, a measurable artifact from their effort. The generation of thatartifact is often backed by computer science, a formalized area of study.Artifacts of effort are reassuring, people like to see the fruits of theirlabors. Tangible results can be measured and quantified, they might even helpestimate future effort.
Joan of Arc
A tester’s value tends to be recognized at the same rate as relationships develop. Testers get characterized as
folk heroes
rather than members of a skilled profession. There’s no trade, education, or craft just individuals saving the day periodically.
The Intuition trap
Evenwhen teams do value testing, people are usually primarily concerned with theprogression or regression of testing efforts. Testers comments in stand-ups caneasily turn into a user story status change report:
Passed this ticket, working this ticket, noblockers
Ifyou aren’t taking the opportunity to discuss your actual actions in stand ups,then you may fall prey to the intuition trap. When you talk about your workthis way, sometimes you end up actually working this way.
Open Source Your Mindset
It’sfun being a hero, but there’s no skill in being
Superman
or
Superwoman
(I was going to say Wonder Woman, but she actually trained and I think everyonerecognized that. So if you are the teams Wonder Woman then kudos, you haveserious skills but I digress).
Whentesting goes smoothly the result is
nothing
.Testing isn’t required for things to go right, it just might help prevent athing going wrong.
Opensourcing your mindset means
teaching
others another way to look at and interact with their code, or the product.This cuts both ways though, it means being willing to accept a pull request ofoutside perspective. Openly integrating new ideas into your
testers mindset
.
Replace Execution with Communication
. Like acting or athletics there might be a script or game plan to start with but there is a lot going on behind the scenes, adapting to circumstance and being thought up on the fly. It’s intuition driven.
Whenactions, practices and motivations are not being verbalized and expressed tothe team and stakeholders it puts testing on the level of superstitions andtoken time sacrifices to the quality gods.
Testersneed to pry open this experience, expose the internals driving motivations.Breaking out of that shell grants visibility into a testers actions andmotivations to provide a context for team members to attribute value.
Replace Isolation with Collaboration
Itcan be uncomfortable or difficult to open up and share what goes through your
testing mindset
.
Theadditional ownership developers have taken in DevOps has changed relationshipdynamics. What was at one point a contentious us vs. them relationship is muchmore open. There just isn’t the time to have as much formal process drivencommunication. Success at scale requires collaboration, pairing and mobbingmore.
When with developers, it’s an opportunity to test at their desk, against their local changes. There are risks testing here, the developers local setup is definitely different than production but so are the testing and staging environments. Let go of the taboo of
it works on my machine
and consider what can be tested here and what needs to be covered on a different stage. Take the opportunity to work together when the code is the freshest in the developers mind and where they are most comfortable.
It’s not only about working with developers. Mob testing with
UX
and
Product
alongside developers is a way to draw the entire team together. When the team is together seeing how the application is working it bypasses all the communication gates. There is no need for a separate follow-up conversations to loop someone in, that person can see and feel how things are working by being a part of the group experience.
Replace Tedium with Stratagem
Teamsare more open to testing and sharing in quality than ever. Testers have beenbuilding quality mindsets for years. Developers are building tooling but teamsare looking for guidance and mentorship in
Quality
.
Testing isn’t just button pressing, by showing that through communication and working together we elevate our status. Draw people together, into the quality circle, and enable the team to be more comfortable and aware of testing. We can set working agreements, talk about overarching strategy for how testing throughout the development cycle builds quality in. We can embrace automated testing as a team activity where unit, integration and end to end coverage is part of a teams strategy for mitigating risk. We can forgo complete control of testing and opt into being a voice of authority and expertise in quality.
Brendan Connolly is an experienced Software Tester, Automator, Speaker, and Blogger. Currently he is a Senior Quality Engineer at Procore Technologies in Santa Barbara, California. He maintains a testing blog @ http://www.brendanconnolly.net
Testers, DevOps, and Enabling Digital Transformation
By Gerie Owen
To be successful in today’s businessenvironment, virtually all organizations must plan for digital transformation,not only to thrive but to survive. Often, when organizations embark on digitaltransformation initiatives, they turn to DevOps to increase business value andimprove customer experience.
Testers, as champions of the customer andguardians of their organizations’ digital reputations, are uniquely positionedto enable digital transformation. Weenable digital transformation by focusing testing on business risk and customervalue as well as transforming our test processes.
DevOps enablesDigital Transformation. The synergy is clear: digital transformation and the principlesof DevOps are closely aligned. Specifically,continuous integration and continuous delivery enable innovation in operations,streamline processes and eliminate bottlenecks.
Continuoustesting plays a key role in the continuous delivery pipeline by enabling bothhigh quality and high velocity and ensuring customer happiness. Let’s examinehow continuous delivery enables digital transformations by streamliningprocesses and focusing on value to the customers and how testers can enable andchampion this process.
DevOps Principles Aligned with Digital Transformation
Digital transformationcan be defined in many ways; however, it is a fundamental shift in the way anorganization approaches its business and operational processes by usingtechnology to increase focus on and improve customer experience. Most digital transformations are focused oncustomer experience, operational processes and business models. DevOps principles described in the
ThreeWays
— systems thinking, amplifying feedback loops and creating a culturalof experimentation and continual learning — can be the basis for creating the majorchanges that are required for digital transformation.
Systemsthinking or focusing on the performance of an entire process is the mosteffective approach to transforming business models and operationalprocesses. If changes are made tofundamental operations in siloed initiatives, they will likely fail. Amplifying feedback loops is critical todigital transformation so that corrections can be made early and often. And creating a culture of continuousexperimentation and learning encourages innovation without fear of consequencesif a new business, process or technological approach fails.
TestersEnabling Digital Transformation by Transforming Test Processes
Testers enable digital transformation not onlyby championing the customer, but also by transforming their testpractices. Transformational testpractices include implementing the practices of continuous testing, including“shift left”, test optimization, as well as the use of service virtualizationand advanced analytics, and “shift right”, i.e., testing in production.
Test Automation Pyramid, from More Agile Testing by Janet Gregory and Lisa Crispin
Continuoustesting is an approach to managing risk by focusing not only on eliminatingtesting bottlenecks but more importantly, increasing the effectiveness of ourtest processes. The test pyramid is a“shift left” pattern whereby most of the automated test cases are concentratedin the unit, API, component and integration layers with fewer automated testcases used to test the GUI layer.
Testersas Champions of the Customer
As testers,meeting the challenges of our role in digital transformation requires acoordinated approach made up of several components. Customers expect a seamless experience acrossall channels; however, they do not necessarily expect to perform the same actionson every channel. Therefore, a digitaltransformation test strategy will require using customer behavior analytics tounderstand customer’s browsing, purchasing and other transactions acrosschannels in order to test efficiently and effectively.
As champions ofthe customers, we must focus on testing the customer experience. This means we must learn to use customeranalytics, personas and customer journeys and combine the results of those withexploratory testing.
Testers and DevOps can be the keys to successful digitaltransformation through by focusing on technological transformation based onbusiness value and customer happiness. Testers, this is your opportunity to goabove and beyond and contribute to your organizations in innovative,customer-centric ways.
Gerie Owen is a Testing Strategist and Evangelist. She is a Certified Scrum Master, Conference Presenter and Author on technology and testing topics. She enjoys mentoring new QA Leads and brings a cohesive team approach to testing. Gerie is the author of many articles on technology including Agile and DevOps topics. She recently developed a curriculum for DevOps 101 training. Gerie chooses her presentation topics based on her experiences in technology, what she has learned from them and what she would like to do to improve them. Gerie can be reached through her website, www.gerieowen.com, her blog, Testing in the Trenches, is at https://testinggirl.wordpress.com/ and @GerieOwen on Twitter and on LinkedIn .
Where you see “DevOps” being practised but the QAs (or Testers) are still relegated to the “checking things corner” and not explicitly and proactively involved in providing input into both applications and infrastructure code, are we perhaps doing the very opposite of what DevOps was meant to be?
Images credit:
Beware the cargo cult
Most of us know the
tale
of how DevOps was conceived and I trust most of us have read The Phoenix Project (Kim et al, 2013) as one of our first DevOps books.
Therein we learned that DevOps is meant to be a collaborative and efficient partnership between all disciplines to achieve a common goal, which ultimately means: working software in a production system, delivering value to customers.
Images credit:
Where apparent ‘DevOps’ excludes other team members, like Security or QA, we have missed the point. DevOps is meant to be development
teams
working with Operations
teams
, not just individuals in developer and operations roles working together.
The QA role is considered to be a member of the Development team. The QA is therefore intrinsically a member of DevOps.
Where we find that QAs have no input into the Operations or Continuous Delivery conversations because they “are not developers” but we think we are doing DevOps, we are in fact practising a cargo cult — i.e. an imitation and approximation of the real thing because the real reasons of WHY have not been fully grasped.
If we are going to do DevOps, it behooves us to know WHY we do it, and not only HOW.
Waste
‘Agile’ and ‘DevOps’ fundamentally assist with reducing waste. (I generally prefer not to use these capitalised nouns as they have become
overloaded
, so use them here cautiously).
Project churn, conflict, friction, slow response, unreliable and unstable systems, needless expenses, lengthy handovers, defects, time delays, failed projects, manual overheads, double-handling and many similar issues, are all waste.
Every time a feature is double-handled (such as the churn when a bug is discovered later), we have incurred waste. Feedback should be sought earlier rather than later so that one may course-correct or remedy issues as early as possible and prevent waste.
QA is feedback
QAs do not provide assurance; they help provide and define analysis and feedback.
Monitoring is a form of feedback of the system’s behaviour.
Just like monitoring is feedback, so is testing. Whether the testing is manual exploratory testing or automated checks, it is feedback.
Testing tells you about components and relationships; monitoring tells you about the system.
To minimise and reduce waste, you want to receive that feedback as early as possible, and that the feedback occurs in meaningful levels and depths in your system.
Early feedback
QA (testing and monitoring) should be done early and continuously. Receiving early feedback enables one to act on that information and improve or course-correct as early as possible.
Your QAs can and should be involved in these conversations – and if they cannot, then as their manager or peer the onus is on you to empower them to become involved.
Images credit:
The days of QAs telling you if you’re OK to ‘go live’, are numbered. The whole team should know that, based on the continuous feedback they have been receiving.
Lean QA
The reduction of waste (rework, defects, friction) by holistically testing the whole system, early and continuously (and into production), including measuring what really matters & incrementally improving thereupon, and using your humans intelligently in doing so, is what I call “Lean QA”.
