A place to share ideas, techniques, resources about all the testing activities needed to feel confident about delivering value to our customers frequently and sustainably.
I am going to share five handy tips for testers in today’s DevOps world from my experience in this post.
Learn Continuous Testing
The first and foremost thing to learn about is: what is
continuous testing
? How do DevOps processes impact the tester’s approach for transitioning from traditional waterfall to agile testing? Before starting with continuous testing, for each sprint, we were testing incrementally and then did the regression testing before deployed to production. Up to now, our sprint cycles for some projects have been weekly or bi-weekly. But this cycle becomes too long to roll out a feature or customer request in a timely way. So, we have adopted DevOps in our process.
Testing over the whole lifecycle by shifting left and shift right, i.e. testing continuously from beginning to end.
Learn about DevOps jargon.
I recommend taking a look at
Lisa Crispin’s course
in Test Automation U to learn basic concepts and terminology around continuous delivery and DevOps. That will help you communicate and collaborate with everyone involved in delivering your product, including operations and infrastructure specialists.
A Culture Shift from an Independent Team to Whole Team Approach
In a waterfall methodology, testing is confined to an independent verification and validation team. In agile development, it becomes a collaborative effort.
People come with baggage: preconceived notions, past experiences, prejudices. The primary responsibility is educating the team about DevOps culture.
In DevOps culture, the focus is on the product, not building fiefdoms and amassing political power. The DevOps slogan is, “Everyone involved with delivering software is responsible for delivering high-quality software.” In short, the delivery becomes a shared responsibility.
Build A Successful Test Strategy
Building the right test strategy is critical in continuous testing. When we have 500 automated tests, we can’t execute all the 500 tests for each change committed in the pipeline. It will take too much execution time. So, we have analyzed and prioritized the test cases by their core functionality as well as using input from the customer support and business teams. In the end, we have come up with 100 automated regression tests which are enough to give us confidence that the latest change hasn’t broken existing features.
Don’t Just Make The Tests Pass, Focus on Efficiency
When we started with DevOps, in our continuous pipeline, our automation tests failed in between or at the last step of the pipeline. We analyzed the test results and the error logs. Most of the errors were due to timeouts or because of the new feature or enhancement changes. We have updated our scripts periodically to take care of this. After deployment, when we cross verified the product/project using manual exploratory testing, we saw a few more failures. Then we did a brainstorming session with our entire product team to make our automated tests more efficient. For this, we have created/updated the automation scripts in the preproduction phase to save time and resources.
Choose Your Tests Wisely
This is related to the points discussed in the previous paragraph. When we started doing the deployment through the DevOps practices, as testers, we need to choose our tests wisely to add those to the pipeline. Along with the regression test suites, 24/7 monitoring should be added. This helps us to understand how the product is to be used by the customers. For choosing the tests, we use input from the user behaviors that we learn from our monitoring tools, as well as discussions with the business stakeholders.
I hope the information from this experience helps you to enhance your testing!
I hold a Certified Software Test Manager and Six Sigma Green Belt. I have a passion for testing and learning, as well as implementing and mentoring the latest automation tools. I spent the last 16+ years working in IT, with a focus on software quality assurance and software testing. I am Testing Guild 2018 conference speaker and a frequent blogger. I manage a fast-growing, dynamic team bringing you
https://www.qatouch.com
. Twitter: . Linkedin: . Blog :
www.bhavaniraj.com
The different pages of this website have links to so many helpful articles, podcasts, videos, webinars, books and more. Here are three that you might want to prioritize to read or listen to first, to build your understanding of devOps, continuous delivery, testing in production, and what makes software delivery teams successful.
Charity Majors on testing in production
Does the phrase “testing in production” scare you? Historically, it was an irresponsible practice – throw our changes into production and let the customers tell us what’s wrong. Today, it is a leading practice. In her recent article “
I test in prod
“, says it’s a superpower!
Most interesting man in the world
Charity points out that we test in production all the time. The key is to do it safely. It takes an investment in tooling and training. This is especially important with the complicated distributed apps we support today. We need to test more than the code – we need to test the whole system, including the environment, the infrastructure, and the customers.
There’s so much to learn in this one article, from Charity’s experiences and insights. She breaks down the three aspects needed to start testing in production in a useful and beneficial way: technical, cultural and managerial. Plus, she works in so many classic memes to hilarious effect! Give it a read and start overcoming the fear of what will happen with each new change released to production.
Theresa Neate: Stop the whisper campaign that devOps means no testers needed!
We’ve been fortunate to have a
guest post
here from , one of the leading testing practitioner who is also a leader in devOps practices. She gives us a preview of her
TestBash Australia
talk in this
Ministry of Testing podcast
with Michael Lang.
Theresa starts with one of the best explanations of devOps and continuous delivery that I have heard. If you are not yet familiar with these terms, listen to the first few minutes of this podcast.
One of the most important messages Theresa delivers here is that we need to stop the “whisper campaign” that devOps means “no testers needed” and “no testers welcome here”. We know we testers can add huge value by participating in devOps practices and helping our teams succeed with continuous delivery. it’s up to us to use our natural curiosity, learn important new skills, and show what we can do.
Photo from Christina Morillo on Pexels.com
This podcast is less than 30 minutes long, and it’s going to be the best half hour you’ve invested in your career in awhile.
I wish I could go to TestBash Australia and participate in Theresa’s workshop “Hands-on Intro to AWS and a Taste of DevOps, for Testers”!
Accelerate: State of DevOps 2019
This led by is building on 6 years of data from a wide group of respondents. The executive summary alone is a gold mine of information about what practices and capabilities correlate with team success.
From the
The report includes insights into testing practices. For example, though previous years found that test automation has a significant impact on continuous delivery (CD), this year they found that automated testing positively impacts continuous integration (CI).
The ability to reproduce and fix failures, gather feedback from tests, improve test quality and iterate test runs quickly also ties into automated testing.
The report also focuses on disaster recovery testing. One finding is that organizations who do disaster recovery testing and create and implement action items based on what they learn from it are more likely to be in the elite performing group.
The report delves into the “Four Key Metrics” of software development and delivery, one of which is culture – too often overlooked by organizations. To improve, organizations need to start with the foundations, and adopt a continuous improvement mindset.
Read the report and help your own team be one of the many teams who are continuing to improve their ability to deploy small changes to production frequently, while enjoying their work.
A continual problem that we must deal with when we run automated tests in our DevOps pipeline is maintenance. I could spend hours sharing posts and blogs about this matter, telling stories about how it affected our team’s capacity and time to market, and giving you several examples of what teams have done to sort this problem out. In fact, all the effort that we invest in maintenance is not visible to the business stakeholders’ eyes. Maintenance consumes our team capacity, but it does not add value to our clients directly.
Test automation challenges at scale
In one of my jobs wefaced this problem on a large scale. We had more an average of more than 1000automated tests running at each code check in, plus one big run at midnightthat we called full regression. Our suites where organized in those levels:
After each run, we noticed that 80% of UI and regression automated tests failed
due
fake failures. After reading my numbers, you can say if you automated crap, you will get crap results and that is correct — but please remember that we were building our automated test suites, with a continuous testing approach and DevOps mindset and pipelines, so we were trying to fail fast, learn and improve. First, we decided to analyze our test results and data, so we collected all the information that we got from our test runs and analyzed, we discovered that our tests failed
because
of those main reasons:
Changes inUI components.
Badcoding.
Exceptionsand errors handling
Time outseverywhere
Data
A new experiment
We had to make a decision. We had invested lot of efforts in testing automation, and we were the product team with more automated tests compare to the rest of product teams globally in the company. This brought us attention from lots of managers, who were keeping their eyes on our DevOps and test automation activities. In addition, we were participating in a DevOps competition which involved all the IT teams of the corporation. So, we decided to invest time to get the failed tests in good shape and have them passing in the pipeline. We called this mindset “Keep them green and running”. We communicated this effort to all the product teams. We asked the Product Owners to consider maintenance time as part of their team capacity, include tasks for this purpose, and keep it part of their definitions of “ready” and “done”.
Unfortunately, we did notachieve the goals we had hoped for. We still had the same test results, withmore than 80% of tests failing. In almost every case, they were fake failuresthat meant that the tests were stopped long before their last steps. To sum up,our rest results were not accurate at all and did not bring confidence to ourteams — which is something critical in testing automation.
Thinking outside the box
We needed to try something different. After some discussion, research, investigation and thinking out of the box, we decided to use gamification to keep our tests in a good shape, running and be green. We named the game “Game of Testing” and we established some critical points about it:
Maturitylevels: The gamers can be developers, testers or anybody who owned an automatedtest running in the pipeline, they were maturing in the game, from a beginnerlevel to a master level.
Visibility:We shared the game boards including ranking, test results and so, on using bigmonitors in the office, allowing everyone to be aware of the gamers andresults.
Weight:This is a very complicated and mathematical part, we needed to stablishformulas and weights for a point system for the game, so we could evaluate thetest results, data per gamer and teams, and gave points and make them get newlevels.
Recognition:We tried to recognize all the gamers taking part. For the best gamers, weshared the results with local and global senior managers, so their achievementscould be known and recognized. Thishelped encourage more team members to participate.
The game was quitesimple:
Gamers: All the team members who owned an automated test running in the pipeline.
Point system: A gamer won
point
s for test running in green and negative points if the test failed for fake results. Later we added negative points of breaking builds.
Rules: You needed to keep all your
test
and team tests running in the pipeline in a good shape for each run. So, you had to invest efforts on “maintenance” and continuous improvement if you want to rock in the game.
The Game of Testing
This approach helps usa lot to decrease false positive runs and keep the tests up and running in thepipeline. We decreased the number of 80% of failed tests to only 20% inapproximately three months and our numbers have continued decreasing. Furthermore, people had fun while playing thegame and they internalized that quality can be fun and not boring like usualmaintenance was.
While we got several criticisms about our failing tests at the beginning of our efforts to improve them, and people lost trust on our quality assurance process, we didn’t give up trying to fix the problems. In fact, it motivated our team to think out of the box and turn this problem into a useful and fun project. We spent several hours investigating continuous testing, scripting, and related topics. Besides that, we wanted to do something different, something motivates other team members to join.
Gamification can provide that motivation. You may not achieve desired results immediately, but if you prepare your environment and involve everyone in trying potential solutions, the game will give you results for sure. If you address the test issues you can get them pass or fail, but if you involve and motivate people to work on problems together, you always win and have fun. I recommend you try gamification to sort your problems out, you may be surprised at the results!
Jorge Luis Castro Toribio
Jorge is an
agilist
who feels passion for Testing Automation, Agile, DevOps and business agility. He enjoys research and learning about business agility and working with cutting-edge technologies. He has been learning continuously for more than 12 years and is still learning new ways to foster business agility and team greatness. He has worked in several roles (developer, tester, manager, agile coach) which helps him to see the big picture and operational and team members dynamics happening inside organizations. This experience lets him help teams design, build and implement digital, DevOps and agile transformation strategies. He encourages focus on: People, Productivity, Continuous improvement, Innovation and having fun to enjoy success in our agile journeys.
One fine day disruption was created inour organisation as part of digital transformation with few people fromoperations joining us – Developer, Tester, Product-owner.
…And we all moved into a NEW world ofDevOps. There were lot of uncertainties about what it would be like? Lot ofquestions whether there will be any role as tester? If there is what that rolewould involve?
As we all moved into DevOps world, allthese questions were getting clear. Our application demanded lots ofnon-functional testing along-with functional. In a DevOps world, it was allabout getting the feedback early and do frequent releases with a good quality.For a tester what it meant for me is to learn the process of continuousintegration and delivery and the testing for not just functional but also fornon-functional. Non-functional testing involved Security, Performance, Visual,Accessibility and reduce dependency on external teams to carry-out thesetesting to delivery faster. In short a Swiss Army Knife’s worth of skills (knowingdifferent types of testing). To top that it should be automated for quick-releases,so enough time is spent in doing exploratory testing and finding the unknowns.
To address that I came-up with the test-automation pyramid model, as illustrated by
to help team visualize & discuss test automation strategy. I will share how I modelled the test-automation pyramid to suit our DevOps world and overview all the testing involved in terms of functional & non-functional in an automated way to deliver faster without compromising on quality.
There are so many forms & interpretations of Test automation pyramid. All point out the same thing it should have more no. of unit tests, followed by service (consisting of API, Integration, Component) tests and finally UI tests.
There’s this another way of looking test pyramid as a Bug-Filter.
But have you noticed it doesn’t talk aboutnon-functional testing. Does that mean non-functional testing are not automatedor part of test automation? For me test automation stands for – tests which areautomated, run by machine/tool scripts and asserted pass/fail on ourexpectations.
At least it should say Functional Testpyramid. But I can see functional test automation is the most widelyimplemented form of automation and it has kind-of become synonymous with test automation.Also, Manual/Exploratory Testing bit is not captured properly. Exploratory Testingcan happen at any level – Unit, API, UI and for different types – Security,Performance, etc.
The test pyramid is an effective visualisationform outlining your testing approach which can be easily explained to others.Just make it visible on a board/wall and you have a very good indicator of howyou want to organise your testing efforts in one picture. It is a better visualreplacement for long test strategy documents.
As earlier mentioned, in our application we have lot of non-functional testing. So, I have taken these existing models and made changes to better suit our application. I will now explain how I view this test pyramid and apply it to testing in DevOps.
Test Automation Pyramid re-visited
So here is my interpretation.
Software testing is about identifyingRISKS and not finding bugs. Based on this, Imagine Test Pyramid to beupside-down as a RISK Filter, having all the different types of testing –Functional in UI, Service, Unit. Non-functional in Performance, Security and Others.
Now imagine the newly developedproduct or feature enters the pyramid for identifying risks, meaning carryingout functional tests in UI, Service, Unit-level and identifying functionalrisks.
Similarly, performance testing foridentifying performance risks, security testing for identifying security risks,and so on for other risks.
All these risks identified are byautomation meaning asserting our expectations or checking known behaviours. Whatabout Unknowns or new information. These will be identified by Exploratorytesting at every level – Unit, API, UI and for different types – Security,Performance, etc.
Do all the identified risks need to befixed? Absolutely NO. The team will prioritise it as high, medium, low etc. andfixed as per priority. Once all the prioritised risks are fixed then releasethe product/feature with known risks.
This is how test pyramid is applied in our case in testing in DevOps, making sure all the testings are done at each and every level from RISK perspective to achieve quality and deliver faster.
Functional Automation
Let’s talk about functional test automation. I will be just covering UI testing. There are so many functional test-automation tools, most popular ones being Selenium, Sahi, Katalon, Ranorex, NightwatchJS. And to implement best practices test there are frameworks such as Page-Object Models, BDD, TDD, Keyword & Data driven, Hybrid and so on. Depending upon your preference select the framework suiting your needs. There are no good or bad frameworks, it’s all about what best suits your application and your team. Same goes with the tools.
Non-Functional Testing
Quality of a software depends not onlyon Functional but Non-functional testing such as Security, Performance, VisualUI, Accessibility and so on.
Security Testing
Securitytesting is a process of identifying weakness in the security mechanisms ofan application that protects data and maintain functionality asintended. Purpose is to make the application immune to SQL injections,cross-site scripting, sensitive data exposure to protect Unauthorized andUnauthenticated users to enter in to your application. All these can bedriven through a tool and tested through test-automation.
Thereare various tools available where you can automate security tests and run aspart of CI/CD process. This automated activity necessarily doesn’t removemanual Penetration testing requirement. Through automation, it is beneficial tofind basic vulnerabilities in our application as much as possible so thatSecurity Specialists/Penetration testers can focus on identifying high-hangingbugs. This helps in delivering at a faster pace by finding basicvulnerabilities early in development phase which in turn saves Penetrationtester’s time so that they can concentrate where it adds value.
Performance Testing
Performance testing is about identifyingperformance issues such as slow response time, support for concurrent users,how much load the application can handle? Does the application crash? Also, it involvesnot only testing the server-side but the client-side as well. As can be seenfrom the chart, 80% of the end-user response time is spent on the front-end.Most of this time is tied up in downloading all the components of the page – images,stylesheets, scripts, Flash, etc.
To verify if an application is fast and efficient enough we perform client-side performance tests. This means checking the response time of a web application from the point of view of a single user. Both server and client side can be automated with pass/fail criterion if it doesn’t match the performance benchmarks. Usually performance tests are run once the functional tests are done and/or just before few weeks before the release. But executing performance tests early whenever new API is built (Server-side) and a new page is built (Client-side) helps to get early feedback and reduce any delays incurring from the issues identified later in the development phase.
Visual Testing
Visual UI testing consists about
Does your UI look the way it should, to allyour users?
Visual UI testing is about verifying that UI appears correctlyto your users. For detecting correct icon-size, padding, colour, layout, text,element position etc. It’s a quick way to check all these through visualautomation. It saves time and eye strain.
Angie Jones:
Even if the application passes functional testing, we still must check the look and feel of UI because the element might be not aligned properly or hidden behind some other element which the end user is not able to see. Visual UI testing is the best way to identify such issues and if integrated with functional tests gives a lot more confidence about the UI. This can be easily seen in the below image. Functional tests for this would pass but not Visual UI testing.
Accessibility Testing
Accessibility testing is a subsetof usability testing, and it is performed to ensure that the applicationbeing tested is usable by all the people and not only for people with disabilities.Accessibility is about ensuring inclusive applications are built for one andall, making an application equally accessible and usable for a normal user too.Purpose is to provide everyone with equal access to the application and meetingthe standards set in web content accessibility guidelines –
WCAG
.
People with disability will use navigation strategies like tabbing through the headings, links on webpages, images and videos with properly described text for accessing. So as a tester we need to test whether it follows the standard or meet the compliance having proper tab flow, recommended headings, alternate text for images, videos etc. Accessibility testing can also be automated to check whether the application has elements well structured, alternate texts and contents are described well or not.
Exploratory Testing
Exploratorytesting is the key to high quality software, it’s highly intellectual and time-consumingactivity. It is an investigative approach and main intent is to uncover newinformation or covering the unknowns in all areas of testing from functional,security, performance, visual and accessibility or any other types of testing. Oftenthis activity is pushed off to end-users.
The image below shows how testing from a risk perspective can be used to bridge the gap between automated testing (asserting expectations) and manual/exploratory testing (uncovering new information), so that lot of unknowns will be explored and converted into automated known behaviours to improve the quality. Exploratory testing helps in creating good automation tests to figure out all the things you want to go and check.
Continuous Testing
Continuous testing
right from requirement to production and taking feedback from production to requirement, this loop of continuous testing is key to achieve highest quality. Below image shows all the different types of testing we perform in various stages of DevOps.
Conclusion
As explained in my DevOps Testingpyramid, testing should always be done from a risk perspective and identifiedrisks should be prioritised and fixed accordingly for delivering qualityproduct faster.
In DevOps, automated testing is key tospeed-up the development cycle. By automation, I mean non-functional testing aswell. In a DevOps world, tester should possess Swiss Army Knife’s worth ofskills, which means as a tester it’s our responsibility to do all otherrelevant types of testing as much as possible apart from functional testing toget early feedback and not depend on external teams. Depending on externalteams might increase the risk for not getting the feedback early and delay therelease. Also, don’t forget to put all the tests including non-functional inCI/CD pipeline to get early and constant feedback on quality.
Continuous testing in all the stages iskey to achieve highest quality. It’s important to be tool aware and practiceidentifying good tools and use them whatever suits to your application and toyour team. Exploratory testing is the key to high quality software and pleasedon’t push it to end-users. Utilize as much time as possible in exploring theapplication for uncovering new information for improving quality.
Lastly, no matter what strategy andprocesses you have the team should have Quality mindset to go extra mile andadd that extra effort to deliver quality to the end user. This is accomplishedby Continuous Testing and doing all the relevant functional and non-functionaltesting at every stage so that
quality is cascaded
down to theend-user.
Testers have many diverse capabilities – just like a Swiss Army Knife!
I would like to end my blog – DevOpsis all about early feedback and frequent releases and for that
Testers should possess a Swiss Army Knife’sworth of skills, versatile and approaching the application from all the differentaspects of testing!!!
About the author
Amit Kulkarni
Amit Kulkarni (@amtoya on Twitter) is a test automation specialist and has over 10 years of experience delivering quality product for clients in Australia, Europe, Japan and USA. Amit is passionate about test automation for both functional and non-functional including security and performance testing. Amit has a diversified experience in Semiconductor technology, Embedded and Software industry in the field of Banking, Healthcare and Consumer Electronics working for companies such as Fujitsu, Mitsubishi, Qualcomm and Telstra. Amit is currently working as a Senior Quality Engineer in Bankwest and driving best quality practices in test automation and continuous testing to deliver quality product faster.
I was recently looking at the world of DevOps: the sheer number of potential changes and practices that shift both the way the company works, and the way things are approached are not easy! There are a few well-known names with a huge following that seem to attend every major conference, and are a great source of knowledge. These folks are there, advocating for the changes that are needed, and are a grand source of ready information for businesses and practitioners that need information on how a specific technique might look in the future.
Photo by
on
And, thankfully, most of them are capable of sharing their energy and excitement — which gets the team and management to be willing to try something new. This is an important skill: any change cannot be done effectively by just a few people. The more you are able to show what a change looks like — what to expect as blockages, things that worked to improve someone else’s situation – makes the whole process still seem magical, but with enough encouragement that things may not go so well at first, but this will work!
The downside to this comes quietly — business and other people have come to view them as “this is what DevOps is”, and try to hire for only the qualities they see in them, but for their specific situation as well. And without a grounding in the current setup and the problems that the company needs to solve (and the things they want to do, which may not be the same at all!), no one can sound expert and confident without a bit of blarney in their presentation. And those that are able to persuade in this fashion might not be the best choice, in the long run.
But, the folks I am looking for to provide guidance are those thousands of folks that are not as well-known: the people that make the cycle of DevOps work for their business, and are successful without making a large splash in terms of followers, or even of videos and blogs. Their companies would be lost without these people — and these are the ones I would love to learn from.
These folks can be at any level in any business. They are the ones that are willing to be a bit stubborn, and not give up after the first time, or even after a few failures, because they have a goal that they want to reach. Finding out options, and getting others to learn about them is a sure sign you have someone I would want to work with – they are focused on getting better.
Improving through small experiments
They are the ones that will make a small change, and see how it goes. Changing too many things at once is both difficult to track effectiveness, and builds resistance to trying something new if some changes make things worse in the short term.
Photo by
on
Rooting out the base cause of a problem is a thankless task at times — it’s so much quicker to put a temporary fix in, and move the real issue to the TODO list: and might cause fewer side effects than fixing the actual problem. But having someone devote the time to finding the issue, or at least systematically eliminating where it is not (and fixing small things as they go through) will allow the gradual improvement.
Working with other sections of the business is a needed skill for DevOps — being able to share problems, options, and ideas is one that helps everyone be aware of the issues. Using a good idea learned from another person may be the best possible outcome — you not only may have cured their issue as well, but you have both one less problem, and a new outlook next time a similar situation occurs.
Practitioners: Thanks for sharing your experiences!
So, for all of you who have taken on the task of making sure that the feedback is fast and accurate, the changes improve quality, and problems big and small are addressed when they occur — thank you. Learning from you, either through informal questioning, or other methods such as blog posts, is a pleasure. Finding a new post by someone who is skilled at these tasks is always a delight, as is a new writer whose knowledge is evidenced in their approach and tone. You may not have the answers, but at least you can ask the questions that will lead to them.
