By Amit Kulkarni
One fine day disruption was created inour organisation as part of digital transformation with few people fromoperations joining us – Developer, Tester, Product-owner.
…And we all moved into a NEW world ofDevOps. There were lot of uncertainties about what it would be like? Lot ofquestions whether there will be any role as tester? If there is what that rolewould involve?
As we all moved into DevOps world, allthese questions were getting clear. Our application demanded lots ofnon-functional testing along-with functional. In a DevOps world, it was allabout getting the feedback early and do frequent releases with a good quality.For a tester what it meant for me is to learn the process of continuousintegration and delivery and the testing for not just functional but also fornon-functional. Non-functional testing involved Security, Performance, Visual,Accessibility and reduce dependency on external teams to carry-out thesetesting to delivery faster. In short a Swiss Army Knife’s worth of skills (knowingdifferent types of testing). To top that it should be automated for quick-releases,so enough time is spent in doing exploratory testing and finding the unknowns.
To address that I came-up with the test-automation pyramid model, as illustrated by to help team visualize & discuss test automation strategy. I will share how I modelled the test-automation pyramid to suit our DevOps world and overview all the testing involved in terms of functional & non-functional in an automated way to deliver faster without compromising on quality.
Test Pyramid:
There are so many forms & interpretations of Test automation pyramid. All point out the same thing it should have more no. of unit tests, followed by service (consisting of API, Integration, Component) tests and finally UI tests.
There’s this another way of looking test pyramid as a Bug-Filter.
But have you noticed it doesn’t talk aboutnon-functional testing. Does that mean non-functional testing are not automatedor part of test automation? For me test automation stands for – tests which areautomated, run by machine/tool scripts and asserted pass/fail on ourexpectations.
At least it should say Functional Testpyramid. But I can see functional test automation is the most widelyimplemented form of automation and it has kind-of become synonymous with test automation.Also, Manual/Exploratory Testing bit is not captured properly. Exploratory Testingcan happen at any level – Unit, API, UI and for different types – Security,Performance, etc.
The test pyramid is an effective visualisationform outlining your testing approach which can be easily explained to others.Just make it visible on a board/wall and you have a very good indicator of howyou want to organise your testing efforts in one picture. It is a better visualreplacement for long test strategy documents.
As earlier mentioned, in our application we have lot of non-functional testing. So, I have taken these existing models and made changes to better suit our application. I will now explain how I view this test pyramid and apply it to testing in DevOps.
Test Automation Pyramid re-visited
So here is my interpretation.
Software testing is about identifyingRISKS and not finding bugs. Based on this, Imagine Test Pyramid to beupside-down as a RISK Filter, having all the different types of testing –Functional in UI, Service, Unit. Non-functional in Performance, Security and Others.
Now imagine the newly developedproduct or feature enters the pyramid for identifying risks, meaning carryingout functional tests in UI, Service, Unit-level and identifying functionalrisks.
Similarly, performance testing foridentifying performance risks, security testing for identifying security risks,and so on for other risks.
All these risks identified are byautomation meaning asserting our expectations or checking known behaviours. Whatabout Unknowns or new information. These will be identified by Exploratorytesting at every level – Unit, API, UI and for different types – Security,Performance, etc.
Do all the identified risks need to befixed? Absolutely NO. The team will prioritise it as high, medium, low etc. andfixed as per priority. Once all the prioritised risks are fixed then releasethe product/feature with known risks.
This is how test pyramid is applied in our case in testing in DevOps, making sure all the testings are done at each and every level from RISK perspective to achieve quality and deliver faster.
Functional Automation
Let’s talk about functional test automation. I will be just covering UI testing. There are so many functional test-automation tools, most popular ones being Selenium, Sahi, Katalon, Ranorex, NightwatchJS. And to implement best practices test there are frameworks such as Page-Object Models, BDD, TDD, Keyword & Data driven, Hybrid and so on. Depending upon your preference select the framework suiting your needs. There are no good or bad frameworks, it’s all about what best suits your application and your team. Same goes with the tools.
Non-Functional Testing
Quality of a software depends not onlyon Functional but Non-functional testing such as Security, Performance, VisualUI, Accessibility and so on.
Security Testing
Securitytesting is a process of identifying weakness in the security mechanisms ofan application that protects data and maintain functionality asintended. Purpose is to make the application immune to SQL injections,cross-site scripting, sensitive data exposure to protect Unauthorized andUnauthenticated users to enter in to your application. All these can bedriven through a tool and tested through test-automation.
Thereare various tools available where you can automate security tests and run aspart of CI/CD process. This automated activity necessarily doesn’t removemanual Penetration testing requirement. Through automation, it is beneficial tofind basic vulnerabilities in our application as much as possible so thatSecurity Specialists/Penetration testers can focus on identifying high-hangingbugs. This helps in delivering at a faster pace by finding basicvulnerabilities early in development phase which in turn saves Penetrationtester’s time so that they can concentrate where it adds value.
Performance Testing
Performance testing is about identifyingperformance issues such as slow response time, support for concurrent users,how much load the application can handle? Does the application crash? Also, it involvesnot only testing the server-side but the client-side as well. As can be seenfrom the chart, 80% of the end-user response time is spent on the front-end.Most of this time is tied up in downloading all the components of the page – images,stylesheets, scripts, Flash, etc.
To verify if an application is fast and efficient enough we perform client-side performance tests. This means checking the response time of a web application from the point of view of a single user. Both server and client side can be automated with pass/fail criterion if it doesn’t match the performance benchmarks. Usually performance tests are run once the functional tests are done and/or just before few weeks before the release. But executing performance tests early whenever new API is built (Server-side) and a new page is built (Client-side) helps to get early feedback and reduce any delays incurring from the issues identified later in the development phase.
Visual Testing
Visual UI testing consists about Does your UI look the way it should, to allyour users? Visual UI testing is about verifying that UI appears correctlyto your users. For detecting correct icon-size, padding, colour, layout, text,element position etc. It’s a quick way to check all these through visualautomation. It saves time and eye strain.
Even if the application passes functional testing, we still must check the look and feel of UI because the element might be not aligned properly or hidden behind some other element which the end user is not able to see. Visual UI testing is the best way to identify such issues and if integrated with functional tests gives a lot more confidence about the UI. This can be easily seen in the below image. Functional tests for this would pass but not Visual UI testing.
Accessibility Testing
Accessibility testing is a subsetof usability testing, and it is performed to ensure that the applicationbeing tested is usable by all the people and not only for people with disabilities.Accessibility is about ensuring inclusive applications are built for one andall, making an application equally accessible and usable for a normal user too.Purpose is to provide everyone with equal access to the application and meetingthe standards set in web content accessibility guidelines – WCAG .
People with disability will use navigation strategies like tabbing through the headings, links on webpages, images and videos with properly described text for accessing. So as a tester we need to test whether it follows the standard or meet the compliance having proper tab flow, recommended headings, alternate text for images, videos etc. Accessibility testing can also be automated to check whether the application has elements well structured, alternate texts and contents are described well or not.
Exploratory Testing
Exploratorytesting is the key to high quality software, it’s highly intellectual and time-consumingactivity. It is an investigative approach and main intent is to uncover newinformation or covering the unknowns in all areas of testing from functional,security, performance, visual and accessibility or any other types of testing. Oftenthis activity is pushed off to end-users.
The image below shows how testing from a risk perspective can be used to bridge the gap between automated testing (asserting expectations) and manual/exploratory testing (uncovering new information), so that lot of unknowns will be explored and converted into automated known behaviours to improve the quality. Exploratory testing helps in creating good automation tests to figure out all the things you want to go and check.
Continuous Testing
Continuous testing right from requirement to production and taking feedback from production to requirement, this loop of continuous testing is key to achieve highest quality. Below image shows all the different types of testing we perform in various stages of DevOps.
Conclusion
As explained in my DevOps Testingpyramid, testing should always be done from a risk perspective and identifiedrisks should be prioritised and fixed accordingly for delivering qualityproduct faster.
In DevOps, automated testing is key tospeed-up the development cycle. By automation, I mean non-functional testing aswell. In a DevOps world, tester should possess Swiss Army Knife’s worth ofskills, which means as a tester it’s our responsibility to do all otherrelevant types of testing as much as possible apart from functional testing toget early feedback and not depend on external teams. Depending on externalteams might increase the risk for not getting the feedback early and delay therelease. Also, don’t forget to put all the tests including non-functional inCI/CD pipeline to get early and constant feedback on quality.
Continuous testing in all the stages iskey to achieve highest quality. It’s important to be tool aware and practiceidentifying good tools and use them whatever suits to your application and toyour team. Exploratory testing is the key to high quality software and pleasedon’t push it to end-users. Utilize as much time as possible in exploring theapplication for uncovering new information for improving quality.
Lastly, no matter what strategy andprocesses you have the team should have Quality mindset to go extra mile andadd that extra effort to deliver quality to the end user. This is accomplishedby Continuous Testing and doing all the relevant functional and non-functionaltesting at every stage so that quality is cascaded down to theend-user.
I would like to end my blog – DevOpsis all about early feedback and frequent releases and for that Testers should possess a Swiss Army Knife’sworth of skills, versatile and approaching the application from all the differentaspects of testing!!!
About the author
Amit Kulkarni (@amtoya on Twitter) is a test automation specialist and has over 10 years of experience delivering quality product for clients in Australia, Europe, Japan and USA. Amit is passionate about test automation for both functional and non-functional including security and performance testing. Amit has a diversified experience in Semiconductor technology, Embedded and Software industry in the field of Banking, Healthcare and Consumer Electronics working for companies such as Fujitsu, Mitsubishi, Qualcomm and Telstra. Amit is currently working as a Senior Quality Engineer in Bankwest and driving best quality practices in test automation and continuous testing to deliver quality product faster.