A place to share ideas, techniques, resources about all the testing activities needed to feel confident about delivering value to our customers frequently and sustainably.
Even when I worked on waterfall teams, we tested early and often. We got ourselves invited to analysis and requirements meetings. We tested requirements docs. We asked questions. We tested as the developers delivered code, even before “code freeze”. We also worked closely with customer support to learn about how customers used the features after they were delivered. (I realize not all teams using a phased-and-gated approach do all that, but that approach should not stop teams from using good practices!)
Extreme Programming (XP) made so much sense to me because it embraced testing and quality as the responsibility of the entire delivery team. In agile development, we focus on building quality in, we test ideas, we automate regression tests, we explore delivered code, we test in production. These same practices apply to teams who use the “DevOps” label as well.
That’s why Dan Ashby’s
post in 2019 on “continuous testing”
resonated so much with me. His visual model captures what is in my brain when I see the term “continuous testing”. With his permission, I often use this model to help others understand the concept:
Dan Ashby’s Continuous Testing Model
But there are different definitions!
WAT!?
Recently, someone posted a link to a short Twitter thread from earlier this year where continuous testing was defined quite differently. Basically, this alternative definition is only about running automated tests in the continuous integration, delivery, deployment pipelines. I started asking around to see how other people define “continuous testing”, and some people shared that definition –
only the automated suites in CI/CD.
A book on “continuous testing” which I had not gotten around to reading also has this definition, and it references the Wikipedia page which says the same thing.
Automated tests in CI are not new!
Continuous integration has been around for decades. My waterfall team back in the early 90s did CI, complete with automated regression tests and automated deployments (to 20+ Unix environments!) We didn’t call the automated regression tests “continuous testing”. We called them automated regression tests. It puzzles me that people would apply the term “continuous testing” to these automated test suites, which teams have been using since long before the term “continuous delivery” appeared.
Going to the source
I learned about continuous delivery by reading the terrific
book
of the same name by Jez Humble and David Farley. If you read it, you will find it is a book about testing. It goes into much detail about many different kinds of testing activities – not only automated regression tests!
So I went to Jez’s website to see how he
defines “continuous testing”
. Please click that link and read it for yourself. To quote Jez,
Our goal is to run many different types of tests—both manual and automated—continually throughout the delivery process.
That is followed by a picture of Brian Marick’s agile testing quadrants, illustrating a range of tests that are business-facing, technology-facing, that guide development and that critique the product. In Jez’s quadrants model, the tests that critique the product include manual tests.
Jez goes on to explain the concept of the deployment pipeline and the importance of getting quick feedback from automated acceptance tests that are in the pipeline. But he does not limit “continuous testing” to only those tests.
Here’s the latest version of the agile testing quadrants model that
Janet Gregory
and I use. We’ve added examples of testing activities we can do today thanks to newer technology and tools, such as monitoring and observability. Those are also part of continuous testing.
Agile Testing Quadrants from
Agile Testing Condensed: A Brief Introduction
by Janet Gregory and Lisa Crispin
Does the terminology matter?
I don’t like to be too pedantic about terms, but we need to be able to communicate with each other. If my idea of continuous delivery was limited to only the pipeline and did not include the concept of breaking features into small slices and deploying small changes frequently, that would not match what most practitioners I know consider to be “continuous delivery”. It would be hard to have a conversation about it with two very different definitions.
Many people use the term “automated regression checks” rather than “automation regression tests” because these tests are checking that requirements are met and that established behavior continues to work as expected. They reserve the term “testing” for activities that are done by humans. Janet suggested that maybe we can refer to the automated regression test suites in CI / CD pipelines as “continuous checking”, and reserve the term “continuous testing” to include those checks PLUS all the testing activities that happen around Dan’s infinite testing loop.
How can we resolve this terminology dilemma? Feel free to comment here. There’s also a discussion topic for this on the
Ministry of Testing Club
.
Meanwhile, I shall see if there are even more definitions of continuous testing I hadn’t heard yet!
A continual problem that we must deal with when we run automated tests in our DevOps pipeline is maintenance. I could spend hours sharing posts and blogs about this matter, telling stories about how it affected our team’s capacity and time to market, and giving you several examples of what teams have done to sort this problem out. In fact, all the effort that we invest in maintenance is not visible to the business stakeholders’ eyes. Maintenance consumes our team capacity, but it does not add value to our clients directly.
Test automation challenges at scale
In one of my jobs wefaced this problem on a large scale. We had more an average of more than 1000automated tests running at each code check in, plus one big run at midnightthat we called full regression. Our suites where organized in those levels:
After each run, we noticed that 80% of UI and regression automated tests failed
due
fake failures. After reading my numbers, you can say if you automated crap, you will get crap results and that is correct — but please remember that we were building our automated test suites, with a continuous testing approach and DevOps mindset and pipelines, so we were trying to fail fast, learn and improve. First, we decided to analyze our test results and data, so we collected all the information that we got from our test runs and analyzed, we discovered that our tests failed
because
of those main reasons:
Changes inUI components.
Badcoding.
Exceptionsand errors handling
Time outseverywhere
Data
A new experiment
We had to make a decision. We had invested lot of efforts in testing automation, and we were the product team with more automated tests compare to the rest of product teams globally in the company. This brought us attention from lots of managers, who were keeping their eyes on our DevOps and test automation activities. In addition, we were participating in a DevOps competition which involved all the IT teams of the corporation. So, we decided to invest time to get the failed tests in good shape and have them passing in the pipeline. We called this mindset “Keep them green and running”. We communicated this effort to all the product teams. We asked the Product Owners to consider maintenance time as part of their team capacity, include tasks for this purpose, and keep it part of their definitions of “ready” and “done”.
Unfortunately, we did notachieve the goals we had hoped for. We still had the same test results, withmore than 80% of tests failing. In almost every case, they were fake failuresthat meant that the tests were stopped long before their last steps. To sum up,our rest results were not accurate at all and did not bring confidence to ourteams — which is something critical in testing automation.
Thinking outside the box
We needed to try something different. After some discussion, research, investigation and thinking out of the box, we decided to use gamification to keep our tests in a good shape, running and be green. We named the game “Game of Testing” and we established some critical points about it:
Maturitylevels: The gamers can be developers, testers or anybody who owned an automatedtest running in the pipeline, they were maturing in the game, from a beginnerlevel to a master level.
Visibility:We shared the game boards including ranking, test results and so, on using bigmonitors in the office, allowing everyone to be aware of the gamers andresults.
Weight:This is a very complicated and mathematical part, we needed to stablishformulas and weights for a point system for the game, so we could evaluate thetest results, data per gamer and teams, and gave points and make them get newlevels.
Recognition:We tried to recognize all the gamers taking part. For the best gamers, weshared the results with local and global senior managers, so their achievementscould be known and recognized. Thishelped encourage more team members to participate.
The game was quitesimple:
Gamers: All the team members who owned an automated test running in the pipeline.
Point system: A gamer won
point
s for test running in green and negative points if the test failed for fake results. Later we added negative points of breaking builds.
Rules: You needed to keep all your
test
and team tests running in the pipeline in a good shape for each run. So, you had to invest efforts on “maintenance” and continuous improvement if you want to rock in the game.
The Game of Testing
This approach helps usa lot to decrease false positive runs and keep the tests up and running in thepipeline. We decreased the number of 80% of failed tests to only 20% inapproximately three months and our numbers have continued decreasing. Furthermore, people had fun while playing thegame and they internalized that quality can be fun and not boring like usualmaintenance was.
While we got several criticisms about our failing tests at the beginning of our efforts to improve them, and people lost trust on our quality assurance process, we didn’t give up trying to fix the problems. In fact, it motivated our team to think out of the box and turn this problem into a useful and fun project. We spent several hours investigating continuous testing, scripting, and related topics. Besides that, we wanted to do something different, something motivates other team members to join.
Gamification can provide that motivation. You may not achieve desired results immediately, but if you prepare your environment and involve everyone in trying potential solutions, the game will give you results for sure. If you address the test issues you can get them pass or fail, but if you involve and motivate people to work on problems together, you always win and have fun. I recommend you try gamification to sort your problems out, you may be surprised at the results!
Jorge Luis Castro Toribio
Jorge is an
agilist
who feels passion for Testing Automation, Agile, DevOps and business agility. He enjoys research and learning about business agility and working with cutting-edge technologies. He has been learning continuously for more than 12 years and is still learning new ways to foster business agility and team greatness. He has worked in several roles (developer, tester, manager, agile coach) which helps him to see the big picture and operational and team members dynamics happening inside organizations. This experience lets him help teams design, build and implement digital, DevOps and agile transformation strategies. He encourages focus on: People, Productivity, Continuous improvement, Innovation and having fun to enjoy success in our agile journeys.
You can find Jorge on and .
Visualizing Pipelines to Help Build a Quality Culture
By Lisa Crispin
At
CAST 2019
, August 12-15 in Cocoa Beach, Florida, and I are pairing to facilitate an all-day workshop called “The Whole Team Approach to Testing in Continuous Delivery”. Our focus will be helping people to find new ways their teams can build a quality culture and succeed with continuous delivery. An important part of this is learning about deployment pipelines. For those of you who can’t attend the live version, here’s an introduction to why deployment pipelines are vital to successful continuous delivery, and a fun exercise your team can use to improve your own pipeline.
Continuous delivery – scary?
Continuous delivery (CD) lets us deliver new value to production safely, quickly and sustainably (to paraphrase Jez Humble). Each new commit to our code base trunk or master is independently verified as a deployable release candidate. We can deliver frequently, which means the changes are small, reducing the risk of problems.
Still, CD can seem like a scary proposition. In the old waterfall days, we didn’t have enough time to complete all the testing activities we wanted. Agile development enabled more bi-weekly or weekly releases, and brought up new testing challenges. And now, how can testing possibly keep up with continuous delivery? How can we feel confident about each release, from a testing/customer support perspective?
New approaches
Following the
principles of continuous delivery
helps us build confidence in our deployments. Following
modern testing
principles can give us even more confidence. When the whole team takes responsibility for building quality into the product and continually improving how we work, we (and our customers) realize the benefits of CD.
This is a huge topic, but let’s look at two key factors for success with CD. Obviously, we need adequate automated regression test coverage to make sure that the things our customers really need are still working. That’s not new. And, it isn’t a complete safety net. We can have great automated regression test coverage, do thorough exploratory testing, verify performance, and all the other many types of testing before we release to production, although we are doing that in test environments. We know that behavior is likely to be different and unexpected in production.
So what is new? Today, we have the ability to capture and analyze all kinds of data (and huge amounts of it) about production use and immediately detect any errors that occur. This is important, but it’s not that helpful unless we can quickly and reliably deploy fixes to production. And to do that, we need a fast, reliable deploy pipeline.
P
ipeline basics
The deployment pipeline consists of steps that changes go through before they can be deployed to production. Every company, every team has their own unique pipeline. Note that whether or not your team has automated tests and automated deploys – you still have a pipeline. Whatever steps a new change goes through to get to production, even if each step is manual, is still a pipeline.
We break our build into stages or steps to speed up feedback. There’s a constant trade-off between fast feedback and confidence. The early steps of a pipeline are usually unit tests. They quickly show us whether anything is broken at that low, granular level. But we need more steps with higher-level, longer-running tests to feel more confident. Later steps in our pipeline exercise more complicated scenarios. We want fast feedback, but we may need comprehensive tests for confidence, and they take a long time to run. Here’s an example deployment pipeline:
In this example, the steps outlined in blue in this example are manual steps. We focus on automation when talking about pipelines, but manual steps are common. As in this example, activities like exploratory testing and security testing may be critical and may require manual work. We should be able to automate deploys to different environments, but perhaps your team just hasn’t gotten to that yet.
Things that slow down the pipeline
“Flaky” or “flappy” tests are a common problem even in completely automated pipelines. We often see this with user interface (UI) tests. Someone makes a change to the UI but forgets to update the automated tests to reflect that. A test environment slows down and causes a test step to time out. The browser version automatically (and unexpectedly) updates in the test environment. Those are just a few common causes of UI test failures.
Dependencies can also cause difficulties. A failure of one stage can stop multiple subsequent steps from running. Test data may fail to be set up properly, causing test failures. A test environment in the cloud might fail to spin up.
More steps in the pipeline give us more confidence, but they take more time. And how do we know all the steps are necessary?
When a pipeline experiences more and more delays, we humans have a tendency to increase the size of the items we’re sending through it to try to get more done. If the tests are failing and people keep committing changes, we’ve now got a big batch of changes which is more likely to include regression failures. It’s a death spiral. If our pipeline is reliable and fast, fewer changes go through it each time, reducing the risk of failures.
Collaborating to improve the pipeline
Making problems more visible is a great first step in making them smaller. Gather your team (or at least a representational group, if you have a large team) together. Take cards or paper and write the name of each step of your pipeline, one per card or paper. Put them on a table or on a wall to create a picture of your pipeline. (This can be done virtually if your team is distributed).
Visualizing a pipeline on a table – picture used with permission from Llewllyn Falco
Next, look at each step in the pipeline and ask “why do we have this step?” What are we learning from it? What business questions does it answer, what risks does it mitigate? Who on the team benefits from this information? You can capture this information on sticky notes to attach to each step.
Then, think about dependencies. For each step, what must in place for it to run successfully? A test environment? Test data? Specific configurations? Also think about how you can optimize the step. Is it currently done manually? Can you automate it?
Identify what triggers each step. Is it triggered automatically by the success of a previous step? Or manually? Can it be run in parallel with one or more other steps? Do you see opportunities to shorten feedback loops? Again, record outcomes of these discussion for each step.
Look at the length of your feedback loops. How much time elapses from code commit to completion of each step? What happens if the step fails – do the right people get notified? How? Is there a better way to communicate alerts? My previous team found that everyone was ignoring the Slack alerts generated by pipeline failures, so they hooked the build up to a police light. That got the team’s attention so that failures were quickly investigated and addressed!
Can any steps be done asynchronously? For example, it’s not realistic to do manual exploratory testing on every change when you are deploying to production frequently. The team can learn to use
release feature toggles
to hide changes from customers until the team has had time to complete any manual or time-consuming testing tasks.
Once you’ve gathered information about each step in the pipeline, choose a pain point to address and think of an experiment to reduce the pain. For example, perhaps your UI test suite has a lot of flaky tests and the team spends a lot of time kicking tests off again and diagnosing the failures that reoccur. Your experiment might be, “We believe that by refactoring the top 10 flaky tests to make them reliable will reduce our pipeline failure rate by 10% in the next month”. Use your retrospectives to evaluate your progress.
Remember the benefits
A fast, reliable pipeline means that you can confidently deploy frequent, small changes to production. And if you introduce problems to production that cause customer pain, you can shorten the time they feel pain by quickly deploying a fix with your fast, reliable pipeline. There are many other factors that go into safe, reliable and sustainable continuous delivery. Fast feedback and the ability to deploy changes quickly are a cornerstone of successful CD – and of a healthy quality culture.
It could be easier, we could have teams for each application we run, that those teams treat their production and non-production application environments the same. That they have fully automated full stack continuous deployment pipelines and the software is so well designed that there is no dependency between each application owner. Teams deliver code by the hour to production and use advance deployment techniques such as blue-green deployment coupled with highly effective monitoring, alerting and automated rollback to identify and resolve issues in production within minutes.
We do have “agile” application teams working within a cloud environment, they have growing autonomy as they build independent features and function. That’s awesome. The problem is that we have not been working that way since we started building technology. We have a sprawling on-premise (we own two data centres) environment where Non-Production environments equates for 60%+ of the overall infrastructure. This is made up of dependent applications running together in an environment.
We have followed traditional programme structure over the last 10 years, with multi-year investments and waterfall based delivery. These have sought to reduce risk through separation rather than integration. This has resulted in multiple non-production environments. Our applications (100’s) have been built tightly coupled and need multiple other applications to work (Commerce systems through to legacy mainframe systems all meshed together), increasing the size of the basic/core environment. These programmes have not invested in automation, leaving a challenging sprawling estate to manage.
Path to Production (Variety of routes followed by teams ensuring lack of consistency in versioning in each environment)
This has created a number of problems. The estate is costly to maintain. There is hidden cost in each application change or infrastructure upgrade. Our people are linked to projects and programmes, when they finish, the applications in non-production fall in to disrepair, they are costly and time consuming to get working again when the next team comes along. A continued desire for separation results in projects hunting down that unused application/environment that has nobody else working in that space. The sprawling nature and lack of ownership results in a small band of “Environment Management” who work to bridge the communication and knowledge gaps across this sprawling estate. These people work for Projects and Programmes and through “best endeavours” get Delivery teams, Operations and Infrastructure teams to stitch the environment back together each time and retain the illusion these environments are good to test in. We effectively have 10 mini-lives beating away with no real Service Management to deal with them.
That long introduction was needed just to give me time to breathe and reflect. The organisation is changing for the better, which is great. We are re-organising and realising the mistakes of the past. We are now building most new systems in a cloud environment, are focused on Micro-service design principles and have teams that own and run the service (well almost). These are small individual successes which highlight the organisation is changing. Our online teams have gone from 10 deployments a year to over 1000 in just over 18 months. It should be a factor of 10 greater by next year. However there is a long way to go till we are fully working in this way, maybe we will never get there. This is where Environment as a Product comes in. How do we change our existing non-production environment landscape for the better.
We have started with two key initiatives. Initiative 1, Let’s get support to introduce an “enabling constraint” through removing the environments that least support delivery. They are the most difficult to maintain, are off the beaten path to production and the most costly to re-enable them. If the project or programme don’t have them they can’t use them. We know they are costly to build which is part of the reason they have been maintained. With the remaining environments we are going to create a paved road of sorts. One of the key things is to not construct the path to production based on your current organisational/delivery construct. We have been very waterfall in nature and have therefore not made change often to production.
This results in us having multiple hops to production for a single change based on the type of testing to be undertaken at each stage. Development (unit, system), Integration, Release (end to end), Performance (performance, operational), Pre-Production (incident and support) and Production. You introduce delay just through deploying to each of these environments and fixing the issues that come up as a result. That’s before you think of data, types of testing, handoff implications. So initially let’s get down to one of each type to support all teams, projects and programmes. Let’s draw it up so it looks like some are optional. Development, Integration, Pre-production, Production (even this is too many).
This was difficult to get going, convince people that we might be able to go faster through having less environments, that there might be more contention initially, however that might force us to look at the organisational shape and how we design software. Whether the full extent of the idea has sunk in, is yet to be seen, however the cost avoidance articulation of maintenance running in to millions of pounds a year was hard to ignore and gave us the initial green light.
Revised simplified Path to Production
Initiative 2, is to consolidate the team we have and build a paved road (common environments, tools and approach). We have taken everyone responsible for non-production “Environment-Release Management” and centralised them in to a single Product team. This wasn’t easy. You have to convince the leadership team that they want to pay for this as a separate service, that the cost is neutral, value adding even and that in doing so you can look at how the team works and make efficiency, as well as improving the path to production. What this doesn’t do is centralise the technical people responsible (not owning) for those environments. These remain fragmented across Operations, Infrastructure and Delivery teams. While the number of different technologies, teams and ways of working make this impractical at a small team level.
It does allow you to put a very clear Service Management approach in place for Non-Production and with leadership support allow you to be more robust in enforcing a better approach. You can start to work with individual teams to support and help them approach their delivery differently, focusing on automation, repeatable process and decoupling of change. The aim is that teams do this themselves and that the Environment-Release team can identify teams that need support. We are also backed by organisational
, which have been established to give teams direction in how they should deliver.
Environment-Release Product Team
Centralising the team has also allowed us to look at how we approach managing the environments. A single Product team with longer term goals, where the value is what we do to the environments, not the end customer outcome that the Project or Programme was focused on, has a very different priority and outlook. What was several ways of working can now be focused in to a single way of working. We have started by making sense of what people were doing daily. This has been done through several value stream mapping workshops but also to get everyone’s work up on a board and try to make sense of the work we do. Get a feel for reactive vs planned work. Upon doing this we can start to build a very clear backlog of priorities for our own ways of working, how we track issues, do we manage problems, how do teams onboard on to our path to production, can we pre-empt issues, what do we do with data, looks at our skills and most importantly clear communication.
Value is now a clear driver, as is ownership. We can now mandate changes to production are part of non-production thinking. We can also start to look at cost of delay and overall costs of supporting teams and drive our own value outcomes. Lead time is also a key measure, whether that that is to measure how quickly teams move through our environments, how long it takes us to respond to incident, downtime within environments etc… We can also help to start to measure teams frequency of deployment/release and to show organisational pace of change. Traditional service management is very much focused on controls but here we want to make it much more about value outcomes and enabling flow within teams.
We also have one foot firmly in the future. Over time the need for this team will be less. Greater ownership, more end to end owning teams, “DevOps” will all drive less need for a Environment-Release Product team. Our life span is based on the value we can show to generate. However we need to develop our skills in to other areas, whether that be Platform management within our Cloud platform or in to more “DevOps” skill sets which can work within teams to help them deliver. For now we are focused on our Product and value. It would be interesting to hear how other companies and people manage non-production environments, especially where there has been ongoing development of a variety of approaches for many years.
Rob Hornby is a Product Engineer working for John Lewis based in London. He has undertaken a number of roles, over the last 20 years, within Testing, Release, Platform and Environments.
Rob is a keen believer in being able to treat all things as Products and be able to evidence the value generated by that team. Ideally everything would be delivered by small teams, delivering continuously and iterating on value. However not all teams start at this point, not all teams are the same or born equal. This is where the challenge and enjoyment comes from.
Outside of work, Rob is a keen runner, father of an amazing young girl, Sofia and works on a number of local community events.
One fine day disruption was created inour organisation as part of digital transformation with few people fromoperations joining us – Developer, Tester, Product-owner.
…And we all moved into a NEW world ofDevOps. There were lot of uncertainties about what it would be like? Lot ofquestions whether there will be any role as tester? If there is what that rolewould involve?
As we all moved into DevOps world, allthese questions were getting clear. Our application demanded lots ofnon-functional testing along-with functional. In a DevOps world, it was allabout getting the feedback early and do frequent releases with a good quality.For a tester what it meant for me is to learn the process of continuousintegration and delivery and the testing for not just functional but also fornon-functional. Non-functional testing involved Security, Performance, Visual,Accessibility and reduce dependency on external teams to carry-out thesetesting to delivery faster. In short a Swiss Army Knife’s worth of skills (knowingdifferent types of testing). To top that it should be automated for quick-releases,so enough time is spent in doing exploratory testing and finding the unknowns.
To address that I came-up with the test-automation pyramid model, as illustrated by
to help team visualize & discuss test automation strategy. I will share how I modelled the test-automation pyramid to suit our DevOps world and overview all the testing involved in terms of functional & non-functional in an automated way to deliver faster without compromising on quality.
There are so many forms & interpretations of Test automation pyramid. All point out the same thing it should have more no. of unit tests, followed by service (consisting of API, Integration, Component) tests and finally UI tests.
There’s this another way of looking test pyramid as a Bug-Filter.
But have you noticed it doesn’t talk aboutnon-functional testing. Does that mean non-functional testing are not automatedor part of test automation? For me test automation stands for – tests which areautomated, run by machine/tool scripts and asserted pass/fail on ourexpectations.
At least it should say Functional Testpyramid. But I can see functional test automation is the most widelyimplemented form of automation and it has kind-of become synonymous with test automation.Also, Manual/Exploratory Testing bit is not captured properly. Exploratory Testingcan happen at any level – Unit, API, UI and for different types – Security,Performance, etc.
The test pyramid is an effective visualisationform outlining your testing approach which can be easily explained to others.Just make it visible on a board/wall and you have a very good indicator of howyou want to organise your testing efforts in one picture. It is a better visualreplacement for long test strategy documents.
As earlier mentioned, in our application we have lot of non-functional testing. So, I have taken these existing models and made changes to better suit our application. I will now explain how I view this test pyramid and apply it to testing in DevOps.
Test Automation Pyramid re-visited
So here is my interpretation.
Software testing is about identifyingRISKS and not finding bugs. Based on this, Imagine Test Pyramid to beupside-down as a RISK Filter, having all the different types of testing –Functional in UI, Service, Unit. Non-functional in Performance, Security and Others.
Now imagine the newly developedproduct or feature enters the pyramid for identifying risks, meaning carryingout functional tests in UI, Service, Unit-level and identifying functionalrisks.
Similarly, performance testing foridentifying performance risks, security testing for identifying security risks,and so on for other risks.
All these risks identified are byautomation meaning asserting our expectations or checking known behaviours. Whatabout Unknowns or new information. These will be identified by Exploratorytesting at every level – Unit, API, UI and for different types – Security,Performance, etc.
Do all the identified risks need to befixed? Absolutely NO. The team will prioritise it as high, medium, low etc. andfixed as per priority. Once all the prioritised risks are fixed then releasethe product/feature with known risks.
This is how test pyramid is applied in our case in testing in DevOps, making sure all the testings are done at each and every level from RISK perspective to achieve quality and deliver faster.
Functional Automation
Let’s talk about functional test automation. I will be just covering UI testing. There are so many functional test-automation tools, most popular ones being Selenium, Sahi, Katalon, Ranorex, NightwatchJS. And to implement best practices test there are frameworks such as Page-Object Models, BDD, TDD, Keyword & Data driven, Hybrid and so on. Depending upon your preference select the framework suiting your needs. There are no good or bad frameworks, it’s all about what best suits your application and your team. Same goes with the tools.
Non-Functional Testing
Quality of a software depends not onlyon Functional but Non-functional testing such as Security, Performance, VisualUI, Accessibility and so on.
Security Testing
Securitytesting is a process of identifying weakness in the security mechanisms ofan application that protects data and maintain functionality asintended. Purpose is to make the application immune to SQL injections,cross-site scripting, sensitive data exposure to protect Unauthorized andUnauthenticated users to enter in to your application. All these can bedriven through a tool and tested through test-automation.
Thereare various tools available where you can automate security tests and run aspart of CI/CD process. This automated activity necessarily doesn’t removemanual Penetration testing requirement. Through automation, it is beneficial tofind basic vulnerabilities in our application as much as possible so thatSecurity Specialists/Penetration testers can focus on identifying high-hangingbugs. This helps in delivering at a faster pace by finding basicvulnerabilities early in development phase which in turn saves Penetrationtester’s time so that they can concentrate where it adds value.
Performance Testing
Performance testing is about identifyingperformance issues such as slow response time, support for concurrent users,how much load the application can handle? Does the application crash? Also, it involvesnot only testing the server-side but the client-side as well. As can be seenfrom the chart, 80% of the end-user response time is spent on the front-end.Most of this time is tied up in downloading all the components of the page – images,stylesheets, scripts, Flash, etc.
To verify if an application is fast and efficient enough we perform client-side performance tests. This means checking the response time of a web application from the point of view of a single user. Both server and client side can be automated with pass/fail criterion if it doesn’t match the performance benchmarks. Usually performance tests are run once the functional tests are done and/or just before few weeks before the release. But executing performance tests early whenever new API is built (Server-side) and a new page is built (Client-side) helps to get early feedback and reduce any delays incurring from the issues identified later in the development phase.
Visual Testing
Visual UI testing consists about
Does your UI look the way it should, to allyour users?
Visual UI testing is about verifying that UI appears correctlyto your users. For detecting correct icon-size, padding, colour, layout, text,element position etc. It’s a quick way to check all these through visualautomation. It saves time and eye strain.
Angie Jones:
Even if the application passes functional testing, we still must check the look and feel of UI because the element might be not aligned properly or hidden behind some other element which the end user is not able to see. Visual UI testing is the best way to identify such issues and if integrated with functional tests gives a lot more confidence about the UI. This can be easily seen in the below image. Functional tests for this would pass but not Visual UI testing.
Accessibility Testing
Accessibility testing is a subsetof usability testing, and it is performed to ensure that the applicationbeing tested is usable by all the people and not only for people with disabilities.Accessibility is about ensuring inclusive applications are built for one andall, making an application equally accessible and usable for a normal user too.Purpose is to provide everyone with equal access to the application and meetingthe standards set in web content accessibility guidelines –
WCAG
.
People with disability will use navigation strategies like tabbing through the headings, links on webpages, images and videos with properly described text for accessing. So as a tester we need to test whether it follows the standard or meet the compliance having proper tab flow, recommended headings, alternate text for images, videos etc. Accessibility testing can also be automated to check whether the application has elements well structured, alternate texts and contents are described well or not.
Exploratory Testing
Exploratorytesting is the key to high quality software, it’s highly intellectual and time-consumingactivity. It is an investigative approach and main intent is to uncover newinformation or covering the unknowns in all areas of testing from functional,security, performance, visual and accessibility or any other types of testing. Oftenthis activity is pushed off to end-users.
The image below shows how testing from a risk perspective can be used to bridge the gap between automated testing (asserting expectations) and manual/exploratory testing (uncovering new information), so that lot of unknowns will be explored and converted into automated known behaviours to improve the quality. Exploratory testing helps in creating good automation tests to figure out all the things you want to go and check.
Continuous Testing
Continuous testing
right from requirement to production and taking feedback from production to requirement, this loop of continuous testing is key to achieve highest quality. Below image shows all the different types of testing we perform in various stages of DevOps.
Conclusion
As explained in my DevOps Testingpyramid, testing should always be done from a risk perspective and identifiedrisks should be prioritised and fixed accordingly for delivering qualityproduct faster.
In DevOps, automated testing is key tospeed-up the development cycle. By automation, I mean non-functional testing aswell. In a DevOps world, tester should possess Swiss Army Knife’s worth ofskills, which means as a tester it’s our responsibility to do all otherrelevant types of testing as much as possible apart from functional testing toget early feedback and not depend on external teams. Depending on externalteams might increase the risk for not getting the feedback early and delay therelease. Also, don’t forget to put all the tests including non-functional inCI/CD pipeline to get early and constant feedback on quality.
Continuous testing in all the stages iskey to achieve highest quality. It’s important to be tool aware and practiceidentifying good tools and use them whatever suits to your application and toyour team. Exploratory testing is the key to high quality software and pleasedon’t push it to end-users. Utilize as much time as possible in exploring theapplication for uncovering new information for improving quality.
Lastly, no matter what strategy andprocesses you have the team should have Quality mindset to go extra mile andadd that extra effort to deliver quality to the end user. This is accomplishedby Continuous Testing and doing all the relevant functional and non-functionaltesting at every stage so that
quality is cascaded
down to theend-user.
Testers have many diverse capabilities – just like a Swiss Army Knife!
I would like to end my blog – DevOpsis all about early feedback and frequent releases and for that
Testers should possess a Swiss Army Knife’sworth of skills, versatile and approaching the application from all the differentaspects of testing!!!
About the author
Amit Kulkarni
Amit Kulkarni (@amtoya on Twitter) is a test automation specialist and has over 10 years of experience delivering quality product for clients in Australia, Europe, Japan and USA. Amit is passionate about test automation for both functional and non-functional including security and performance testing. Amit has a diversified experience in Semiconductor technology, Embedded and Software industry in the field of Banking, Healthcare and Consumer Electronics working for companies such as Fujitsu, Mitsubishi, Qualcomm and Telstra. Amit is currently working as a Senior Quality Engineer in Bankwest and driving best quality practices in test automation and continuous testing to deliver quality product faster.
I was at
TestBash Brighton
recently, and a common theme kept coming up:
“You can be technical without knowing how to code!”
, TestBash organiser, even did a 99 second talk on the topic. He listed various skills – git, debugging, changing the microwave clock. If you have these skills as a tester, there are plenty of uses for these skills. If you have an agile or DevOps culture – where roles are less set in stone, and teams swarm on problems – you’ll get loads of opportunities to use them.
Well, apart from the microwave clock.
Learning the ecosystem
A yak dominates an ecosystem
It’s important to realise most developer skills aren’t coding. If you’ve tried to learn coding, you’ll know lots of other skills can get in the way. You start writing a simple application and lots of good practice pops up. Your developer friend says you should set up a git repository. Then instead of installing dependencies manually, it’s better to set up a dependency manager. Then you learn how to compile and run the code. The list goes on. This can be very daunting, and quickly turns into
yak-shaving
.
Learning doesn’t stop with developer skills
This seemingly never ending list of things to learn doesn’t stop with a developer’s skillset. Run a production-grade environment, and SRE/Ops skills will get in the way. Ops ensure that users can use your environments. They do all kinds of things. Backups. User and authentication control. Familiarity with the Linux CLI to debug issues. Log monitoring and analysis. Failover processes like blue/green. Containerisation like
Docker
, or container orchestration like
Kubernetes
. Your workplace may differ in the specifics, but production environments have lots of additional complex functionality. Complex functionality you can’t replicate by running the code on your laptop, or even sometimes in a test environment.
Broad skills are valuable too
Your broad skills are valuable
There are specialists in these subjects. Nobody is expecting testers to be experts with all of these skills. Much like testers don’t have to have all the coding skills of developers. That said, knowing about these things will help us understand our application. It will help understand facets of our application otherwise overlooked. If testing is about understanding the quality of our application, then we need to look beyond the code. As far as your application is concerned, code is the tip of the iceberg.
Let’s take an example: logging in to an application. Your users are not only expecting that they can login successfully. They’re expecting that the database stores their user credentials. They’re expecting that their data is backed up in the event of disaster. They’re expecting that they can still login if lots of other people are. They’re expecting their confidential data is securely stored and transported.
Much of that functionality is not done in code by developers. It is often the responsibility of ops, or sysadmins, or database admins. They configure databases and networks and servers that ensure all these things happen.
Ops opportunities for testers
Before DevOps, it was difficult for testers to have influence upon this. Much was siloed away from us. We test the code, then let them deal with that stuff. Out of sight, out of mind.
Cross-functional teams include Ops specialists
But now, Ops specialists are being moved into our teams, or working closely with our teams (hopefully not as a permanent ‘DevOps team’!). We see the work they do, and it needs testing too. That’s because it’s part of the application. It’s part of what users expect. We don’t just deliver code, we deliver a service.
Pair with an ops expert to assess risks
So how can testers start helping? To test code, we didn’t need to be developers. But learning related skills helped us empathise and co-operate. It helped us ask the right questions. So try pairing with an ops expert on your team and work out what risks might arise. Or doing some shaping of some infrastructure tickets, and thinking about who the users are (with pipelines, they’re often developers!). Or try writing some automation scripts to ensure that associated problems get caught before production. Or go to a devops meetup/conference. Learn about bad practice and security/infrastructure smells. Start treating ops like developers – to be assisted and questioned.
Because our application is more than just code. It’s all the processes, and the infrastructure too. That stuff needs testing too.
Matt Long is a QA & devops consultant for OpenCredo, an open source software consultancy based in London.
He has worked as a QA consultant for many years, and built test automation frameworks in half a dozen programming languages. After working on many cloud infrastructure projects, he’s become particularly interested in the intersection of modern cloud infrastructure and testing.
Asdevelopment has shifted from waterfall to agile and now DevOps, at each stageyou see developers taking on more responsibility and accountability.
Waterfallwas isolationist, building silos around different groups in softwaredevelopment. Developers owned the act of coding and in the stereotypicaldeveloper way, had little to no interest what happened before or after theycoded. Other roles on teams in many ways acted as insulators to keep externalforces away from developers.
Agilewas developers pulling that insulation away, shifting themselves left andtaking more ownership of the product creation process. Developers found theyrequired a real time feedback loop to build the right things and successfullyreact to change.
DevOpsis essentially the same process on the right side of the spectrum. Developmenttakes more ownership of deployment, monitoring and supporting the applicationshealth. It’s just too slow and cumbersome to react to the requirements ofmaintaining applications by filtering feedback through separate teams at scale.
Thebenefits of integration outweigh the cost of the additional responsibility.
Code Continuous
Tocope with the additional responsibilities, developers have turned to theirtrusty friend
code
.
Withagile we see rise of tooling developed for
ContinuousIntegration
to alleviate the pain and reduce the risk in iterativedevelopment.
InDevOps we see the rise of code and tools supporting
Continuous Deployment
. Even when many teams already had productinstallers, or automated deployments what was missing was the integration ofthat information.
Ifdevelopers are owning more of the product as well as release and deployments,then testing naturally gets swept up in the process. In fact we see thispattern emerging with the concept of
ContinuousTesting
. Code is being written, tools created to help solve the issue ofintegrating test execution and reporting into teams release pipeline. Thefeedback loop is extended to provide rapid information to make it easier andfaster for development to be responsive to change.
Continuous Testing
Automatedtesting is a loaded term. In developer circles it almost exclusively means unitand possibly integration tests. In QA it almost almost exclusively means end toend (e2e) tests, and is often synonymous with
Selenium
.
Onteams where both sets of tests exist, their worlds may seldom interact.Developers may be aware of the
e2e
tests but they probably aren’t writing them and even less likely running them.
ContinuousTesting has nothing to do with the testing that most QA team members are mostfamiliar with, exploratory testing. The purpose of the code and tooling thatcomprises a pipeline isn’t for there to be some manual intervention requiredpart of the way through it. Putting exploratory testing somewhere in the laterstages for the pipeline is akin to pointing a fire hose at someone and askingthem to direct the flow to the next stage.
Ifthe
speed of DevOps
wasn’t enough, the presence of
continuous testing
only puts morepressure on testers, simply because more attention is being drawn to testing.
Automationengineers and product engineers are being drawn closer together. The importanceand relevance of end to end automated tests is becoming more of a reality forproduct developers as they become run regularly as a post deploy verificationtests.
Whilecode is being thrown at testing with newfound passion, exploratory testers needto be prepared for their time in the spotlight. If automated tests are runningall the time what tests are you doing? What makes them special? Why are theyrequired before we can deploy?
What do you do different than Selenium?
Beforeyou dismiss that question, pause and consider what all selenium and the testsit drives truly provide.
Seleniumis cross browser, scalable, schedulable, repeatable, and can provide logs andscreenshots of its tests. Can you compete with that?
Flaky,slow, and wrong sometimes, you say? And you and all other humans aren’t?
Conversationsthat try to explain using people instead of machines for testing software canend up justifying them under the guise of
thereare some tests that aren’t good candidates for automation
.
Don’t settle for executing tests that don’t have enough ROIto automate.
Thistype of conversation relies on the belief that testing is comprised to twoelements:
test case creation
and
test execution
. Perhaps at its most baselevel testing is just that, the value proposition of the role is in the nuancethat a skilled tester can provide to teams through those elements.
Testersdo more than a fill a gap in automated test coverage. The problem is manytesters express this nuance solely through actions.
Developersproduce code, a measurable artifact from their effort. The generation of thatartifact is often backed by computer science, a formalized area of study.Artifacts of effort are reassuring, people like to see the fruits of theirlabors. Tangible results can be measured and quantified, they might even helpestimate future effort.
Joan of Arc
A tester’s value tends to be recognized at the same rate as relationships develop. Testers get characterized as
folk heroes
rather than members of a skilled profession. There’s no trade, education, or craft just individuals saving the day periodically.
The Intuition trap
Evenwhen teams do value testing, people are usually primarily concerned with theprogression or regression of testing efforts. Testers comments in stand-ups caneasily turn into a user story status change report:
Passed this ticket, working this ticket, noblockers
Ifyou aren’t taking the opportunity to discuss your actual actions in stand ups,then you may fall prey to the intuition trap. When you talk about your workthis way, sometimes you end up actually working this way.
Open Source Your Mindset
It’sfun being a hero, but there’s no skill in being
Superman
or
Superwoman
(I was going to say Wonder Woman, but she actually trained and I think everyonerecognized that. So if you are the teams Wonder Woman then kudos, you haveserious skills but I digress).
Whentesting goes smoothly the result is
nothing
.Testing isn’t required for things to go right, it just might help prevent athing going wrong.
Opensourcing your mindset means
teaching
others another way to look at and interact with their code, or the product.This cuts both ways though, it means being willing to accept a pull request ofoutside perspective. Openly integrating new ideas into your
testers mindset
.
Replace Execution with Communication
. Like acting or athletics there might be a script or game plan to start with but there is a lot going on behind the scenes, adapting to circumstance and being thought up on the fly. It’s intuition driven.
Whenactions, practices and motivations are not being verbalized and expressed tothe team and stakeholders it puts testing on the level of superstitions andtoken time sacrifices to the quality gods.
Testersneed to pry open this experience, expose the internals driving motivations.Breaking out of that shell grants visibility into a testers actions andmotivations to provide a context for team members to attribute value.
Replace Isolation with Collaboration
Itcan be uncomfortable or difficult to open up and share what goes through your
testing mindset
.
Theadditional ownership developers have taken in DevOps has changed relationshipdynamics. What was at one point a contentious us vs. them relationship is muchmore open. There just isn’t the time to have as much formal process drivencommunication. Success at scale requires collaboration, pairing and mobbingmore.
When with developers, it’s an opportunity to test at their desk, against their local changes. There are risks testing here, the developers local setup is definitely different than production but so are the testing and staging environments. Let go of the taboo of
it works on my machine
and consider what can be tested here and what needs to be covered on a different stage. Take the opportunity to work together when the code is the freshest in the developers mind and where they are most comfortable.
It’s not only about working with developers. Mob testing with
UX
and
Product
alongside developers is a way to draw the entire team together. When the team is together seeing how the application is working it bypasses all the communication gates. There is no need for a separate follow-up conversations to loop someone in, that person can see and feel how things are working by being a part of the group experience.
Replace Tedium with Stratagem
Teamsare more open to testing and sharing in quality than ever. Testers have beenbuilding quality mindsets for years. Developers are building tooling but teamsare looking for guidance and mentorship in
Quality
.
Testing isn’t just button pressing, by showing that through communication and working together we elevate our status. Draw people together, into the quality circle, and enable the team to be more comfortable and aware of testing. We can set working agreements, talk about overarching strategy for how testing throughout the development cycle builds quality in. We can embrace automated testing as a team activity where unit, integration and end to end coverage is part of a teams strategy for mitigating risk. We can forgo complete control of testing and opt into being a voice of authority and expertise in quality.
Brendan Connolly is an experienced Software Tester, Automator, Speaker, and Blogger. Currently he is a Senior Quality Engineer at Procore Technologies in Santa Barbara, California. He maintains a testing blog @ http://www.brendanconnolly.net
Where you see “DevOps” being practised but the QAs (or Testers) are still relegated to the “checking things corner” and not explicitly and proactively involved in providing input into both applications and infrastructure code, are we perhaps doing the very opposite of what DevOps was meant to be?
Images credit:
Beware the cargo cult
Most of us know the
tale
of how DevOps was conceived and I trust most of us have read The Phoenix Project (Kim et al, 2013) as one of our first DevOps books.
Therein we learned that DevOps is meant to be a collaborative and efficient partnership between all disciplines to achieve a common goal, which ultimately means: working software in a production system, delivering value to customers.
Images credit:
Where apparent ‘DevOps’ excludes other team members, like Security or QA, we have missed the point. DevOps is meant to be development
teams
working with Operations
teams
, not just individuals in developer and operations roles working together.
The QA role is considered to be a member of the Development team. The QA is therefore intrinsically a member of DevOps.
Where we find that QAs have no input into the Operations or Continuous Delivery conversations because they “are not developers” but we think we are doing DevOps, we are in fact practising a cargo cult — i.e. an imitation and approximation of the real thing because the real reasons of WHY have not been fully grasped.
If we are going to do DevOps, it behooves us to know WHY we do it, and not only HOW.
Waste
‘Agile’ and ‘DevOps’ fundamentally assist with reducing waste. (I generally prefer not to use these capitalised nouns as they have become
overloaded
, so use them here cautiously).
Project churn, conflict, friction, slow response, unreliable and unstable systems, needless expenses, lengthy handovers, defects, time delays, failed projects, manual overheads, double-handling and many similar issues, are all waste.
Every time a feature is double-handled (such as the churn when a bug is discovered later), we have incurred waste. Feedback should be sought earlier rather than later so that one may course-correct or remedy issues as early as possible and prevent waste.
QA is feedback
QAs do not provide assurance; they help provide and define analysis and feedback.
Monitoring is a form of feedback of the system’s behaviour.
Just like monitoring is feedback, so is testing. Whether the testing is manual exploratory testing or automated checks, it is feedback.
Testing tells you about components and relationships; monitoring tells you about the system.
To minimise and reduce waste, you want to receive that feedback as early as possible, and that the feedback occurs in meaningful levels and depths in your system.
Early feedback
QA (testing and monitoring) should be done early and continuously. Receiving early feedback enables one to act on that information and improve or course-correct as early as possible.
Your QAs can and should be involved in these conversations – and if they cannot, then as their manager or peer the onus is on you to empower them to become involved.
Images credit:
The days of QAs telling you if you’re OK to ‘go live’, are numbered. The whole team should know that, based on the continuous feedback they have been receiving.
Lean QA
The reduction of waste (rework, defects, friction) by holistically testing the whole system, early and continuously (and into production), including measuring what really matters & incrementally improving thereupon, and using your humans intelligently in doing so, is what I call “Lean QA”.
And what I consider
essential
to DevOps.
DevTestOps: It’s alive, but what can testers do to make it alive AND kicking into overdrive!
The journey of the ‘typical tester’ has been a pretty bumpy ride over the past few years, more so recently with the massive uptake of DevOps. Apart from the many new and intimidating terms thrown our way, we are now required to do anything from digging in to business case details to monitoring how applications are performing in Production. It’s become a world which is very different to the one a tester would have experienced around the year 2005 or so.This world has incorporated testing to be at the heart of it, and not just testing but full on quality! So its only natural for the tester to play a massive role here.
But what can we do?
As mentioned, the tester should be at the very heart of every activity happening in the software delivery process. Below I highlight some of the areas where testers can directly influence and should influence if the process is to go from just being alive, to becoming Alive and Kicking!
Help Sculpt a Team Culture (Team player)
When one speaks about DevOps, technical aspects are usually at the front of the conversation whereas the softer aspects like Team culture and communication are very rarely mentioned.
Everyone loves quality products, testers love quality. A good healthy, dedicated team can help make better quality products. So naturally a tester should be one of the key individuals to drive a dedicated team. Testers should keep their critical thinking quality as their most prized asset by asking the right question at key moments during team discussions/meetings and interactions. This gets the team thinking and hopefully to a point where this starts to become second nature to all team members. Quality should be a driving force of every team member in everything they do, this should be a non-negotiable principle.
Another key area around team culture is by re-iterating the culture of ‘safe to fail’ and the ability to recover quickly. Without these important concepts inherent in a team the ability to deliver faster might prove to be a huge challenge.
Promote Visibility and Transparency (Seeing is believing)
I cannot stress how key this aspect is to a solid DevTestOps model. Gone are the days of working in isolation. A tester should look at some of the key artefacts that traditionally would fall within his/her space and how these can be made visible and transparent to the entire team.
Some of these include Test coverage visibility, defect statuses per sprint and release, test automation coverage, test automation scripts (functional and non-functional). One can make these visible in the form of dashboards or even repositories for test checklist or automation scripts.
At any point in time any team member should be able to give you the current status of your testing per sprint and/or release. After all if it is a difficult and an intensive process to decide on whether you are ready to deploy to production, then you have to question what you as the tester can do to change this.
Increase your skillset (Up your game)
Learn to use tools that help you contribute more (Photo by
on
A modern tester needs a modern armoury. Without it, I battle to see how the transition to the DevTestOps model can be easily made.There is so much to mention under this area, up-skilling should span these key areas:
Automation tools
From backend to front end focused automation tools, a tester should not only know these tools but know when and where to use them. Tools learned should include those which at least cover API testing, UI Testing, Performance/Load Testing, Security and Databases.
Context switching between these tools and types of testing is a rather intimidating thought for those not usually exposed to modern ways of software delivery. My advice here is to take baby steps in learning these tools in order to not just learn something for the sake of it, but rather for the cause of true DevTestOps.
Technical Tools
Technical type tools that equip a tester to contribute effectively in a DevOps environment would once again span the entire spectrum of the application solution. Tools like Postman & Swagger for
API
testing, knowing your way around
version control
applications (GitHub, GitLab etc),
SQL and NoSql
interfaces (like DBKoda, Sql Server etc),
CI/CD
apps (TravisCI, GoCD, Jenkins etc) ,
Logging
applications (GrayLog, ElasticSearch,ElkStack etc) and
Monitoring
tools (NewRelic, Dynatrace, Instana etc).
Contribute to your CI/CD (its in the pipeline)
Testers have a crucial part to play in defining the ideal pipeline for your applications deployment. Being part of discussions around composition of the pipeline, ideal duration of pipelines, pre and post deployment tests as part of the pipeline.
A point to consider even further is where do your automation tests fit in these pipelines? Should it be part of it?
CI/CD tools also promote to the point on visibility and transparency, if constructed properly the entire team should know exactly at what stage the deployment is at, what tests were run and the overall status of the deployments.
Monitor it (What’s trending?)
Dashboards help us steer (Photo by
on
To know what your application is doing both in PROD and Test environments, contributes to how you approach your testing. Findings and trends even contribute to what scenarios you end up testing and has massive input regarding determining what load you put your system under during Performance testing as an example.
Traditionally testers would steer clear of any tool or application that falls outside the scope of testing, but in a fast paced delivery world, monitoring tools can be a testers best friend. A tester’s system or application knowledge should also push the tester to not just use existing monitoring metrics but to also determine what to include as part of monitoring.
In summary
The evolution into DevTestOps has meant that the tester now plays a much more involved role in the entire delivery process. The day to day tasks of a tester spans between a balance of soft skills, analytical, technical and even an investigative mindset sometimes seamlessly crossing between these tasks without even knowing. By following some of the pointers above the initial ground work to successful DevTestOps is well underway.
Go on Testers, get DevTestOps Alive and Kicking in full swing!
Toyer Mamoojee
Toyer Mamoojee is a ‘die-hard’ passionate Agile Automation Test Analyst with almost 16 years testing experience across many industries. From Waterfall to Agile approaches, Commercial to Open Source Automation tools and Manual to Automation Testing he has been through the wars of Software Delivery. He is the co-founder of Cape Town Automation User group and co-organizer of Cape Town Testing Meetup back in his home country of South Africa. In his spare time he loves exploring the great outdoors, taking on the world playing online video games , major football fan, adopting the title of technological husband and playing super dad to 2 pre-schoolers!
Toyer Mamoojee @tottiLfc
A Tester’s Story: Adapting to the new world of pipelines
By , Guest Author
As a tester, how do you adapt when a new technology or process is introduced within your team? Do you panic? Do you accept it as a challenge? Do you research to find a solution along with the team? Do you reach out to people in the community to see how they approached it? Or do you just expect it to magically work for you without any effort on your part?
I was in similar situation when our team decided to adopt Continuous Integration (CI), as we were transitioning to a DevOps culture. I would like to share my experience of how, as a tester, it was to go through the journey.
It had been a while since we started to work in agile from waterfall. We took baby steps and evolved slowly. We were all happy, comfortable, and settled into the way we were approaching agile with our scrum process. Just when we were happy that we went through the agile transition from waterfall, now it was a new challenge of transitioning to DevOps. We had to embrace another new change of process, and it was the introduction to the world of pipelines – it was the CI/CD world!
As a tester you always like challenges, and this was challenging because I had to understand a new way of how testing fits in continuous integration, and embrace the new type of value it could provide while working on it.
As a tester I was very curious to know:
What it brings to my world of testing
How would it affect current test process
What do I have to change
What benefits do the pipelines get?
What do they look like?
How do they work?
And on and on. It was pretty confusing and scary at the beginning to visualize the change.
Why is our first instant emotion towards any change ‘Scary’? Why can’t it be an excited feeling or enthusiastic feeling for trying something new – I asked myself this question. Then I thought to myself, it’s as simple as this – When you don’t know about something, you are always scared about the change, but if you feel safe and see it as a new challenge, you will definitely get excited.
Then I started to explore and read more about pipelines. It wasn’t surprising that the more I was reading, the more I was confused. Then I decided to draw those pipelines and visualize them.
I sat with my colleagues and said: “Let’s please visualize the pipeline by drawing it on a big white board.” They agreed and we had a whole discussion around these pipelines. And we finally agreed on this and it looked something like this initially:
That’s where it started to make more sense to me, and I got a better understanding about what exactly this continuous integration is – but all in theory though. What I understood at this point was that it is simply nothing but a step by step process to release the code once there’s a commit.
Continuous Integration
is a practice for integrating the code several times to receive a quick feedback without waiting long. Whenever the developer finishes development task and pushes the changes to the repository, that’s where the pipeline stage starts. Because it has stages, it is called as a pipeline I thought to myself .
Now, a few more questions came to my mind:
What are those stages?
Who defines those stages?
Who defines what should go in the pipeline and when?
Will I have any place as a tester to suggest something?
The biggest question was – will there be no manual testing at all?
We as a team had discussions and came up with the our own version of the pipeline because there’s no definitive structure for a Continuous Integration (CI) pipeline. We decided we shall run static code analysis, unit tests and integration tests as our initial stages of our pipeline. This stage would create a unique artifact which could be deployed on a test environment to run additional tests. As my understanding got better about pipelines, I started to suggest that we add more valuable tests to the pipeline.
Initially we added smoke tests to the pipeline which would run alongside the existing tests. The reason why I suggested to add smoke tests as part of the pipeline was to make sure the build is testable before it is deployed on a test environment for further testing. I just called it as smoke test, but we could call it a Build Verification Test or Build Acceptance Test too. If the test didn’t pass, the build would be rejected and a new build would be released once the failed tests were fixed.
I could see each stage of tests running one after the other. It would stop going further if there was a failure at any of the tests or stages. Isn’t that magical!!!
It was just the beginning for us and especially for me. I continued to understand and observe more and more, and then I discussed with my team if we could add regression testing to the continuous integration pipeline. But initially we did not wanted to add any long running tests to the pipeline as we thought the whole point of having these pipelines was for quick feedback. So we decided to run the regression tests as nightly run task instead of running it with every commit. But we had the goal to run these regression tests with every commit once our pipeline was mature enough, when we would decide to run those tests in parallel. But that was a long term goal.
Each morning I could see the results of which tests where passed and which were failed and the difference was that I could add the build number while raising the issue, so it got easier for a developer to find the root cause. It’s not to say that it tells you where exactly where the issue is but they could easily check the recent code related to that build to see what went wrong. Instead of searching a needle in a haystack, it’s like you know which part of haystack you need to look to find the needle.
I got to spent most of my time as tester defining the test strategy, exploratory testing, being part of the 3 amigos to refine user stories and acceptance criteria, checking the test feedback from the pipeline, and adding more valuable tests to regression tests.
As we got along and had more clarity and saw the benefits for having these pipelines in place, we decided to discuss a long term goal, where I suggested to have visual testing as part of the pipeline as well, from a testing point of view.
This was my actual pipeline journey into the world of testing! It was very challenging and interesting!!
