Month: July 2019

What is continuous testing?

By Lisa Crispin

Even when I worked on waterfall teams, we tested early and often. We got ourselves invited to analysis and requirements meetings. We tested requirements docs. We asked questions. We tested as the developers delivered code, even before “code freeze”. We also worked closely with customer support to learn about how customers used the features after they were delivered. (I realize not all teams using a phased-and-gated approach do all that, but that approach should not stop teams from using good practices!)

Extreme Programming (XP) made so much sense to me because it embraced testing and quality as the responsibility of the entire delivery team. In agile development, we focus on building quality in, we test ideas, we automate regression tests, we explore delivered code, we test in production. These same practices apply to teams who use the “DevOps” label as well.

That’s why Dan Ashby’s post in 2019 on “continuous testing” resonated so much with me. His visual model captures what is in my brain when I see the term “continuous testing”. With his permission, I often use this model to help others understand the concept:


Dan Ashby’s Continuous Testing Model

But there are different definitions!

WAT!?

Recently, someone posted a link to a short Twitter thread from earlier this year where continuous testing was defined quite differently. Basically, this alternative definition is only about running automated tests in the continuous integration, delivery, deployment pipelines. I started asking around to see how other people define “continuous testing”, and some people shared that definition – only the automated suites in CI/CD.

A book on “continuous testing” which I had not gotten around to reading also has this definition, and it references the Wikipedia page which says the same thing.

Automated tests in CI are not new!

Continuous integration has been around for decades. My waterfall team back in the early 90s did CI, complete with automated regression tests and automated deployments (to 20+ Unix environments!) We didn’t call the automated regression tests “continuous testing”. We called them automated regression tests. It puzzles me that people would apply the term “continuous testing” to these automated test suites, which teams have been using since long before the term “continuous delivery” appeared.

Going to the source

I learned about continuous delivery by reading the terrific book of the same name by Jez Humble and David Farley. If you read it, you will find it is a book about testing. It goes into much detail about many different kinds of testing activities – not only automated regression tests!

So I went to Jez’s website to see how he defines “continuous testing” . Please click that link and read it for yourself. To quote Jez,

Our goal is to run many different types of tests—both manual and automated—continually throughout the delivery process.

That is followed by a picture of Brian Marick’s agile testing quadrants, illustrating a range of tests that are business-facing, technology-facing, that guide development and that critique the product. In Jez’s quadrants model, the tests that critique the product include manual tests.

Jez goes on to explain the concept of the deployment pipeline and the importance of getting quick feedback from automated acceptance tests that are in the pipeline. But he does not limit “continuous testing” to only those tests.

Here’s the latest version of the agile testing quadrants model that Janet Gregory and I use. We’ve added examples of testing activities we can do today thanks to newer technology and tools, such as monitoring and observability. Those are also part of continuous testing.

Agile Testing Quadrants from Agile Testing Condensed: A Brief Introduction by Janet Gregory and Lisa Crispin

Does the terminology matter?

I don’t like to be too pedantic about terms, but we need to be able to communicate with each other. If my idea of continuous delivery was limited to only the pipeline and did not include the concept of breaking features into small slices and deploying small changes frequently, that would not match what most practitioners I know consider to be “continuous delivery”. It would be hard to have a conversation about it with two very different definitions.

Many people use the term “automated regression checks” rather than “automation regression tests” because these tests are checking that requirements are met and that established behavior continues to work as expected. They reserve the term “testing” for activities that are done by humans. Janet suggested that maybe we can refer to the automated regression test suites in CI / CD pipelines as “continuous checking”, and reserve the term “continuous testing” to include those checks PLUS all the testing activities that happen around Dan’s infinite testing loop.

How can we resolve this terminology dilemma? Feel free to comment here. There’s also a discussion topic for this on the Ministry of Testing Club .

Meanwhile, I shall see if there are even more definitions of continuous testing I hadn’t heard yet!

Continuous Testing in DevOps: A gamification approach

By Jorge Luis Castro Toribio

A continual problem that we must deal with when we run automated tests in our DevOps pipeline is maintenance. I could spend hours sharing posts and blogs about this matter, telling stories about how it affected our team’s capacity and time to market, and giving you several examples of what teams have done to sort this problem out. In fact, all the effort that we invest in maintenance is not visible to the business stakeholders’ eyes. Maintenance consumes our team capacity, but it does not add value to our clients directly.

Test automation challenges at scale

In one of my jobs wefaced this problem on a large scale. We had more an average of more than 1000automated tests running at each code check in, plus one big run at midnightthat we called full regression. Our suites where organized in those levels:

  • L1: Unit testing – Development Stage
  • L2: Integration Testing – Commit Stage
  • L3: UI Functional Test MVP Suite – QA Stage
  • L4: Regression suites – Full package – UAT Stage

Those testing levelswere our creation, but we used the great lessons shared in this book https://www.amazon.com/Practical-Approach-Large-Scale-Agile-Development/dp/0321821726 that I recommend for any team looking to buildtheir testing strategy.

After each run, we noticed that 80% of UI and regression automated tests failed due fake failures. After reading my numbers, you can say if you automated crap, you will get crap results and that is correct — but please remember that we were building our automated test suites, with a continuous testing approach and DevOps mindset and pipelines, so we were trying to fail fast, learn and improve. First, we decided to analyze our test results and data, so we collected all the information that we got from our test runs and analyzed, we discovered that our tests failed because of those main reasons:

  1. Changes inUI components.
  2. Badcoding.
  3. Exceptionsand errors handling
  4. Time outseverywhere
  5. Data

A new experiment

We had to make a decision. We had invested lot of efforts in testing automation, and we were the product team with more automated tests compare to the rest of product teams globally in the company.  This brought us attention from lots of managers, who were keeping their eyes on our DevOps and test automation activities. In addition, we were participating in a DevOps competition which involved all the IT teams of the corporation.  So, we decided to invest time to get the failed tests in good shape and have them passing in the pipeline. We called this mindset  “Keep them green and running”. We communicated this effort to all the product teams. We asked the Product Owners to consider maintenance time as part of their team capacity, include tasks for this purpose, and keep it part of their definitions of “ready” and “done”.

Unfortunately, we did notachieve the goals we had hoped for. We still had the same test results, withmore than 80% of tests failing. In almost every case, they were fake failuresthat meant that the tests were stopped long before their last steps. To sum up,our rest results were not accurate at all and did not bring confidence to ourteams — which is something critical in testing automation.

Thinking outside the box

We needed to try something different. After some discussion, research, investigation and thinking out of the box, we decided to use gamification to keep our tests in a good shape, running and be green. We named the game “Game of Testing” and we established some critical points about it:

  1. Maturitylevels: The gamers can be developers, testers or anybody who owned an automatedtest running in the pipeline, they were maturing in the game, from a beginnerlevel to a master level.
  2. Visibility:We shared the game boards including ranking, test results and so, on using bigmonitors in the office, allowing everyone to be aware of the gamers andresults.
  3. Weight:This is a very complicated and mathematical part, we needed to stablishformulas and weights for a point system for the game, so we could evaluate thetest results, data per gamer and teams, and gave points and make them get newlevels.
  4. Recognition:We tried to recognize all the gamers taking part. For the best gamers, weshared the results with local and global senior managers, so their achievementscould be known and recognized.  Thishelped encourage more team members to participate.

The game was quitesimple:

  1. Gamers: All the team members who owned an automated test running in the pipeline.
  2. Point system: A gamer won point s for test running in green and negative points if the test failed for fake results. Later we added negative points of breaking builds.
  3. Rules: You needed to keep all your test and team tests running in the pipeline in a good shape for each run. So, you had to invest efforts on “maintenance” and continuous improvement if you want to rock in the game.
The Game of Testing

This approach helps usa lot to decrease false positive runs and keep the tests up and running in thepipeline. We decreased the number of 80% of failed tests to only 20% inapproximately three months and our numbers have continued decreasing.  Furthermore, people had fun while playing thegame and they internalized that quality can be fun and not boring like usualmaintenance was.

While we got several criticisms about our failing tests at the beginning of our efforts to improve them, and people lost trust on our quality assurance process, we didn’t give up trying to fix the problems. In fact, it motivated our team to think out of the box and turn this problem into a useful and fun project. We spent several hours investigating continuous testing, scripting, and related topics. Besides that, we wanted to do something different, something motivates other team members to join.

Gamification can provide that motivation. You may not achieve desired results immediately, but if you prepare your environment and involve everyone in trying potential solutions, the game will give you results for sure. If you address the test issues you can get them pass or fail, but if you involve and motivate people to work on problems together, you always win and have fun. I recommend you try gamification to sort your problems out, you may be surprised at the results!

Jorge Luis Castro Toribio

Jorge is an agilist who feels passion for Testing Automation, Agile, DevOps and business agility. He enjoys research and learning about business agility and working with cutting-edge technologies. He has been learning continuously for more than 12 years and is still learning new ways to foster business agility and team greatness. He has worked in several roles (developer, tester, manager, agile coach) which helps him to see the big picture and operational and team members dynamics happening inside organizations. This experience lets him help teams design, build and implement digital, DevOps and agile transformation strategies. He encourages focus on: People, Productivity, Continuous improvement, Innovation and having fun to enjoy success in our agile journeys.

You can find Jorge on and .

Visualizing Pipelines to Help Build a Quality Culture

By Lisa Crispin

At CAST 2019 , August 12-15 in Cocoa Beach, Florida, and I are pairing to facilitate an all-day workshop called “The Whole Team Approach to Testing in Continuous Delivery”. Our focus will be helping people to find new ways their teams can build a quality culture and succeed with continuous delivery. An important part of this is learning about deployment pipelines. For those of you who can’t attend the live version, here’s an introduction to why deployment pipelines are vital to successful continuous delivery, and a fun exercise your team can use to improve your own pipeline.

Continuous delivery – scary?

Continuous delivery (CD) lets us deliver new value to production safely, quickly and sustainably (to paraphrase Jez Humble). Each new commit to our code base trunk or master is independently verified as a deployable release candidate. We can deliver frequently, which means the changes are small, reducing the risk of problems. 

Still, CD can seem like a scary proposition. In the old waterfall days, we didn’t have enough time to complete all the testing activities we wanted. Agile development enabled more bi-weekly or weekly releases, and brought up new testing challenges. And now, how can testing possibly keep up with continuous delivery? How can we feel confident about each release, from a testing/customer support perspective?

New approaches

Following the principles of continuous delivery helps us build confidence in our deployments. Following modern testing principles can give us even more confidence. When the whole team takes responsibility for building quality into the product and continually improving how we work, we (and our customers) realize the benefits of CD. 

This is a huge topic, but let’s look at two key factors for success with CD. Obviously, we need adequate automated regression test coverage to make sure that the things our customers really need are still working. That’s not new. And, it isn’t a complete safety net. We can have great automated regression test coverage, do thorough exploratory testing, verify performance, and all the other many types of testing before we release to production, although we are doing that in test environments. We know that behavior is likely to be different and unexpected in production. 

So what is new? Today, we have the ability to capture and analyze all kinds of data (and huge amounts of it) about production use and immediately detect any errors that occur. This is important, but it’s not that helpful unless we can quickly and reliably deploy fixes to production. And to do that, we need a fast, reliable deploy pipeline.

P ipeline basics

The deployment pipeline consists of steps that changes go through before they can be deployed to production. Every company, every team has their own unique pipeline. Note that whether or not your team has automated tests and automated deploys – you still have a pipeline. Whatever steps a new change goes through to get to production, even if each step is manual, is still a pipeline.

We break our build into stages or steps to speed up feedback. There’s a constant trade-off between fast feedback and confidence. The early steps of a pipeline are usually unit tests. They quickly show us whether anything is broken at that low, granular level. But we need more steps with higher-level, longer-running tests to feel more confident. Later steps in our pipeline exercise more complicated scenarios. We want fast feedback, but we may need comprehensive tests for confidence, and they take a long time to run. Here’s an example deployment pipeline:

In this example, the steps outlined in blue in this example are manual steps. We focus on automation when talking about pipelines, but manual steps are common. As in this example, activities like exploratory testing and security testing may be critical and may require manual work. We should be able to automate deploys to different environments, but perhaps your team just hasn’t gotten to that yet. 

Things that slow down the pipeline

“Flaky” or “flappy” tests are a common problem even in completely automated pipelines. We often see this with user interface (UI) tests. Someone makes a change to the UI but forgets to update the automated tests to reflect that. A test environment slows down and causes a test step to time out. The browser version automatically (and unexpectedly) updates in the test environment. Those are just a few common causes of UI test failures. 

Dependencies can also cause difficulties. A failure of one stage can stop multiple subsequent steps from running. Test data may fail to be set up properly, causing test failures. A test environment in the cloud might fail to spin up. 

More steps in the pipeline give us more confidence, but they take more time. And how do we know all the steps are necessary? 

When a pipeline experiences more and more delays, we humans have a tendency to increase the size of the items we’re sending through it to try to get more done. If the tests are failing and people keep committing changes, we’ve now got a big batch of changes which is more likely to include regression failures. It’s a death spiral. If our pipeline is reliable and fast, fewer changes go through it each time, reducing the risk of failures.

Collaborating to improve the pipeline

Making problems more visible is a great first step in making them smaller. Gather your team (or at least a representational group, if you have a large team) together. Take cards or paper and write the name of each step of your pipeline, one per card or paper. Put them on a table or on a wall to create a picture of your pipeline. (This can be done virtually if your team is distributed).

Visualizing a pipeline on a table – picture used with permission from Llewllyn Falco 

Next, look at each step in the pipeline and ask “why do we have this step?” What are we learning from it? What business questions does it answer, what risks does it mitigate? Who on the team benefits from this information? You can capture this information on sticky notes to attach to each step.

Then, think about dependencies. For each step, what must in place for it to run successfully? A test environment? Test data? Specific configurations? Also think about how you can optimize the step. Is it currently done manually? Can you automate it? 

Identify what triggers each step. Is it triggered automatically by the success of a previous step? Or manually? Can it be run in parallel with one or more other steps? Do you see opportunities to shorten feedback loops? Again, record outcomes of these discussion for each step.

Look at the length of your feedback loops. How much time elapses from code commit to completion of each step? What happens if the step fails – do the right people get notified? How? Is there a better way to communicate alerts? My previous team found that everyone was ignoring the Slack alerts generated by pipeline failures, so they hooked the build up to a police light. That got the team’s attention so that failures were quickly investigated and addressed!

Can any steps be done asynchronously? For example, it’s not realistic to do manual exploratory testing on every change when you are deploying to production frequently. The team can learn to use release feature toggles to hide changes from customers until the team has had time to complete any manual or time-consuming testing tasks.

Once you’ve gathered information about each step in the pipeline, choose a pain point to address and think of an experiment to reduce the pain. For example, perhaps your UI test suite has a lot of flaky tests and the team spends a lot of time kicking tests off again and diagnosing the failures that reoccur. Your experiment might be, “We believe that by refactoring the top 10 flaky tests to make them reliable will reduce our pipeline failure rate by 10% in the next month”. Use your retrospectives to evaluate your progress.

Remember the benefits

A fast, reliable pipeline means that you can confidently deploy frequent, small changes to production. And if you introduce problems to production that cause customer pain, you can shorten the time they feel pain by quickly deploying a fix with your fast, reliable pipeline. There are many other factors that go into safe, reliable and sustainable continuous delivery. Fast feedback and the ability to deploy changes quickly are a cornerstone of successful CD – and of a healthy quality culture.

References: 

Lisa Crispin is a testing advocate with . With Janet Gregory , she has co-authored Agile Testing , More Agile Testing , and the upcoming Agile Testing Condensed . Lisa and Janet also have a 5 hour video course, Agile Testing Essentials . Through the Agile Testing Fellowship , they offer a three day live course, “Agile Testing for the Whole Team”.