Tag: continuous integration

What is continuous testing?

By Lisa Crispin

Even when I worked on waterfall teams, we tested early and often. We got ourselves invited to analysis and requirements meetings. We tested requirements docs. We asked questions. We tested as the developers delivered code, even before “code freeze”. We also worked closely with customer support to learn about how customers used the features after they were delivered. (I realize not all teams using a phased-and-gated approach do all that, but that approach should not stop teams from using good practices!)

Extreme Programming (XP) made so much sense to me because it embraced testing and quality as the responsibility of the entire delivery team. In agile development, we focus on building quality in, we test ideas, we automate regression tests, we explore delivered code, we test in production. These same practices apply to teams who use the “DevOps” label as well.

That’s why Dan Ashby’s post in 2019 on “continuous testing” resonated so much with me. His visual model captures what is in my brain when I see the term “continuous testing”. With his permission, I often use this model to help others understand the concept:


Dan Ashby’s Continuous Testing Model

But there are different definitions!

WAT!?

Recently, someone posted a link to a short Twitter thread from earlier this year where continuous testing was defined quite differently. Basically, this alternative definition is only about running automated tests in the continuous integration, delivery, deployment pipelines. I started asking around to see how other people define “continuous testing”, and some people shared that definition – only the automated suites in CI/CD.

A book on “continuous testing” which I had not gotten around to reading also has this definition, and it references the Wikipedia page which says the same thing.

Automated tests in CI are not new!

Continuous integration has been around for decades. My waterfall team back in the early 90s did CI, complete with automated regression tests and automated deployments (to 20+ Unix environments!) We didn’t call the automated regression tests “continuous testing”. We called them automated regression tests. It puzzles me that people would apply the term “continuous testing” to these automated test suites, which teams have been using since long before the term “continuous delivery” appeared.

Going to the source

I learned about continuous delivery by reading the terrific book of the same name by Jez Humble and David Farley. If you read it, you will find it is a book about testing. It goes into much detail about many different kinds of testing activities – not only automated regression tests!

So I went to Jez’s website to see how he defines “continuous testing” . Please click that link and read it for yourself. To quote Jez,

Our goal is to run many different types of tests—both manual and automated—continually throughout the delivery process.

That is followed by a picture of Brian Marick’s agile testing quadrants, illustrating a range of tests that are business-facing, technology-facing, that guide development and that critique the product. In Jez’s quadrants model, the tests that critique the product include manual tests.

Jez goes on to explain the concept of the deployment pipeline and the importance of getting quick feedback from automated acceptance tests that are in the pipeline. But he does not limit “continuous testing” to only those tests.

Here’s the latest version of the agile testing quadrants model that Janet Gregory and I use. We’ve added examples of testing activities we can do today thanks to newer technology and tools, such as monitoring and observability. Those are also part of continuous testing.

Agile Testing Quadrants from Agile Testing Condensed: A Brief Introduction by Janet Gregory and Lisa Crispin

Does the terminology matter?

I don’t like to be too pedantic about terms, but we need to be able to communicate with each other. If my idea of continuous delivery was limited to only the pipeline and did not include the concept of breaking features into small slices and deploying small changes frequently, that would not match what most practitioners I know consider to be “continuous delivery”. It would be hard to have a conversation about it with two very different definitions.

Many people use the term “automated regression checks” rather than “automation regression tests” because these tests are checking that requirements are met and that established behavior continues to work as expected. They reserve the term “testing” for activities that are done by humans. Janet suggested that maybe we can refer to the automated regression test suites in CI / CD pipelines as “continuous checking”, and reserve the term “continuous testing” to include those checks PLUS all the testing activities that happen around Dan’s infinite testing loop.

How can we resolve this terminology dilemma? Feel free to comment here. There’s also a discussion topic for this on the Ministry of Testing Club .

Meanwhile, I shall see if there are even more definitions of continuous testing I hadn’t heard yet!

A Tester’s Story: Adapting to the new world of pipelines

A Tester’s Story: Adapting to the new world of pipelines

By , Guest Author

As a tester, how do you adapt when a new technology or process is introduced within your team? Do you panic? Do you accept it as a challenge? Do you research to find a solution along with the team? Do you reach out to people in the community to see how they approached it? Or do you just expect it to magically work for you without any effort on your part?

I was in similar situation when our team decided to adopt Continuous Integration (CI), as we were transitioning to a DevOps culture. I would like to share my experience of how, as a tester, it was to go through the journey.

It had been a while since we started to work in agile from waterfall. We took baby steps and evolved slowly. We were all happy, comfortable, and settled into the way we were approaching agile with our scrum process. Just when we were happy that we went through the agile transition from waterfall, now it was a new challenge of transitioning to DevOps. We had to embrace another new change of process, and it was the introduction to the world of pipelines – it was the CI/CD world!

As a tester you always like challenges, and this was challenging because I had to understand a new way of how testing fits in continuous integration, and embrace the new type of value it could provide while working on it.

ContinuousTestingNow

As a tester I was very curious to know: 

  • What it brings to my world of testing
  • How would it affect current test  process
  • What do I have to change
  • What benefits do the pipelines get?
  • What do they look like?
  • How do they work?

And on and on. It was pretty confusing and scary at the beginning to visualize the change.

Why is our first instant emotion towards any change ‘Scary’? Why can’t it be an excited feeling or enthusiastic feeling for trying something new – I asked myself this question. Then I thought to myself, it’s as simple as this – When you don’t know about something, you are always scared about the change, but if you feel safe and see it as a new challenge, you will definitely get excited.

Then I started to explore and read more about pipelines. It wasn’t surprising that the more I was reading, the more I was confused. Then I decided to draw those pipelines and visualize them.

I sat with my colleagues and said: “Let’s please visualize the pipeline by drawing it on a big white board.” They  agreed and we had a whole discussion around these pipelines. And we finally agreed on this and it looked something like this initially:

That’s where it started to make more sense to me, and I got a better understanding about what exactly this continuous integration is – but all in theory though. What I understood at this point was that it is simply nothing but a step by step process to release the code once there’s a commit. Continuous Integration is a practice for integrating the code several times to receive a quick feedback without waiting long. Whenever the developer finishes development task and pushes the changes to the repository, that’s where the pipeline stage starts. Because it has stages, it is called as a pipeline I thought to myself .

Now, a few more questions came to my mind:

  • What are those stages?
  • Who defines those stages?
  • Who defines what should go in the pipeline and when?
  • Will I have any place as a tester to suggest something?
  • The biggest question was – will there be no manual testing at all?


We as a team had discussions and came up with the our own version of the pipeline because there’s no definitive structure for a Continuous Integration (CI) pipeline. We decided we shall run static code analysis, unit tests and integration tests as our initial stages of our pipeline. This stage would create a unique artifact which could be deployed on a test environment to run additional tests. As my understanding got better about pipelines, I started to suggest that we add more valuable tests to the pipeline.

Initially we added smoke tests to the pipeline which would run alongside the existing tests. The reason why I suggested to add smoke tests as part of the pipeline was to make sure the build is testable before it is deployed on a test environment for further testing. I just called it as smoke test, but we could call it a Build Verification Test or Build Acceptance Test too. If the test didn’t pass, the build would be rejected and a new build would be released once the failed tests were fixed.

I could see each stage of tests running one after the other. It would stop going further if there was a failure at any of the tests or stages. Isn’t that magical!!!

It was just the beginning for us and especially for me. I continued to understand and observe more and more, and then I discussed with my team if we could add regression testing to the continuous integration pipeline. But initially we did not wanted to add any long running tests to the pipeline as we thought the whole point of having these pipelines was for quick feedback. So we decided to run the regression tests as nightly run task instead of running it with every commit. But we had the goal to run these regression tests with every commit once our pipeline was mature enough, when we would decide to run those tests in parallel. But that was a long term goal.

Each morning I could see the results of which tests where passed and which were failed and the difference was that I could add the build number while raising the issue, so it got easier for a developer to find the root cause. It’s not to say that it tells you where exactly where the issue is but they could easily check the recent code related to that build to see what went wrong. Instead of searching a needle in a haystack, it’s like you know which part of haystack you need to look to find the needle.

I got to spent most of my time as tester defining the test strategy, exploratory testing, being part of the 3 amigos to refine user stories and acceptance criteria, checking the test feedback from the pipeline, and adding more valuable tests to regression tests.

As we got along and had more clarity and saw the benefits for having these pipelines in place, we decided to discuss a long term goal, where I suggested to have visual testing as part of the pipeline as well, from a testing point of view.

This was my actual pipeline journey into the world of testing! It was very challenging and interesting!!

Thanks for reading my story!!  Happy Testing!!!!