A place to share ideas, techniques, resources about all the testing activities needed to feel confident about delivering value to our customers frequently and sustainably.
Author:
Lisa Crispin
Continuous Testing in a seasonal business
For many countries around the world, the heaviest retail season is already starting. In the U.S., Black Friday, the heaviest day for brick-and-mortar retail sales, and Cyber Monday, when online retailers get the most traffic, are almost upon us. The delivery teams responsible for the software that powers those retail engines must be busy.
Or… are they on a steady cadence, practicing continuous delivery, and not feeling stressed? Teams that continually release small changes to production should be able to do so in a sustainable way. Hmm.
Most business domains revolve around some kind of seasonal pressure. Some industries have a big trade fair, where everyone shows off their new product. All development is aimed at that big event. Tax preparation products will be used more heavily when people have to file their returns. Government regulations may change annually, requiring software updates to remain compliant.
How do teams achieve continuous delivery, including testing at all parts of the DevOps continuous loop, with seasonal spikes in demand? Please join me along with , ,
,
and
as we share our own experiences with these challenges in a webinar panel hosted by
, Thursday, 14 November. We hope you can join us live and bring your questions. Please
to attend or to get the link to the recording after the event.
DevOps, DevTestOps, Dev*Ops… what’s in a label?
by Lisa Crispin
The catalyst for this
community website
was a “DevTestOps Manifesto” dreamed up originally by a teammate and myself, then with input from the testing and DevOps community. After lots more discussions with practitioners, we realized that what we had were some
guiding principles
rather than a “manifesto”.
We also had a lot of discussions about the “DevTestOps” term. I liked it because when a lot of people see the word “DevOps”, they think of programmers/coders plus operations specialists, and don’t include testers or other specialists. Having “Test” in the label could help make testing and testers more visible. After all, testing is the heart of DevOps and continuous delivery. Others pointed out to me that using the term “DevTestOps” could also be construed as promoting silos again, which is the opposite of what agile development and DevOps are all about.
Who’s included in “Dev”?
Everyone who helps to create, deliver and maintain a software product is a developer, and that includes testers. The term DevOps came about 10 years ago because a lot of organizations said they had adopted “agile”, but they were still throwing code over the wall to operations teams to deploy it, nurture it in production, and carry the pager. After “DevOps” was established, we started hearing about “DevSecOps”, which has become an established term and specialty. I’ve seen other labels merged with Dev and Ops, like DevAIOps and DevUXOps.
The Whole Team Approach to building quality into software has to include operations folks, along with product people, designers, business stakeholders, sales and marketing, business analysts, database experts, data scientists… we are all in this together. It may be useful to insert another label between Dev and Ops, to highlight an activity which some people may not consider in terms of the practices needed to succeed with continuous integration and delivery, but “DevOps” is a fine, all-inclusive term.
Labels and goals
Personally, I wish we did not have to use any kind of labels like “agile” and “DevOps”. Our aim is to continually improve how we build software so that we can delight our customers while enjoying the process ourselves. But it seems that we humans love to label things.
Our goal here at TestingInDevOps.org is to get testing and QA professionals interested in learning about DevOps and continuous delivery, and provide resources to help them learn. Naturally, we welcome people in any specialty or role to learn how testing activities fit into and add value to DevOps culture and practices. We especially want to make testers visible in this DevOps and CD world. People with deep testing skills and experience help everyone on their teams learn testing skills and find ways to bake quality into their product during the whole continuous loop of discovering what customers want, building delivering those features.
Let’s find more great ways to deliver small, valuable changes to production frequently, at a sustainable pace that lets our teams enjoy the journey!
State of DevTestOps survey
by Lisa Crispin
Yes, the State of DevOps survey is the gold standard for truly scientific and valuable surveys, and you should both fill it out every year and read the reports and the book
Accelerate
! And, there are lots of testing-related surveys out there. They all provide interesting information that help us think about our profession and where we need to grow. Why yet another testing-related and DevOps-related survey? Because we have so much more to learn!
It’s the 2nd year of the
survey. I was happy to help with it again, because last year’s survey provided some valuable and thought-provoking
. It generated some interesting conversations and led to productive panel discussions for our Testing in DevOps community here as well as other webinars (see the
webinars and videos page
here for links to those).
Testers need to learn about DevOps and continuous delivery, and engage with people in other roles in their organizations to help their own teams shorten cycle times and improve their ability to build quality in. DevOps needs testers –
testing is the heart of DevOps
. Getting some insights as to how teams are progressing in DevOps practices and culture, as well as how they have incorporated testing activities and testers, is vital so that we can find more ways to add value as testers/testing coaches/testing consultants.
Whether your team has started to embrace DevOps and continuous delivery or not, please take eight minutes (or less!) to fill out the
, and share your team’s experiences with what many people now refer to as shifting testing to the right – testing activities during the release process and in production. You don’t have to give your email or any other personal information.
We’re adding webinars to our Testing in DevOps community learning opportunities! Our first one is a panel discussion with awesome leading practitioners , , , and I will also join! Many thanks to
for hosting the webinar (they host this site as well).
So what will we talk about? As many of our guest posts here have pointed out, DevOps is about
the whole team
– including operations specialists – working together to deliver value to customers frequently, without drama or stress. It’s that continual loop: together we identify the most valuable capability to delivery next.
In the webinar, we’ll talk about how we build and test it, test and build it, together. We have created a pipeline to deploy it to production, together. We’ll probably get into topics such as monitoring and observability to learn how customers use the product and respond to problems, and get ideas for the next valuable feature slice to build. We will answer your questions!
For more details and to register, please go to my
(I am too lazy to copy it all over here!)
We welcome your questions in advance, just tweet them to @! Or, simply ask on the day in the Zoom chat. We look forward to an insightful and fun discussion!
The different pages of this website have links to so many helpful articles, podcasts, videos, webinars, books and more. Here are three that you might want to prioritize to read or listen to first, to build your understanding of devOps, continuous delivery, testing in production, and what makes software delivery teams successful.
Charity Majors on testing in production
Does the phrase “testing in production” scare you? Historically, it was an irresponsible practice – throw our changes into production and let the customers tell us what’s wrong. Today, it is a leading practice. In her recent article “
I test in prod
“, says it’s a superpower!
Most interesting man in the world
Charity points out that we test in production all the time. The key is to do it safely. It takes an investment in tooling and training. This is especially important with the complicated distributed apps we support today. We need to test more than the code – we need to test the whole system, including the environment, the infrastructure, and the customers.
There’s so much to learn in this one article, from Charity’s experiences and insights. She breaks down the three aspects needed to start testing in production in a useful and beneficial way: technical, cultural and managerial. Plus, she works in so many classic memes to hilarious effect! Give it a read and start overcoming the fear of what will happen with each new change released to production.
Theresa Neate: Stop the whisper campaign that devOps means no testers needed!
We’ve been fortunate to have a
guest post
here from , one of the leading testing practitioner who is also a leader in devOps practices. She gives us a preview of her
TestBash Australia
talk in this
Ministry of Testing podcast
with Michael Lang.
Theresa starts with one of the best explanations of devOps and continuous delivery that I have heard. If you are not yet familiar with these terms, listen to the first few minutes of this podcast.
One of the most important messages Theresa delivers here is that we need to stop the “whisper campaign” that devOps means “no testers needed” and “no testers welcome here”. We know we testers can add huge value by participating in devOps practices and helping our teams succeed with continuous delivery. it’s up to us to use our natural curiosity, learn important new skills, and show what we can do.
Photo from Christina Morillo on Pexels.com
This podcast is less than 30 minutes long, and it’s going to be the best half hour you’ve invested in your career in awhile.
I wish I could go to TestBash Australia and participate in Theresa’s workshop “Hands-on Intro to AWS and a Taste of DevOps, for Testers”!
Accelerate: State of DevOps 2019
This led by is building on 6 years of data from a wide group of respondents. The executive summary alone is a gold mine of information about what practices and capabilities correlate with team success.
From the
The report includes insights into testing practices. For example, though previous years found that test automation has a significant impact on continuous delivery (CD), this year they found that automated testing positively impacts continuous integration (CI).
The ability to reproduce and fix failures, gather feedback from tests, improve test quality and iterate test runs quickly also ties into automated testing.
The report also focuses on disaster recovery testing. One finding is that organizations who do disaster recovery testing and create and implement action items based on what they learn from it are more likely to be in the elite performing group.
The report delves into the “Four Key Metrics” of software development and delivery, one of which is culture – too often overlooked by organizations. To improve, organizations need to start with the foundations, and adopt a continuous improvement mindset.
Read the report and help your own team be one of the many teams who are continuing to improve their ability to deploy small changes to production frequently, while enjoying their work.
Even when I worked on waterfall teams, we tested early and often. We got ourselves invited to analysis and requirements meetings. We tested requirements docs. We asked questions. We tested as the developers delivered code, even before “code freeze”. We also worked closely with customer support to learn about how customers used the features after they were delivered. (I realize not all teams using a phased-and-gated approach do all that, but that approach should not stop teams from using good practices!)
Extreme Programming (XP) made so much sense to me because it embraced testing and quality as the responsibility of the entire delivery team. In agile development, we focus on building quality in, we test ideas, we automate regression tests, we explore delivered code, we test in production. These same practices apply to teams who use the “DevOps” label as well.
That’s why Dan Ashby’s
post in 2019 on “continuous testing”
resonated so much with me. His visual model captures what is in my brain when I see the term “continuous testing”. With his permission, I often use this model to help others understand the concept:
Dan Ashby’s Continuous Testing Model
But there are different definitions!
WAT!?
Recently, someone posted a link to a short Twitter thread from earlier this year where continuous testing was defined quite differently. Basically, this alternative definition is only about running automated tests in the continuous integration, delivery, deployment pipelines. I started asking around to see how other people define “continuous testing”, and some people shared that definition –
only the automated suites in CI/CD.
A book on “continuous testing” which I had not gotten around to reading also has this definition, and it references the Wikipedia page which says the same thing.
Automated tests in CI are not new!
Continuous integration has been around for decades. My waterfall team back in the early 90s did CI, complete with automated regression tests and automated deployments (to 20+ Unix environments!) We didn’t call the automated regression tests “continuous testing”. We called them automated regression tests. It puzzles me that people would apply the term “continuous testing” to these automated test suites, which teams have been using since long before the term “continuous delivery” appeared.
Going to the source
I learned about continuous delivery by reading the terrific
book
of the same name by Jez Humble and David Farley. If you read it, you will find it is a book about testing. It goes into much detail about many different kinds of testing activities – not only automated regression tests!
So I went to Jez’s website to see how he
defines “continuous testing”
. Please click that link and read it for yourself. To quote Jez,
Our goal is to run many different types of tests—both manual and automated—continually throughout the delivery process.
That is followed by a picture of Brian Marick’s agile testing quadrants, illustrating a range of tests that are business-facing, technology-facing, that guide development and that critique the product. In Jez’s quadrants model, the tests that critique the product include manual tests.
Jez goes on to explain the concept of the deployment pipeline and the importance of getting quick feedback from automated acceptance tests that are in the pipeline. But he does not limit “continuous testing” to only those tests.
Here’s the latest version of the agile testing quadrants model that
Janet Gregory
and I use. We’ve added examples of testing activities we can do today thanks to newer technology and tools, such as monitoring and observability. Those are also part of continuous testing.
Agile Testing Quadrants from
Agile Testing Condensed: A Brief Introduction
by Janet Gregory and Lisa Crispin
Does the terminology matter?
I don’t like to be too pedantic about terms, but we need to be able to communicate with each other. If my idea of continuous delivery was limited to only the pipeline and did not include the concept of breaking features into small slices and deploying small changes frequently, that would not match what most practitioners I know consider to be “continuous delivery”. It would be hard to have a conversation about it with two very different definitions.
Many people use the term “automated regression checks” rather than “automation regression tests” because these tests are checking that requirements are met and that established behavior continues to work as expected. They reserve the term “testing” for activities that are done by humans. Janet suggested that maybe we can refer to the automated regression test suites in CI / CD pipelines as “continuous checking”, and reserve the term “continuous testing” to include those checks PLUS all the testing activities that happen around Dan’s infinite testing loop.
How can we resolve this terminology dilemma? Feel free to comment here. There’s also a discussion topic for this on the
Ministry of Testing Club
.
Meanwhile, I shall see if there are even more definitions of continuous testing I hadn’t heard yet!
Visualizing Pipelines to Help Build a Quality Culture
By Lisa Crispin
At
CAST 2019
, August 12-15 in Cocoa Beach, Florida, and I are pairing to facilitate an all-day workshop called “The Whole Team Approach to Testing in Continuous Delivery”. Our focus will be helping people to find new ways their teams can build a quality culture and succeed with continuous delivery. An important part of this is learning about deployment pipelines. For those of you who can’t attend the live version, here’s an introduction to why deployment pipelines are vital to successful continuous delivery, and a fun exercise your team can use to improve your own pipeline.
Continuous delivery – scary?
Continuous delivery (CD) lets us deliver new value to production safely, quickly and sustainably (to paraphrase Jez Humble). Each new commit to our code base trunk or master is independently verified as a deployable release candidate. We can deliver frequently, which means the changes are small, reducing the risk of problems.
Still, CD can seem like a scary proposition. In the old waterfall days, we didn’t have enough time to complete all the testing activities we wanted. Agile development enabled more bi-weekly or weekly releases, and brought up new testing challenges. And now, how can testing possibly keep up with continuous delivery? How can we feel confident about each release, from a testing/customer support perspective?
New approaches
Following the
principles of continuous delivery
helps us build confidence in our deployments. Following
modern testing
principles can give us even more confidence. When the whole team takes responsibility for building quality into the product and continually improving how we work, we (and our customers) realize the benefits of CD.
This is a huge topic, but let’s look at two key factors for success with CD. Obviously, we need adequate automated regression test coverage to make sure that the things our customers really need are still working. That’s not new. And, it isn’t a complete safety net. We can have great automated regression test coverage, do thorough exploratory testing, verify performance, and all the other many types of testing before we release to production, although we are doing that in test environments. We know that behavior is likely to be different and unexpected in production.
So what is new? Today, we have the ability to capture and analyze all kinds of data (and huge amounts of it) about production use and immediately detect any errors that occur. This is important, but it’s not that helpful unless we can quickly and reliably deploy fixes to production. And to do that, we need a fast, reliable deploy pipeline.
P
ipeline basics
The deployment pipeline consists of steps that changes go through before they can be deployed to production. Every company, every team has their own unique pipeline. Note that whether or not your team has automated tests and automated deploys – you still have a pipeline. Whatever steps a new change goes through to get to production, even if each step is manual, is still a pipeline.
We break our build into stages or steps to speed up feedback. There’s a constant trade-off between fast feedback and confidence. The early steps of a pipeline are usually unit tests. They quickly show us whether anything is broken at that low, granular level. But we need more steps with higher-level, longer-running tests to feel more confident. Later steps in our pipeline exercise more complicated scenarios. We want fast feedback, but we may need comprehensive tests for confidence, and they take a long time to run. Here’s an example deployment pipeline:
In this example, the steps outlined in blue in this example are manual steps. We focus on automation when talking about pipelines, but manual steps are common. As in this example, activities like exploratory testing and security testing may be critical and may require manual work. We should be able to automate deploys to different environments, but perhaps your team just hasn’t gotten to that yet.
Things that slow down the pipeline
“Flaky” or “flappy” tests are a common problem even in completely automated pipelines. We often see this with user interface (UI) tests. Someone makes a change to the UI but forgets to update the automated tests to reflect that. A test environment slows down and causes a test step to time out. The browser version automatically (and unexpectedly) updates in the test environment. Those are just a few common causes of UI test failures.
Dependencies can also cause difficulties. A failure of one stage can stop multiple subsequent steps from running. Test data may fail to be set up properly, causing test failures. A test environment in the cloud might fail to spin up.
More steps in the pipeline give us more confidence, but they take more time. And how do we know all the steps are necessary?
When a pipeline experiences more and more delays, we humans have a tendency to increase the size of the items we’re sending through it to try to get more done. If the tests are failing and people keep committing changes, we’ve now got a big batch of changes which is more likely to include regression failures. It’s a death spiral. If our pipeline is reliable and fast, fewer changes go through it each time, reducing the risk of failures.
Collaborating to improve the pipeline
Making problems more visible is a great first step in making them smaller. Gather your team (or at least a representational group, if you have a large team) together. Take cards or paper and write the name of each step of your pipeline, one per card or paper. Put them on a table or on a wall to create a picture of your pipeline. (This can be done virtually if your team is distributed).
Visualizing a pipeline on a table – picture used with permission from Llewllyn Falco
Next, look at each step in the pipeline and ask “why do we have this step?” What are we learning from it? What business questions does it answer, what risks does it mitigate? Who on the team benefits from this information? You can capture this information on sticky notes to attach to each step.
Then, think about dependencies. For each step, what must in place for it to run successfully? A test environment? Test data? Specific configurations? Also think about how you can optimize the step. Is it currently done manually? Can you automate it?
Identify what triggers each step. Is it triggered automatically by the success of a previous step? Or manually? Can it be run in parallel with one or more other steps? Do you see opportunities to shorten feedback loops? Again, record outcomes of these discussion for each step.
Look at the length of your feedback loops. How much time elapses from code commit to completion of each step? What happens if the step fails – do the right people get notified? How? Is there a better way to communicate alerts? My previous team found that everyone was ignoring the Slack alerts generated by pipeline failures, so they hooked the build up to a police light. That got the team’s attention so that failures were quickly investigated and addressed!
Can any steps be done asynchronously? For example, it’s not realistic to do manual exploratory testing on every change when you are deploying to production frequently. The team can learn to use
release feature toggles
to hide changes from customers until the team has had time to complete any manual or time-consuming testing tasks.
Once you’ve gathered information about each step in the pipeline, choose a pain point to address and think of an experiment to reduce the pain. For example, perhaps your UI test suite has a lot of flaky tests and the team spends a lot of time kicking tests off again and diagnosing the failures that reoccur. Your experiment might be, “We believe that by refactoring the top 10 flaky tests to make them reliable will reduce our pipeline failure rate by 10% in the next month”. Use your retrospectives to evaluate your progress.
Remember the benefits
A fast, reliable pipeline means that you can confidently deploy frequent, small changes to production. And if you introduce problems to production that cause customer pain, you can shorten the time they feel pain by quickly deploying a fix with your fast, reliable pipeline. There are many other factors that go into safe, reliable and sustainable continuous delivery. Fast feedback and the ability to deploy changes quickly are a cornerstone of successful CD – and of a healthy quality culture.
Please contribute your links to your favorite content!
Welcome to our community site to help raise the visibility of testing and testers in DevOps, and help us all learn ways to succeed with testing in continuous delivery.
We’re already receiving links to useful articles, videos and podcasts related to testing in DevOps and continuous delivery. Please check out the pages in the sidebar on the right for this helpful material. Please your favorite links to articles, videos, podcasts, books that have helped you learn how testing fits into continuous delivery, how testers engage in DevOps practices, testing in production, monitoring, observability, this is a wide field!
Please contact with any feedback, comments or questions. We want to make this your site for learning more about testing in DevOps, CD, observability, monitoring and more.
Welcome to the Dev Test Ops community
Welcome to an experiment in building a community around testing and testers in DevOps! Do you work on a team with a DevOps culture? Would your team like to adopt DevOps practices to support efforts such as continuous integration and continuous delivery? How can we help each other?
Our idea is to build this community site so that we can share ideas, practices, tools and resources that help us put the Test in DevOps. DevOps helps our teams feel confident to deploy changes to production weekly, daily, many times a day. It’s a journey, we’re not all in the same place. Those of us who are a bit further along can help those of us who are newer to the concepts.
Before we go too crazy, we’d love your input. Please leave comments as to the kind of thing you would find helpful and whether it would fit on a community site such as this. We can make the site be whatever we want it to become. (Especially if I get some technical help, as I’m not a website admin or dev!)
Just to get us started, here’s the type of article I think that a lot of people might find helpful, an
about the challenges they see around testing in DevOps.
