A place to share ideas, techniques, resources about all the testing activities needed to feel confident about delivering value to our customers frequently and sustainably.
Tag: continuous testing
Continuous Testing in a seasonal business
For many countries around the world, the heaviest retail season is already starting. In the U.S., Black Friday, the heaviest day for brick-and-mortar retail sales, and Cyber Monday, when online retailers get the most traffic, are almost upon us. The delivery teams responsible for the software that powers those retail engines must be busy.
Or… are they on a steady cadence, practicing continuous delivery, and not feeling stressed? Teams that continually release small changes to production should be able to do so in a sustainable way. Hmm.
Most business domains revolve around some kind of seasonal pressure. Some industries have a big trade fair, where everyone shows off their new product. All development is aimed at that big event. Tax preparation products will be used more heavily when people have to file their returns. Government regulations may change annually, requiring software updates to remain compliant.
How do teams achieve continuous delivery, including testing at all parts of the DevOps continuous loop, with seasonal spikes in demand? Please join me along with , ,
,
and
as we share our own experiences with these challenges in a webinar panel hosted by
, Thursday, 14 November. We hope you can join us live and bring your questions. Please
to attend or to get the link to the recording after the event.
I am going to share five handy tips for testers in today’s DevOps world from my experience in this post.
Learn Continuous Testing
The first and foremost thing to learn about is: what is
continuous testing
? How do DevOps processes impact the tester’s approach for transitioning from traditional waterfall to agile testing? Before starting with continuous testing, for each sprint, we were testing incrementally and then did the regression testing before deployed to production. Up to now, our sprint cycles for some projects have been weekly or bi-weekly. But this cycle becomes too long to roll out a feature or customer request in a timely way. So, we have adopted DevOps in our process.
Testing over the whole lifecycle by shifting left and shift right, i.e. testing continuously from beginning to end.
Learn about DevOps jargon.
I recommend taking a look at
Lisa Crispin’s course
in Test Automation U to learn basic concepts and terminology around continuous delivery and DevOps. That will help you communicate and collaborate with everyone involved in delivering your product, including operations and infrastructure specialists.
A Culture Shift from an Independent Team to Whole Team Approach
In a waterfall methodology, testing is confined to an independent verification and validation team. In agile development, it becomes a collaborative effort.
People come with baggage: preconceived notions, past experiences, prejudices. The primary responsibility is educating the team about DevOps culture.
In DevOps culture, the focus is on the product, not building fiefdoms and amassing political power. The DevOps slogan is, “Everyone involved with delivering software is responsible for delivering high-quality software.” In short, the delivery becomes a shared responsibility.
Build A Successful Test Strategy
Building the right test strategy is critical in continuous testing. When we have 500 automated tests, we can’t execute all the 500 tests for each change committed in the pipeline. It will take too much execution time. So, we have analyzed and prioritized the test cases by their core functionality as well as using input from the customer support and business teams. In the end, we have come up with 100 automated regression tests which are enough to give us confidence that the latest change hasn’t broken existing features.
Don’t Just Make The Tests Pass, Focus on Efficiency
When we started with DevOps, in our continuous pipeline, our automation tests failed in between or at the last step of the pipeline. We analyzed the test results and the error logs. Most of the errors were due to timeouts or because of the new feature or enhancement changes. We have updated our scripts periodically to take care of this. After deployment, when we cross verified the product/project using manual exploratory testing, we saw a few more failures. Then we did a brainstorming session with our entire product team to make our automated tests more efficient. For this, we have created/updated the automation scripts in the preproduction phase to save time and resources.
Choose Your Tests Wisely
This is related to the points discussed in the previous paragraph. When we started doing the deployment through the DevOps practices, as testers, we need to choose our tests wisely to add those to the pipeline. Along with the regression test suites, 24/7 monitoring should be added. This helps us to understand how the product is to be used by the customers. For choosing the tests, we use input from the user behaviors that we learn from our monitoring tools, as well as discussions with the business stakeholders.
I hope the information from this experience helps you to enhance your testing!
I hold a Certified Software Test Manager and Six Sigma Green Belt. I have a passion for testing and learning, as well as implementing and mentoring the latest automation tools. I spent the last 16+ years working in IT, with a focus on software quality assurance and software testing. I am Testing Guild 2018 conference speaker and a frequent blogger. I manage a fast-growing, dynamic team bringing you
https://www.qatouch.com
. Twitter: . Linkedin: . Blog :
www.bhavaniraj.com
Even when I worked on waterfall teams, we tested early and often. We got ourselves invited to analysis and requirements meetings. We tested requirements docs. We asked questions. We tested as the developers delivered code, even before “code freeze”. We also worked closely with customer support to learn about how customers used the features after they were delivered. (I realize not all teams using a phased-and-gated approach do all that, but that approach should not stop teams from using good practices!)
Extreme Programming (XP) made so much sense to me because it embraced testing and quality as the responsibility of the entire delivery team. In agile development, we focus on building quality in, we test ideas, we automate regression tests, we explore delivered code, we test in production. These same practices apply to teams who use the “DevOps” label as well.
That’s why Dan Ashby’s
post in 2019 on “continuous testing”
resonated so much with me. His visual model captures what is in my brain when I see the term “continuous testing”. With his permission, I often use this model to help others understand the concept:
Dan Ashby’s Continuous Testing Model
But there are different definitions!
WAT!?
Recently, someone posted a link to a short Twitter thread from earlier this year where continuous testing was defined quite differently. Basically, this alternative definition is only about running automated tests in the continuous integration, delivery, deployment pipelines. I started asking around to see how other people define “continuous testing”, and some people shared that definition –
only the automated suites in CI/CD.
A book on “continuous testing” which I had not gotten around to reading also has this definition, and it references the Wikipedia page which says the same thing.
Automated tests in CI are not new!
Continuous integration has been around for decades. My waterfall team back in the early 90s did CI, complete with automated regression tests and automated deployments (to 20+ Unix environments!) We didn’t call the automated regression tests “continuous testing”. We called them automated regression tests. It puzzles me that people would apply the term “continuous testing” to these automated test suites, which teams have been using since long before the term “continuous delivery” appeared.
Going to the source
I learned about continuous delivery by reading the terrific
book
of the same name by Jez Humble and David Farley. If you read it, you will find it is a book about testing. It goes into much detail about many different kinds of testing activities – not only automated regression tests!
So I went to Jez’s website to see how he
defines “continuous testing”
. Please click that link and read it for yourself. To quote Jez,
Our goal is to run many different types of tests—both manual and automated—continually throughout the delivery process.
That is followed by a picture of Brian Marick’s agile testing quadrants, illustrating a range of tests that are business-facing, technology-facing, that guide development and that critique the product. In Jez’s quadrants model, the tests that critique the product include manual tests.
Jez goes on to explain the concept of the deployment pipeline and the importance of getting quick feedback from automated acceptance tests that are in the pipeline. But he does not limit “continuous testing” to only those tests.
Here’s the latest version of the agile testing quadrants model that
Janet Gregory
and I use. We’ve added examples of testing activities we can do today thanks to newer technology and tools, such as monitoring and observability. Those are also part of continuous testing.
Agile Testing Quadrants from
Agile Testing Condensed: A Brief Introduction
by Janet Gregory and Lisa Crispin
Does the terminology matter?
I don’t like to be too pedantic about terms, but we need to be able to communicate with each other. If my idea of continuous delivery was limited to only the pipeline and did not include the concept of breaking features into small slices and deploying small changes frequently, that would not match what most practitioners I know consider to be “continuous delivery”. It would be hard to have a conversation about it with two very different definitions.
Many people use the term “automated regression checks” rather than “automation regression tests” because these tests are checking that requirements are met and that established behavior continues to work as expected. They reserve the term “testing” for activities that are done by humans. Janet suggested that maybe we can refer to the automated regression test suites in CI / CD pipelines as “continuous checking”, and reserve the term “continuous testing” to include those checks PLUS all the testing activities that happen around Dan’s infinite testing loop.
How can we resolve this terminology dilemma? Feel free to comment here. There’s also a discussion topic for this on the
Ministry of Testing Club
.
Meanwhile, I shall see if there are even more definitions of continuous testing I hadn’t heard yet!
Open Sourcing Your Mindset
By Brendan Connolly
Asdevelopment has shifted from waterfall to agile and now DevOps, at each stageyou see developers taking on more responsibility and accountability.
Waterfallwas isolationist, building silos around different groups in softwaredevelopment. Developers owned the act of coding and in the stereotypicaldeveloper way, had little to no interest what happened before or after theycoded. Other roles on teams in many ways acted as insulators to keep externalforces away from developers.
Agilewas developers pulling that insulation away, shifting themselves left andtaking more ownership of the product creation process. Developers found theyrequired a real time feedback loop to build the right things and successfullyreact to change.
DevOpsis essentially the same process on the right side of the spectrum. Developmenttakes more ownership of deployment, monitoring and supporting the applicationshealth. It’s just too slow and cumbersome to react to the requirements ofmaintaining applications by filtering feedback through separate teams at scale.
Thebenefits of integration outweigh the cost of the additional responsibility.
Code Continuous
Tocope with the additional responsibilities, developers have turned to theirtrusty friend
code
.
Withagile we see rise of tooling developed for
ContinuousIntegration
to alleviate the pain and reduce the risk in iterativedevelopment.
InDevOps we see the rise of code and tools supporting
Continuous Deployment
. Even when many teams already had productinstallers, or automated deployments what was missing was the integration ofthat information.
Ifdevelopers are owning more of the product as well as release and deployments,then testing naturally gets swept up in the process. In fact we see thispattern emerging with the concept of
ContinuousTesting
. Code is being written, tools created to help solve the issue ofintegrating test execution and reporting into teams release pipeline. Thefeedback loop is extended to provide rapid information to make it easier andfaster for development to be responsive to change.
Continuous Testing
Automatedtesting is a loaded term. In developer circles it almost exclusively means unitand possibly integration tests. In QA it almost almost exclusively means end toend (e2e) tests, and is often synonymous with
Selenium
.
Onteams where both sets of tests exist, their worlds may seldom interact.Developers may be aware of the
e2e
tests but they probably aren’t writing them and even less likely running them.
ContinuousTesting has nothing to do with the testing that most QA team members are mostfamiliar with, exploratory testing. The purpose of the code and tooling thatcomprises a pipeline isn’t for there to be some manual intervention requiredpart of the way through it. Putting exploratory testing somewhere in the laterstages for the pipeline is akin to pointing a fire hose at someone and askingthem to direct the flow to the next stage.
Ifthe
speed of DevOps
wasn’t enough, the presence of
continuous testing
only puts morepressure on testers, simply because more attention is being drawn to testing.
Automationengineers and product engineers are being drawn closer together. The importanceand relevance of end to end automated tests is becoming more of a reality forproduct developers as they become run regularly as a post deploy verificationtests.
Whilecode is being thrown at testing with newfound passion, exploratory testers needto be prepared for their time in the spotlight. If automated tests are runningall the time what tests are you doing? What makes them special? Why are theyrequired before we can deploy?
What do you do different than Selenium?
Beforeyou dismiss that question, pause and consider what all selenium and the testsit drives truly provide.
Seleniumis cross browser, scalable, schedulable, repeatable, and can provide logs andscreenshots of its tests. Can you compete with that?
Flaky,slow, and wrong sometimes, you say? And you and all other humans aren’t?
Conversationsthat try to explain using people instead of machines for testing software canend up justifying them under the guise of
thereare some tests that aren’t good candidates for automation
.
Don’t settle for executing tests that don’t have enough ROIto automate.
Thistype of conversation relies on the belief that testing is comprised to twoelements:
test case creation
and
test execution
. Perhaps at its most baselevel testing is just that, the value proposition of the role is in the nuancethat a skilled tester can provide to teams through those elements.
Testersdo more than a fill a gap in automated test coverage. The problem is manytesters express this nuance solely through actions.
Developersproduce code, a measurable artifact from their effort. The generation of thatartifact is often backed by computer science, a formalized area of study.Artifacts of effort are reassuring, people like to see the fruits of theirlabors. Tangible results can be measured and quantified, they might even helpestimate future effort.
Joan of Arc
A tester’s value tends to be recognized at the same rate as relationships develop. Testers get characterized as
folk heroes
rather than members of a skilled profession. There’s no trade, education, or craft just individuals saving the day periodically.
The Intuition trap
Evenwhen teams do value testing, people are usually primarily concerned with theprogression or regression of testing efforts. Testers comments in stand-ups caneasily turn into a user story status change report:
Passed this ticket, working this ticket, noblockers
Ifyou aren’t taking the opportunity to discuss your actual actions in stand ups,then you may fall prey to the intuition trap. When you talk about your workthis way, sometimes you end up actually working this way.
Open Source Your Mindset
It’sfun being a hero, but there’s no skill in being
Superman
or
Superwoman
(I was going to say Wonder Woman, but she actually trained and I think everyonerecognized that. So if you are the teams Wonder Woman then kudos, you haveserious skills but I digress).
Whentesting goes smoothly the result is
nothing
.Testing isn’t required for things to go right, it just might help prevent athing going wrong.
Opensourcing your mindset means
teaching
others another way to look at and interact with their code, or the product.This cuts both ways though, it means being willing to accept a pull request ofoutside perspective. Openly integrating new ideas into your
testers mindset
.
Replace Execution with Communication
. Like acting or athletics there might be a script or game plan to start with but there is a lot going on behind the scenes, adapting to circumstance and being thought up on the fly. It’s intuition driven.
Whenactions, practices and motivations are not being verbalized and expressed tothe team and stakeholders it puts testing on the level of superstitions andtoken time sacrifices to the quality gods.
Testersneed to pry open this experience, expose the internals driving motivations.Breaking out of that shell grants visibility into a testers actions andmotivations to provide a context for team members to attribute value.
Replace Isolation with Collaboration
Itcan be uncomfortable or difficult to open up and share what goes through your
testing mindset
.
Theadditional ownership developers have taken in DevOps has changed relationshipdynamics. What was at one point a contentious us vs. them relationship is muchmore open. There just isn’t the time to have as much formal process drivencommunication. Success at scale requires collaboration, pairing and mobbingmore.
When with developers, it’s an opportunity to test at their desk, against their local changes. There are risks testing here, the developers local setup is definitely different than production but so are the testing and staging environments. Let go of the taboo of
it works on my machine
and consider what can be tested here and what needs to be covered on a different stage. Take the opportunity to work together when the code is the freshest in the developers mind and where they are most comfortable.
It’s not only about working with developers. Mob testing with
UX
and
Product
alongside developers is a way to draw the entire team together. When the team is together seeing how the application is working it bypasses all the communication gates. There is no need for a separate follow-up conversations to loop someone in, that person can see and feel how things are working by being a part of the group experience.
Replace Tedium with Stratagem
Teamsare more open to testing and sharing in quality than ever. Testers have beenbuilding quality mindsets for years. Developers are building tooling but teamsare looking for guidance and mentorship in
Quality
.
Testing isn’t just button pressing, by showing that through communication and working together we elevate our status. Draw people together, into the quality circle, and enable the team to be more comfortable and aware of testing. We can set working agreements, talk about overarching strategy for how testing throughout the development cycle builds quality in. We can embrace automated testing as a team activity where unit, integration and end to end coverage is part of a teams strategy for mitigating risk. We can forgo complete control of testing and opt into being a voice of authority and expertise in quality.
Brendan Connolly is an experienced Software Tester, Automator, Speaker, and Blogger. Currently he is a Senior Quality Engineer at Procore Technologies in Santa Barbara, California. He maintains a testing blog @ http://www.brendanconnolly.net
Testers, DevOps, and Enabling Digital Transformation
By Gerie Owen
To be successful in today’s businessenvironment, virtually all organizations must plan for digital transformation,not only to thrive but to survive. Often, when organizations embark on digitaltransformation initiatives, they turn to DevOps to increase business value andimprove customer experience.
Testers, as champions of the customer andguardians of their organizations’ digital reputations, are uniquely positionedto enable digital transformation. Weenable digital transformation by focusing testing on business risk and customervalue as well as transforming our test processes.
DevOps enablesDigital Transformation. The synergy is clear: digital transformation and the principlesof DevOps are closely aligned. Specifically,continuous integration and continuous delivery enable innovation in operations,streamline processes and eliminate bottlenecks.
Continuoustesting plays a key role in the continuous delivery pipeline by enabling bothhigh quality and high velocity and ensuring customer happiness. Let’s examinehow continuous delivery enables digital transformations by streamliningprocesses and focusing on value to the customers and how testers can enable andchampion this process.
DevOps Principles Aligned with Digital Transformation
Digital transformationcan be defined in many ways; however, it is a fundamental shift in the way anorganization approaches its business and operational processes by usingtechnology to increase focus on and improve customer experience. Most digital transformations are focused oncustomer experience, operational processes and business models. DevOps principles described in the
ThreeWays
— systems thinking, amplifying feedback loops and creating a culturalof experimentation and continual learning — can be the basis for creating the majorchanges that are required for digital transformation.
Systemsthinking or focusing on the performance of an entire process is the mosteffective approach to transforming business models and operationalprocesses. If changes are made tofundamental operations in siloed initiatives, they will likely fail. Amplifying feedback loops is critical todigital transformation so that corrections can be made early and often. And creating a culture of continuousexperimentation and learning encourages innovation without fear of consequencesif a new business, process or technological approach fails.
TestersEnabling Digital Transformation by Transforming Test Processes
Testers enable digital transformation not onlyby championing the customer, but also by transforming their testpractices. Transformational testpractices include implementing the practices of continuous testing, including“shift left”, test optimization, as well as the use of service virtualizationand advanced analytics, and “shift right”, i.e., testing in production.
Test Automation Pyramid, from More Agile Testing by Janet Gregory and Lisa Crispin
Continuoustesting is an approach to managing risk by focusing not only on eliminatingtesting bottlenecks but more importantly, increasing the effectiveness of ourtest processes. The test pyramid is a“shift left” pattern whereby most of the automated test cases are concentratedin the unit, API, component and integration layers with fewer automated testcases used to test the GUI layer.
Testersas Champions of the Customer
As testers,meeting the challenges of our role in digital transformation requires acoordinated approach made up of several components. Customers expect a seamless experience acrossall channels; however, they do not necessarily expect to perform the same actionson every channel. Therefore, a digitaltransformation test strategy will require using customer behavior analytics tounderstand customer’s browsing, purchasing and other transactions acrosschannels in order to test efficiently and effectively.
As champions ofthe customers, we must focus on testing the customer experience. This means we must learn to use customeranalytics, personas and customer journeys and combine the results of those withexploratory testing.
Testers and DevOps can be the keys to successful digitaltransformation through by focusing on technological transformation based onbusiness value and customer happiness. Testers, this is your opportunity to goabove and beyond and contribute to your organizations in innovative,customer-centric ways.
Gerie Owen is a Testing Strategist and Evangelist. She is a Certified Scrum Master, Conference Presenter and Author on technology and testing topics. She enjoys mentoring new QA Leads and brings a cohesive team approach to testing. Gerie is the author of many articles on technology including Agile and DevOps topics. She recently developed a curriculum for DevOps 101 training. Gerie chooses her presentation topics based on her experiences in technology, what she has learned from them and what she would like to do to improve them. Gerie can be reached through her website, www.gerieowen.com, her blog, Testing in the Trenches, is at https://testinggirl.wordpress.com/ and @GerieOwen on Twitter and on LinkedIn .
