A place to share ideas, techniques, resources about all the testing activities needed to feel confident about delivering value to our customers frequently and sustainably.
Even when I worked on waterfall teams, we tested early and often. We got ourselves invited to analysis and requirements meetings. We tested requirements docs. We asked questions. We tested as the developers delivered code, even before “code freeze”. We also worked closely with customer support to learn about how customers used the features after they were delivered. (I realize not all teams using a phased-and-gated approach do all that, but that approach should not stop teams from using good practices!)
Extreme Programming (XP) made so much sense to me because it embraced testing and quality as the responsibility of the entire delivery team. In agile development, we focus on building quality in, we test ideas, we automate regression tests, we explore delivered code, we test in production. These same practices apply to teams who use the “DevOps” label as well.
That’s why Dan Ashby’s
post in 2019 on “continuous testing”
resonated so much with me. His visual model captures what is in my brain when I see the term “continuous testing”. With his permission, I often use this model to help others understand the concept:
Dan Ashby’s Continuous Testing Model
But there are different definitions!
WAT!?
Recently, someone posted a link to a short Twitter thread from earlier this year where continuous testing was defined quite differently. Basically, this alternative definition is only about running automated tests in the continuous integration, delivery, deployment pipelines. I started asking around to see how other people define “continuous testing”, and some people shared that definition –
only the automated suites in CI/CD.
A book on “continuous testing” which I had not gotten around to reading also has this definition, and it references the Wikipedia page which says the same thing.
Automated tests in CI are not new!
Continuous integration has been around for decades. My waterfall team back in the early 90s did CI, complete with automated regression tests and automated deployments (to 20+ Unix environments!) We didn’t call the automated regression tests “continuous testing”. We called them automated regression tests. It puzzles me that people would apply the term “continuous testing” to these automated test suites, which teams have been using since long before the term “continuous delivery” appeared.
Going to the source
I learned about continuous delivery by reading the terrific
book
of the same name by Jez Humble and David Farley. If you read it, you will find it is a book about testing. It goes into much detail about many different kinds of testing activities – not only automated regression tests!
So I went to Jez’s website to see how he
defines “continuous testing”
. Please click that link and read it for yourself. To quote Jez,
Our goal is to run many different types of tests—both manual and automated—continually throughout the delivery process.
That is followed by a picture of Brian Marick’s agile testing quadrants, illustrating a range of tests that are business-facing, technology-facing, that guide development and that critique the product. In Jez’s quadrants model, the tests that critique the product include manual tests.
Jez goes on to explain the concept of the deployment pipeline and the importance of getting quick feedback from automated acceptance tests that are in the pipeline. But he does not limit “continuous testing” to only those tests.
Here’s the latest version of the agile testing quadrants model that
Janet Gregory
and I use. We’ve added examples of testing activities we can do today thanks to newer technology and tools, such as monitoring and observability. Those are also part of continuous testing.
Agile Testing Quadrants from
Agile Testing Condensed: A Brief Introduction
by Janet Gregory and Lisa Crispin
Does the terminology matter?
I don’t like to be too pedantic about terms, but we need to be able to communicate with each other. If my idea of continuous delivery was limited to only the pipeline and did not include the concept of breaking features into small slices and deploying small changes frequently, that would not match what most practitioners I know consider to be “continuous delivery”. It would be hard to have a conversation about it with two very different definitions.
Many people use the term “automated regression checks” rather than “automation regression tests” because these tests are checking that requirements are met and that established behavior continues to work as expected. They reserve the term “testing” for activities that are done by humans. Janet suggested that maybe we can refer to the automated regression test suites in CI / CD pipelines as “continuous checking”, and reserve the term “continuous testing” to include those checks PLUS all the testing activities that happen around Dan’s infinite testing loop.
How can we resolve this terminology dilemma? Feel free to comment here. There’s also a discussion topic for this on the
Ministry of Testing Club
.
Meanwhile, I shall see if there are even more definitions of continuous testing I hadn’t heard yet!
A continual problem that we must deal with when we run automated tests in our DevOps pipeline is maintenance. I could spend hours sharing posts and blogs about this matter, telling stories about how it affected our team’s capacity and time to market, and giving you several examples of what teams have done to sort this problem out. In fact, all the effort that we invest in maintenance is not visible to the business stakeholders’ eyes. Maintenance consumes our team capacity, but it does not add value to our clients directly.
Test automation challenges at scale
In one of my jobs wefaced this problem on a large scale. We had more an average of more than 1000automated tests running at each code check in, plus one big run at midnightthat we called full regression. Our suites where organized in those levels:
After each run, we noticed that 80% of UI and regression automated tests failed
due
fake failures. After reading my numbers, you can say if you automated crap, you will get crap results and that is correct — but please remember that we were building our automated test suites, with a continuous testing approach and DevOps mindset and pipelines, so we were trying to fail fast, learn and improve. First, we decided to analyze our test results and data, so we collected all the information that we got from our test runs and analyzed, we discovered that our tests failed
because
of those main reasons:
Changes inUI components.
Badcoding.
Exceptionsand errors handling
Time outseverywhere
Data
A new experiment
We had to make a decision. We had invested lot of efforts in testing automation, and we were the product team with more automated tests compare to the rest of product teams globally in the company. This brought us attention from lots of managers, who were keeping their eyes on our DevOps and test automation activities. In addition, we were participating in a DevOps competition which involved all the IT teams of the corporation. So, we decided to invest time to get the failed tests in good shape and have them passing in the pipeline. We called this mindset “Keep them green and running”. We communicated this effort to all the product teams. We asked the Product Owners to consider maintenance time as part of their team capacity, include tasks for this purpose, and keep it part of their definitions of “ready” and “done”.
Unfortunately, we did notachieve the goals we had hoped for. We still had the same test results, withmore than 80% of tests failing. In almost every case, they were fake failuresthat meant that the tests were stopped long before their last steps. To sum up,our rest results were not accurate at all and did not bring confidence to ourteams — which is something critical in testing automation.
Thinking outside the box
We needed to try something different. After some discussion, research, investigation and thinking out of the box, we decided to use gamification to keep our tests in a good shape, running and be green. We named the game “Game of Testing” and we established some critical points about it:
Maturitylevels: The gamers can be developers, testers or anybody who owned an automatedtest running in the pipeline, they were maturing in the game, from a beginnerlevel to a master level.
Visibility:We shared the game boards including ranking, test results and so, on using bigmonitors in the office, allowing everyone to be aware of the gamers andresults.
Weight:This is a very complicated and mathematical part, we needed to stablishformulas and weights for a point system for the game, so we could evaluate thetest results, data per gamer and teams, and gave points and make them get newlevels.
Recognition:We tried to recognize all the gamers taking part. For the best gamers, weshared the results with local and global senior managers, so their achievementscould be known and recognized. Thishelped encourage more team members to participate.
The game was quitesimple:
Gamers: All the team members who owned an automated test running in the pipeline.
Point system: A gamer won
point
s for test running in green and negative points if the test failed for fake results. Later we added negative points of breaking builds.
Rules: You needed to keep all your
test
and team tests running in the pipeline in a good shape for each run. So, you had to invest efforts on “maintenance” and continuous improvement if you want to rock in the game.
The Game of Testing
This approach helps usa lot to decrease false positive runs and keep the tests up and running in thepipeline. We decreased the number of 80% of failed tests to only 20% inapproximately three months and our numbers have continued decreasing. Furthermore, people had fun while playing thegame and they internalized that quality can be fun and not boring like usualmaintenance was.
While we got several criticisms about our failing tests at the beginning of our efforts to improve them, and people lost trust on our quality assurance process, we didn’t give up trying to fix the problems. In fact, it motivated our team to think out of the box and turn this problem into a useful and fun project. We spent several hours investigating continuous testing, scripting, and related topics. Besides that, we wanted to do something different, something motivates other team members to join.
Gamification can provide that motivation. You may not achieve desired results immediately, but if you prepare your environment and involve everyone in trying potential solutions, the game will give you results for sure. If you address the test issues you can get them pass or fail, but if you involve and motivate people to work on problems together, you always win and have fun. I recommend you try gamification to sort your problems out, you may be surprised at the results!
Jorge Luis Castro Toribio
Jorge is an
agilist
who feels passion for Testing Automation, Agile, DevOps and business agility. He enjoys research and learning about business agility and working with cutting-edge technologies. He has been learning continuously for more than 12 years and is still learning new ways to foster business agility and team greatness. He has worked in several roles (developer, tester, manager, agile coach) which helps him to see the big picture and operational and team members dynamics happening inside organizations. This experience lets him help teams design, build and implement digital, DevOps and agile transformation strategies. He encourages focus on: People, Productivity, Continuous improvement, Innovation and having fun to enjoy success in our agile journeys.
You can find Jorge on and .
Visualizing Pipelines to Help Build a Quality Culture
By Lisa Crispin
At
CAST 2019
, August 12-15 in Cocoa Beach, Florida, and I are pairing to facilitate an all-day workshop called “The Whole Team Approach to Testing in Continuous Delivery”. Our focus will be helping people to find new ways their teams can build a quality culture and succeed with continuous delivery. An important part of this is learning about deployment pipelines. For those of you who can’t attend the live version, here’s an introduction to why deployment pipelines are vital to successful continuous delivery, and a fun exercise your team can use to improve your own pipeline.
Continuous delivery – scary?
Continuous delivery (CD) lets us deliver new value to production safely, quickly and sustainably (to paraphrase Jez Humble). Each new commit to our code base trunk or master is independently verified as a deployable release candidate. We can deliver frequently, which means the changes are small, reducing the risk of problems.
Still, CD can seem like a scary proposition. In the old waterfall days, we didn’t have enough time to complete all the testing activities we wanted. Agile development enabled more bi-weekly or weekly releases, and brought up new testing challenges. And now, how can testing possibly keep up with continuous delivery? How can we feel confident about each release, from a testing/customer support perspective?
New approaches
Following the
principles of continuous delivery
helps us build confidence in our deployments. Following
modern testing
principles can give us even more confidence. When the whole team takes responsibility for building quality into the product and continually improving how we work, we (and our customers) realize the benefits of CD.
This is a huge topic, but let’s look at two key factors for success with CD. Obviously, we need adequate automated regression test coverage to make sure that the things our customers really need are still working. That’s not new. And, it isn’t a complete safety net. We can have great automated regression test coverage, do thorough exploratory testing, verify performance, and all the other many types of testing before we release to production, although we are doing that in test environments. We know that behavior is likely to be different and unexpected in production.
So what is new? Today, we have the ability to capture and analyze all kinds of data (and huge amounts of it) about production use and immediately detect any errors that occur. This is important, but it’s not that helpful unless we can quickly and reliably deploy fixes to production. And to do that, we need a fast, reliable deploy pipeline.
P
ipeline basics
The deployment pipeline consists of steps that changes go through before they can be deployed to production. Every company, every team has their own unique pipeline. Note that whether or not your team has automated tests and automated deploys – you still have a pipeline. Whatever steps a new change goes through to get to production, even if each step is manual, is still a pipeline.
We break our build into stages or steps to speed up feedback. There’s a constant trade-off between fast feedback and confidence. The early steps of a pipeline are usually unit tests. They quickly show us whether anything is broken at that low, granular level. But we need more steps with higher-level, longer-running tests to feel more confident. Later steps in our pipeline exercise more complicated scenarios. We want fast feedback, but we may need comprehensive tests for confidence, and they take a long time to run. Here’s an example deployment pipeline:
In this example, the steps outlined in blue in this example are manual steps. We focus on automation when talking about pipelines, but manual steps are common. As in this example, activities like exploratory testing and security testing may be critical and may require manual work. We should be able to automate deploys to different environments, but perhaps your team just hasn’t gotten to that yet.
Things that slow down the pipeline
“Flaky” or “flappy” tests are a common problem even in completely automated pipelines. We often see this with user interface (UI) tests. Someone makes a change to the UI but forgets to update the automated tests to reflect that. A test environment slows down and causes a test step to time out. The browser version automatically (and unexpectedly) updates in the test environment. Those are just a few common causes of UI test failures.
Dependencies can also cause difficulties. A failure of one stage can stop multiple subsequent steps from running. Test data may fail to be set up properly, causing test failures. A test environment in the cloud might fail to spin up.
More steps in the pipeline give us more confidence, but they take more time. And how do we know all the steps are necessary?
When a pipeline experiences more and more delays, we humans have a tendency to increase the size of the items we’re sending through it to try to get more done. If the tests are failing and people keep committing changes, we’ve now got a big batch of changes which is more likely to include regression failures. It’s a death spiral. If our pipeline is reliable and fast, fewer changes go through it each time, reducing the risk of failures.
Collaborating to improve the pipeline
Making problems more visible is a great first step in making them smaller. Gather your team (or at least a representational group, if you have a large team) together. Take cards or paper and write the name of each step of your pipeline, one per card or paper. Put them on a table or on a wall to create a picture of your pipeline. (This can be done virtually if your team is distributed).
Visualizing a pipeline on a table – picture used with permission from Llewllyn Falco
Next, look at each step in the pipeline and ask “why do we have this step?” What are we learning from it? What business questions does it answer, what risks does it mitigate? Who on the team benefits from this information? You can capture this information on sticky notes to attach to each step.
Then, think about dependencies. For each step, what must in place for it to run successfully? A test environment? Test data? Specific configurations? Also think about how you can optimize the step. Is it currently done manually? Can you automate it?
Identify what triggers each step. Is it triggered automatically by the success of a previous step? Or manually? Can it be run in parallel with one or more other steps? Do you see opportunities to shorten feedback loops? Again, record outcomes of these discussion for each step.
Look at the length of your feedback loops. How much time elapses from code commit to completion of each step? What happens if the step fails – do the right people get notified? How? Is there a better way to communicate alerts? My previous team found that everyone was ignoring the Slack alerts generated by pipeline failures, so they hooked the build up to a police light. That got the team’s attention so that failures were quickly investigated and addressed!
Can any steps be done asynchronously? For example, it’s not realistic to do manual exploratory testing on every change when you are deploying to production frequently. The team can learn to use
release feature toggles
to hide changes from customers until the team has had time to complete any manual or time-consuming testing tasks.
Once you’ve gathered information about each step in the pipeline, choose a pain point to address and think of an experiment to reduce the pain. For example, perhaps your UI test suite has a lot of flaky tests and the team spends a lot of time kicking tests off again and diagnosing the failures that reoccur. Your experiment might be, “We believe that by refactoring the top 10 flaky tests to make them reliable will reduce our pipeline failure rate by 10% in the next month”. Use your retrospectives to evaluate your progress.
Remember the benefits
A fast, reliable pipeline means that you can confidently deploy frequent, small changes to production. And if you introduce problems to production that cause customer pain, you can shorten the time they feel pain by quickly deploying a fix with your fast, reliable pipeline. There are many other factors that go into safe, reliable and sustainable continuous delivery. Fast feedback and the ability to deploy changes quickly are a cornerstone of successful CD – and of a healthy quality culture.
It could be easier, we could have teams for each application we run, that those teams treat their production and non-production application environments the same. That they have fully automated full stack continuous deployment pipelines and the software is so well designed that there is no dependency between each application owner. Teams deliver code by the hour to production and use advance deployment techniques such as blue-green deployment coupled with highly effective monitoring, alerting and automated rollback to identify and resolve issues in production within minutes.
We do have “agile” application teams working within a cloud environment, they have growing autonomy as they build independent features and function. That’s awesome. The problem is that we have not been working that way since we started building technology. We have a sprawling on-premise (we own two data centres) environment where Non-Production environments equates for 60%+ of the overall infrastructure. This is made up of dependent applications running together in an environment.
We have followed traditional programme structure over the last 10 years, with multi-year investments and waterfall based delivery. These have sought to reduce risk through separation rather than integration. This has resulted in multiple non-production environments. Our applications (100’s) have been built tightly coupled and need multiple other applications to work (Commerce systems through to legacy mainframe systems all meshed together), increasing the size of the basic/core environment. These programmes have not invested in automation, leaving a challenging sprawling estate to manage.
Path to Production (Variety of routes followed by teams ensuring lack of consistency in versioning in each environment)
This has created a number of problems. The estate is costly to maintain. There is hidden cost in each application change or infrastructure upgrade. Our people are linked to projects and programmes, when they finish, the applications in non-production fall in to disrepair, they are costly and time consuming to get working again when the next team comes along. A continued desire for separation results in projects hunting down that unused application/environment that has nobody else working in that space. The sprawling nature and lack of ownership results in a small band of “Environment Management” who work to bridge the communication and knowledge gaps across this sprawling estate. These people work for Projects and Programmes and through “best endeavours” get Delivery teams, Operations and Infrastructure teams to stitch the environment back together each time and retain the illusion these environments are good to test in. We effectively have 10 mini-lives beating away with no real Service Management to deal with them.
That long introduction was needed just to give me time to breathe and reflect. The organisation is changing for the better, which is great. We are re-organising and realising the mistakes of the past. We are now building most new systems in a cloud environment, are focused on Micro-service design principles and have teams that own and run the service (well almost). These are small individual successes which highlight the organisation is changing. Our online teams have gone from 10 deployments a year to over 1000 in just over 18 months. It should be a factor of 10 greater by next year. However there is a long way to go till we are fully working in this way, maybe we will never get there. This is where Environment as a Product comes in. How do we change our existing non-production environment landscape for the better.
We have started with two key initiatives. Initiative 1, Let’s get support to introduce an “enabling constraint” through removing the environments that least support delivery. They are the most difficult to maintain, are off the beaten path to production and the most costly to re-enable them. If the project or programme don’t have them they can’t use them. We know they are costly to build which is part of the reason they have been maintained. With the remaining environments we are going to create a paved road of sorts. One of the key things is to not construct the path to production based on your current organisational/delivery construct. We have been very waterfall in nature and have therefore not made change often to production.
This results in us having multiple hops to production for a single change based on the type of testing to be undertaken at each stage. Development (unit, system), Integration, Release (end to end), Performance (performance, operational), Pre-Production (incident and support) and Production. You introduce delay just through deploying to each of these environments and fixing the issues that come up as a result. That’s before you think of data, types of testing, handoff implications. So initially let’s get down to one of each type to support all teams, projects and programmes. Let’s draw it up so it looks like some are optional. Development, Integration, Pre-production, Production (even this is too many).
This was difficult to get going, convince people that we might be able to go faster through having less environments, that there might be more contention initially, however that might force us to look at the organisational shape and how we design software. Whether the full extent of the idea has sunk in, is yet to be seen, however the cost avoidance articulation of maintenance running in to millions of pounds a year was hard to ignore and gave us the initial green light.
Revised simplified Path to Production
Initiative 2, is to consolidate the team we have and build a paved road (common environments, tools and approach). We have taken everyone responsible for non-production “Environment-Release Management” and centralised them in to a single Product team. This wasn’t easy. You have to convince the leadership team that they want to pay for this as a separate service, that the cost is neutral, value adding even and that in doing so you can look at how the team works and make efficiency, as well as improving the path to production. What this doesn’t do is centralise the technical people responsible (not owning) for those environments. These remain fragmented across Operations, Infrastructure and Delivery teams. While the number of different technologies, teams and ways of working make this impractical at a small team level.
It does allow you to put a very clear Service Management approach in place for Non-Production and with leadership support allow you to be more robust in enforcing a better approach. You can start to work with individual teams to support and help them approach their delivery differently, focusing on automation, repeatable process and decoupling of change. The aim is that teams do this themselves and that the Environment-Release team can identify teams that need support. We are also backed by organisational
, which have been established to give teams direction in how they should deliver.
Environment-Release Product Team
Centralising the team has also allowed us to look at how we approach managing the environments. A single Product team with longer term goals, where the value is what we do to the environments, not the end customer outcome that the Project or Programme was focused on, has a very different priority and outlook. What was several ways of working can now be focused in to a single way of working. We have started by making sense of what people were doing daily. This has been done through several value stream mapping workshops but also to get everyone’s work up on a board and try to make sense of the work we do. Get a feel for reactive vs planned work. Upon doing this we can start to build a very clear backlog of priorities for our own ways of working, how we track issues, do we manage problems, how do teams onboard on to our path to production, can we pre-empt issues, what do we do with data, looks at our skills and most importantly clear communication.
Value is now a clear driver, as is ownership. We can now mandate changes to production are part of non-production thinking. We can also start to look at cost of delay and overall costs of supporting teams and drive our own value outcomes. Lead time is also a key measure, whether that that is to measure how quickly teams move through our environments, how long it takes us to respond to incident, downtime within environments etc… We can also help to start to measure teams frequency of deployment/release and to show organisational pace of change. Traditional service management is very much focused on controls but here we want to make it much more about value outcomes and enabling flow within teams.
We also have one foot firmly in the future. Over time the need for this team will be less. Greater ownership, more end to end owning teams, “DevOps” will all drive less need for a Environment-Release Product team. Our life span is based on the value we can show to generate. However we need to develop our skills in to other areas, whether that be Platform management within our Cloud platform or in to more “DevOps” skill sets which can work within teams to help them deliver. For now we are focused on our Product and value. It would be interesting to hear how other companies and people manage non-production environments, especially where there has been ongoing development of a variety of approaches for many years.
Rob Hornby is a Product Engineer working for John Lewis based in London. He has undertaken a number of roles, over the last 20 years, within Testing, Release, Platform and Environments.
Rob is a keen believer in being able to treat all things as Products and be able to evidence the value generated by that team. Ideally everything would be delivered by small teams, delivering continuously and iterating on value. However not all teams start at this point, not all teams are the same or born equal. This is where the challenge and enjoyment comes from.
Outside of work, Rob is a keen runner, father of an amazing young girl, Sofia and works on a number of local community events.
One fine day disruption was created inour organisation as part of digital transformation with few people fromoperations joining us – Developer, Tester, Product-owner.
…And we all moved into a NEW world ofDevOps. There were lot of uncertainties about what it would be like? Lot ofquestions whether there will be any role as tester? If there is what that rolewould involve?
As we all moved into DevOps world, allthese questions were getting clear. Our application demanded lots ofnon-functional testing along-with functional. In a DevOps world, it was allabout getting the feedback early and do frequent releases with a good quality.For a tester what it meant for me is to learn the process of continuousintegration and delivery and the testing for not just functional but also fornon-functional. Non-functional testing involved Security, Performance, Visual,Accessibility and reduce dependency on external teams to carry-out thesetesting to delivery faster. In short a Swiss Army Knife’s worth of skills (knowingdifferent types of testing). To top that it should be automated for quick-releases,so enough time is spent in doing exploratory testing and finding the unknowns.
To address that I came-up with the test-automation pyramid model, as illustrated by
to help team visualize & discuss test automation strategy. I will share how I modelled the test-automation pyramid to suit our DevOps world and overview all the testing involved in terms of functional & non-functional in an automated way to deliver faster without compromising on quality.
There are so many forms & interpretations of Test automation pyramid. All point out the same thing it should have more no. of unit tests, followed by service (consisting of API, Integration, Component) tests and finally UI tests.
There’s this another way of looking test pyramid as a Bug-Filter.
But have you noticed it doesn’t talk aboutnon-functional testing. Does that mean non-functional testing are not automatedor part of test automation? For me test automation stands for – tests which areautomated, run by machine/tool scripts and asserted pass/fail on ourexpectations.
At least it should say Functional Testpyramid. But I can see functional test automation is the most widelyimplemented form of automation and it has kind-of become synonymous with test automation.Also, Manual/Exploratory Testing bit is not captured properly. Exploratory Testingcan happen at any level – Unit, API, UI and for different types – Security,Performance, etc.
The test pyramid is an effective visualisationform outlining your testing approach which can be easily explained to others.Just make it visible on a board/wall and you have a very good indicator of howyou want to organise your testing efforts in one picture. It is a better visualreplacement for long test strategy documents.
As earlier mentioned, in our application we have lot of non-functional testing. So, I have taken these existing models and made changes to better suit our application. I will now explain how I view this test pyramid and apply it to testing in DevOps.
Test Automation Pyramid re-visited
So here is my interpretation.
Software testing is about identifyingRISKS and not finding bugs. Based on this, Imagine Test Pyramid to beupside-down as a RISK Filter, having all the different types of testing –Functional in UI, Service, Unit. Non-functional in Performance, Security and Others.
Now imagine the newly developedproduct or feature enters the pyramid for identifying risks, meaning carryingout functional tests in UI, Service, Unit-level and identifying functionalrisks.
Similarly, performance testing foridentifying performance risks, security testing for identifying security risks,and so on for other risks.
All these risks identified are byautomation meaning asserting our expectations or checking known behaviours. Whatabout Unknowns or new information. These will be identified by Exploratorytesting at every level – Unit, API, UI and for different types – Security,Performance, etc.
Do all the identified risks need to befixed? Absolutely NO. The team will prioritise it as high, medium, low etc. andfixed as per priority. Once all the prioritised risks are fixed then releasethe product/feature with known risks.
This is how test pyramid is applied in our case in testing in DevOps, making sure all the testings are done at each and every level from RISK perspective to achieve quality and deliver faster.
Functional Automation
Let’s talk about functional test automation. I will be just covering UI testing. There are so many functional test-automation tools, most popular ones being Selenium, Sahi, Katalon, Ranorex, NightwatchJS. And to implement best practices test there are frameworks such as Page-Object Models, BDD, TDD, Keyword & Data driven, Hybrid and so on. Depending upon your preference select the framework suiting your needs. There are no good or bad frameworks, it’s all about what best suits your application and your team. Same goes with the tools.
Non-Functional Testing
Quality of a software depends not onlyon Functional but Non-functional testing such as Security, Performance, VisualUI, Accessibility and so on.
Security Testing
Securitytesting is a process of identifying weakness in the security mechanisms ofan application that protects data and maintain functionality asintended. Purpose is to make the application immune to SQL injections,cross-site scripting, sensitive data exposure to protect Unauthorized andUnauthenticated users to enter in to your application. All these can bedriven through a tool and tested through test-automation.
Thereare various tools available where you can automate security tests and run aspart of CI/CD process. This automated activity necessarily doesn’t removemanual Penetration testing requirement. Through automation, it is beneficial tofind basic vulnerabilities in our application as much as possible so thatSecurity Specialists/Penetration testers can focus on identifying high-hangingbugs. This helps in delivering at a faster pace by finding basicvulnerabilities early in development phase which in turn saves Penetrationtester’s time so that they can concentrate where it adds value.
Performance Testing
Performance testing is about identifyingperformance issues such as slow response time, support for concurrent users,how much load the application can handle? Does the application crash? Also, it involvesnot only testing the server-side but the client-side as well. As can be seenfrom the chart, 80% of the end-user response time is spent on the front-end.Most of this time is tied up in downloading all the components of the page – images,stylesheets, scripts, Flash, etc.
To verify if an application is fast and efficient enough we perform client-side performance tests. This means checking the response time of a web application from the point of view of a single user. Both server and client side can be automated with pass/fail criterion if it doesn’t match the performance benchmarks. Usually performance tests are run once the functional tests are done and/or just before few weeks before the release. But executing performance tests early whenever new API is built (Server-side) and a new page is built (Client-side) helps to get early feedback and reduce any delays incurring from the issues identified later in the development phase.
Visual Testing
Visual UI testing consists about
Does your UI look the way it should, to allyour users?
Visual UI testing is about verifying that UI appears correctlyto your users. For detecting correct icon-size, padding, colour, layout, text,element position etc. It’s a quick way to check all these through visualautomation. It saves time and eye strain.
Angie Jones:
Even if the application passes functional testing, we still must check the look and feel of UI because the element might be not aligned properly or hidden behind some other element which the end user is not able to see. Visual UI testing is the best way to identify such issues and if integrated with functional tests gives a lot more confidence about the UI. This can be easily seen in the below image. Functional tests for this would pass but not Visual UI testing.
Accessibility Testing
Accessibility testing is a subsetof usability testing, and it is performed to ensure that the applicationbeing tested is usable by all the people and not only for people with disabilities.Accessibility is about ensuring inclusive applications are built for one andall, making an application equally accessible and usable for a normal user too.Purpose is to provide everyone with equal access to the application and meetingthe standards set in web content accessibility guidelines –
WCAG
.
People with disability will use navigation strategies like tabbing through the headings, links on webpages, images and videos with properly described text for accessing. So as a tester we need to test whether it follows the standard or meet the compliance having proper tab flow, recommended headings, alternate text for images, videos etc. Accessibility testing can also be automated to check whether the application has elements well structured, alternate texts and contents are described well or not.
Exploratory Testing
Exploratorytesting is the key to high quality software, it’s highly intellectual and time-consumingactivity. It is an investigative approach and main intent is to uncover newinformation or covering the unknowns in all areas of testing from functional,security, performance, visual and accessibility or any other types of testing. Oftenthis activity is pushed off to end-users.
The image below shows how testing from a risk perspective can be used to bridge the gap between automated testing (asserting expectations) and manual/exploratory testing (uncovering new information), so that lot of unknowns will be explored and converted into automated known behaviours to improve the quality. Exploratory testing helps in creating good automation tests to figure out all the things you want to go and check.
Continuous Testing
Continuous testing
right from requirement to production and taking feedback from production to requirement, this loop of continuous testing is key to achieve highest quality. Below image shows all the different types of testing we perform in various stages of DevOps.
Conclusion
As explained in my DevOps Testingpyramid, testing should always be done from a risk perspective and identifiedrisks should be prioritised and fixed accordingly for delivering qualityproduct faster.
In DevOps, automated testing is key tospeed-up the development cycle. By automation, I mean non-functional testing aswell. In a DevOps world, tester should possess Swiss Army Knife’s worth ofskills, which means as a tester it’s our responsibility to do all otherrelevant types of testing as much as possible apart from functional testing toget early feedback and not depend on external teams. Depending on externalteams might increase the risk for not getting the feedback early and delay therelease. Also, don’t forget to put all the tests including non-functional inCI/CD pipeline to get early and constant feedback on quality.
Continuous testing in all the stages iskey to achieve highest quality. It’s important to be tool aware and practiceidentifying good tools and use them whatever suits to your application and toyour team. Exploratory testing is the key to high quality software and pleasedon’t push it to end-users. Utilize as much time as possible in exploring theapplication for uncovering new information for improving quality.
Lastly, no matter what strategy andprocesses you have the team should have Quality mindset to go extra mile andadd that extra effort to deliver quality to the end user. This is accomplishedby Continuous Testing and doing all the relevant functional and non-functionaltesting at every stage so that
quality is cascaded
down to theend-user.
Testers have many diverse capabilities – just like a Swiss Army Knife!
I would like to end my blog – DevOpsis all about early feedback and frequent releases and for that
Testers should possess a Swiss Army Knife’sworth of skills, versatile and approaching the application from all the differentaspects of testing!!!
About the author
Amit Kulkarni
Amit Kulkarni (@amtoya on Twitter) is a test automation specialist and has over 10 years of experience delivering quality product for clients in Australia, Europe, Japan and USA. Amit is passionate about test automation for both functional and non-functional including security and performance testing. Amit has a diversified experience in Semiconductor technology, Embedded and Software industry in the field of Banking, Healthcare and Consumer Electronics working for companies such as Fujitsu, Mitsubishi, Qualcomm and Telstra. Amit is currently working as a Senior Quality Engineer in Bankwest and driving best quality practices in test automation and continuous testing to deliver quality product faster.
I was recently looking at the world of DevOps: the sheer number of potential changes and practices that shift both the way the company works, and the way things are approached are not easy! There are a few well-known names with a huge following that seem to attend every major conference, and are a great source of knowledge. These folks are there, advocating for the changes that are needed, and are a grand source of ready information for businesses and practitioners that need information on how a specific technique might look in the future.
Photo by
on
And, thankfully, most of them are capable of sharing their energy and excitement — which gets the team and management to be willing to try something new. This is an important skill: any change cannot be done effectively by just a few people. The more you are able to show what a change looks like — what to expect as blockages, things that worked to improve someone else’s situation – makes the whole process still seem magical, but with enough encouragement that things may not go so well at first, but this will work!
The downside to this comes quietly — business and other people have come to view them as “this is what DevOps is”, and try to hire for only the qualities they see in them, but for their specific situation as well. And without a grounding in the current setup and the problems that the company needs to solve (and the things they want to do, which may not be the same at all!), no one can sound expert and confident without a bit of blarney in their presentation. And those that are able to persuade in this fashion might not be the best choice, in the long run.
But, the folks I am looking for to provide guidance are those thousands of folks that are not as well-known: the people that make the cycle of DevOps work for their business, and are successful without making a large splash in terms of followers, or even of videos and blogs. Their companies would be lost without these people — and these are the ones I would love to learn from.
These folks can be at any level in any business. They are the ones that are willing to be a bit stubborn, and not give up after the first time, or even after a few failures, because they have a goal that they want to reach. Finding out options, and getting others to learn about them is a sure sign you have someone I would want to work with – they are focused on getting better.
Improving through small experiments
They are the ones that will make a small change, and see how it goes. Changing too many things at once is both difficult to track effectiveness, and builds resistance to trying something new if some changes make things worse in the short term.
Photo by
on
Rooting out the base cause of a problem is a thankless task at times — it’s so much quicker to put a temporary fix in, and move the real issue to the TODO list: and might cause fewer side effects than fixing the actual problem. But having someone devote the time to finding the issue, or at least systematically eliminating where it is not (and fixing small things as they go through) will allow the gradual improvement.
Working with other sections of the business is a needed skill for DevOps — being able to share problems, options, and ideas is one that helps everyone be aware of the issues. Using a good idea learned from another person may be the best possible outcome — you not only may have cured their issue as well, but you have both one less problem, and a new outlook next time a similar situation occurs.
Practitioners: Thanks for sharing your experiences!
So, for all of you who have taken on the task of making sure that the feedback is fast and accurate, the changes improve quality, and problems big and small are addressed when they occur — thank you. Learning from you, either through informal questioning, or other methods such as blog posts, is a pleasure. Finding a new post by someone who is skilled at these tasks is always a delight, as is a new writer whose knowledge is evidenced in their approach and tone. You may not have the answers, but at least you can ask the questions that will lead to them.
I was at
TestBash Brighton
recently, and a common theme kept coming up:
“You can be technical without knowing how to code!”
, TestBash organiser, even did a 99 second talk on the topic. He listed various skills – git, debugging, changing the microwave clock. If you have these skills as a tester, there are plenty of uses for these skills. If you have an agile or DevOps culture – where roles are less set in stone, and teams swarm on problems – you’ll get loads of opportunities to use them.
Well, apart from the microwave clock.
Learning the ecosystem
A yak dominates an ecosystem
It’s important to realise most developer skills aren’t coding. If you’ve tried to learn coding, you’ll know lots of other skills can get in the way. You start writing a simple application and lots of good practice pops up. Your developer friend says you should set up a git repository. Then instead of installing dependencies manually, it’s better to set up a dependency manager. Then you learn how to compile and run the code. The list goes on. This can be very daunting, and quickly turns into
yak-shaving
.
Learning doesn’t stop with developer skills
This seemingly never ending list of things to learn doesn’t stop with a developer’s skillset. Run a production-grade environment, and SRE/Ops skills will get in the way. Ops ensure that users can use your environments. They do all kinds of things. Backups. User and authentication control. Familiarity with the Linux CLI to debug issues. Log monitoring and analysis. Failover processes like blue/green. Containerisation like
Docker
, or container orchestration like
Kubernetes
. Your workplace may differ in the specifics, but production environments have lots of additional complex functionality. Complex functionality you can’t replicate by running the code on your laptop, or even sometimes in a test environment.
Broad skills are valuable too
Your broad skills are valuable
There are specialists in these subjects. Nobody is expecting testers to be experts with all of these skills. Much like testers don’t have to have all the coding skills of developers. That said, knowing about these things will help us understand our application. It will help understand facets of our application otherwise overlooked. If testing is about understanding the quality of our application, then we need to look beyond the code. As far as your application is concerned, code is the tip of the iceberg.
Let’s take an example: logging in to an application. Your users are not only expecting that they can login successfully. They’re expecting that the database stores their user credentials. They’re expecting that their data is backed up in the event of disaster. They’re expecting that they can still login if lots of other people are. They’re expecting their confidential data is securely stored and transported.
Much of that functionality is not done in code by developers. It is often the responsibility of ops, or sysadmins, or database admins. They configure databases and networks and servers that ensure all these things happen.
Ops opportunities for testers
Before DevOps, it was difficult for testers to have influence upon this. Much was siloed away from us. We test the code, then let them deal with that stuff. Out of sight, out of mind.
Cross-functional teams include Ops specialists
But now, Ops specialists are being moved into our teams, or working closely with our teams (hopefully not as a permanent ‘DevOps team’!). We see the work they do, and it needs testing too. That’s because it’s part of the application. It’s part of what users expect. We don’t just deliver code, we deliver a service.
Pair with an ops expert to assess risks
So how can testers start helping? To test code, we didn’t need to be developers. But learning related skills helped us empathise and co-operate. It helped us ask the right questions. So try pairing with an ops expert on your team and work out what risks might arise. Or doing some shaping of some infrastructure tickets, and thinking about who the users are (with pipelines, they’re often developers!). Or try writing some automation scripts to ensure that associated problems get caught before production. Or go to a devops meetup/conference. Learn about bad practice and security/infrastructure smells. Start treating ops like developers – to be assisted and questioned.
Because our application is more than just code. It’s all the processes, and the infrastructure too. That stuff needs testing too.
Matt Long is a QA & devops consultant for OpenCredo, an open source software consultancy based in London.
He has worked as a QA consultant for many years, and built test automation frameworks in half a dozen programming languages. After working on many cloud infrastructure projects, he’s become particularly interested in the intersection of modern cloud infrastructure and testing.
Similarities between SRE and testing: Alert/automation suite maintenance
By Abby Bangser
I want to take a few minutes to share with youhow I have grown to once again realise how common so many of our challenges arein tech. This time the commonality is between the world of test automation andoperational alerting. Before I dive into these details though, let me quicklyexplain my career journey which led me here.
From the beginning of 2012 to the middle of2018 I worked as a test consultant in organisations ranging from small non-profitsthrough to large multinational financial institutions. During this time I sawthe scale, financing, and cultures change fairly drastically, however thechallenges were often quite similar when it came down to the team level ofdelivering software. Most notably, over these years I realised that being aspecialist in quality often meant my biggest contribution was in how we delivercode through Continuous Integration and Delivery processes rather than just how
I
test things. This led me to working with many DevOps and infrastructure relatedteams and activities and eventually on to joining MOO as a Test Engineer ontheir Platform Engineering team.
Introduction to our platformengineering team
At MOO the Platform Engineering team isresponsible for the infrastructure of both production and pre-productionenvironments as well as the running of internal tooling like source control andbacklog management tooling. As you may expect this includes being on call forthese systems 24 hours a day, 7 days a week. From what I can see around theindustry our team is not particularly unique in how we handle this.
We have a rota which makes sure we share theresponsibility and our team generates both warning and critical alerts to makesure we respond if issues occur. These alerts provide us huge opportunity forspeeding up our Mean Time To Recovery (MTTR) by decreasing our time todiscovery. However when I joined the team, they were also generating a lot ofstress which is why one of my first tasks was to review our alerts for valueand how to remove that stress.
Tackling our alerting challenges
Our alert stress was generated for a few majorreasons. First of all, some of our alerts were things that our MOO team memberscould do nothing about. For example, if our Google authentication is failingthere is an impact to our internal users which could trigger an alert to our oncall engineer, however there is nothing we can do but wait until Google sortsout their incident. Another issue with our alerts was critical alerts (meaningsomeone is woken up in the middle of the night) that resolved themselves withinminutes. Often these were issues with machine health in AWS which has its ownself-healing mechanisms and our thresholds were just too low. Finally, we hadsome alerts that made people wonder if they were worthwhile as they had nevertriggered once in over a year of tracking.
When I joined the team there was a ticket onthe backlog to “Improve team alert fatigue”. This was intentionally a prettyvague topic and was meant to be an analysis placeholder as there wasn’t justone solution to all these concerns with our alerts. Alert fatigue can bedescribed as “exposed to a large number of frequent alarms and consequentlybecomes desensitized to them. Desensitization can lead to longer response timesor to missing important alarms.” and our team was definitely showing the earlysymptoms. After spending some time with each team member we decided thatdefinition of done for this ticket would be to:
Identify the shape andrequirements of a “good” alert
Review our alerts against this definition
Set up future work to bring ourcurrent alerts in line with our desired state
Achieving a working solution byputting it to the test
To achieve these goals I went about puttingtogether a straw man definition of what a warning and a critical alert wouldneed to have. Then scheduled reviews of all of our alerts against thesedefinitions to let them be hardened and evolved by our real context. This was a(very) long process but by grouping alarms together, sharing the load acrossall team members in small groups and doing no more than 30 minute meetings wewere able to get through the review in less than a month and with only alimited amount of meeting fatigue!
In the end these became our team’s workingdefinitions:
All critical alerts should have…
…an immediate and quantifiable end user impact which must require a specific human interaction to rectify
…and should be paged on immediately
Whereas
All warning alerts should have…
…no short term action required, but do have proven critical impact to the system if left unmanaged
…and should be paged on after 72 hours in a persistent state
We of course dove into each of the detailslike how to define “short term” but seeing this high level framework canhopefully demonstrate how we also uncovered the need to discuss the informationprovided by an alert (if it requires human interaction, shouldn’t that actionbe clear from the get go?) and also how we handle different types of services(grafana which just displays data and is used most heavily in office hours maynot need to be paged on Sunday morning).
Outcomes and learnings from theprocess
By the end of the review we identified somealerts that could be removed, some alerts which should be retired throughretiring the systems they were attached to, and many alerts which could usesome sprucing up. In all these cases we created story cards with the reviewcontext and have been regularly feeding them into our workload since then.
As we are now about four months on, I thought it may be a good time to retrospect on the experience and the value of the work done. My biggest takeaway is that this experience closely mirrored my experience with test automation suite reviews.
In particular, stepping away from a singlestory card or a single feature helped our team realise the bigger picture ofalerts and come up with a framework to keep our alerts in order. Nothing wecame up with was novel to our team, it is just that we had not been able to sitdown and agree while simultaneously being able to apply or theories againstreal world scenarios. The alert review is what really helped cement agreementover our alert framework through discussion and debate about possible outliers.
Another similarity to test automation is howthe team cared deeply about quality alerts, but given the intermittent pain ofpoor alerts and lack of customer facing impact, it was hard to spend the timeto improve them. To be honest, this continues to be a challenge as we try andstrike a balance between interrupt driven work (incidents or alerts on a givenday), feature delivery, and long term maintenance. That being said, having thethoughtfully created and thorough documentation of fixes in places has meantwhen a “mildly annoying” alert goes off
again
we can often pluck that card from the backlog and prioritise it quickly.
Finally, the test community doesn’t seem tohave as well known a term for red build fatigue as the operations community hasfor alert fatigue, but trust me they are the same thing! Our team started torely on their instincts and experience to decide which warning alerts
actually
mattered and more often thannot just wanted to remove the culprit alerts because it was the only way theysaw any chance of improving the situation.
I am not suggesting we are in an ideal placenow, but by introducing some practices for bringing the pain forward likeescalating all warnings after 72 hours, it has increased our on call engineerand product owners focus on alert quality for the sake of team sanity.
Abby Bangser is a software tester with a keen interest in working on products where fellow engineers are the users. Abby brings the techniques of analysing and testing customer facing products to tools like delivery pipelines and logging so as to generate clearer feedback and greater value. Currently Abby is a Test Engineer on the Platform Engineering team at MOO which supports the shared infrastructure and tooling needs of the organisation.
Open Sourcing Your Mindset
By Brendan Connolly
Asdevelopment has shifted from waterfall to agile and now DevOps, at each stageyou see developers taking on more responsibility and accountability.
Waterfallwas isolationist, building silos around different groups in softwaredevelopment. Developers owned the act of coding and in the stereotypicaldeveloper way, had little to no interest what happened before or after theycoded. Other roles on teams in many ways acted as insulators to keep externalforces away from developers.
Agilewas developers pulling that insulation away, shifting themselves left andtaking more ownership of the product creation process. Developers found theyrequired a real time feedback loop to build the right things and successfullyreact to change.
DevOpsis essentially the same process on the right side of the spectrum. Developmenttakes more ownership of deployment, monitoring and supporting the applicationshealth. It’s just too slow and cumbersome to react to the requirements ofmaintaining applications by filtering feedback through separate teams at scale.
Thebenefits of integration outweigh the cost of the additional responsibility.
Code Continuous
Tocope with the additional responsibilities, developers have turned to theirtrusty friend
code
.
Withagile we see rise of tooling developed for
ContinuousIntegration
to alleviate the pain and reduce the risk in iterativedevelopment.
InDevOps we see the rise of code and tools supporting
Continuous Deployment
. Even when many teams already had productinstallers, or automated deployments what was missing was the integration ofthat information.
Ifdevelopers are owning more of the product as well as release and deployments,then testing naturally gets swept up in the process. In fact we see thispattern emerging with the concept of
ContinuousTesting
. Code is being written, tools created to help solve the issue ofintegrating test execution and reporting into teams release pipeline. Thefeedback loop is extended to provide rapid information to make it easier andfaster for development to be responsive to change.
Continuous Testing
Automatedtesting is a loaded term. In developer circles it almost exclusively means unitand possibly integration tests. In QA it almost almost exclusively means end toend (e2e) tests, and is often synonymous with
Selenium
.
Onteams where both sets of tests exist, their worlds may seldom interact.Developers may be aware of the
e2e
tests but they probably aren’t writing them and even less likely running them.
ContinuousTesting has nothing to do with the testing that most QA team members are mostfamiliar with, exploratory testing. The purpose of the code and tooling thatcomprises a pipeline isn’t for there to be some manual intervention requiredpart of the way through it. Putting exploratory testing somewhere in the laterstages for the pipeline is akin to pointing a fire hose at someone and askingthem to direct the flow to the next stage.
Ifthe
speed of DevOps
wasn’t enough, the presence of
continuous testing
only puts morepressure on testers, simply because more attention is being drawn to testing.
Automationengineers and product engineers are being drawn closer together. The importanceand relevance of end to end automated tests is becoming more of a reality forproduct developers as they become run regularly as a post deploy verificationtests.
Whilecode is being thrown at testing with newfound passion, exploratory testers needto be prepared for their time in the spotlight. If automated tests are runningall the time what tests are you doing? What makes them special? Why are theyrequired before we can deploy?
What do you do different than Selenium?
Beforeyou dismiss that question, pause and consider what all selenium and the testsit drives truly provide.
Seleniumis cross browser, scalable, schedulable, repeatable, and can provide logs andscreenshots of its tests. Can you compete with that?
Flaky,slow, and wrong sometimes, you say? And you and all other humans aren’t?
Conversationsthat try to explain using people instead of machines for testing software canend up justifying them under the guise of
thereare some tests that aren’t good candidates for automation
.
Don’t settle for executing tests that don’t have enough ROIto automate.
Thistype of conversation relies on the belief that testing is comprised to twoelements:
test case creation
and
test execution
. Perhaps at its most baselevel testing is just that, the value proposition of the role is in the nuancethat a skilled tester can provide to teams through those elements.
Testersdo more than a fill a gap in automated test coverage. The problem is manytesters express this nuance solely through actions.
Developersproduce code, a measurable artifact from their effort. The generation of thatartifact is often backed by computer science, a formalized area of study.Artifacts of effort are reassuring, people like to see the fruits of theirlabors. Tangible results can be measured and quantified, they might even helpestimate future effort.
Joan of Arc
A tester’s value tends to be recognized at the same rate as relationships develop. Testers get characterized as
folk heroes
rather than members of a skilled profession. There’s no trade, education, or craft just individuals saving the day periodically.
The Intuition trap
Evenwhen teams do value testing, people are usually primarily concerned with theprogression or regression of testing efforts. Testers comments in stand-ups caneasily turn into a user story status change report:
Passed this ticket, working this ticket, noblockers
Ifyou aren’t taking the opportunity to discuss your actual actions in stand ups,then you may fall prey to the intuition trap. When you talk about your workthis way, sometimes you end up actually working this way.
Open Source Your Mindset
It’sfun being a hero, but there’s no skill in being
Superman
or
Superwoman
(I was going to say Wonder Woman, but she actually trained and I think everyonerecognized that. So if you are the teams Wonder Woman then kudos, you haveserious skills but I digress).
Whentesting goes smoothly the result is
nothing
.Testing isn’t required for things to go right, it just might help prevent athing going wrong.
Opensourcing your mindset means
teaching
others another way to look at and interact with their code, or the product.This cuts both ways though, it means being willing to accept a pull request ofoutside perspective. Openly integrating new ideas into your
testers mindset
.
Replace Execution with Communication
. Like acting or athletics there might be a script or game plan to start with but there is a lot going on behind the scenes, adapting to circumstance and being thought up on the fly. It’s intuition driven.
Whenactions, practices and motivations are not being verbalized and expressed tothe team and stakeholders it puts testing on the level of superstitions andtoken time sacrifices to the quality gods.
Testersneed to pry open this experience, expose the internals driving motivations.Breaking out of that shell grants visibility into a testers actions andmotivations to provide a context for team members to attribute value.
Replace Isolation with Collaboration
Itcan be uncomfortable or difficult to open up and share what goes through your
testing mindset
.
Theadditional ownership developers have taken in DevOps has changed relationshipdynamics. What was at one point a contentious us vs. them relationship is muchmore open. There just isn’t the time to have as much formal process drivencommunication. Success at scale requires collaboration, pairing and mobbingmore.
When with developers, it’s an opportunity to test at their desk, against their local changes. There are risks testing here, the developers local setup is definitely different than production but so are the testing and staging environments. Let go of the taboo of
it works on my machine
and consider what can be tested here and what needs to be covered on a different stage. Take the opportunity to work together when the code is the freshest in the developers mind and where they are most comfortable.
It’s not only about working with developers. Mob testing with
UX
and
Product
alongside developers is a way to draw the entire team together. When the team is together seeing how the application is working it bypasses all the communication gates. There is no need for a separate follow-up conversations to loop someone in, that person can see and feel how things are working by being a part of the group experience.
Replace Tedium with Stratagem
Teamsare more open to testing and sharing in quality than ever. Testers have beenbuilding quality mindsets for years. Developers are building tooling but teamsare looking for guidance and mentorship in
Quality
.
Testing isn’t just button pressing, by showing that through communication and working together we elevate our status. Draw people together, into the quality circle, and enable the team to be more comfortable and aware of testing. We can set working agreements, talk about overarching strategy for how testing throughout the development cycle builds quality in. We can embrace automated testing as a team activity where unit, integration and end to end coverage is part of a teams strategy for mitigating risk. We can forgo complete control of testing and opt into being a voice of authority and expertise in quality.
Brendan Connolly is an experienced Software Tester, Automator, Speaker, and Blogger. Currently he is a Senior Quality Engineer at Procore Technologies in Santa Barbara, California. He maintains a testing blog @ http://www.brendanconnolly.net
Testers, DevOps, and Enabling Digital Transformation
By Gerie Owen
To be successful in today’s businessenvironment, virtually all organizations must plan for digital transformation,not only to thrive but to survive. Often, when organizations embark on digitaltransformation initiatives, they turn to DevOps to increase business value andimprove customer experience.
Testers, as champions of the customer andguardians of their organizations’ digital reputations, are uniquely positionedto enable digital transformation. Weenable digital transformation by focusing testing on business risk and customervalue as well as transforming our test processes.
DevOps enablesDigital Transformation. The synergy is clear: digital transformation and the principlesof DevOps are closely aligned. Specifically,continuous integration and continuous delivery enable innovation in operations,streamline processes and eliminate bottlenecks.
Continuoustesting plays a key role in the continuous delivery pipeline by enabling bothhigh quality and high velocity and ensuring customer happiness. Let’s examinehow continuous delivery enables digital transformations by streamliningprocesses and focusing on value to the customers and how testers can enable andchampion this process.
DevOps Principles Aligned with Digital Transformation
Digital transformationcan be defined in many ways; however, it is a fundamental shift in the way anorganization approaches its business and operational processes by usingtechnology to increase focus on and improve customer experience. Most digital transformations are focused oncustomer experience, operational processes and business models. DevOps principles described in the
ThreeWays
— systems thinking, amplifying feedback loops and creating a culturalof experimentation and continual learning — can be the basis for creating the majorchanges that are required for digital transformation.
Systemsthinking or focusing on the performance of an entire process is the mosteffective approach to transforming business models and operationalprocesses. If changes are made tofundamental operations in siloed initiatives, they will likely fail. Amplifying feedback loops is critical todigital transformation so that corrections can be made early and often. And creating a culture of continuousexperimentation and learning encourages innovation without fear of consequencesif a new business, process or technological approach fails.
TestersEnabling Digital Transformation by Transforming Test Processes
Testers enable digital transformation not onlyby championing the customer, but also by transforming their testpractices. Transformational testpractices include implementing the practices of continuous testing, including“shift left”, test optimization, as well as the use of service virtualizationand advanced analytics, and “shift right”, i.e., testing in production.
Test Automation Pyramid, from More Agile Testing by Janet Gregory and Lisa Crispin
Continuoustesting is an approach to managing risk by focusing not only on eliminatingtesting bottlenecks but more importantly, increasing the effectiveness of ourtest processes. The test pyramid is a“shift left” pattern whereby most of the automated test cases are concentratedin the unit, API, component and integration layers with fewer automated testcases used to test the GUI layer.
Testersas Champions of the Customer
As testers,meeting the challenges of our role in digital transformation requires acoordinated approach made up of several components. Customers expect a seamless experience acrossall channels; however, they do not necessarily expect to perform the same actionson every channel. Therefore, a digitaltransformation test strategy will require using customer behavior analytics tounderstand customer’s browsing, purchasing and other transactions acrosschannels in order to test efficiently and effectively.
As champions ofthe customers, we must focus on testing the customer experience. This means we must learn to use customeranalytics, personas and customer journeys and combine the results of those withexploratory testing.
Testers and DevOps can be the keys to successful digitaltransformation through by focusing on technological transformation based onbusiness value and customer happiness. Testers, this is your opportunity to goabove and beyond and contribute to your organizations in innovative,customer-centric ways.
Gerie Owen is a Testing Strategist and Evangelist. She is a Certified Scrum Master, Conference Presenter and Author on technology and testing topics. She enjoys mentoring new QA Leads and brings a cohesive team approach to testing. Gerie is the author of many articles on technology including Agile and DevOps topics. She recently developed a curriculum for DevOps 101 training. Gerie chooses her presentation topics based on her experiences in technology, what she has learned from them and what she would like to do to improve them. Gerie can be reached through her website, www.gerieowen.com, her blog, Testing in the Trenches, is at https://testinggirl.wordpress.com/ and @GerieOwen on Twitter and on LinkedIn .
