team culture Archives - DevTestOps Community

Continuous Testing in DevOps: A gamification approach

By Jorge Luis Castro Toribio

A continual problem that we must deal with when we run automated tests in our DevOps pipeline is maintenance. I could spend hours sharing posts and blogs about this matter, telling stories about how it affected our team’s capacity and time to market, and giving you several examples of what teams have done to sort this problem out. In fact, all the effort that we invest in maintenance is not visible to the business stakeholders’ eyes. Maintenance consumes our team capacity, but it does not add value to our clients directly.

Test automation challenges at scale

In one of my jobs wefaced this problem on a large scale. We had more an average of more than 1000automated tests running at each code check in, plus one big run at midnightthat we called full regression. Our suites where organized in those levels:

L1: Unit testing – Development Stage
L2: Integration Testing – Commit Stage
L3: UI Functional Test MVP Suite – QA Stage
L4: Regression suites – Full package – UAT Stage

Those testing levelswere our creation, but we used the great lessons shared in this book https://www.amazon.com/Practical-Approach-Large-Scale-Agile-Development/dp/0321821726 that I recommend for any team looking to buildtheir testing strategy.

After each run, we noticed that 80% of UI and regression automated tests failed due fake failures. After reading my numbers, you can say if you automated crap, you will get crap results and that is correct — but please remember that we were building our automated test suites, with a continuous testing approach and DevOps mindset and pipelines, so we were trying to fail fast, learn and improve. First, we decided to analyze our test results and data, so we collected all the information that we got from our test runs and analyzed, we discovered that our tests failed because of those main reasons:

Changes inUI components.
Badcoding.
Exceptionsand errors handling
Time outseverywhere
Data

A new experiment

We had to make a decision. We had invested lot of efforts in testing automation, and we were the product team with more automated tests compare to the rest of product teams globally in the company. This brought us attention from lots of managers, who were keeping their eyes on our DevOps and test automation activities. In addition, we were participating in a DevOps competition which involved all the IT teams of the corporation. So, we decided to invest time to get the failed tests in good shape and have them passing in the pipeline. We called this mindset “Keep them green and running”. We communicated this effort to all the product teams. We asked the Product Owners to consider maintenance time as part of their team capacity, include tasks for this purpose, and keep it part of their definitions of “ready” and “done”.

Unfortunately, we did notachieve the goals we had hoped for. We still had the same test results, withmore than 80% of tests failing. In almost every case, they were fake failuresthat meant that the tests were stopped long before their last steps. To sum up,our rest results were not accurate at all and did not bring confidence to ourteams — which is something critical in testing automation.

Thinking outside the box

We needed to try something different. After some discussion, research, investigation and thinking out of the box, we decided to use gamification to keep our tests in a good shape, running and be green. We named the game “Game of Testing” and we established some critical points about it:

Maturitylevels: The gamers can be developers, testers or anybody who owned an automatedtest running in the pipeline, they were maturing in the game, from a beginnerlevel to a master level.
Visibility:We shared the game boards including ranking, test results and so, on using bigmonitors in the office, allowing everyone to be aware of the gamers andresults.
Weight:This is a very complicated and mathematical part, we needed to stablishformulas and weights for a point system for the game, so we could evaluate thetest results, data per gamer and teams, and gave points and make them get newlevels.
Recognition:We tried to recognize all the gamers taking part. For the best gamers, weshared the results with local and global senior managers, so their achievementscould be known and recognized. Thishelped encourage more team members to participate.

The game was quitesimple:

Gamers: All the team members who owned an automated test running in the pipeline.
Point system: A gamer won point s for test running in green and negative points if the test failed for fake results. Later we added negative points of breaking builds.
Rules: You needed to keep all your test and team tests running in the pipeline in a good shape for each run. So, you had to invest efforts on “maintenance” and continuous improvement if you want to rock in the game.

The Game of Testing

This approach helps usa lot to decrease false positive runs and keep the tests up and running in thepipeline. We decreased the number of 80% of failed tests to only 20% inapproximately three months and our numbers have continued decreasing. Furthermore, people had fun while playing thegame and they internalized that quality can be fun and not boring like usualmaintenance was.

While we got several criticisms about our failing tests at the beginning of our efforts to improve them, and people lost trust on our quality assurance process, we didn’t give up trying to fix the problems. In fact, it motivated our team to think out of the box and turn this problem into a useful and fun project. We spent several hours investigating continuous testing, scripting, and related topics. Besides that, we wanted to do something different, something motivates other team members to join.

Gamification can provide that motivation. You may not achieve desired results immediately, but if you prepare your environment and involve everyone in trying potential solutions, the game will give you results for sure. If you address the test issues you can get them pass or fail, but if you involve and motivate people to work on problems together, you always win and have fun. I recommend you try gamification to sort your problems out, you may be surprised at the results!

Jorge Luis Castro Toribio

Jorge is an agilist who feels passion for Testing Automation, Agile, DevOps and business agility. He enjoys research and learning about business agility and working with cutting-edge technologies. He has been learning continuously for more than 12 years and is still learning new ways to foster business agility and team greatness. He has worked in several roles (developer, tester, manager, agile coach) which helps him to see the big picture and operational and team members dynamics happening inside organizations. This experience lets him help teams design, build and implement digital, DevOps and agile transformation strategies. He encourages focus on: People, Productivity, Continuous improvement, Innovation and having fun to enjoy success in our agile journeys.

You can find Jorge on and .

Visualizing Pipelines to Help Build a Quality Culture

By Lisa Crispin

At CAST 2019 , August 12-15 in Cocoa Beach, Florida, and I are pairing to facilitate an all-day workshop called “The Whole Team Approach to Testing in Continuous Delivery”. Our focus will be helping people to find new ways their teams can build a quality culture and succeed with continuous delivery. An important part of this is learning about deployment pipelines. For those of you who can’t attend the live version, here’s an introduction to why deployment pipelines are vital to successful continuous delivery, and a fun exercise your team can use to improve your own pipeline.

Continuous delivery – scary?

Continuous delivery (CD) lets us deliver new value to production safely, quickly and sustainably (to paraphrase Jez Humble). Each new commit to our code base trunk or master is independently verified as a deployable release candidate. We can deliver frequently, which means the changes are small, reducing the risk of problems.

Still, CD can seem like a scary proposition. In the old waterfall days, we didn’t have enough time to complete all the testing activities we wanted. Agile development enabled more bi-weekly or weekly releases, and brought up new testing challenges. And now, how can testing possibly keep up with continuous delivery? How can we feel confident about each release, from a testing/customer support perspective?

New approaches

Following the principles of continuous delivery helps us build confidence in our deployments. Following modern testing principles can give us even more confidence. When the whole team takes responsibility for building quality into the product and continually improving how we work, we (and our customers) realize the benefits of CD.

This is a huge topic, but let’s look at two key factors for success with CD. Obviously, we need adequate automated regression test coverage to make sure that the things our customers really need are still working. That’s not new. And, it isn’t a complete safety net. We can have great automated regression test coverage, do thorough exploratory testing, verify performance, and all the other many types of testing before we release to production, although we are doing that in test environments. We know that behavior is likely to be different and unexpected in production.

So what is new? Today, we have the ability to capture and analyze all kinds of data (and huge amounts of it) about production use and immediately detect any errors that occur. This is important, but it’s not that helpful unless we can quickly and reliably deploy fixes to production. And to do that, we need a fast, reliable deploy pipeline.

P ipeline basics

The deployment pipeline consists of steps that changes go through before they can be deployed to production. Every company, every team has their own unique pipeline. Note that whether or not your team has automated tests and automated deploys – you still have a pipeline. Whatever steps a new change goes through to get to production, even if each step is manual, is still a pipeline.

We break our build into stages or steps to speed up feedback. There’s a constant trade-off between fast feedback and confidence. The early steps of a pipeline are usually unit tests. They quickly show us whether anything is broken at that low, granular level. But we need more steps with higher-level, longer-running tests to feel more confident. Later steps in our pipeline exercise more complicated scenarios. We want fast feedback, but we may need comprehensive tests for confidence, and they take a long time to run. Here’s an example deployment pipeline:

In this example, the steps outlined in blue in this example are manual steps. We focus on automation when talking about pipelines, but manual steps are common. As in this example, activities like exploratory testing and security testing may be critical and may require manual work. We should be able to automate deploys to different environments, but perhaps your team just hasn’t gotten to that yet.

Things that slow down the pipeline

“Flaky” or “flappy” tests are a common problem even in completely automated pipelines. We often see this with user interface (UI) tests. Someone makes a change to the UI but forgets to update the automated tests to reflect that. A test environment slows down and causes a test step to time out. The browser version automatically (and unexpectedly) updates in the test environment. Those are just a few common causes of UI test failures.

Dependencies can also cause difficulties. A failure of one stage can stop multiple subsequent steps from running. Test data may fail to be set up properly, causing test failures. A test environment in the cloud might fail to spin up.

More steps in the pipeline give us more confidence, but they take more time. And how do we know all the steps are necessary?

When a pipeline experiences more and more delays, we humans have a tendency to increase the size of the items we’re sending through it to try to get more done. If the tests are failing and people keep committing changes, we’ve now got a big batch of changes which is more likely to include regression failures. It’s a death spiral. If our pipeline is reliable and fast, fewer changes go through it each time, reducing the risk of failures.

Collaborating to improve the pipeline

Making problems more visible is a great first step in making them smaller. Gather your team (or at least a representational group, if you have a large team) together. Take cards or paper and write the name of each step of your pipeline, one per card or paper. Put them on a table or on a wall to create a picture of your pipeline. (This can be done virtually if your team is distributed).

Visualizing a pipeline on a table – picture used with permission from Llewllyn Falco

Next, look at each step in the pipeline and ask “why do we have this step?” What are we learning from it? What business questions does it answer, what risks does it mitigate? Who on the team benefits from this information? You can capture this information on sticky notes to attach to each step.

Then, think about dependencies. For each step, what must in place for it to run successfully? A test environment? Test data? Specific configurations? Also think about how you can optimize the step. Is it currently done manually? Can you automate it?

Identify what triggers each step. Is it triggered automatically by the success of a previous step? Or manually? Can it be run in parallel with one or more other steps? Do you see opportunities to shorten feedback loops? Again, record outcomes of these discussion for each step.

Look at the length of your feedback loops. How much time elapses from code commit to completion of each step? What happens if the step fails – do the right people get notified? How? Is there a better way to communicate alerts? My previous team found that everyone was ignoring the Slack alerts generated by pipeline failures, so they hooked the build up to a police light. That got the team’s attention so that failures were quickly investigated and addressed!

Can any steps be done asynchronously? For example, it’s not realistic to do manual exploratory testing on every change when you are deploying to production frequently. The team can learn to use release feature toggles to hide changes from customers until the team has had time to complete any manual or time-consuming testing tasks.

Once you’ve gathered information about each step in the pipeline, choose a pain point to address and think of an experiment to reduce the pain. For example, perhaps your UI test suite has a lot of flaky tests and the team spends a lot of time kicking tests off again and diagnosing the failures that reoccur. Your experiment might be, “We believe that by refactoring the top 10 flaky tests to make them reliable will reduce our pipeline failure rate by 10% in the next month”. Use your retrospectives to evaluate your progress.

Remember the benefits

A fast, reliable pipeline means that you can confidently deploy frequent, small changes to production. And if you introduce problems to production that cause customer pain, you can shorten the time they feel pain by quickly deploying a fix with your fast, reliable pipeline. There are many other factors that go into safe, reliable and sustainable continuous delivery. Fast feedback and the ability to deploy changes quickly are a cornerstone of successful CD – and of a healthy quality culture.

References:

A Practical Guide to Testing in DevOps , Katrina Clokie, 2017
Continuous Delivery , Jez Humble and David Farley, 2010 (also see their website )
, Chapter 23, Lisa Crispin and Janet Gregory
“ Feature Toggles, aka Feature Flags “, Pete Hodgson

Lisa Crispin is a testing advocate with . With Janet Gregory , she has co-authored Agile Testing , More Agile Testing , and the upcoming Agile Testing Condensed . Lisa and Janet also have a 5 hour video course, Agile Testing Essentials . Through the Agile Testing Fellowship , they offer a three day live course, “Agile Testing for the Whole Team”.

Path to Production – Test Environments as a Product

By Rob Hornby

It could be easier, we could have teams for each application we run, that those teams treat their production and non-production application environments the same. That they have fully automated full stack continuous deployment pipelines and the software is so well designed that there is no dependency between each application owner. Teams deliver code by the hour to production and use advance deployment techniques such as blue-green deployment coupled with highly effective monitoring, alerting and automated rollback to identify and resolve issues in production within minutes.

We do have “agile” application teams working within a cloud environment, they have growing autonomy as they build independent features and function. That’s awesome. The problem is that we have not been working that way since we started building technology. We have a sprawling on-premise (we own two data centres) environment where Non-Production environments equates for 60%+ of the overall infrastructure. This is made up of dependent applications running together in an environment.

We have followed traditional programme structure over the last 10 years, with multi-year investments and waterfall based delivery. These have sought to reduce risk through separation rather than integration. This has resulted in multiple non-production environments. Our applications (100’s) have been built tightly coupled and need multiple other applications to work (Commerce systems through to legacy mainframe systems all meshed together), increasing the size of the basic/core environment. These programmes have not invested in automation, leaving a challenging sprawling estate to manage.

Path to Production (Variety of routes followed by teams ensuring lack of consistency in versioning in each environment)

This has created a number of problems. The estate is costly to maintain. There is hidden cost in each application change or infrastructure upgrade. Our people are linked to projects and programmes, when they finish, the applications in non-production fall in to disrepair, they are costly and time consuming to get working again when the next team comes along. A continued desire for separation results in projects hunting down that unused application/environment that has nobody else working in that space. The sprawling nature and lack of ownership results in a small band of “Environment Management” who work to bridge the communication and knowledge gaps across this sprawling estate. These people work for Projects and Programmes and through “best endeavours” get Delivery teams, Operations and Infrastructure teams to stitch the environment back together each time and retain the illusion these environments are good to test in. We effectively have 10 mini-lives beating away with no real Service Management to deal with them.

That long introduction was needed just to give me time to breathe and reflect. The organisation is changing for the better, which is great. We are re-organising and realising the mistakes of the past. We are now building most new systems in a cloud environment, are focused on Micro-service design principles and have teams that own and run the service (well almost). These are small individual successes which highlight the organisation is changing. Our online teams have gone from 10 deployments a year to over 1000 in just over 18 months. It should be a factor of 10 greater by next year. However there is a long way to go till we are fully working in this way, maybe we will never get there. This is where Environment as a Product comes in. How do we change our existing non-production environment landscape for the better.

We have started with two key initiatives. Initiative 1, Let’s get support to introduce an “enabling constraint” through removing the environments that least support delivery. They are the most difficult to maintain, are off the beaten path to production and the most costly to re-enable them. If the project or programme don’t have them they can’t use them. We know they are costly to build which is part of the reason they have been maintained. With the remaining environments we are going to create a paved road of sorts. One of the key things is to not construct the path to production based on your current organisational/delivery construct. We have been very waterfall in nature and have therefore not made change often to production.

This results in us having multiple hops to production for a single change based on the type of testing to be undertaken at each stage. Development (unit, system), Integration, Release (end to end), Performance (performance, operational), Pre-Production (incident and support) and Production. You introduce delay just through deploying to each of these environments and fixing the issues that come up as a result. That’s before you think of data, types of testing, handoff implications. So initially let’s get down to one of each type to support all teams, projects and programmes. Let’s draw it up so it looks like some are optional. Development, Integration, Pre-production, Production (even this is too many).

This was difficult to get going, convince people that we might be able to go faster through having less environments, that there might be more contention initially, however that might force us to look at the organisational shape and how we design software. Whether the full extent of the idea has sunk in, is yet to be seen, however the cost avoidance articulation of maintenance running in to millions of pounds a year was hard to ignore and gave us the initial green light.

Initiative 2, is to consolidate the team we have and build a paved road (common environments, tools and approach). We have taken everyone responsible for non-production “Environment-Release Management” and centralised them in to a single Product team. This wasn’t easy. You have to convince the leadership team that they want to pay for this as a separate service, that the cost is neutral, value adding even and that in doing so you can look at how the team works and make efficiency, as well as improving the path to production. What this doesn’t do is centralise the technical people responsible (not owning) for those environments. These remain fragmented across Operations, Infrastructure and Delivery teams. While the number of different technologies, teams and ways of working make this impractical at a small team level.

It does allow you to put a very clear Service Management approach in place for Non-Production and with leadership support allow you to be more robust in enforcing a better approach. You can start to work with individual teams to support and help them approach their delivery differently, focusing on automation, repeatable process and decoupling of change. The aim is that teams do this themselves and that the Environment-Release team can identify teams that need support. We are also backed by organisational , which have been established to give teams direction in how they should deliver.

Centralising the team has also allowed us to look at how we approach managing the environments. A single Product team with longer term goals, where the value is what we do to the environments, not the end customer outcome that the Project or Programme was focused on, has a very different priority and outlook. What was several ways of working can now be focused in to a single way of working. We have started by making sense of what people were doing daily. This has been done through several value stream mapping workshops but also to get everyone’s work up on a board and try to make sense of the work we do. Get a feel for reactive vs planned work. Upon doing this we can start to build a very clear backlog of priorities for our own ways of working, how we track issues, do we manage problems, how do teams onboard on to our path to production, can we pre-empt issues, what do we do with data, looks at our skills and most importantly clear communication.

Value is now a clear driver, as is ownership. We can now mandate changes to production are part of non-production thinking. We can also start to look at cost of delay and overall costs of supporting teams and drive our own value outcomes. Lead time is also a key measure, whether that that is to measure how quickly teams move through our environments, how long it takes us to respond to incident, downtime within environments etc… We can also help to start to measure teams frequency of deployment/release and to show organisational pace of change. Traditional service management is very much focused on controls but here we want to make it much more about value outcomes and enabling flow within teams.

We also have one foot firmly in the future. Over time the need for this team will be less. Greater ownership, more end to end owning teams, “DevOps” will all drive less need for a Environment-Release Product team. Our life span is based on the value we can show to generate. However we need to develop our skills in to other areas, whether that be Platform management within our Cloud platform or in to more “DevOps” skill sets which can work within teams to help them deliver. For now we are focused on our Product and value. It would be interesting to hear how other companies and people manage non-production environments, especially where there has been ongoing development of a variety of approaches for many years.

Rob Hornby is a Product Engineer working for John Lewis based in London. He has undertaken a number of roles, over the last 20 years, within Testing, Release, Platform and Environments.

Rob is a keen believer in being able to treat all things as Products and be able to evidence the value generated by that team. Ideally everything would be delivered by small teams, delivering continuously and iterating on value. However not all teams start at this point, not all teams are the same or born equal. This is where the challenge and enjoyment comes from.

Outside of work, Rob is a keen runner, father of an amazing young girl, Sofia and works on a number of local community events.

Applications are more than just code

By Matt Long

I was at TestBash Brighton recently, and a common theme kept coming up:

“You can be technical without knowing how to code!”

, TestBash organiser, even did a 99 second talk on the topic. He listed various skills – git, debugging, changing the microwave clock. If you have these skills as a tester, there are plenty of uses for these skills. If you have an agile or DevOps culture – where roles are less set in stone, and teams swarm on problems – you’ll get loads of opportunities to use them.

Well, apart from the microwave clock.

Learning the ecosystem

A yak dominates an ecosystem

It’s important to realise most developer skills aren’t coding. If you’ve tried to learn coding, you’ll know lots of other skills can get in the way. You start writing a simple application and lots of good practice pops up. Your developer friend says you should set up a git repository. Then instead of installing dependencies manually, it’s better to set up a dependency manager. Then you learn how to compile and run the code. The list goes on. This can be very daunting, and quickly turns into yak-shaving .

Learning doesn’t stop with developer skills

This seemingly never ending list of things to learn doesn’t stop with a developer’s skillset. Run a production-grade environment, and SRE/Ops skills will get in the way. Ops ensure that users can use your environments. They do all kinds of things. Backups. User and authentication control. Familiarity with the Linux CLI to debug issues. Log monitoring and analysis. Failover processes like blue/green. Containerisation like Docker , or container orchestration like Kubernetes . Your workplace may differ in the specifics, but production environments have lots of additional complex functionality. Complex functionality you can’t replicate by running the code on your laptop, or even sometimes in a test environment.

Broad skills are valuable too

Your broad skills are valuable

There are specialists in these subjects. Nobody is expecting testers to be experts with all of these skills. Much like testers don’t have to have all the coding skills of developers. That said, knowing about these things will help us understand our application. It will help understand facets of our application otherwise overlooked. If testing is about understanding the quality of our application, then we need to look beyond the code. As far as your application is concerned, code is the tip of the iceberg.

Let’s take an example: logging in to an application. Your users are not only expecting that they can login successfully. They’re expecting that the database stores their user credentials. They’re expecting that their data is backed up in the event of disaster. They’re expecting that they can still login if lots of other people are. They’re expecting their confidential data is securely stored and transported.

Much of that functionality is not done in code by developers. It is often the responsibility of ops, or sysadmins, or database admins. They configure databases and networks and servers that ensure all these things happen.

Ops opportunities for testers

Before DevOps, it was difficult for testers to have influence upon this. Much was siloed away from us. We test the code, then let them deal with that stuff. Out of sight, out of mind.

Cross-functional teams include Ops specialists

But now, Ops specialists are being moved into our teams, or working closely with our teams (hopefully not as a permanent ‘DevOps team’!). We see the work they do, and it needs testing too. That’s because it’s part of the application. It’s part of what users expect. We don’t just deliver code, we deliver a service.

Pairing — Pair with an ops expert to assess risks

So how can testers start helping? To test code, we didn’t need to be developers. But learning related skills helped us empathise and co-operate. It helped us ask the right questions. So try pairing with an ops expert on your team and work out what risks might arise. Or doing some shaping of some infrastructure tickets, and thinking about who the users are (with pipelines, they’re often developers!). Or try writing some automation scripts to ensure that associated problems get caught before production. Or go to a devops meetup/conference. Learn about bad practice and security/infrastructure smells. Start treating ops like developers – to be assisted and questioned.

Because our application is more than just code. It’s all the processes, and the infrastructure too. That stuff needs testing too.

Matt Long is a QA & devops consultant for OpenCredo , an open source software consultancy based in London.

He has worked as a QA consultant for many years, and built test automation frameworks in half a dozen programming languages. After working on many cloud infrastructure projects, he’s become particularly interested in the intersection of modern cloud infrastructure and testing.

BuryTheHammer.com ,

Open Sourcing Your Mindset

By Brendan Connolly

Asdevelopment has shifted from waterfall to agile and now DevOps, at each stageyou see developers taking on more responsibility and accountability.

Waterfallwas isolationist, building silos around different groups in softwaredevelopment. Developers owned the act of coding and in the stereotypicaldeveloper way, had little to no interest what happened before or after theycoded. Other roles on teams in many ways acted as insulators to keep externalforces away from developers.

Agilewas developers pulling that insulation away, shifting themselves left andtaking more ownership of the product creation process. Developers found theyrequired a real time feedback loop to build the right things and successfullyreact to change.

DevOpsis essentially the same process on the right side of the spectrum. Developmenttakes more ownership of deployment, monitoring and supporting the applicationshealth. It’s just too slow and cumbersome to react to the requirements ofmaintaining applications by filtering feedback through separate teams at scale.

Thebenefits of integration outweigh the cost of the additional responsibility.

Code Continuous

Tocope with the additional responsibilities, developers have turned to theirtrusty friend code .

Withagile we see rise of tooling developed for ContinuousIntegration to alleviate the pain and reduce the risk in iterativedevelopment.

InDevOps we see the rise of code and tools supporting Continuous Deployment . Even when many teams already had productinstallers, or automated deployments what was missing was the integration ofthat information.

Ifdevelopers are owning more of the product as well as release and deployments,then testing naturally gets swept up in the process. In fact we see thispattern emerging with the concept of ContinuousTesting . Code is being written, tools created to help solve the issue ofintegrating test execution and reporting into teams release pipeline. Thefeedback loop is extended to provide rapid information to make it easier andfaster for development to be responsive to change.

Continuous Testing

Automatedtesting is a loaded term. In developer circles it almost exclusively means unitand possibly integration tests. In QA it almost almost exclusively means end toend (e2e) tests, and is often synonymous with Selenium .

Onteams where both sets of tests exist, their worlds may seldom interact.Developers may be aware of the e2e tests but they probably aren’t writing them and even less likely running them.

*Illustration from “Continuous testing in DevOps”, Dan Ashby, https://danashby.co.uk/2016/10/19/continuous-testing-in-devops/*

ContinuousTesting has nothing to do with the testing that most QA team members are mostfamiliar with, exploratory testing. The purpose of the code and tooling thatcomprises a pipeline isn’t for there to be some manual intervention requiredpart of the way through it. Putting exploratory testing somewhere in the laterstages for the pipeline is akin to pointing a fire hose at someone and askingthem to direct the flow to the next stage.

Ifthe speed of DevOps wasn’t enough, the presence of continuous testing only puts morepressure on testers, simply because more attention is being drawn to testing.

Automationengineers and product engineers are being drawn closer together. The importanceand relevance of end to end automated tests is becoming more of a reality forproduct developers as they become run regularly as a post deploy verificationtests.

Whilecode is being thrown at testing with newfound passion, exploratory testers needto be prepared for their time in the spotlight. If automated tests are runningall the time what tests are you doing? What makes them special? Why are theyrequired before we can deploy?

What do you do different than Selenium?

Beforeyou dismiss that question, pause and consider what all selenium and the testsit drives truly provide.

Seleniumis cross browser, scalable, schedulable, repeatable, and can provide logs andscreenshots of its tests. Can you compete with that?

Flaky,slow, and wrong sometimes, you say? And you and all other humans aren’t?

Conversationsthat try to explain using people instead of machines for testing software canend up justifying them under the guise of thereare some tests that aren’t good candidates for automation .

Don’t settle for executing tests that don’t have enough ROIto automate.

Thistype of conversation relies on the belief that testing is comprised to twoelements: test case creation and test execution . Perhaps at its most baselevel testing is just that, the value proposition of the role is in the nuancethat a skilled tester can provide to teams through those elements.

Testersdo more than a fill a gap in automated test coverage. The problem is manytesters express this nuance solely through actions.

Developersproduce code, a measurable artifact from their effort. The generation of thatartifact is often backed by computer science, a formalized area of study.Artifacts of effort are reassuring, people like to see the fruits of theirlabors. Tangible results can be measured and quantified, they might even helpestimate future effort.

A tester’s value tends to be recognized at the same rate as relationships develop. Testers get characterized as folk heroes rather than members of a skilled profession. There’s no trade, education, or craft just individuals saving the day periodically.

The Intuition trap

Evenwhen teams do value testing, people are usually primarily concerned with theprogression or regression of testing efforts. Testers comments in stand-ups caneasily turn into a user story status change report:

Passed this ticket, working this ticket, noblockers

Ifyou aren’t taking the opportunity to discuss your actual actions in stand ups,then you may fall prey to the intuition trap. When you talk about your workthis way, sometimes you end up actually working this way.

Open Source Your Mindset

It’sfun being a hero, but there’s no skill in being Superman or Superwoman (I was going to say Wonder Woman, but she actually trained and I think everyonerecognized that. So if you are the teams Wonder Woman then kudos, you haveserious skills but I digress).

Whentesting goes smoothly the result is nothing .Testing isn’t required for things to go right, it just might help prevent athing going wrong.

Opensourcing your mindset means teaching others another way to look at and interact with their code, or the product.This cuts both ways though, it means being willing to accept a pull request ofoutside perspective. Openly integrating new ideas into your testers mindset .

Replace Execution with Communication

. Like acting or athletics there might be a script or game plan to start with but there is a lot going on behind the scenes, adapting to circumstance and being thought up on the fly. It’s intuition driven.

Whenactions, practices and motivations are not being verbalized and expressed tothe team and stakeholders it puts testing on the level of superstitions andtoken time sacrifices to the quality gods.

Testersneed to pry open this experience, expose the internals driving motivations.Breaking out of that shell grants visibility into a testers actions andmotivations to provide a context for team members to attribute value.

Replace Isolation with Collaboration

Itcan be uncomfortable or difficult to open up and share what goes through your testing mindset .

Theadditional ownership developers have taken in DevOps has changed relationshipdynamics. What was at one point a contentious us vs. them relationship is muchmore open. There just isn’t the time to have as much formal process drivencommunication. Success at scale requires collaboration, pairing and mobbingmore.

When with developers, it’s an opportunity to test at their desk, against their local changes. There are risks testing here, the developers local setup is definitely different than production but so are the testing and staging environments. Let go of the taboo of it works on my machine and consider what can be tested here and what needs to be covered on a different stage. Take the opportunity to work together when the code is the freshest in the developers mind and where they are most comfortable.

It’s not only about working with developers. Mob testing with UX and Product alongside developers is a way to draw the entire team together. When the team is together seeing how the application is working it bypasses all the communication gates. There is no need for a separate follow-up conversations to loop someone in, that person can see and feel how things are working by being a part of the group experience.

Replace Tedium with Stratagem

Teamsare more open to testing and sharing in quality than ever. Testers have beenbuilding quality mindsets for years. Developers are building tooling but teamsare looking for guidance and mentorship in Quality .

Testing isn’t just button pressing, by showing that through communication and working together we elevate our status. Draw people together, into the quality circle, and enable the team to be more comfortable and aware of testing. We can set working agreements, talk about overarching strategy for how testing throughout the development cycle builds quality in. We can embrace automated testing as a team activity where unit, integration and end to end coverage is part of a teams strategy for mitigating risk. We can forgo complete control of testing and opt into being a voice of authority and expertise in quality.

Brendan Connolly is an experienced Software Tester, Automator, Speaker, and Blogger. Currently he is a Senior Quality Engineer at Procore Technologies in Santa Barbara, California. He maintains a testing blog @ http://www.brendanconnolly.net

DevTestOps: It’s alive, but what can testers do to make it alive AND kicking into overdrive!

By Toyer Mamojee

The journey of the ‘typical tester’ has been a pretty bumpy ride over the past few years, more so recently with the massive uptake of DevOps. Apart from the many new and intimidating terms thrown our way, we are now required to do anything from digging in to business case details to monitoring how applications are performing in Production. It’s become a world which is very different to the one a tester would have experienced around the year 2005 or so.This world has incorporated testing to be at the heart of it, and not just testing but full on quality! So its only natural for the tester to play a massive role here.

But what can we do?

As mentioned, the tester should be at the very heart of every activity happening in the software delivery process. Below I highlight some of the areas where testers can directly influence and should influence if the process is to go from just being alive, to becoming Alive and Kicking!

Help Sculpt a Team Culture (Team player)

When one speaks about DevOps, technical aspects are usually at the front of the conversation whereas the softer aspects like Team culture and communication are very rarely mentioned.

Everyone loves quality products, testers love quality. A good healthy, dedicated team can help make better quality products. So naturally a tester should be one of the key individuals to drive a dedicated team. Testers should keep their critical thinking quality as their most prized asset by asking the right question at key moments during team discussions/meetings and interactions. This gets the team thinking and hopefully to a point where this starts to become second nature to all team members. Quality should be a driving force of every team member in everything they do, this should be a non-negotiable principle.

Another key area around team culture is by re-iterating the culture of ‘safe to fail’ and the ability to recover quickly. Without these important concepts inherent in a team the ability to deliver faster might prove to be a huge challenge.

Promote Visibility and Transparency (Seeing is believing)

I cannot stress how key this aspect is to a solid DevTestOps model. Gone are the days of working in isolation. A tester should look at some of the key artefacts that traditionally would fall within his/her space and how these can be made visible and transparent to the entire team.

Some of these include Test coverage visibility, defect statuses per sprint and release, test automation coverage, test automation scripts (functional and non-functional). One can make these visible in the form of dashboards or even repositories for test checklist or automation scripts.

At any point in time any team member should be able to give you the current status of your testing per sprint and/or release. After all if it is a difficult and an intensive process to decide on whether you are ready to deploy to production, then you have to question what you as the tester can do to change this.

Increase your skillset (Up your game)

Learn to use tools that help you work better — Learn to use tools that help you contribute more (Photo by on

A modern tester needs a modern armoury. Without it, I battle to see how the transition to the DevTestOps model can be easily made.There is so much to mention under this area, up-skilling should span these key areas:

Automation tools

From backend to front end focused automation tools, a tester should not only know these tools but know when and where to use them. Tools learned should include those which at least cover API testing, UI Testing, Performance/Load Testing, Security and Databases.

Context switching between these tools and types of testing is a rather intimidating thought for those not usually exposed to modern ways of software delivery. My advice here is to take baby steps in learning these tools in order to not just learn something for the sake of it, but rather for the cause of true DevTestOps.

Technical Tools

Technical type tools that equip a tester to contribute effectively in a DevOps environment would once again span the entire spectrum of the application solution. Tools like Postman & Swagger for API testing, knowing your way around version control applications (GitHub, GitLab etc), SQL and NoSql interfaces (like DBKoda, Sql Server etc), CI/CD apps (TravisCI, GoCD, Jenkins etc) , Logging applications (GrayLog, ElasticSearch,ElkStack etc) and Monitoring tools (NewRelic, Dynatrace, Instana etc).

Contribute to your CI/CD (its in the pipeline)

Testers have a crucial part to play in defining the ideal pipeline for your applications deployment. Being part of discussions around composition of the pipeline, ideal duration of pipelines, pre and post deployment tests as part of the pipeline.
A point to consider even further is where do your automation tests fit in these pipelines? Should it be part of it?
CI/CD tools also promote to the point on visibility and transparency, if constructed properly the entire team should know exactly at what stage the deployment is at, what tests were run and the overall status of the deployments.

Monitor it (What’s trending?)

To know what your application is doing both in PROD and Test environments, contributes to how you approach your testing. Findings and trends even contribute to what scenarios you end up testing and has massive input regarding determining what load you put your system under during Performance testing as an example.
Traditionally testers would steer clear of any tool or application that falls outside the scope of testing, but in a fast paced delivery world, monitoring tools can be a testers best friend. A tester’s system or application knowledge should also push the tester to not just use existing monitoring metrics but to also determine what to include as part of monitoring.

In summary

The evolution into DevTestOps has meant that the tester now plays a much more involved role in the entire delivery process. The day to day tasks of a tester spans between a balance of soft skills, analytical, technical and even an investigative mindset sometimes seamlessly crossing between these tasks without even knowing. By following some of the pointers above the initial ground work to successful DevTestOps is well underway. Go on Testers, get DevTestOps Alive and Kicking in full swing!

Toyer Mamoojee is a ‘die-hard’ passionate Agile Automation Test Analyst with almost 16 years testing experience across many industries. From Waterfall to Agile approaches, Commercial to Open Source Automation tools and Manual to Automation Testing he has been through the wars of Software Delivery. He is the co-founder of Cape Town Automation User group and co-organizer of Cape Town Testing Meetup back in his home country of South Africa. In his spare time he loves exploring the great outdoors, taking on the world playing online video games , major football fan, adopting the title of technological husband and playing super dad to 2 pre-schoolers!

Toyer Mamoojee @tottiLfc