pipelines Archives - DevTestOps Community

Running API Smoke Tests in Your Continuous Deployment Pipeline

By Kristin Jackvony

Each year, more software companies switch to a microservices architecture using Application Programming Interfaces (APIs), because APIs make it easy for different teams on a project to access each other’s resources. APIs also enable software companies or individuals to get data from each other: for example, Twitter has an API that is used by many small businesses to retrieve tweets related to their business for display on the business’s webpage.

Another great feature of APIs is that they are smaller than a monolithic application, which means that they can be deployed frequently. At the company where I work, my team has eight different APIs, and we have eleven different environments that we deploy to. If we were relying solely on manual testing to verify that the APIs had been deployed correctly, we’d have to run eleven sets of tests for each API release. And if we were deploying more than one API at a time, which we often do because our APIs work together, we could be running 88 sets of tests. And if we deploy every two weeks, those 88 sets of tests will amount to a lot of tedious, repetitive testing!

We’ve solved this problem by setting up automated API smoke tests that run with every deployment, to every environment. In this article, I’ll describe how we decided what to test and how we set up our smoke tests. We used Postman, Newman , Powershell , and Octopus to set up our automated smoke tests, so I’ll be describing what we did with those tools. However, these strategies could be adopted to any continuous deployment (CD) pipeline.

Step One: Decide What to Test

The main purpose of smoke tests is to do some simple, high-level tests to ensure that the deployment was successful. This is not the place to test every single thing that your API can do. When we chose what to put in our smoke tests, here is what we considered:

We set up one “Happy Path” test for each endpoint- that is, one test that is expected to be successful. For example, a GET request for a resource with a specific ID would be expected to return that resource. You’ll want to test each endpoint once to verify that each one is working as expected.
If there were major variations in how a specific endpoint would be used, we added one or more tests to check those variations. For instance, we have a file retrieval API where a file can be retrieved by two different methods. The endpoint looks the same, but the bodies of the requests are different. So we added one test for each method.
For endpoints which required a certain level of security, we added a test to validate that a request without the appropriate authentication would NOT return information, and would instead return a 400-level error.
We did not add any other negative tests, such as a POST request with data outside the allowed parameters, because we didn’t feel that those were necessary to ensure that the API was working. We instead ran those tests as part of a nightly regression suite.

Step Two: Export Your Tests

We used Postman to create our API tests, so when the tests were created, we exported both the test collection and the test environment file into JSON files. For those who are not familiar with Postman, a test environment is a group of variables that can be referenced in a collection of tests.

Step Three: Write a Script to Run Your Tests

Postman has a command-line tool called Newman that can be used to run a test collection, so this is what we used to run our tests. We created a Powershell script that would run the Newman command.

Step Four: Create a Deployment Step to Call Your Script

Once the Powershell script was ready, we set up a deployment step in each Octopus API deployment project that would run the test script. If the script failed, the deployment would fail.

Step Five: Organize Your Variables

Variables are often a consideration when running smoke tests in different environments. For example, in your QA environment, your test user might have an ID of 340, but in your Staging environment, your test user might have an idea of 620. Another issue with variables is that sometimes for security purposes, you might not have access to passwords or keys in Production environments. This was the case for our team, but fortunately Octopus has the values we need to run our tests in Production, so we could solve the problem by having Octopus pass the values that we needed to our test script.

There were three different kinds of variables needed for our smoke test:

Type 1: Variables that were unchanging for each test environment, such as “firstName”: “Prunella”. These variables could be put directly into the Postman environment, so nothing more needed to be done with them.

Type 2: Variables that changed for each test environment, but did not need to be kept secure, such as “userId”: 340. These variables were added as Octopus variables in this fashion: “smoke.userId”, and the value of the variable was set for each environment; for example, QA was set to 340, Staging was set to 620, and Production was set to 450.

We then used the variables like this:v

When we called the Powershell script in Octopus, we sent the variables set in Octopus to the script.
In the Powershell script, we accepted the Octopus variables sent in and assigned them to Powershell variables.
When we used the Newman command in the Powershell script, we sent the variables in the script to the Postman environment.
Newman used the variables sent in the Powershell script, combined with the variables in the Postman environment to run the Postman collection.

With our API smoke tests running in Octopus for every API and in every environment, we can feel confident that any big problems with an API deployment will be detected right away. Furthermore, we can test in Production environments even when we don’t have access to sensitive API keys. Having automatic deployment smoke tests frees us up to do other types of testing, such as manual exploratory testing and security testing, as well as write more test automation for nightly regressions.

About Kristin:

Kristin Jackvony discovered her passion for software testing after working as a music educator for nearly two decades. She has been a QA engineer, manager, and lead for the last ten years and is currently working as a QA Lead at Paylocity. Find me on and .

Continuous Testing in DevOps: A gamification approach

By Jorge Luis Castro Toribio

A continual problem that we must deal with when we run automated tests in our DevOps pipeline is maintenance. I could spend hours sharing posts and blogs about this matter, telling stories about how it affected our team’s capacity and time to market, and giving you several examples of what teams have done to sort this problem out. In fact, all the effort that we invest in maintenance is not visible to the business stakeholders’ eyes. Maintenance consumes our team capacity, but it does not add value to our clients directly.

Test automation challenges at scale

In one of my jobs wefaced this problem on a large scale. We had more an average of more than 1000automated tests running at each code check in, plus one big run at midnightthat we called full regression. Our suites where organized in those levels:

L1: Unit testing – Development Stage
L2: Integration Testing – Commit Stage
L3: UI Functional Test MVP Suite – QA Stage
L4: Regression suites – Full package – UAT Stage

Those testing levelswere our creation, but we used the great lessons shared in this book https://www.amazon.com/Practical-Approach-Large-Scale-Agile-Development/dp/0321821726 that I recommend for any team looking to buildtheir testing strategy.

After each run, we noticed that 80% of UI and regression automated tests failed due fake failures. After reading my numbers, you can say if you automated crap, you will get crap results and that is correct — but please remember that we were building our automated test suites, with a continuous testing approach and DevOps mindset and pipelines, so we were trying to fail fast, learn and improve. First, we decided to analyze our test results and data, so we collected all the information that we got from our test runs and analyzed, we discovered that our tests failed because of those main reasons:

Changes inUI components.
Badcoding.
Exceptionsand errors handling
Time outseverywhere
Data

A new experiment

We had to make a decision. We had invested lot of efforts in testing automation, and we were the product team with more automated tests compare to the rest of product teams globally in the company. This brought us attention from lots of managers, who were keeping their eyes on our DevOps and test automation activities. In addition, we were participating in a DevOps competition which involved all the IT teams of the corporation. So, we decided to invest time to get the failed tests in good shape and have them passing in the pipeline. We called this mindset “Keep them green and running”. We communicated this effort to all the product teams. We asked the Product Owners to consider maintenance time as part of their team capacity, include tasks for this purpose, and keep it part of their definitions of “ready” and “done”.

Unfortunately, we did notachieve the goals we had hoped for. We still had the same test results, withmore than 80% of tests failing. In almost every case, they were fake failuresthat meant that the tests were stopped long before their last steps. To sum up,our rest results were not accurate at all and did not bring confidence to ourteams — which is something critical in testing automation.

Thinking outside the box

We needed to try something different. After some discussion, research, investigation and thinking out of the box, we decided to use gamification to keep our tests in a good shape, running and be green. We named the game “Game of Testing” and we established some critical points about it:

Maturitylevels: The gamers can be developers, testers or anybody who owned an automatedtest running in the pipeline, they were maturing in the game, from a beginnerlevel to a master level.
Visibility:We shared the game boards including ranking, test results and so, on using bigmonitors in the office, allowing everyone to be aware of the gamers andresults.
Weight:This is a very complicated and mathematical part, we needed to stablishformulas and weights for a point system for the game, so we could evaluate thetest results, data per gamer and teams, and gave points and make them get newlevels.
Recognition:We tried to recognize all the gamers taking part. For the best gamers, weshared the results with local and global senior managers, so their achievementscould be known and recognized. Thishelped encourage more team members to participate.

The game was quitesimple:

Gamers: All the team members who owned an automated test running in the pipeline.
Point system: A gamer won point s for test running in green and negative points if the test failed for fake results. Later we added negative points of breaking builds.
Rules: You needed to keep all your test and team tests running in the pipeline in a good shape for each run. So, you had to invest efforts on “maintenance” and continuous improvement if you want to rock in the game.

The Game of Testing

This approach helps usa lot to decrease false positive runs and keep the tests up and running in thepipeline. We decreased the number of 80% of failed tests to only 20% inapproximately three months and our numbers have continued decreasing. Furthermore, people had fun while playing thegame and they internalized that quality can be fun and not boring like usualmaintenance was.

While we got several criticisms about our failing tests at the beginning of our efforts to improve them, and people lost trust on our quality assurance process, we didn’t give up trying to fix the problems. In fact, it motivated our team to think out of the box and turn this problem into a useful and fun project. We spent several hours investigating continuous testing, scripting, and related topics. Besides that, we wanted to do something different, something motivates other team members to join.

Gamification can provide that motivation. You may not achieve desired results immediately, but if you prepare your environment and involve everyone in trying potential solutions, the game will give you results for sure. If you address the test issues you can get them pass or fail, but if you involve and motivate people to work on problems together, you always win and have fun. I recommend you try gamification to sort your problems out, you may be surprised at the results!

Jorge Luis Castro Toribio

Jorge is an agilist who feels passion for Testing Automation, Agile, DevOps and business agility. He enjoys research and learning about business agility and working with cutting-edge technologies. He has been learning continuously for more than 12 years and is still learning new ways to foster business agility and team greatness. He has worked in several roles (developer, tester, manager, agile coach) which helps him to see the big picture and operational and team members dynamics happening inside organizations. This experience lets him help teams design, build and implement digital, DevOps and agile transformation strategies. He encourages focus on: People, Productivity, Continuous improvement, Innovation and having fun to enjoy success in our agile journeys.

You can find Jorge on and .

Visualizing Pipelines to Help Build a Quality Culture

By Lisa Crispin

At CAST 2019 , August 12-15 in Cocoa Beach, Florida, and I are pairing to facilitate an all-day workshop called “The Whole Team Approach to Testing in Continuous Delivery”. Our focus will be helping people to find new ways their teams can build a quality culture and succeed with continuous delivery. An important part of this is learning about deployment pipelines. For those of you who can’t attend the live version, here’s an introduction to why deployment pipelines are vital to successful continuous delivery, and a fun exercise your team can use to improve your own pipeline.

Continuous delivery – scary?

Continuous delivery (CD) lets us deliver new value to production safely, quickly and sustainably (to paraphrase Jez Humble). Each new commit to our code base trunk or master is independently verified as a deployable release candidate. We can deliver frequently, which means the changes are small, reducing the risk of problems.

Still, CD can seem like a scary proposition. In the old waterfall days, we didn’t have enough time to complete all the testing activities we wanted. Agile development enabled more bi-weekly or weekly releases, and brought up new testing challenges. And now, how can testing possibly keep up with continuous delivery? How can we feel confident about each release, from a testing/customer support perspective?

New approaches

Following the principles of continuous delivery helps us build confidence in our deployments. Following modern testing principles can give us even more confidence. When the whole team takes responsibility for building quality into the product and continually improving how we work, we (and our customers) realize the benefits of CD.

This is a huge topic, but let’s look at two key factors for success with CD. Obviously, we need adequate automated regression test coverage to make sure that the things our customers really need are still working. That’s not new. And, it isn’t a complete safety net. We can have great automated regression test coverage, do thorough exploratory testing, verify performance, and all the other many types of testing before we release to production, although we are doing that in test environments. We know that behavior is likely to be different and unexpected in production.

So what is new? Today, we have the ability to capture and analyze all kinds of data (and huge amounts of it) about production use and immediately detect any errors that occur. This is important, but it’s not that helpful unless we can quickly and reliably deploy fixes to production. And to do that, we need a fast, reliable deploy pipeline.

P ipeline basics

The deployment pipeline consists of steps that changes go through before they can be deployed to production. Every company, every team has their own unique pipeline. Note that whether or not your team has automated tests and automated deploys – you still have a pipeline. Whatever steps a new change goes through to get to production, even if each step is manual, is still a pipeline.

We break our build into stages or steps to speed up feedback. There’s a constant trade-off between fast feedback and confidence. The early steps of a pipeline are usually unit tests. They quickly show us whether anything is broken at that low, granular level. But we need more steps with higher-level, longer-running tests to feel more confident. Later steps in our pipeline exercise more complicated scenarios. We want fast feedback, but we may need comprehensive tests for confidence, and they take a long time to run. Here’s an example deployment pipeline:

In this example, the steps outlined in blue in this example are manual steps. We focus on automation when talking about pipelines, but manual steps are common. As in this example, activities like exploratory testing and security testing may be critical and may require manual work. We should be able to automate deploys to different environments, but perhaps your team just hasn’t gotten to that yet.

Things that slow down the pipeline

“Flaky” or “flappy” tests are a common problem even in completely automated pipelines. We often see this with user interface (UI) tests. Someone makes a change to the UI but forgets to update the automated tests to reflect that. A test environment slows down and causes a test step to time out. The browser version automatically (and unexpectedly) updates in the test environment. Those are just a few common causes of UI test failures.

Dependencies can also cause difficulties. A failure of one stage can stop multiple subsequent steps from running. Test data may fail to be set up properly, causing test failures. A test environment in the cloud might fail to spin up.

More steps in the pipeline give us more confidence, but they take more time. And how do we know all the steps are necessary?

When a pipeline experiences more and more delays, we humans have a tendency to increase the size of the items we’re sending through it to try to get more done. If the tests are failing and people keep committing changes, we’ve now got a big batch of changes which is more likely to include regression failures. It’s a death spiral. If our pipeline is reliable and fast, fewer changes go through it each time, reducing the risk of failures.

Collaborating to improve the pipeline

Making problems more visible is a great first step in making them smaller. Gather your team (or at least a representational group, if you have a large team) together. Take cards or paper and write the name of each step of your pipeline, one per card or paper. Put them on a table or on a wall to create a picture of your pipeline. (This can be done virtually if your team is distributed).

Visualizing a pipeline on a table – picture used with permission from Llewllyn Falco

Next, look at each step in the pipeline and ask “why do we have this step?” What are we learning from it? What business questions does it answer, what risks does it mitigate? Who on the team benefits from this information? You can capture this information on sticky notes to attach to each step.

Then, think about dependencies. For each step, what must in place for it to run successfully? A test environment? Test data? Specific configurations? Also think about how you can optimize the step. Is it currently done manually? Can you automate it?

Identify what triggers each step. Is it triggered automatically by the success of a previous step? Or manually? Can it be run in parallel with one or more other steps? Do you see opportunities to shorten feedback loops? Again, record outcomes of these discussion for each step.

Look at the length of your feedback loops. How much time elapses from code commit to completion of each step? What happens if the step fails – do the right people get notified? How? Is there a better way to communicate alerts? My previous team found that everyone was ignoring the Slack alerts generated by pipeline failures, so they hooked the build up to a police light. That got the team’s attention so that failures were quickly investigated and addressed!

Can any steps be done asynchronously? For example, it’s not realistic to do manual exploratory testing on every change when you are deploying to production frequently. The team can learn to use release feature toggles to hide changes from customers until the team has had time to complete any manual or time-consuming testing tasks.

Once you’ve gathered information about each step in the pipeline, choose a pain point to address and think of an experiment to reduce the pain. For example, perhaps your UI test suite has a lot of flaky tests and the team spends a lot of time kicking tests off again and diagnosing the failures that reoccur. Your experiment might be, “We believe that by refactoring the top 10 flaky tests to make them reliable will reduce our pipeline failure rate by 10% in the next month”. Use your retrospectives to evaluate your progress.

Remember the benefits

A fast, reliable pipeline means that you can confidently deploy frequent, small changes to production. And if you introduce problems to production that cause customer pain, you can shorten the time they feel pain by quickly deploying a fix with your fast, reliable pipeline. There are many other factors that go into safe, reliable and sustainable continuous delivery. Fast feedback and the ability to deploy changes quickly are a cornerstone of successful CD – and of a healthy quality culture.

References:

A Practical Guide to Testing in DevOps , Katrina Clokie, 2017
Continuous Delivery , Jez Humble and David Farley, 2010 (also see their website )
, Chapter 23, Lisa Crispin and Janet Gregory
“ Feature Toggles, aka Feature Flags “, Pete Hodgson

Lisa Crispin is a testing advocate with . With Janet Gregory , she has co-authored Agile Testing , More Agile Testing , and the upcoming Agile Testing Condensed . Lisa and Janet also have a 5 hour video course, Agile Testing Essentials . Through the Agile Testing Fellowship , they offer a three day live course, “Agile Testing for the Whole Team”.

Path to Production – Test Environments as a Product

By Rob Hornby

It could be easier, we could have teams for each application we run, that those teams treat their production and non-production application environments the same. That they have fully automated full stack continuous deployment pipelines and the software is so well designed that there is no dependency between each application owner. Teams deliver code by the hour to production and use advance deployment techniques such as blue-green deployment coupled with highly effective monitoring, alerting and automated rollback to identify and resolve issues in production within minutes.

We do have “agile” application teams working within a cloud environment, they have growing autonomy as they build independent features and function. That’s awesome. The problem is that we have not been working that way since we started building technology. We have a sprawling on-premise (we own two data centres) environment where Non-Production environments equates for 60%+ of the overall infrastructure. This is made up of dependent applications running together in an environment.

We have followed traditional programme structure over the last 10 years, with multi-year investments and waterfall based delivery. These have sought to reduce risk through separation rather than integration. This has resulted in multiple non-production environments. Our applications (100’s) have been built tightly coupled and need multiple other applications to work (Commerce systems through to legacy mainframe systems all meshed together), increasing the size of the basic/core environment. These programmes have not invested in automation, leaving a challenging sprawling estate to manage.

Path to Production (Variety of routes followed by teams ensuring lack of consistency in versioning in each environment)

This has created a number of problems. The estate is costly to maintain. There is hidden cost in each application change or infrastructure upgrade. Our people are linked to projects and programmes, when they finish, the applications in non-production fall in to disrepair, they are costly and time consuming to get working again when the next team comes along. A continued desire for separation results in projects hunting down that unused application/environment that has nobody else working in that space. The sprawling nature and lack of ownership results in a small band of “Environment Management” who work to bridge the communication and knowledge gaps across this sprawling estate. These people work for Projects and Programmes and through “best endeavours” get Delivery teams, Operations and Infrastructure teams to stitch the environment back together each time and retain the illusion these environments are good to test in. We effectively have 10 mini-lives beating away with no real Service Management to deal with them.

That long introduction was needed just to give me time to breathe and reflect. The organisation is changing for the better, which is great. We are re-organising and realising the mistakes of the past. We are now building most new systems in a cloud environment, are focused on Micro-service design principles and have teams that own and run the service (well almost). These are small individual successes which highlight the organisation is changing. Our online teams have gone from 10 deployments a year to over 1000 in just over 18 months. It should be a factor of 10 greater by next year. However there is a long way to go till we are fully working in this way, maybe we will never get there. This is where Environment as a Product comes in. How do we change our existing non-production environment landscape for the better.

We have started with two key initiatives. Initiative 1, Let’s get support to introduce an “enabling constraint” through removing the environments that least support delivery. They are the most difficult to maintain, are off the beaten path to production and the most costly to re-enable them. If the project or programme don’t have them they can’t use them. We know they are costly to build which is part of the reason they have been maintained. With the remaining environments we are going to create a paved road of sorts. One of the key things is to not construct the path to production based on your current organisational/delivery construct. We have been very waterfall in nature and have therefore not made change often to production.

This results in us having multiple hops to production for a single change based on the type of testing to be undertaken at each stage. Development (unit, system), Integration, Release (end to end), Performance (performance, operational), Pre-Production (incident and support) and Production. You introduce delay just through deploying to each of these environments and fixing the issues that come up as a result. That’s before you think of data, types of testing, handoff implications. So initially let’s get down to one of each type to support all teams, projects and programmes. Let’s draw it up so it looks like some are optional. Development, Integration, Pre-production, Production (even this is too many).

This was difficult to get going, convince people that we might be able to go faster through having less environments, that there might be more contention initially, however that might force us to look at the organisational shape and how we design software. Whether the full extent of the idea has sunk in, is yet to be seen, however the cost avoidance articulation of maintenance running in to millions of pounds a year was hard to ignore and gave us the initial green light.

Initiative 2, is to consolidate the team we have and build a paved road (common environments, tools and approach). We have taken everyone responsible for non-production “Environment-Release Management” and centralised them in to a single Product team. This wasn’t easy. You have to convince the leadership team that they want to pay for this as a separate service, that the cost is neutral, value adding even and that in doing so you can look at how the team works and make efficiency, as well as improving the path to production. What this doesn’t do is centralise the technical people responsible (not owning) for those environments. These remain fragmented across Operations, Infrastructure and Delivery teams. While the number of different technologies, teams and ways of working make this impractical at a small team level.

It does allow you to put a very clear Service Management approach in place for Non-Production and with leadership support allow you to be more robust in enforcing a better approach. You can start to work with individual teams to support and help them approach their delivery differently, focusing on automation, repeatable process and decoupling of change. The aim is that teams do this themselves and that the Environment-Release team can identify teams that need support. We are also backed by organisational , which have been established to give teams direction in how they should deliver.

Centralising the team has also allowed us to look at how we approach managing the environments. A single Product team with longer term goals, where the value is what we do to the environments, not the end customer outcome that the Project or Programme was focused on, has a very different priority and outlook. What was several ways of working can now be focused in to a single way of working. We have started by making sense of what people were doing daily. This has been done through several value stream mapping workshops but also to get everyone’s work up on a board and try to make sense of the work we do. Get a feel for reactive vs planned work. Upon doing this we can start to build a very clear backlog of priorities for our own ways of working, how we track issues, do we manage problems, how do teams onboard on to our path to production, can we pre-empt issues, what do we do with data, looks at our skills and most importantly clear communication.

Value is now a clear driver, as is ownership. We can now mandate changes to production are part of non-production thinking. We can also start to look at cost of delay and overall costs of supporting teams and drive our own value outcomes. Lead time is also a key measure, whether that that is to measure how quickly teams move through our environments, how long it takes us to respond to incident, downtime within environments etc… We can also help to start to measure teams frequency of deployment/release and to show organisational pace of change. Traditional service management is very much focused on controls but here we want to make it much more about value outcomes and enabling flow within teams.

We also have one foot firmly in the future. Over time the need for this team will be less. Greater ownership, more end to end owning teams, “DevOps” will all drive less need for a Environment-Release Product team. Our life span is based on the value we can show to generate. However we need to develop our skills in to other areas, whether that be Platform management within our Cloud platform or in to more “DevOps” skill sets which can work within teams to help them deliver. For now we are focused on our Product and value. It would be interesting to hear how other companies and people manage non-production environments, especially where there has been ongoing development of a variety of approaches for many years.

Rob Hornby is a Product Engineer working for John Lewis based in London. He has undertaken a number of roles, over the last 20 years, within Testing, Release, Platform and Environments.

Rob is a keen believer in being able to treat all things as Products and be able to evidence the value generated by that team. Ideally everything would be delivered by small teams, delivering continuously and iterating on value. However not all teams start at this point, not all teams are the same or born equal. This is where the challenge and enjoyment comes from.

Outside of work, Rob is a keen runner, father of an amazing young girl, Sofia and works on a number of local community events.

DevTestOps: It’s alive, but what can testers do to make it alive AND kicking into overdrive!

By Toyer Mamojee

The journey of the ‘typical tester’ has been a pretty bumpy ride over the past few years, more so recently with the massive uptake of DevOps. Apart from the many new and intimidating terms thrown our way, we are now required to do anything from digging in to business case details to monitoring how applications are performing in Production. It’s become a world which is very different to the one a tester would have experienced around the year 2005 or so.This world has incorporated testing to be at the heart of it, and not just testing but full on quality! So its only natural for the tester to play a massive role here.

But what can we do?

As mentioned, the tester should be at the very heart of every activity happening in the software delivery process. Below I highlight some of the areas where testers can directly influence and should influence if the process is to go from just being alive, to becoming Alive and Kicking!

Help Sculpt a Team Culture (Team player)

When one speaks about DevOps, technical aspects are usually at the front of the conversation whereas the softer aspects like Team culture and communication are very rarely mentioned.

Everyone loves quality products, testers love quality. A good healthy, dedicated team can help make better quality products. So naturally a tester should be one of the key individuals to drive a dedicated team. Testers should keep their critical thinking quality as their most prized asset by asking the right question at key moments during team discussions/meetings and interactions. This gets the team thinking and hopefully to a point where this starts to become second nature to all team members. Quality should be a driving force of every team member in everything they do, this should be a non-negotiable principle.

Another key area around team culture is by re-iterating the culture of ‘safe to fail’ and the ability to recover quickly. Without these important concepts inherent in a team the ability to deliver faster might prove to be a huge challenge.

Promote Visibility and Transparency (Seeing is believing)

I cannot stress how key this aspect is to a solid DevTestOps model. Gone are the days of working in isolation. A tester should look at some of the key artefacts that traditionally would fall within his/her space and how these can be made visible and transparent to the entire team.

Some of these include Test coverage visibility, defect statuses per sprint and release, test automation coverage, test automation scripts (functional and non-functional). One can make these visible in the form of dashboards or even repositories for test checklist or automation scripts.

At any point in time any team member should be able to give you the current status of your testing per sprint and/or release. After all if it is a difficult and an intensive process to decide on whether you are ready to deploy to production, then you have to question what you as the tester can do to change this.

Increase your skillset (Up your game)

Learn to use tools that help you work better — Learn to use tools that help you contribute more (Photo by on

A modern tester needs a modern armoury. Without it, I battle to see how the transition to the DevTestOps model can be easily made.There is so much to mention under this area, up-skilling should span these key areas:

Automation tools

From backend to front end focused automation tools, a tester should not only know these tools but know when and where to use them. Tools learned should include those which at least cover API testing, UI Testing, Performance/Load Testing, Security and Databases.

Context switching between these tools and types of testing is a rather intimidating thought for those not usually exposed to modern ways of software delivery. My advice here is to take baby steps in learning these tools in order to not just learn something for the sake of it, but rather for the cause of true DevTestOps.

Technical Tools

Technical type tools that equip a tester to contribute effectively in a DevOps environment would once again span the entire spectrum of the application solution. Tools like Postman & Swagger for API testing, knowing your way around version control applications (GitHub, GitLab etc), SQL and NoSql interfaces (like DBKoda, Sql Server etc), CI/CD apps (TravisCI, GoCD, Jenkins etc) , Logging applications (GrayLog, ElasticSearch,ElkStack etc) and Monitoring tools (NewRelic, Dynatrace, Instana etc).

Contribute to your CI/CD (its in the pipeline)

Testers have a crucial part to play in defining the ideal pipeline for your applications deployment. Being part of discussions around composition of the pipeline, ideal duration of pipelines, pre and post deployment tests as part of the pipeline.
A point to consider even further is where do your automation tests fit in these pipelines? Should it be part of it?
CI/CD tools also promote to the point on visibility and transparency, if constructed properly the entire team should know exactly at what stage the deployment is at, what tests were run and the overall status of the deployments.

Monitor it (What’s trending?)

To know what your application is doing both in PROD and Test environments, contributes to how you approach your testing. Findings and trends even contribute to what scenarios you end up testing and has massive input regarding determining what load you put your system under during Performance testing as an example.
Traditionally testers would steer clear of any tool or application that falls outside the scope of testing, but in a fast paced delivery world, monitoring tools can be a testers best friend. A tester’s system or application knowledge should also push the tester to not just use existing monitoring metrics but to also determine what to include as part of monitoring.

In summary

The evolution into DevTestOps has meant that the tester now plays a much more involved role in the entire delivery process. The day to day tasks of a tester spans between a balance of soft skills, analytical, technical and even an investigative mindset sometimes seamlessly crossing between these tasks without even knowing. By following some of the pointers above the initial ground work to successful DevTestOps is well underway. Go on Testers, get DevTestOps Alive and Kicking in full swing!

Toyer Mamoojee is a ‘die-hard’ passionate Agile Automation Test Analyst with almost 16 years testing experience across many industries. From Waterfall to Agile approaches, Commercial to Open Source Automation tools and Manual to Automation Testing he has been through the wars of Software Delivery. He is the co-founder of Cape Town Automation User group and co-organizer of Cape Town Testing Meetup back in his home country of South Africa. In his spare time he loves exploring the great outdoors, taking on the world playing online video games , major football fan, adopting the title of technological husband and playing super dad to 2 pre-schoolers!

Toyer Mamoojee @tottiLfc

A Tester’s Story: Adapting to the new world of pipelines

By , Guest Author

As a tester, how do you adapt when a new technology or process is introduced within your team? Do you panic? Do you accept it as a challenge? Do you research to find a solution along with the team? Do you reach out to people in the community to see how they approached it? Or do you just expect it to magically work for you without any effort on your part?

I was in similar situation when our team decided to adopt Continuous Integration (CI), as we were transitioning to a DevOps culture. I would like to share my experience of how, as a tester, it was to go through the journey.

It had been a while since we started to work in agile from waterfall. We took baby steps and evolved slowly. We were all happy, comfortable, and settled into the way we were approaching agile with our scrum process. Just when we were happy that we went through the agile transition from waterfall, now it was a new challenge of transitioning to DevOps. We had to embrace another new change of process, and it was the introduction to the world of pipelines – it was the CI/CD world!

As a tester you always like challenges, and this was challenging because I had to understand a new way of how testing fits in continuous integration, and embrace the new type of value it could provide while working on it.

As a tester I was very curious to know:

What it brings to my world of testing
How would it affect current test process
What do I have to change
What benefits do the pipelines get?
What do they look like?
How do they work?

And on and on. It was pretty confusing and scary at the beginning to visualize the change.

Why is our first instant emotion towards any change ‘Scary’? Why can’t it be an excited feeling or enthusiastic feeling for trying something new – I asked myself this question. Then I thought to myself, it’s as simple as this – When you don’t know about something, you are always scared about the change, but if you feel safe and see it as a new challenge, you will definitely get excited.

Then I started to explore and read more about pipelines. It wasn’t surprising that the more I was reading, the more I was confused. Then I decided to draw those pipelines and visualize them.

I sat with my colleagues and said: “Let’s please visualize the pipeline by drawing it on a big white board.” They agreed and we had a whole discussion around these pipelines. And we finally agreed on this and it looked something like this initially:

That’s where it started to make more sense to me, and I got a better understanding about what exactly this continuous integration is – but all in theory though. What I understood at this point was that it is simply nothing but a step by step process to release the code once there’s a commit. Continuous Integration is a practice for integrating the code several times to receive a quick feedback without waiting long. Whenever the developer finishes development task and pushes the changes to the repository, that’s where the pipeline stage starts. Because it has stages, it is called as a pipeline I thought to myself .

Now, a few more questions came to my mind:

What are those stages?
Who defines those stages?
Who defines what should go in the pipeline and when?
Will I have any place as a tester to suggest something?
The biggest question was – will there be no manual testing at all?

We as a team had discussions and came up with the our own version of the pipeline because there’s no definitive structure for a Continuous Integration (CI) pipeline. We decided we shall run static code analysis, unit tests and integration tests as our initial stages of our pipeline. This stage would create a unique artifact which could be deployed on a test environment to run additional tests. As my understanding got better about pipelines, I started to suggest that we add more valuable tests to the pipeline.

Initially we added smoke tests to the pipeline which would run alongside the existing tests. The reason why I suggested to add smoke tests as part of the pipeline was to make sure the build is testable before it is deployed on a test environment for further testing. I just called it as smoke test, but we could call it a Build Verification Test or Build Acceptance Test too. If the test didn’t pass, the build would be rejected and a new build would be released once the failed tests were fixed.

I could see each stage of tests running one after the other. It would stop going further if there was a failure at any of the tests or stages. Isn’t that magical!!!

It was just the beginning for us and especially for me. I continued to understand and observe more and more, and then I discussed with my team if we could add regression testing to the continuous integration pipeline. But initially we did not wanted to add any long running tests to the pipeline as we thought the whole point of having these pipelines was for quick feedback. So we decided to run the regression tests as nightly run task instead of running it with every commit. But we had the goal to run these regression tests with every commit once our pipeline was mature enough, when we would decide to run those tests in parallel. But that was a long term goal.

Each morning I could see the results of which tests where passed and which were failed and the difference was that I could add the build number while raising the issue, so it got easier for a developer to find the root cause. It’s not to say that it tells you where exactly where the issue is but they could easily check the recent code related to that build to see what went wrong. Instead of searching a needle in a haystack, it’s like you know which part of haystack you need to look to find the needle.

I got to spent most of my time as tester defining the test strategy, exploratory testing, being part of the 3 amigos to refine user stories and acceptance criteria, checking the test feedback from the pipeline, and adding more valuable tests to regression tests.

As we got along and had more clarity and saw the benefits for having these pipelines in place, we decided to discuss a long term goal, where I suggested to have visual testing as part of the pipeline as well, from a testing point of view.

This was my actual pipeline journey into the world of testing! It was very challenging and interesting!!

Thanks for reading my story!! Happy Testing!!!!