Shifting towards Omni Testing – Part 2

By Sérgio Freire

In Part 1 of this series, Sérgio introduced his idea of “Omni Testing” as a term that better expresses the concept of continually testing throughout the DevOps infinite loop of discovery and delivery. In this second and final part, he explains why he likes to use this term to describe the multitude of testing activities teams do to succeed with continuous delivery and deployment.

We all heard about Shift-Left Testing and Shift-Right Testing in recent years. Probably you may have got the idea that it was the thing, that single idea that was going to massively improve your testing. Due to this misconception, for example, Janet Gregory prefers to think about testing as something always happening and holistic (i.e. considering all parts of the whole and their interconnection).

Let’s be clear: you cannot shift something that is always there and happens all the time; however, what you’re focused on at a given moment in time can, and probably should, change. Within a team, testing happens continuously and in parallel, so different people may be performing quite different testing activities and from different angles.

If we have pure waterfall development in mind, testing happens as a phase after implementation. However, in real-life, testing is not constrained anymore. Of course, some teams perform more intensive testing before releasing; in that sense, people may refer to shift-left testing efforts as a way to clarify requirements altogether from the start and introduce more unit kind of tests. Similarly, with DevOps, you may “shift-right” testing efforts to production and use as a feedback loop to product development.

However, testing itself is not shifted-left nor right; testing efforts can shift though. Having “ Agile Testing Quadrants ” (by & ) and previous work from (i.e. testing matrix ), we can see that there are different types of tests, with different goals. As Lisa and Janet point in their book “ Agile Testing – A Practical Guide for Testers and Agile Teams “, risks are an inherent and important part of projects where testing is used as a mitigation strategy.

As you plan each epic, release, or iteration, work with your customer team to understand the business priorities and analyze risks. Use the quadrants to help identify all of the different types of testing that will be needed and when they should be performed.

Thus, the type and level of testing performed, where and how it is performed, are dynamic.

Testing is not a specific task assignable to one person and delimited in time; everyone is involved in testing, it happens in different ways, at the same time and continuously; and this leads us to Continuous Testing.

From Continuous Testing to Omni Testing

Continuous Testing has been around for some time; the “Continuous” wording is not new. Unfortunately, it is being pushed by some tool vendors and some people to sell it as a way to continuously run “automated tests”.

Well, first there aren’t automated tests in a strict sense; you have automated checks but testing frameworks having been calling them as such from the start. Second, “continuous” in testing is not related to automated scripts/checks being continuously run or not. Even if it was, it wouldn’t make sense, as in CI “automated tests” are triggered on code commits, pull-requests, merges or due to some predefined schedule.

So what does Continuous Testing mean? I advise you to start by this post from Lisa Crispin and complement it with this course from .

We know that automation plays a key role in DevOps and also, though to a minor scale, on Continuous Integration (CI) practices which are closely related to Continuous Testing.

The CI concept was depicted by and largely widespread with Extreme Programming as one of the 12 core XP practices (, , , et al).

During Continuous Integration, “automated tests” (i.e. automated checks to be more precise: code that runs and performs checks to validate expected behavior) are run. These will help assure, to a point, that code that has been integrated isn’t breaking anything and previous expectations are still met.

As teams grow and code is committed more often, CI provides the means for immediate feedback so the team knows right away when a possible “bug has been added”. It can quickly be fixed before anyone builds on top of it. These automated scripts perform unit and integration tests but can also cover other types of tests such as security tests and more.

As builds are continuously being started by the CI process, these kinds of tests are also constantly being run.

However, “Continuous Testing” is not limited to these “tests”. Lisa Crispin and Janet Gregory put it like this instead:

Collaborative testing practices that occur continuously, from inception to delivery and beyond, supporting frequent delivery of value for our customers. Testing activities focus on building quality into the product, using fast feedback loops to validate our understanding. The practices strengthen and support the idea of whole team responsibility for quality.

In my opinion, this is the real reason behind the “Continuous” wording before “Testing”.

Atom Symbol representing Omni Testing with a brain icon, bug icon, other testing tools

Well, at first sight, “Continuous ” makes sense whenever everything else is using the “Continuous” prefix. But you don’t hear about “Continuous Coding”, do you?

Thus, a better adjective should be used instead: I think Omni (Testing) may describe Testing in a more complete way. Do you agree?

Continuously evolve our way of understanding the world around us

We moved from testing in waterfall, to testing in Agile , to testing in a DevOps context. Testing, similarly to other processes, evolved along the way; its essence hasn’t.

We still use testing for the same reasons, because we aim to understand what we test and use that information to make decisions.

Whenever we go to a shop and buy something, whenever we download an app and try it out, whenever we meet or go out with someone: we’re trying to somehow understand what’s in front of us. We’re information seekers because ultimately we want to take actions leveraged on data, i.e. informed decisions.

In software development, which may also include hardware development, by the way, testing is done as a means to ensure that the deliverables produced on our products/solutions iterations have a certain level of quality. And this includes evaluating not only expected behavior but also that “customer” needs are covered in the best way, thus we look at certain quality attributes.

Testing is centered around humans and it requires them. However, machines, tools, algorithms, and AI can leverage the unique power we have within us to provide better testing. That’s around 3.5 billion years of continuous evolution and accumulated learning about life on Earth.

I know what you may be thinking: maybe there is no Agile Life, no Continuous Life, and we also don’t call it Omni Life right? Well, true; similarly to what happened with life, testing evolved, it’s widespread, it lives from the outskirts of software development. But, life is restricted in many ways: to certain conditions, to our planet (as far as we know).

Testing, on the other hand, is a broader concept; no limits attached… and being Omni is part of its real essence.

Sérgio Freire’s background is in electronics and Telecommunications. Throughout the years, he has worked with teams in developing real-time software, web and mobile applications and complex hardware systems. He has worked and interacted with teams in waterfall and Agile contexts. As Head of Solution Architecture for Xray, he works closely with many different teams worldwide to help them achieve high-quality, testable products. Follow Sérgio on and .

Shifting Towards Omni Testing – Part 1

By Sérgio Freire

Testing, as we foreseek, is something not limited to time, not limited to space, not limited to a specific team member, not limited to a given type, technique or approach; it is a process we use to understand something, to obtain more information about it so that we can take actions on it.

Testing is not just about continuously “running” (checks); testing is always there, it is part of the whole (e.g. product development), it encompasses different kinds of activities and involves everyone.

That’s why Omni Testing can, perhaps, be a better way of describing Testing as we aim it to be.

Testing is universal. It’s one thing every single human has performed in their life.

For many of us working in software or hardware product development, testing is even more “natural.” However, we sometimes misinterpret it or see it just from a narrow perspective.

Lately, we have been hearing about Continuous Testing, Agile Testing, and DevTestOps/DevTestSecOps. Before that, Shift-Left and Shift-Right Testing and “Test Automation” were also around.

This tells us that Testing is:

broad
important
always present
intermingled with what we do
used interchangeably, in different contexts

Because of all of this, I think that Testing is a process unbound, in time, location, people and context; yet, it always has a purpose – actionable information. No matter if you’re using it to verify if you retained knowledge after attending a course, to evaluate if you like some food or a perfume, or to understand how a product works and if it meets some customer needs. Even before “software was invented, testing was already part of our lives and helping us shape our decisions.

Omni Testing

Why Omni? Is it a “new thing” or a “new kind” of testing?

What types of tests and approaches does it embrace? Who is involved in it?

“Omni” comes from the Latin word omnis and means “all,” “the whole of, “all things,” “everything.”

More and more, teams everywhere are aiming to have a holistic approach to testing. We’re aiming to have on-going conversations, on-going experiments and investigations, on-going checks, on-going information, and on-going feedback. The whole team is involved in testing and has quality in mind, from the start. For that, the team starts by assessing all risks, performs all sorts of tests, using all sorts of techniques and approaches.

Mind map of omni testing showing people, machines, multifaceted, active, embracing modern approaches — Omni Testing mind map

(Software) Testing is multi-faceted; it can be used in so many different ways, although in Omni Testing we try to perform it in a way where:

it produces relevant information ASAP (i.e. accordingly with the agreed test strategy and the overall agreed meaning of quality)
it focuses on what matters first
it is efficient and quickly adapts

Thus, it is not dependent/limited to:

specific roles
specific assets
specific test levels, types, techniques and approaches
specific methodologies
specific phases
specific tools

Defining Testing, again

Defining “testing” although may seem simple at first sight, it isn’t. Why? Because it is a broad concept and sometimes used incorrectly or in a very restricted sense.

Recently, I had an opportunity of attending a workshop on exploratory testing with at .

To the question “why do we perform testing?” the group came to these results.

Spend time/resourcesUnderstanding the systemUncover gapsProvide valuable info to team membersGive everyone feedbackFinding risksMake the end user happyTranslate information to othersReady to ship?

Probably, others would come to different conclusions. For some, testing is just about checking and “finding bugs.” However, it’s much more than that.

I do agree with that pictures such as the following one won’t help, as they depict “test” as a phase (which besides not being true is also not very agile).

DevOps infinite loop of plan, code, build, test, release, dep;oy, operate, monitor, plan...

There’s an in-depth from , where he goes over the definition from Cem Kaner ( The Ongoing Revolution in Software Testing, 2007 ):

Testing is an empirical, technical investigation conducted to provide quality-related information about a software product to a stakeholder.

In testing, as Alan Richardson correctly points, we build models and we compare them with reality. has a short, yet clear video on which may be useful to you in this context.

When focusing on software testing, you’ll probably find some terms: “Traditional”, Waterfall, Agile, Continuous and Modern Testing (you may want to look at and “ Modern Testing Principles “; there’s a course on MoT).

Let’s look at this from James Bach & Michael Bolton:

“Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes: questioning, study, modeling, observation and inference, output checking, etc.”

I would rephrase this slightly, adding my perspective.

Testing is a targeted process of challenging something (e.g. a product) and our understanding about it , using a set of activities leveraged by specific human skills.

Where,

“targeted process” – a set of activities, having a common goal (i.e. quality attributes in mind), that will produce a result
“challenging” – includes trial/experimentation, checking, …
“activities” – build models, create test ideas, execute actions in the SUT, investigation, ask questions, provide feedback, …
“human skills” – creativity, curiosity, perseverance, analytical thinking, communication, courage, continuous learning mindset, …

Testing is by nature; we try things and test ideas, we explore to seek understanding, to obtain information as states in his adapted model of James Bach’s . We also use other approaches, such as script-based tests (i.e. checks) which sometimes are automated to verify “compliance” with predefined rules. We use all the outputs we collect as data that we process. This data becomes information, which is then used to improve our learning of the system/product and provide feedback to “stakeholders” (e.g. business, product team).

Whenever testing something we have in mind certain quality attributes (e.g. correctness, usability, scalability) – “requirements” we demand or expect from it.

To perform it, we test at different levels and use a bunch of different techniques that help us shape what we call “tests”.

We test everything, software, hardware, specifications, established ideas, documents, and specifications.

Thus, testing is a complete, omni process that uses best-of-breed human skills to answer tailored questions to clarify our understanding about something and that can help shape it before, during and after it is built.

That being said, how does it relate with Shift-Left/Shift-Right and Continuous Testing? We’ll dive further into this topic on the next blog post of this series related to Omni Testing.

Check next week for Part 2 of this series about Omni Testing!

Running API Smoke Tests in Your Continuous Deployment Pipeline

By Kristin Jackvony

Each year, more software companies switch to a microservices architecture using Application Programming Interfaces (APIs), because APIs make it easy for different teams on a project to access each other’s resources. APIs also enable software companies or individuals to get data from each other: for example, Twitter has an API that is used by many small businesses to retrieve tweets related to their business for display on the business’s webpage.

Another great feature of APIs is that they are smaller than a monolithic application, which means that they can be deployed frequently. At the company where I work, my team has eight different APIs, and we have eleven different environments that we deploy to. If we were relying solely on manual testing to verify that the APIs had been deployed correctly, we’d have to run eleven sets of tests for each API release. And if we were deploying more than one API at a time, which we often do because our APIs work together, we could be running 88 sets of tests. And if we deploy every two weeks, those 88 sets of tests will amount to a lot of tedious, repetitive testing!

We’ve solved this problem by setting up automated API smoke tests that run with every deployment, to every environment. In this article, I’ll describe how we decided what to test and how we set up our smoke tests. We used Postman, Newman , Powershell , and Octopus to set up our automated smoke tests, so I’ll be describing what we did with those tools. However, these strategies could be adopted to any continuous deployment (CD) pipeline.

Step One: Decide What to Test

The main purpose of smoke tests is to do some simple, high-level tests to ensure that the deployment was successful. This is not the place to test every single thing that your API can do. When we chose what to put in our smoke tests, here is what we considered:

We set up one “Happy Path” test for each endpoint- that is, one test that is expected to be successful. For example, a GET request for a resource with a specific ID would be expected to return that resource. You’ll want to test each endpoint once to verify that each one is working as expected.
If there were major variations in how a specific endpoint would be used, we added one or more tests to check those variations. For instance, we have a file retrieval API where a file can be retrieved by two different methods. The endpoint looks the same, but the bodies of the requests are different. So we added one test for each method.
For endpoints which required a certain level of security, we added a test to validate that a request without the appropriate authentication would NOT return information, and would instead return a 400-level error.
We did not add any other negative tests, such as a POST request with data outside the allowed parameters, because we didn’t feel that those were necessary to ensure that the API was working. We instead ran those tests as part of a nightly regression suite.

Step Two: Export Your Tests

We used Postman to create our API tests, so when the tests were created, we exported both the test collection and the test environment file into JSON files. For those who are not familiar with Postman, a test environment is a group of variables that can be referenced in a collection of tests.

Step Three: Write a Script to Run Your Tests

Postman has a command-line tool called Newman that can be used to run a test collection, so this is what we used to run our tests. We created a Powershell script that would run the Newman command.

Step Four: Create a Deployment Step to Call Your Script

Once the Powershell script was ready, we set up a deployment step in each Octopus API deployment project that would run the test script. If the script failed, the deployment would fail.

Step Five: Organize Your Variables

Variables are often a consideration when running smoke tests in different environments. For example, in your QA environment, your test user might have an ID of 340, but in your Staging environment, your test user might have an idea of 620. Another issue with variables is that sometimes for security purposes, you might not have access to passwords or keys in Production environments. This was the case for our team, but fortunately Octopus has the values we need to run our tests in Production, so we could solve the problem by having Octopus pass the values that we needed to our test script.

There were three different kinds of variables needed for our smoke test:

Type 1: Variables that were unchanging for each test environment, such as “firstName”: “Prunella”. These variables could be put directly into the Postman environment, so nothing more needed to be done with them.

Type 2: Variables that changed for each test environment, but did not need to be kept secure, such as “userId”: 340. These variables were added as Octopus variables in this fashion: “smoke.userId”, and the value of the variable was set for each environment; for example, QA was set to 340, Staging was set to 620, and Production was set to 450.

We then used the variables like this:v

When we called the Powershell script in Octopus, we sent the variables set in Octopus to the script.
In the Powershell script, we accepted the Octopus variables sent in and assigned them to Powershell variables.
When we used the Newman command in the Powershell script, we sent the variables in the script to the Postman environment.
Newman used the variables sent in the Powershell script, combined with the variables in the Postman environment to run the Postman collection.

With our API smoke tests running in Octopus for every API and in every environment, we can feel confident that any big problems with an API deployment will be detected right away. Furthermore, we can test in Production environments even when we don’t have access to sensitive API keys. Having automatic deployment smoke tests frees us up to do other types of testing, such as manual exploratory testing and security testing, as well as write more test automation for nightly regressions.

About Kristin:

Kristin Jackvony discovered her passion for software testing after working as a music educator for nearly two decades. She has been a QA engineer, manager, and lead for the last ten years and is currently working as a QA Lead at Paylocity. Find me on and .

5 Handy Tips for Testers in DevOps

By Bhavani R

I am going to share five handy tips for testers in today’s DevOps world from my experience in this post.

Learn Continuous Testing

The first and foremost thing to learn about is: what is continuous testing ? How do DevOps processes impact the tester’s approach for transitioning from traditional waterfall to agile testing? Before starting with continuous testing, for each sprint, we were testing incrementally and then did the regression testing before deployed to production. Up to now, our sprint cycles for some projects have been weekly or bi-weekly. But this cycle becomes too long to roll out a feature or customer request in a timely way. So, we have adopted DevOps in our process. Testing over the whole lifecycle by shifting left and shift right, i.e. testing continuously from beginning to end.

Learn about DevOps jargon.

I recommend taking a look at Lisa Crispin’s course in Test Automation U to learn basic concepts and terminology around continuous delivery and DevOps. That will help you communicate and collaborate with everyone involved in delivering your product, including operations and infrastructure specialists.

A Culture Shift from an Independent Team to Whole Team Approach

In a waterfall methodology, testing is confined to an independent verification and validation team. In agile development, it becomes a collaborative effort.

People come with baggage: preconceived notions, past experiences, prejudices. The primary responsibility is educating the team about DevOps culture.

In DevOps culture, the focus is on the product, not building fiefdoms and amassing political power. The DevOps slogan is, “Everyone involved with delivering software is responsible for delivering high-quality software.” In short, the delivery becomes a shared responsibility.

Build A Successful Test Strategy

Building the right test strategy is critical in continuous testing. When we have 500 automated tests, we can’t execute all the 500 tests for each change committed in the pipeline. It will take too much execution time. So, we have analyzed and prioritized the test cases by their core functionality as well as using input from the customer support and business teams. In the end, we have come up with 100 automated regression tests which are enough to give us confidence that the latest change hasn’t broken existing features.

Don’t Just Make The Tests Pass, Focus on Efficiency

When we started with DevOps, in our continuous pipeline, our automation tests failed in between or at the last step of the pipeline. We analyzed the test results and the error logs. Most of the errors were due to timeouts or because of the new feature or enhancement changes. We have updated our scripts periodically to take care of this. After deployment, when we cross verified the product/project using manual exploratory testing, we saw a few more failures. Then we did a brainstorming session with our entire product team to make our automated tests more efficient. For this, we have created/updated the automation scripts in the preproduction phase to save time and resources.

Choose Your Tests Wisely

This is related to the points discussed in the previous paragraph. When we started doing the deployment through the DevOps practices, as testers, we need to choose our tests wisely to add those to the pipeline. Along with the regression test suites, 24/7 monitoring should be added. This helps us to understand how the product is to be used by the customers. For choosing the tests, we use input from the user behaviors that we learn from our monitoring tools, as well as discussions with the business stakeholders.

I hope the information from this experience helps you to enhance your testing!

References:

“Building a DevOps Culture ” – Mandi Walls

?” Lisa Crispin

Test Automation in DevOps , Test Automation U, Lisa Crispin

About the Author:

Bhavani R

I hold a Certified Software Test Manager and Six Sigma Green Belt. I have a passion for testing and learning, as well as implementing and mentoring the latest automation tools. I spent the last 16+ years working in IT, with a focus on software quality assurance and software testing. I am Testing Guild 2018 conference speaker and a frequent blogger. I manage a fast-growing, dynamic team bringing you https://www.qatouch.com . Twitter: . Linkedin: . Blog : www.bhavaniraj.com