Shifting Towards Omni Testing – Part 1

By Sérgio Freire

Testing, as we foreseek, is something not limited to time, not limited to space, not limited to a specific team member, not limited to a given type, technique or approach; it is a process we use to understand something, to obtain more information about it so that we can take actions on it.

Testing is not just about continuously “running” (checks); testing is always there, it is part of the whole (e.g. product development), it encompasses different kinds of activities and involves everyone.

That’s why Omni Testing can, perhaps, be a better way of describing Testing as we aim it to be.

Testing is universal. It’s one thing every single human has performed in their life.

For many of us working in software or hardware product development, testing is even more “natural.” However, we sometimes misinterpret it or see it just from a narrow perspective.

Lately, we have been hearing about Continuous Testing, Agile Testing, and DevTestOps/DevTestSecOps. Before that, Shift-Left and Shift-Right Testing and “Test Automation” were also around.

This tells us that Testing is:

broad
important
always present
intermingled with what we do
used interchangeably, in different contexts

Because of all of this, I think that Testing is a process unbound, in time, location, people and context; yet, it always has a purpose – actionable information. No matter if you’re using it to verify if you retained knowledge after attending a course, to evaluate if you like some food or a perfume, or to understand how a product works and if it meets some customer needs. Even before “software was invented, testing was already part of our lives and helping us shape our decisions.

Omni Testing

Why Omni? Is it a “new thing” or a “new kind” of testing?

What types of tests and approaches does it embrace? Who is involved in it?

“Omni” comes from the Latin word omnis and means “all,” “the whole of, “all things,” “everything.”

More and more, teams everywhere are aiming to have a holistic approach to testing. We’re aiming to have on-going conversations, on-going experiments and investigations, on-going checks, on-going information, and on-going feedback. The whole team is involved in testing and has quality in mind, from the start. For that, the team starts by assessing all risks, performs all sorts of tests, using all sorts of techniques and approaches.

Mind map of omni testing showing people, machines, multifaceted, active, embracing modern approaches — Omni Testing mind map

(Software) Testing is multi-faceted; it can be used in so many different ways, although in Omni Testing we try to perform it in a way where:

it produces relevant information ASAP (i.e. accordingly with the agreed test strategy and the overall agreed meaning of quality)
it focuses on what matters first
it is efficient and quickly adapts

Thus, it is not dependent/limited to:

specific roles
specific assets
specific test levels, types, techniques and approaches
specific methodologies
specific phases
specific tools

Defining Testing, again

Defining “testing” although may seem simple at first sight, it isn’t. Why? Because it is a broad concept and sometimes used incorrectly or in a very restricted sense.

Recently, I had an opportunity of attending a workshop on exploratory testing with at .

To the question “why do we perform testing?” the group came to these results.

Spend time/resourcesUnderstanding the systemUncover gapsProvide valuable info to team membersGive everyone feedbackFinding risksMake the end user happyTranslate information to othersReady to ship?

Probably, others would come to different conclusions. For some, testing is just about checking and “finding bugs.” However, it’s much more than that.

I do agree with that pictures such as the following one won’t help, as they depict “test” as a phase (which besides not being true is also not very agile).

DevOps infinite loop of plan, code, build, test, release, dep;oy, operate, monitor, plan...

There’s an in-depth from , where he goes over the definition from Cem Kaner ( The Ongoing Revolution in Software Testing, 2007 ):

Testing is an empirical, technical investigation conducted to provide quality-related information about a software product to a stakeholder.

In testing, as Alan Richardson correctly points, we build models and we compare them with reality. has a short, yet clear video on which may be useful to you in this context.

When focusing on software testing, you’ll probably find some terms: “Traditional”, Waterfall, Agile, Continuous and Modern Testing (you may want to look at and “ Modern Testing Principles “; there’s a course on MoT).

Let’s look at this from James Bach & Michael Bolton:

“Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes: questioning, study, modeling, observation and inference, output checking, etc.”

I would rephrase this slightly, adding my perspective.

Testing is a targeted process of challenging something (e.g. a product) and our understanding about it , using a set of activities leveraged by specific human skills.

Where,

“targeted process” – a set of activities, having a common goal (i.e. quality attributes in mind), that will produce a result
“challenging” – includes trial/experimentation, checking, …
“activities” – build models, create test ideas, execute actions in the SUT, investigation, ask questions, provide feedback, …
“human skills” – creativity, curiosity, perseverance, analytical thinking, communication, courage, continuous learning mindset, …

Testing is by nature; we try things and test ideas, we explore to seek understanding, to obtain information as states in his adapted model of James Bach’s . We also use other approaches, such as script-based tests (i.e. checks) which sometimes are automated to verify “compliance” with predefined rules. We use all the outputs we collect as data that we process. This data becomes information, which is then used to improve our learning of the system/product and provide feedback to “stakeholders” (e.g. business, product team).

Whenever testing something we have in mind certain quality attributes (e.g. correctness, usability, scalability) – “requirements” we demand or expect from it.

To perform it, we test at different levels and use a bunch of different techniques that help us shape what we call “tests”.

We test everything, software, hardware, specifications, established ideas, documents, and specifications.

Thus, testing is a complete, omni process that uses best-of-breed human skills to answer tailored questions to clarify our understanding about something and that can help shape it before, during and after it is built.

That being said, how does it relate with Shift-Left/Shift-Right and Continuous Testing? We’ll dive further into this topic on the next blog post of this series related to Omni Testing.

Check next week for Part 2 of this series about Omni Testing!

Sérgio Freire’s background is in electronics and Telecommunications. Throughout the years, he has worked with teams in developing real-time software, web and mobile applications and complex hardware systems. He has worked and interacted with teams in waterfall and Agile contexts. As Head of Solution Architecture for Xray, he works closely with many different teams worldwide to help them achieve high-quality, testable products. Follow Sérgio on and .

Continuous Testing in a seasonal business

For many countries around the world, the heaviest retail season is already starting. In the U.S., Black Friday, the heaviest day for brick-and-mortar retail sales, and Cyber Monday, when online retailers get the most traffic, are almost upon us. The delivery teams responsible for the software that powers those retail engines must be busy.

Or… are they on a steady cadence, practicing continuous delivery, and not feeling stressed? Teams that continually release small changes to production should be able to do so in a sustainable way. Hmm.

Most business domains revolve around some kind of seasonal pressure. Some industries have a big trade fair, where everyone shows off their new product. All development is aimed at that big event. Tax preparation products will be used more heavily when people have to file their returns. Government regulations may change annually, requiring software updates to remain compliant.

How do teams achieve continuous delivery, including testing at all parts of the DevOps continuous loop, with seasonal spikes in demand? Please join me along with , , , and as we share our own experiences with these challenges in a webinar panel hosted by , Thursday, 14 November. We hope you can join us live and bring your questions. Please to attend or to get the link to the recording after the event.

Running API Smoke Tests in Your Continuous Deployment Pipeline

By Kristin Jackvony

Each year, more software companies switch to a microservices architecture using Application Programming Interfaces (APIs), because APIs make it easy for different teams on a project to access each other’s resources. APIs also enable software companies or individuals to get data from each other: for example, Twitter has an API that is used by many small businesses to retrieve tweets related to their business for display on the business’s webpage.

Another great feature of APIs is that they are smaller than a monolithic application, which means that they can be deployed frequently. At the company where I work, my team has eight different APIs, and we have eleven different environments that we deploy to. If we were relying solely on manual testing to verify that the APIs had been deployed correctly, we’d have to run eleven sets of tests for each API release. And if we were deploying more than one API at a time, which we often do because our APIs work together, we could be running 88 sets of tests. And if we deploy every two weeks, those 88 sets of tests will amount to a lot of tedious, repetitive testing!

We’ve solved this problem by setting up automated API smoke tests that run with every deployment, to every environment. In this article, I’ll describe how we decided what to test and how we set up our smoke tests. We used Postman, Newman , Powershell , and Octopus to set up our automated smoke tests, so I’ll be describing what we did with those tools. However, these strategies could be adopted to any continuous deployment (CD) pipeline.

Step One: Decide What to Test

The main purpose of smoke tests is to do some simple, high-level tests to ensure that the deployment was successful. This is not the place to test every single thing that your API can do. When we chose what to put in our smoke tests, here is what we considered:

We set up one “Happy Path” test for each endpoint- that is, one test that is expected to be successful. For example, a GET request for a resource with a specific ID would be expected to return that resource. You’ll want to test each endpoint once to verify that each one is working as expected.
If there were major variations in how a specific endpoint would be used, we added one or more tests to check those variations. For instance, we have a file retrieval API where a file can be retrieved by two different methods. The endpoint looks the same, but the bodies of the requests are different. So we added one test for each method.
For endpoints which required a certain level of security, we added a test to validate that a request without the appropriate authentication would NOT return information, and would instead return a 400-level error.
We did not add any other negative tests, such as a POST request with data outside the allowed parameters, because we didn’t feel that those were necessary to ensure that the API was working. We instead ran those tests as part of a nightly regression suite.

Step Two: Export Your Tests

We used Postman to create our API tests, so when the tests were created, we exported both the test collection and the test environment file into JSON files. For those who are not familiar with Postman, a test environment is a group of variables that can be referenced in a collection of tests.

Step Three: Write a Script to Run Your Tests

Postman has a command-line tool called Newman that can be used to run a test collection, so this is what we used to run our tests. We created a Powershell script that would run the Newman command.

Step Four: Create a Deployment Step to Call Your Script

Once the Powershell script was ready, we set up a deployment step in each Octopus API deployment project that would run the test script. If the script failed, the deployment would fail.

Step Five: Organize Your Variables

Variables are often a consideration when running smoke tests in different environments. For example, in your QA environment, your test user might have an ID of 340, but in your Staging environment, your test user might have an idea of 620. Another issue with variables is that sometimes for security purposes, you might not have access to passwords or keys in Production environments. This was the case for our team, but fortunately Octopus has the values we need to run our tests in Production, so we could solve the problem by having Octopus pass the values that we needed to our test script.

There were three different kinds of variables needed for our smoke test:

Type 1: Variables that were unchanging for each test environment, such as “firstName”: “Prunella”. These variables could be put directly into the Postman environment, so nothing more needed to be done with them.

Type 2: Variables that changed for each test environment, but did not need to be kept secure, such as “userId”: 340. These variables were added as Octopus variables in this fashion: “smoke.userId”, and the value of the variable was set for each environment; for example, QA was set to 340, Staging was set to 620, and Production was set to 450.

We then used the variables like this:v

When we called the Powershell script in Octopus, we sent the variables set in Octopus to the script.
In the Powershell script, we accepted the Octopus variables sent in and assigned them to Powershell variables.
When we used the Newman command in the Powershell script, we sent the variables in the script to the Postman environment.
Newman used the variables sent in the Powershell script, combined with the variables in the Postman environment to run the Postman collection.

With our API smoke tests running in Octopus for every API and in every environment, we can feel confident that any big problems with an API deployment will be detected right away. Furthermore, we can test in Production environments even when we don’t have access to sensitive API keys. Having automatic deployment smoke tests frees us up to do other types of testing, such as manual exploratory testing and security testing, as well as write more test automation for nightly regressions.

About Kristin:

Kristin Jackvony discovered her passion for software testing after working as a music educator for nearly two decades. She has been a QA engineer, manager, and lead for the last ten years and is currently working as a QA Lead at Paylocity. Find me on and .

DevOps, DevTestOps, Dev*Ops… what’s in a label?

by Lisa Crispin

The catalyst for this community website was a “DevTestOps Manifesto” dreamed up originally by a teammate and myself, then with input from the testing and DevOps community. After lots more discussions with practitioners, we realized that what we had were some guiding principles rather than a “manifesto”.

We also had a lot of discussions about the “DevTestOps” term. I liked it because when a lot of people see the word “DevOps”, they think of programmers/coders plus operations specialists, and don’t include testers or other specialists. Having “Test” in the label could help make testing and testers more visible. After all, testing is the heart of DevOps and continuous delivery. Others pointed out to me that using the term “DevTestOps” could also be construed as promoting silos again, which is the opposite of what agile development and DevOps are all about.

Who’s included in “Dev”?

Everyone who helps to create, deliver and maintain a software product is a developer, and that includes testers. The term DevOps came about 10 years ago because a lot of organizations said they had adopted “agile”, but they were still throwing code over the wall to operations teams to deploy it, nurture it in production, and carry the pager. After “DevOps” was established, we started hearing about “DevSecOps”, which has become an established term and specialty. I’ve seen other labels merged with Dev and Ops, like DevAIOps and DevUXOps.

The Whole Team Approach to building quality into software has to include operations folks, along with product people, designers, business stakeholders, sales and marketing, business analysts, database experts, data scientists… we are all in this together. It may be useful to insert another label between Dev and Ops, to highlight an activity which some people may not consider in terms of the practices needed to succeed with continuous integration and delivery, but “DevOps” is a fine, all-inclusive term.

Labels and goals

Personally, I wish we did not have to use any kind of labels like “agile” and “DevOps”. Our aim is to continually improve how we build software so that we can delight our customers while enjoying the process ourselves. But it seems that we humans love to label things.

Our goal here at TestingInDevOps.org is to get testing and QA professionals interested in learning about DevOps and continuous delivery, and provide resources to help them learn. Naturally, we welcome people in any specialty or role to learn how testing activities fit into and add value to DevOps culture and practices. We especially want to make testers visible in this DevOps and CD world. People with deep testing skills and experience help everyone on their teams learn testing skills and find ways to bake quality into their product during the whole continuous loop of discovering what customers want, building delivering those features.

Let’s find more great ways to deliver small, valuable changes to production frequently, at a sustainable pace that lets our teams enjoy the journey!

State of DevTestOps survey

by Lisa Crispin

Yes, the State of DevOps survey is the gold standard for truly scientific and valuable surveys, and you should both fill it out every year and read the reports and the book Accelerate ! And, there are lots of testing-related surveys out there. They all provide interesting information that help us think about our profession and where we need to grow. Why yet another testing-related and DevOps-related survey? Because we have so much more to learn!

It’s the 2nd year of the survey. I was happy to help with it again, because last year’s survey provided some valuable and thought-provoking . It generated some interesting conversations and led to productive panel discussions for our Testing in DevOps community here as well as other webinars (see the webinars and videos page here for links to those).

Testers need to learn about DevOps and continuous delivery, and engage with people in other roles in their organizations to help their own teams shorten cycle times and improve their ability to build quality in. DevOps needs testers – testing is the heart of DevOps . Getting some insights as to how teams are progressing in DevOps practices and culture, as well as how they have incorporated testing activities and testers, is vital so that we can find more ways to add value as testers/testing coaches/testing consultants.

Whether your team has started to embrace DevOps and continuous delivery or not, please take eight minutes (or less!) to fill out the , and share your team’s experiences with what many people now refer to as shifting testing to the right – testing activities during the release process and in production. You don’t have to give your email or any other personal information.

Keeping Up with Compliance in a DevOps World

By Areti Panou

About 5years ago I was working in a small product that was doing (almost) everythingthe DevOps way. We had the works: code and tests versioned in git, trunk baseddevelopment, building the code once (more or less), feature toggles, dailydeployments, the ability to fix issues fast, monitoring and alerting of ourproduction system.

But therewas one tiny problem.

We stillneeded to adhere to the company’s Quality Management System (QMS) whichincluded compliance to ISO 9001 certification as well as other regulatoryrequirements. While development was adapting to the new world in a fast pace,the same could not be said for the company’s processes at that time.

So hereare some things we did to address the situation.

Identify the “must haves”

Yes, the QMS consisted of a huge list of requirements but our common sense told us that they didn’t all have the same weight and importance. Minor performance issues didn’t seem as serious as upholding security and licensing requirements that could prompt legal ramifications. With the help of our company’s QMS owners, we understood the severity of each requirement and created 2 lists: One with all the requirements that had to be fulfilled with every deployment and one with the ones that had to be fulfilled for the release of new functionality. For example we were scanning for any OWASP Top Ten vulnerabilities before each deployment but running and documenting performance tests on a new feature only before its release to customers. Respecting the requirements on these 2 lists meant that our product and the company were safe from any wrongdoing.

Customise to your product’s needs

Once weknew what was important for the company, we needed to implement therequirements’ checks in a manner, that they indeed improved the quality of ourproduct. Instead of blindly using a recommended tool, we checked if there werealso other options that would better support our technology stack. We even cameup with our own tools , as wedid for example for complying with the traceability requirements. When weneeded to only take care of priority 1 issues for breaking a certainrequirement, sometimes we would decide to not allow priority 2 issues getthrough either. We evaluated which of the non-essential requirements would be beneficialfor us and added their checking to our to-do list as well. Our goal was to takecare of the regulations, not because the company imposed it on us, but insteadbecause we saw true value in them.

Add requirements’ checks in the pipeline

What naturallycame next, was to automate as many of the requirements’ checks as possible andinclude them in our deployment pipeline. The priority was to have the onesnecessary for deployment running right away once something was pushed by thedevelopers to the main trunk, while requirements’ checks needed for releasing newfunctionality run daily. Our aim was to get feedback on the status of corporaterequirements fast, since any violation could either block us from deliveringsoftware or take a great amount of time to resolve as QMS representativesneeded to get involved. Feeling confident about compliance gave us more time tofocus on additional activities that would further improve the quality of ourproduct.

Share with the company

The lastaction we took was to keep an open channel of communication with the QMSowners. We showed them our implementation of requirements’ checks and got theirfeedback whether they were enough to ensure compliance. We adopted theirsuggestions and equally they understood and took note of what our pain pointswere. What I found to be the most surprising thing, is that they were respondingpretty positively to our proposals and they were eager to learn more about howthings could be even better. Because the truth of the matter was that for along time the QMS has been used as the bogeyman to do things right, so nobodywas eager to talk to its owners, out of fear they were doing something wrong.

Of coursenot everything was ideal and sometimes we needed to debate about significantpoints, for example regarding the separation of responsibilities. How candevelopers test their own code? But once we discussed about code reviews orpair programming and the multiple layers of testing taking place before anycode ended up in production, their concerns were at least alleviated.

In the spring of 2018 some of theleading figures in the DevOps community wrote a love letter to auditors, titled “Dear Auditor” , anattempt to apologise for leaving them behind and not adequately explaining howa product can be compliant in this fast-paced world. The letter includes a where audit concerns are listed along with methods to address them.

Looking back now I realise that we treated compliance requirements like any other Agile project: identified the essentials, refined them, implemented them and asked for feedback from our stakeholders, the QMS owners. And from their side they saw the need to adapt to the times and facilitate us by revisiting policies and processes. In the end, the biggest lesson we learned was that only through an honest conversation between development and internal auditors were we able to accomplish what we both wanted: deliver quality software to our customers fast.

About Areti:

Amathematician by vocation, a software tester by profession, Areti Panou is nowworking as a cloud quality coach. Her job is to help development teams withinSAP come up with test and quality strategies that best fit their needs.Before that, she was the sole tester in one of the first products at SAP, toput a full Continuous Delivery approach into action and the first one in SAPhistory to daily deliver to its customers.

She blogs about questions that occupy her mindregarding software development, testing, and quality and shares random thoughtson Twitter as .

5 Handy Tips for Testers in DevOps

By Bhavani R

I am going to share five handy tips for testers in today’s DevOps world from my experience in this post.

Learn Continuous Testing

The first and foremost thing to learn about is: what is continuous testing ? How do DevOps processes impact the tester’s approach for transitioning from traditional waterfall to agile testing? Before starting with continuous testing, for each sprint, we were testing incrementally and then did the regression testing before deployed to production. Up to now, our sprint cycles for some projects have been weekly or bi-weekly. But this cycle becomes too long to roll out a feature or customer request in a timely way. So, we have adopted DevOps in our process. Testing over the whole lifecycle by shifting left and shift right, i.e. testing continuously from beginning to end.

Learn about DevOps jargon.

I recommend taking a look at Lisa Crispin’s course in Test Automation U to learn basic concepts and terminology around continuous delivery and DevOps. That will help you communicate and collaborate with everyone involved in delivering your product, including operations and infrastructure specialists.

A Culture Shift from an Independent Team to Whole Team Approach

In a waterfall methodology, testing is confined to an independent verification and validation team. In agile development, it becomes a collaborative effort.

People come with baggage: preconceived notions, past experiences, prejudices. The primary responsibility is educating the team about DevOps culture.

In DevOps culture, the focus is on the product, not building fiefdoms and amassing political power. The DevOps slogan is, “Everyone involved with delivering software is responsible for delivering high-quality software.” In short, the delivery becomes a shared responsibility.

Build A Successful Test Strategy

Building the right test strategy is critical in continuous testing. When we have 500 automated tests, we can’t execute all the 500 tests for each change committed in the pipeline. It will take too much execution time. So, we have analyzed and prioritized the test cases by their core functionality as well as using input from the customer support and business teams. In the end, we have come up with 100 automated regression tests which are enough to give us confidence that the latest change hasn’t broken existing features.

Don’t Just Make The Tests Pass, Focus on Efficiency

When we started with DevOps, in our continuous pipeline, our automation tests failed in between or at the last step of the pipeline. We analyzed the test results and the error logs. Most of the errors were due to timeouts or because of the new feature or enhancement changes. We have updated our scripts periodically to take care of this. After deployment, when we cross verified the product/project using manual exploratory testing, we saw a few more failures. Then we did a brainstorming session with our entire product team to make our automated tests more efficient. For this, we have created/updated the automation scripts in the preproduction phase to save time and resources.

Choose Your Tests Wisely

This is related to the points discussed in the previous paragraph. When we started doing the deployment through the DevOps practices, as testers, we need to choose our tests wisely to add those to the pipeline. Along with the regression test suites, 24/7 monitoring should be added. This helps us to understand how the product is to be used by the customers. For choosing the tests, we use input from the user behaviors that we learn from our monitoring tools, as well as discussions with the business stakeholders.

I hope the information from this experience helps you to enhance your testing!

References:

“Building a DevOps Culture ” – Mandi Walls

?” Lisa Crispin

Test Automation in DevOps , Test Automation U, Lisa Crispin

About the Author:

Bhavani R

I hold a Certified Software Test Manager and Six Sigma Green Belt. I have a passion for testing and learning, as well as implementing and mentoring the latest automation tools. I spent the last 16+ years working in IT, with a focus on software quality assurance and software testing. I am Testing Guild 2018 conference speaker and a frequent blogger. I manage a fast-growing, dynamic team bringing you https://www.qatouch.com . Twitter: . Linkedin: . Blog : www.bhavaniraj.com

Anticipating Problems with Infrastructure as Code in DevOps

By Nilanjan Bhattacharya

There are different interpretations of theterm, ‘DevOps’. However, most peoplewill endorse the idea – ‘DevOps also focuses on the organization change tosupport great collaboration between the many functions involved’.

Despite the focus on collaboration andorganization change, a critical part of DevOps is the enabling technology. There are different aspects of thetechnology. Continuous deliveryautomates the entire process of delivery, including building the software,validating the build and progressing through other steps in the form of apipeline. A much more elaborate part ofthe technology is Infrastructure as code (IaC). Infrastructure as code was pioneered by Amazon Web Services (AWS). Thereare newer entrants such as Microsoft Azure and Google Cloud.

The main concept of an IaC service, likeAWS, is to access infrastructure such as servers, databases and networks, asvirtual resources. It’s critical to beable to create and access the infrastructure programmatically . You can then treat the scripts which interactwith the infrastructure as software and use software development principles, todevelop and manage your code/infrastructure, such as source code control andcontinuous delivery (of infrastructure). IaC provides the ability to scale up and scale down components at will,greatly increasing the economy while meeting customer/market demands.

DevOps isoften introduced with examples from Netflix and Facebook. However, the software industry is vast. It includes both software product companieslike Netflix, but also includes enterprise software development in banks,telecom, government. It also includesservice providers. Given the universalavailability of cloud-based infrastructure and the accessibility due to lowcost and the pay-as-you-go model, the size of teams adopting IaC can varywidely. The two-person team creating amobile app, or the government department creating a health portal, is verydifferent from a team in Netflix. Thechallenge for all teams is preventing problems when deploying IaC as part ofDevOps.

How can you anticipate problems?

One way to anticipate problems is to askquestions.

A database as a service, like AWS RDS, isvery appealing both from a technical and business point of view. However, what happens when there is downtime,even for a few seconds, due to routine maintenance? For non-critical applications it may notmatter (you may still want to find out if the application is affected). For critical applications, you may want tocreate a highly available solution.

If you are working on an existingapplication, how do you replace the single instance of the database with ahighly available solution? Are there anymaintenance windows? What is the natureof the business? Is the business seasonal?

Here are some questions we might ask:

For a database as a service, what happens whenthere is downtime, due to maintenance?
Does the business have a maintenance window forreplacing the database?
What is the nature of the business? Can they afford downtime?
Is the business seasonal? Are there peak times when they can’t afforddowntime?

You could generalize the first question as,‘For any cloud-based service, what happens when there is downtime?’ You could also ask, ‘For any cloud-based service,what happens when you need to upgrade the version?’

Here is a question for you to practice (my response given below):

What happens when you need to upgrade thelanguage version (e.g., node.js) of a function as a service?

As an aside: Is this a problem with a cloud-basedservice/Infrastructure as code? Wouldthis have not been a problem if you controlled the server/environment, i.e., with physical servers?

How do you ask questions?

You want to ask questions which are open ended . You could ask, ‘What cancause the storage to become full on an EC2 instance?’ or ‘What could cause theinstance to become unresponsive?’ Theanswer will depend on your application and specific circumstances. (You will probably set up alerts for diskutilization reaching a threshold. However, alerts don’t replace the thinkingabout what might go wrong.)

You want to take asking open endedquestions a step further. What you arereally doing is looking for questionsthat you hadn’t thought about .

The idea of asking questions might seemvery unstructured. You can structure your thinking by using anapproach to generate ideas. One suchapproach is using the mnemonic, SFDiPOT, to structure your questions. SFDiPOTstands for structure, functions, data, interfaces, platform, operations,time. You could use S-structure to thinkabout the different components of the system such as the database, lambdas and instances and how they mightpose a risk. In the case of IaC, itisn’t a physical product, but an entire platform.

You can use another mnemonic CIDTESTD(Customers, Information, Developer relations, Test Team, Equipment & Tools,Schedules, Test Items, Deliverables) to generate ideas. If you are working with an enterpriseapplication like Adobe AEM, do your DevOps engineers have enough knowledge ofthe product? If not, does that representa risk? (We used the letter T, which stands for (Test) Team, to think aboutrisks related to the nature of the team).

When you ask questions, you want to holdyourself and your team accountable andincorporate feedback. Retrospectives andpost-mortems can identify questions that weren’t asked. You also want to identify how you could have generated ideas which would have led tothose questions.

Here is what I would ask when thinkingabout upgrades to the environment (e.g. node.js) for a function as a service,like AWS Lambda:

When we look at the Lambda code a few years fromnow, will it make sense to a new person/team?
Do we have automated tests for the larger functions?
Are there functions used for infrastructure, asopposed to, being used by the application?
Do we have any automated tests for the Lambdafunctions?
Are there differences in the quality of codewritten by the infrastructure team as opposed to the application team?
Will we be able to roll back the functions afterupgrade, in case there are problems?
Is there a development environment for theLambda functions, i.e., does the dev environment include the Lambda functionsespecially the ones used for Infrastructure?
Will the development and production environmentsbe kept in sync (for the next few years)?
Can we track differences in the development andproduction environment?

Here are a few more general questions youcould think about when working with infrastructure as code:

What can cause a component to not respond?
When can an instance not respond?
When can a component run out of capacity, e.g.,storage?
Can you upgrade a component to a highercapacity?
Are higher capacity components always availablein the region that you are working?
Does the region that you are working in, impactyour available options? Does that impactyour choices for increasing capacity in the future?

Beyond questions

There are other activities which cangenerate ideas on risk, for example, stress testing a component or creating arealistic customer environment, including real time data. These activities can also be used to generatequestions. Asking questions is part ofthe much broader activity of critical thinking .

For a start, I recommend beginning with asking questions in a work situation(see example on functions as a service above). You should accompany that by using retrospectives to evaluate questionsthat weren’t asked and identifying the ideas or activities which could have ledto those questions.

The idea of asking questions related to IaCcan form a foundation for thinking about your model for anticipatingproblems/risk in DevOps. It may alsolead the team to make inroads into critical thinking.

Notes

Thedescription of the core of DevOps is from Wikipedia: https://en.wikipedia.org/wiki/DevOps
SFDiPOT and CIDTESTD are part of the Heuristic Test StrategyModel created by James Bach ( )
The idea ofusing questions to identify risk has been used earlier, e.g., ‘a test case is aquestion that you ask of the program’, in ‘What is a Good Test Case?, StareastConference, May 2003,

Nilanjan Bhattacharya

Nilanjan works as a DevOps Delivery coordinator for AC3 in Sydney. AC3 is a multi-cloud service provider and Australia’s leading AWS Premier Consulting Partner and AWS Managed Services Provider.

Nilanjan has a background in software testing and project management. He has worked with a variety of software teams, including software product teams. His experience in analyzing customer support issues and working with successful software products with a multi-year lifecycle, provides a unique perspective on software quality and defects.

Webinar panel discussion, 11 September, 1:00 PM Eastern

We’re adding webinars to our Testing in DevOps community learning opportunities! Our first one is a panel discussion with awesome leading practitioners , , , and I will also join! Many thanks to for hosting the webinar (they host this site as well).

So what will we talk about? As many of our guest posts here have pointed out, DevOps is about the whole team – including operations specialists – working together to deliver value to customers frequently, without drama or stress. It’s that continual loop: together we identify the most valuable capability to delivery next.

In the webinar, we’ll talk about how we build and test it, test and build it, together. We have created a pipeline to deploy it to production, together. We’ll probably get into topics such as monitoring and observability to learn how customers use the product and respond to problems, and get ideas for the next valuable feature slice to build. We will answer your questions!

For more details and to register, please go to my (I am too lazy to copy it all over here!)

We welcome your questions in advance, just tweet them to @! Or, simply ask on the day in the Zoom chat. We look forward to an insightful and fun discussion!

Read, listen and learn today!

shared by Lisa Crispin

Read, listen, learn — Photo by freestocks.org from Pexels

The different pages of this website have links to so many helpful articles, podcasts, videos, webinars, books and more. Here are three that you might want to prioritize to read or listen to first, to build your understanding of devOps, continuous delivery, testing in production, and what makes software delivery teams successful.

Charity Majors on testing in production

Does the phrase “testing in production” scare you? Historically, it was an irresponsible practice – throw our changes into production and let the customers tell us what’s wrong. Today, it is a leading practice. In her recent article “ I test in prod “, says it’s a superpower!

Most interesting man in the world meme — Most interesting man in the world

Charity points out that we test in production all the time. The key is to do it safely. It takes an investment in tooling and training. This is especially important with the complicated distributed apps we support today. We need to test more than the code – we need to test the whole system, including the environment, the infrastructure, and the customers.

There’s so much to learn in this one article, from Charity’s experiences and insights. She breaks down the three aspects needed to start testing in production in a useful and beneficial way: technical, cultural and managerial. Plus, she works in so many classic memes to hilarious effect! Give it a read and start overcoming the fear of what will happen with each new change released to production.

Theresa Neate: Stop the whisper campaign that devOps means no testers needed!

We’ve been fortunate to have a guest post here from , one of the leading testing practitioner who is also a leader in devOps practices. She gives us a preview of her TestBash Australia talk in this Ministry of Testing podcast with Michael Lang.

Theresa starts with one of the best explanations of devOps and continuous delivery that I have heard. If you are not yet familiar with these terms, listen to the first few minutes of this podcast.

One of the most important messages Theresa delivers here is that we need to stop the “whisper campaign” that devOps means “no testers needed” and “no testers welcome here”. We know we testers can add huge value by participating in devOps practices and helping our teams succeed with continuous delivery. it’s up to us to use our natural curiosity, learn important new skills, and show what we can do.

Photo from Christina Morillo on Pexels.com

This podcast is less than 30 minutes long, and it’s going to be the best half hour you’ve invested in your career in awhile.

I wish I could go to TestBash Australia and participate in Theresa’s workshop “Hands-on Intro to AWS and a Taste of DevOps, for Testers”!

Accelerate: State of DevOps 2019

This led by is building on 6 years of data from a wide group of respondents. The executive summary alone is a gold mine of information about what practices and capabilities correlate with team success.

From the

The report includes insights into testing practices. For example, though previous years found that test automation has a significant impact on continuous delivery (CD), this year they found that automated testing positively impacts continuous integration (CI).

The ability to reproduce and fix failures, gather feedback from tests, improve test quality and iterate test runs quickly also ties into automated testing.

The report also focuses on disaster recovery testing. One finding is that organizations who do disaster recovery testing and create and implement action items based on what they learn from it are more likely to be in the elite performing group.

The report delves into the “Four Key Metrics” of software development and delivery, one of which is culture – too often overlooked by organizations. To improve, organizations need to start with the foundations, and adopt a continuous improvement mindset.

Read the report and help your own team be one of the many teams who are continuing to improve their ability to deploy small changes to production frequently, while enjoying their work.