Shifting towards Omni Testing – Part 2

By Sérgio Freire

In Part 1 of this series, Sérgio introduced his idea of “Omni Testing” as a term that better expresses the concept of continually testing throughout the DevOps infinite loop of discovery and delivery. In this second and final part, he explains why he likes to use this term to describe the multitude of testing activities teams do to succeed with continuous delivery and deployment.

We all heard about Shift-Left Testing and Shift-Right Testing in recent years. Probably you may have got the idea that it was the thing, that single idea that was going to massively improve your testing. Due to this misconception, for example, Janet Gregory prefers to think about testing as something always happening and holistic (i.e. considering all parts of the whole and their interconnection).

Let’s be clear: you cannot shift something that is always there and happens all the time; however, what you’re focused on at a given moment in time can, and probably should, change. Within a team, testing happens continuously and in parallel, so different people may be performing quite different testing activities and from different angles.

If we have pure waterfall development in mind, testing happens as a phase after implementation. However, in real-life, testing is not constrained anymore. Of course, some teams perform more intensive testing before releasing; in that sense, people may refer to shift-left testing efforts as a way to clarify requirements altogether from the start and introduce more unit kind of tests. Similarly, with DevOps, you may “shift-right” testing efforts to production and use as a feedback loop to product development.

However, testing itself is not shifted-left nor right; testing efforts can shift though. Having “ Agile Testing Quadrants ” (by & ) and previous work from (i.e. testing matrix ), we can see that there are different types of tests, with different goals. As Lisa and Janet point in their book “ Agile Testing – A Practical Guide for Testers and Agile Teams “, risks are an inherent and important part of projects where testing is used as a mitigation strategy.

As you plan each epic, release, or iteration, work with your customer team to understand the business priorities and analyze risks. Use the quadrants to help identify all of the different types of testing that will be needed and when they should be performed.

Thus, the type and level of testing performed, where and how it is performed, are dynamic.

Testing is not a specific task assignable to one person and delimited in time; everyone is involved in testing, it happens in different ways, at the same time and continuously; and this leads us to Continuous Testing.

From Continuous Testing to Omni Testing

Continuous Testing has been around for some time; the “Continuous” wording is not new. Unfortunately, it is being pushed by some tool vendors and some people to sell it as a way to continuously run “automated tests”.

Well, first there aren’t automated tests in a strict sense; you have automated checks but testing frameworks having been calling them as such from the start. Second, “continuous” in testing is not related to automated scripts/checks being continuously run or not. Even if it was, it wouldn’t make sense, as in CI “automated tests” are triggered on code commits, pull-requests, merges or due to some predefined schedule.

So what does Continuous Testing mean? I advise you to start by this post from Lisa Crispin and complement it with this course from .

We know that automation plays a key role in DevOps and also, though to a minor scale, on Continuous Integration (CI) practices which are closely related to Continuous Testing.

The CI concept was depicted by and largely widespread with Extreme Programming as one of the 12 core XP practices (, , , et al).

During Continuous Integration, “automated tests” (i.e. automated checks to be more precise: code that runs and performs checks to validate expected behavior) are run. These will help assure, to a point, that code that has been integrated isn’t breaking anything and previous expectations are still met.

As teams grow and code is committed more often, CI provides the means for immediate feedback so the team knows right away when a possible “bug has been added”. It can quickly be fixed before anyone builds on top of it. These automated scripts perform unit and integration tests but can also cover other types of tests such as security tests and more.

As builds are continuously being started by the CI process, these kinds of tests are also constantly being run.

However, “Continuous Testing” is not limited to these “tests”. Lisa Crispin and Janet Gregory put it like this instead:

Collaborative testing practices that occur continuously, from inception to delivery and beyond, supporting frequent delivery of value for our customers. Testing activities focus on building quality into the product, using fast feedback loops to validate our understanding. The practices strengthen and support the idea of whole team responsibility for quality.

In my opinion, this is the real reason behind the “Continuous” wording before “Testing”.

Atom Symbol representing Omni Testing with a brain icon, bug icon, other testing tools

Well, at first sight, “Continuous ” makes sense whenever everything else is using the “Continuous” prefix. But you don’t hear about “Continuous Coding”, do you?

Thus, a better adjective should be used instead: I think Omni (Testing) may describe Testing in a more complete way. Do you agree?

Continuously evolve our way of understanding the world around us

We moved from testing in waterfall, to testing in Agile , to testing in a DevOps context. Testing, similarly to other processes, evolved along the way; its essence hasn’t.

We still use testing for the same reasons, because we aim to understand what we test and use that information to make decisions.

Whenever we go to a shop and buy something, whenever we download an app and try it out, whenever we meet or go out with someone: we’re trying to somehow understand what’s in front of us. We’re information seekers because ultimately we want to take actions leveraged on data, i.e. informed decisions.

In software development, which may also include hardware development, by the way, testing is done as a means to ensure that the deliverables produced on our products/solutions iterations have a certain level of quality. And this includes evaluating not only expected behavior but also that “customer” needs are covered in the best way, thus we look at certain quality attributes.

Testing is centered around humans and it requires them. However, machines, tools, algorithms, and AI can leverage the unique power we have within us to provide better testing. That’s around 3.5 billion years of continuous evolution and accumulated learning about life on Earth.

I know what you may be thinking: maybe there is no Agile Life, no Continuous Life, and we also don’t call it Omni Life right? Well, true; similarly to what happened with life, testing evolved, it’s widespread, it lives from the outskirts of software development. But, life is restricted in many ways: to certain conditions, to our planet (as far as we know).

Testing, on the other hand, is a broader concept; no limits attached… and being Omni is part of its real essence.

Sérgio Freire’s background is in electronics and Telecommunications. Throughout the years, he has worked with teams in developing real-time software, web and mobile applications and complex hardware systems. He has worked and interacted with teams in waterfall and Agile contexts. As Head of Solution Architecture for Xray, he works closely with many different teams worldwide to help them achieve high-quality, testable products. Follow Sérgio on and .

Shifting Towards Omni Testing – Part 1

By Sérgio Freire

Testing, as we foreseek, is something not limited to time, not limited to space, not limited to a specific team member, not limited to a given type, technique or approach; it is a process we use to understand something, to obtain more information about it so that we can take actions on it.

Testing is not just about continuously “running” (checks); testing is always there, it is part of the whole (e.g. product development), it encompasses different kinds of activities and involves everyone.

That’s why Omni Testing can, perhaps, be a better way of describing Testing as we aim it to be.

Testing is universal. It’s one thing every single human has performed in their life.

For many of us working in software or hardware product development, testing is even more “natural.” However, we sometimes misinterpret it or see it just from a narrow perspective.

Lately, we have been hearing about Continuous Testing, Agile Testing, and DevTestOps/DevTestSecOps. Before that, Shift-Left and Shift-Right Testing and “Test Automation” were also around.

This tells us that Testing is:

broad
important
always present
intermingled with what we do
used interchangeably, in different contexts

Because of all of this, I think that Testing is a process unbound, in time, location, people and context; yet, it always has a purpose – actionable information. No matter if you’re using it to verify if you retained knowledge after attending a course, to evaluate if you like some food or a perfume, or to understand how a product works and if it meets some customer needs. Even before “software was invented, testing was already part of our lives and helping us shape our decisions.

Omni Testing

Why Omni? Is it a “new thing” or a “new kind” of testing?

What types of tests and approaches does it embrace? Who is involved in it?

“Omni” comes from the Latin word omnis and means “all,” “the whole of, “all things,” “everything.”

More and more, teams everywhere are aiming to have a holistic approach to testing. We’re aiming to have on-going conversations, on-going experiments and investigations, on-going checks, on-going information, and on-going feedback. The whole team is involved in testing and has quality in mind, from the start. For that, the team starts by assessing all risks, performs all sorts of tests, using all sorts of techniques and approaches.

Mind map of omni testing showing people, machines, multifaceted, active, embracing modern approaches — Omni Testing mind map

(Software) Testing is multi-faceted; it can be used in so many different ways, although in Omni Testing we try to perform it in a way where:

it produces relevant information ASAP (i.e. accordingly with the agreed test strategy and the overall agreed meaning of quality)
it focuses on what matters first
it is efficient and quickly adapts

Thus, it is not dependent/limited to:

specific roles
specific assets
specific test levels, types, techniques and approaches
specific methodologies
specific phases
specific tools

Defining Testing, again

Defining “testing” although may seem simple at first sight, it isn’t. Why? Because it is a broad concept and sometimes used incorrectly or in a very restricted sense.

Recently, I had an opportunity of attending a workshop on exploratory testing with at .

To the question “why do we perform testing?” the group came to these results.

Spend time/resourcesUnderstanding the systemUncover gapsProvide valuable info to team membersGive everyone feedbackFinding risksMake the end user happyTranslate information to othersReady to ship?

Probably, others would come to different conclusions. For some, testing is just about checking and “finding bugs.” However, it’s much more than that.

I do agree with that pictures such as the following one won’t help, as they depict “test” as a phase (which besides not being true is also not very agile).

DevOps infinite loop of plan, code, build, test, release, dep;oy, operate, monitor, plan...

There’s an in-depth from , where he goes over the definition from Cem Kaner ( The Ongoing Revolution in Software Testing, 2007 ):

Testing is an empirical, technical investigation conducted to provide quality-related information about a software product to a stakeholder.

In testing, as Alan Richardson correctly points, we build models and we compare them with reality. has a short, yet clear video on which may be useful to you in this context.

When focusing on software testing, you’ll probably find some terms: “Traditional”, Waterfall, Agile, Continuous and Modern Testing (you may want to look at and “ Modern Testing Principles “; there’s a course on MoT).

Let’s look at this from James Bach & Michael Bolton:

“Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes: questioning, study, modeling, observation and inference, output checking, etc.”

I would rephrase this slightly, adding my perspective.

Testing is a targeted process of challenging something (e.g. a product) and our understanding about it , using a set of activities leveraged by specific human skills.

Where,

“targeted process” – a set of activities, having a common goal (i.e. quality attributes in mind), that will produce a result
“challenging” – includes trial/experimentation, checking, …
“activities” – build models, create test ideas, execute actions in the SUT, investigation, ask questions, provide feedback, …
“human skills” – creativity, curiosity, perseverance, analytical thinking, communication, courage, continuous learning mindset, …

Testing is by nature; we try things and test ideas, we explore to seek understanding, to obtain information as states in his adapted model of James Bach’s . We also use other approaches, such as script-based tests (i.e. checks) which sometimes are automated to verify “compliance” with predefined rules. We use all the outputs we collect as data that we process. This data becomes information, which is then used to improve our learning of the system/product and provide feedback to “stakeholders” (e.g. business, product team).

Whenever testing something we have in mind certain quality attributes (e.g. correctness, usability, scalability) – “requirements” we demand or expect from it.

To perform it, we test at different levels and use a bunch of different techniques that help us shape what we call “tests”.

We test everything, software, hardware, specifications, established ideas, documents, and specifications.

Thus, testing is a complete, omni process that uses best-of-breed human skills to answer tailored questions to clarify our understanding about something and that can help shape it before, during and after it is built.

That being said, how does it relate with Shift-Left/Shift-Right and Continuous Testing? We’ll dive further into this topic on the next blog post of this series related to Omni Testing.

Check next week for Part 2 of this series about Omni Testing!

Continuous Testing in a seasonal business

For many countries around the world, the heaviest retail season is already starting. In the U.S., Black Friday, the heaviest day for brick-and-mortar retail sales, and Cyber Monday, when online retailers get the most traffic, are almost upon us. The delivery teams responsible for the software that powers those retail engines must be busy.

Or… are they on a steady cadence, practicing continuous delivery, and not feeling stressed? Teams that continually release small changes to production should be able to do so in a sustainable way. Hmm.

Most business domains revolve around some kind of seasonal pressure. Some industries have a big trade fair, where everyone shows off their new product. All development is aimed at that big event. Tax preparation products will be used more heavily when people have to file their returns. Government regulations may change annually, requiring software updates to remain compliant.

How do teams achieve continuous delivery, including testing at all parts of the DevOps continuous loop, with seasonal spikes in demand? Please join me along with , , , and as we share our own experiences with these challenges in a webinar panel hosted by , Thursday, 14 November. We hope you can join us live and bring your questions. Please to attend or to get the link to the recording after the event.

Running API Smoke Tests in Your Continuous Deployment Pipeline

By Kristin Jackvony

Each year, more software companies switch to a microservices architecture using Application Programming Interfaces (APIs), because APIs make it easy for different teams on a project to access each other’s resources. APIs also enable software companies or individuals to get data from each other: for example, Twitter has an API that is used by many small businesses to retrieve tweets related to their business for display on the business’s webpage.

Another great feature of APIs is that they are smaller than a monolithic application, which means that they can be deployed frequently. At the company where I work, my team has eight different APIs, and we have eleven different environments that we deploy to. If we were relying solely on manual testing to verify that the APIs had been deployed correctly, we’d have to run eleven sets of tests for each API release. And if we were deploying more than one API at a time, which we often do because our APIs work together, we could be running 88 sets of tests. And if we deploy every two weeks, those 88 sets of tests will amount to a lot of tedious, repetitive testing!

We’ve solved this problem by setting up automated API smoke tests that run with every deployment, to every environment. In this article, I’ll describe how we decided what to test and how we set up our smoke tests. We used Postman, Newman , Powershell , and Octopus to set up our automated smoke tests, so I’ll be describing what we did with those tools. However, these strategies could be adopted to any continuous deployment (CD) pipeline.

Step One: Decide What to Test

The main purpose of smoke tests is to do some simple, high-level tests to ensure that the deployment was successful. This is not the place to test every single thing that your API can do. When we chose what to put in our smoke tests, here is what we considered:

We set up one “Happy Path” test for each endpoint- that is, one test that is expected to be successful. For example, a GET request for a resource with a specific ID would be expected to return that resource. You’ll want to test each endpoint once to verify that each one is working as expected.
If there were major variations in how a specific endpoint would be used, we added one or more tests to check those variations. For instance, we have a file retrieval API where a file can be retrieved by two different methods. The endpoint looks the same, but the bodies of the requests are different. So we added one test for each method.
For endpoints which required a certain level of security, we added a test to validate that a request without the appropriate authentication would NOT return information, and would instead return a 400-level error.
We did not add any other negative tests, such as a POST request with data outside the allowed parameters, because we didn’t feel that those were necessary to ensure that the API was working. We instead ran those tests as part of a nightly regression suite.

Step Two: Export Your Tests

We used Postman to create our API tests, so when the tests were created, we exported both the test collection and the test environment file into JSON files. For those who are not familiar with Postman, a test environment is a group of variables that can be referenced in a collection of tests.

Step Three: Write a Script to Run Your Tests

Postman has a command-line tool called Newman that can be used to run a test collection, so this is what we used to run our tests. We created a Powershell script that would run the Newman command.

Step Four: Create a Deployment Step to Call Your Script

Once the Powershell script was ready, we set up a deployment step in each Octopus API deployment project that would run the test script. If the script failed, the deployment would fail.

Step Five: Organize Your Variables

Variables are often a consideration when running smoke tests in different environments. For example, in your QA environment, your test user might have an ID of 340, but in your Staging environment, your test user might have an idea of 620. Another issue with variables is that sometimes for security purposes, you might not have access to passwords or keys in Production environments. This was the case for our team, but fortunately Octopus has the values we need to run our tests in Production, so we could solve the problem by having Octopus pass the values that we needed to our test script.

There were three different kinds of variables needed for our smoke test:

Type 1: Variables that were unchanging for each test environment, such as “firstName”: “Prunella”. These variables could be put directly into the Postman environment, so nothing more needed to be done with them.

Type 2: Variables that changed for each test environment, but did not need to be kept secure, such as “userId”: 340. These variables were added as Octopus variables in this fashion: “smoke.userId”, and the value of the variable was set for each environment; for example, QA was set to 340, Staging was set to 620, and Production was set to 450.

We then used the variables like this:v

When we called the Powershell script in Octopus, we sent the variables set in Octopus to the script.
In the Powershell script, we accepted the Octopus variables sent in and assigned them to Powershell variables.
When we used the Newman command in the Powershell script, we sent the variables in the script to the Postman environment.
Newman used the variables sent in the Powershell script, combined with the variables in the Postman environment to run the Postman collection.

With our API smoke tests running in Octopus for every API and in every environment, we can feel confident that any big problems with an API deployment will be detected right away. Furthermore, we can test in Production environments even when we don’t have access to sensitive API keys. Having automatic deployment smoke tests frees us up to do other types of testing, such as manual exploratory testing and security testing, as well as write more test automation for nightly regressions.

About Kristin:

Kristin Jackvony discovered her passion for software testing after working as a music educator for nearly two decades. She has been a QA engineer, manager, and lead for the last ten years and is currently working as a QA Lead at Paylocity. Find me on and .

Anticipating Problems with Infrastructure as Code in DevOps

By Nilanjan Bhattacharya

There are different interpretations of theterm, ‘DevOps’. However, most peoplewill endorse the idea – ‘DevOps also focuses on the organization change tosupport great collaboration between the many functions involved’.

Despite the focus on collaboration andorganization change, a critical part of DevOps is the enabling technology. There are different aspects of thetechnology. Continuous deliveryautomates the entire process of delivery, including building the software,validating the build and progressing through other steps in the form of apipeline. A much more elaborate part ofthe technology is Infrastructure as code (IaC). Infrastructure as code was pioneered by Amazon Web Services (AWS). Thereare newer entrants such as Microsoft Azure and Google Cloud.

The main concept of an IaC service, likeAWS, is to access infrastructure such as servers, databases and networks, asvirtual resources. It’s critical to beable to create and access the infrastructure programmatically . You can then treat the scripts which interactwith the infrastructure as software and use software development principles, todevelop and manage your code/infrastructure, such as source code control andcontinuous delivery (of infrastructure). IaC provides the ability to scale up and scale down components at will,greatly increasing the economy while meeting customer/market demands.

DevOps isoften introduced with examples from Netflix and Facebook. However, the software industry is vast. It includes both software product companieslike Netflix, but also includes enterprise software development in banks,telecom, government. It also includesservice providers. Given the universalavailability of cloud-based infrastructure and the accessibility due to lowcost and the pay-as-you-go model, the size of teams adopting IaC can varywidely. The two-person team creating amobile app, or the government department creating a health portal, is verydifferent from a team in Netflix. Thechallenge for all teams is preventing problems when deploying IaC as part ofDevOps.

How can you anticipate problems?

One way to anticipate problems is to askquestions.

A database as a service, like AWS RDS, isvery appealing both from a technical and business point of view. However, what happens when there is downtime,even for a few seconds, due to routine maintenance? For non-critical applications it may notmatter (you may still want to find out if the application is affected). For critical applications, you may want tocreate a highly available solution.

If you are working on an existingapplication, how do you replace the single instance of the database with ahighly available solution? Are there anymaintenance windows? What is the natureof the business? Is the business seasonal?

Here are some questions we might ask:

For a database as a service, what happens whenthere is downtime, due to maintenance?
Does the business have a maintenance window forreplacing the database?
What is the nature of the business? Can they afford downtime?
Is the business seasonal? Are there peak times when they can’t afforddowntime?

You could generalize the first question as,‘For any cloud-based service, what happens when there is downtime?’ You could also ask, ‘For any cloud-based service,what happens when you need to upgrade the version?’

Here is a question for you to practice (my response given below):

What happens when you need to upgrade thelanguage version (e.g., node.js) of a function as a service?

As an aside: Is this a problem with a cloud-basedservice/Infrastructure as code? Wouldthis have not been a problem if you controlled the server/environment, i.e., with physical servers?

How do you ask questions?

You want to ask questions which are open ended . You could ask, ‘What cancause the storage to become full on an EC2 instance?’ or ‘What could cause theinstance to become unresponsive?’ Theanswer will depend on your application and specific circumstances. (You will probably set up alerts for diskutilization reaching a threshold. However, alerts don’t replace the thinkingabout what might go wrong.)

You want to take asking open endedquestions a step further. What you arereally doing is looking for questionsthat you hadn’t thought about .

The idea of asking questions might seemvery unstructured. You can structure your thinking by using anapproach to generate ideas. One suchapproach is using the mnemonic, SFDiPOT, to structure your questions. SFDiPOTstands for structure, functions, data, interfaces, platform, operations,time. You could use S-structure to thinkabout the different components of the system such as the database, lambdas and instances and how they mightpose a risk. In the case of IaC, itisn’t a physical product, but an entire platform.

You can use another mnemonic CIDTESTD(Customers, Information, Developer relations, Test Team, Equipment & Tools,Schedules, Test Items, Deliverables) to generate ideas. If you are working with an enterpriseapplication like Adobe AEM, do your DevOps engineers have enough knowledge ofthe product? If not, does that representa risk? (We used the letter T, which stands for (Test) Team, to think aboutrisks related to the nature of the team).

When you ask questions, you want to holdyourself and your team accountable andincorporate feedback. Retrospectives andpost-mortems can identify questions that weren’t asked. You also want to identify how you could have generated ideas which would have led tothose questions.

Here is what I would ask when thinkingabout upgrades to the environment (e.g. node.js) for a function as a service,like AWS Lambda:

When we look at the Lambda code a few years fromnow, will it make sense to a new person/team?
Do we have automated tests for the larger functions?
Are there functions used for infrastructure, asopposed to, being used by the application?
Do we have any automated tests for the Lambdafunctions?
Are there differences in the quality of codewritten by the infrastructure team as opposed to the application team?
Will we be able to roll back the functions afterupgrade, in case there are problems?
Is there a development environment for theLambda functions, i.e., does the dev environment include the Lambda functionsespecially the ones used for Infrastructure?
Will the development and production environmentsbe kept in sync (for the next few years)?
Can we track differences in the development andproduction environment?

Here are a few more general questions youcould think about when working with infrastructure as code:

What can cause a component to not respond?
When can an instance not respond?
When can a component run out of capacity, e.g.,storage?
Can you upgrade a component to a highercapacity?
Are higher capacity components always availablein the region that you are working?
Does the region that you are working in, impactyour available options? Does that impactyour choices for increasing capacity in the future?

Beyond questions

There are other activities which cangenerate ideas on risk, for example, stress testing a component or creating arealistic customer environment, including real time data. These activities can also be used to generatequestions. Asking questions is part ofthe much broader activity of critical thinking .

For a start, I recommend beginning with asking questions in a work situation(see example on functions as a service above). You should accompany that by using retrospectives to evaluate questionsthat weren’t asked and identifying the ideas or activities which could have ledto those questions.

The idea of asking questions related to IaCcan form a foundation for thinking about your model for anticipatingproblems/risk in DevOps. It may alsolead the team to make inroads into critical thinking.

Notes

Thedescription of the core of DevOps is from Wikipedia: https://en.wikipedia.org/wiki/DevOps
SFDiPOT and CIDTESTD are part of the Heuristic Test StrategyModel created by James Bach ( )
The idea ofusing questions to identify risk has been used earlier, e.g., ‘a test case is aquestion that you ask of the program’, in ‘What is a Good Test Case?, StareastConference, May 2003,

Nilanjan Bhattacharya

Nilanjan works as a DevOps Delivery coordinator for AC3 in Sydney. AC3 is a multi-cloud service provider and Australia’s leading AWS Premier Consulting Partner and AWS Managed Services Provider.

Nilanjan has a background in software testing and project management. He has worked with a variety of software teams, including software product teams. His experience in analyzing customer support issues and working with successful software products with a multi-year lifecycle, provides a unique perspective on software quality and defects.

Measuring Our Continuous Delivery Progress

by Rob Hornby

We’ve embarked on work to move off our legacy monolithic Oracle Commerce platform and are creating a microservice -based architecture hosted in . As part of this we have created a Platform ( John Lewis Digital Platform, JLDP) which enables teams to get started quickly through providing key capabilities that are needed by all teams e.g. Google Cloud environment, code repository, path to production, telemetry. We call this our “ ”. We started out on this journey as it was observed that we were not delivering business value quickly enough and technology was believed to be a key impediment to doing this.

One of the key initiatives we undertook as part of building the platform was being able to measure the progress of teams working on the platform through their approach to Continuous Delivery . Previously we had lacked this information and could not make data-driven decisions about our service/application health. We wanted to avoid this and build it in to the platform from the start.

We started capturing much of the data through spreadsheets based on data visible in Google’s logs. We’ve iterated on this and have developed an application which we call the “Service Catalogue”, that brings together the various metrics we’ve collected on health of a Service (one or more microservices to achieve a particular outcome), along with key information on each service/application e.g. Search.

Lead Time to Value

Our initial focus was to show that access to the Platform was no longer a constraint in getting a service live to customer. We wanted to capture some information that proved this was no longer the constraint. We were able to quickly show our service creation lead time took hours not months as it had been previously.

We then looked at capturing the lead time of the first service deployment to production, we called this our onboarding lead time . For the platform this evidenced that teams had managed to navigate our paved road and deployed at least a “hello world” to production. This confirmation also reduced the platforms reactive work load and was a key message to teams when onboarding them, get to production early.

The final addition was to measure of lead time to first value . This was our final measure of success and confirmed the team had made their service live to customers.

JLDP v2 Services – Lead to value

Deployment Indicators

Once a team had deployed and were live to customer, we wanted to continue to understand their continuous delivery on the platform. We set out to capture more information: deployment intervals , deployment lead time and deployment throughput. We have again used systems data to capture this information, looking at git commit history through to container deployment to derive the data. We are looking to observe teams’ approach to agile principles and lean methodology through their continuous delivery practice.

The deployment interval captured is the time between deployments to our production cluster. We want to see a low interval and consistency which suggests a team is then deploying frequently and continuing to work on the Service.

The deployment lead time is then how quickly a code change gets through the pipeline to production. This tells us how long it takes a team to get a change live and again looks for consistency and a short time from commit to production. This tells us that teams can get change through to production quickly through their pipeline and their ways of working.

Delivery Indicators (Interval Time, Lead Time)

Deployment throughput then helps us confirm overall deployment volumes across the platform helping us to evidence our overall deployment trends within teams and services. We have the data for this but have yet to fully integrate this in to our Service Catalogue, though we can present it on a on-demand basis.

Live to Customer — Monthly Service Deployments

What did we learn?

These data points were then critical for our initial conversations with teams which we called platform research. We used the data to talk to teams that were taking time to get to their first customer and understand what might be constraints in either the platform or the wider delivery process. We are also trying to build a product not just a platform and so the capabilities we had will be informed through the customers (service teams) running on our platform.

We learnt quickly that we’d started to unblock some key constraints.

The access to the platform has reduced the provision of infrastructure to teams from months to hours.
Our overall deployment throughput was increasing substantially from 10’s of deployments a years to 1000’s.

Total Deployments Throughout the Year — Total deployment throughout the year

Continuous Delivery patterns (Lead Time: Blue, Interval Time: Red) - so much to read into these graphs. — Continuous Delivery patterns (Lead Time: Blue, Interval Time: Red) – so much to read in these graphs

Through further ongoing research we learnt that there were common service constraints

Teams were struggling with the security process which was designed for large waterfall projects and hadn’t caught up with a more agile, microservice-based architecture.
Teams had too much work in progress. We had teams which were requesting multiple services without getting their first service live to customer.
Teams had external dependencies, usually around integrations outside the platform ingesting data from other external systems.
Teams were attempting to get to feature parity before going live to customer.
Teams were still attempting to undertake end-to-end functional or performance testing with external dependencies.
Change freeze still plays a part in reducing consistency and increasing batch size.

These were all things we could now start initiatives on to improve lead time. For example, we have created a Platform Security team to help engagement with teams in to our current security process with a goal of changing and developing a pattern more consistent with our delivery approach.

It also helped us inform the direction for the platform. We increment the version of our platform quarterly (although deploy daily). Each increment we aim to delivery new capabilities such as improved service creation, monitoring and logging capabilities and resilience, all focused on building a product that helps the teams running on our platform go faster or improves their stability.

What Next?

So far we have focused on measuring our continuous delivery indicators to show throughput to production. Operability is the next key focus. Teams are now starting to support their services, and the traditional operational structures which supported many of the operability aspects are now transferring their knowledge on processes to teams.

Continue to develop the platform with operability as a focal point. We have already provided teams with indicators of health e.g. based on Operability patterns outlined in .
We want to combine this with alerting focused around and .
We will look to expand our Service Catalogue to include deployment stability indicators. Deployment Failure Rate (number of failed production deployments and duration between), Recovery time from failure (time taken to recover the service after a failure).

The aim is to combine our continuous delivery and operability metrics together in one place. The future will then be tying this data together with business data to truly show how deployment frequency relates to value. The platform and being able to evolve it as a product will continue based on our ability to evidence and communicate the value it can add.

Reference:

For more information, we recommend reading Measuring Continuous Delivery by Steve Smith ().

Rob Hornby is a Product Engineer working for John Lewis based in London. He has undertaken a number of roles, over the last 20 years, within Testing, Release, Platform and Environments.

Rob is a keen believer in being able to treat all things as Products and be able to evidence the value generated by that team. Ideally everything would be delivered by small teams, delivering continuously and iterating on value. However not all teams start at this point, not all teams are the same or born equal. This is where the challenge and enjoyment comes from.

Outside of work, Rob is a keen runner, father of an amazing young girl, Sofia and works on a number of local community events.

What is continuous testing?

By Lisa Crispin

Even when I worked on waterfall teams, we tested early and often. We got ourselves invited to analysis and requirements meetings. We tested requirements docs. We asked questions. We tested as the developers delivered code, even before “code freeze”. We also worked closely with customer support to learn about how customers used the features after they were delivered. (I realize not all teams using a phased-and-gated approach do all that, but that approach should not stop teams from using good practices!)

Extreme Programming (XP) made so much sense to me because it embraced testing and quality as the responsibility of the entire delivery team. In agile development, we focus on building quality in, we test ideas, we automate regression tests, we explore delivered code, we test in production. These same practices apply to teams who use the “DevOps” label as well.

That’s why Dan Ashby’s post in 2019 on “continuous testing” resonated so much with me. His visual model captures what is in my brain when I see the term “continuous testing”. With his permission, I often use this model to help others understand the concept:

Dan Ashby’s Continuous Testing Model

But there are different definitions!

Recently, someone posted a link to a short Twitter thread from earlier this year where continuous testing was defined quite differently. Basically, this alternative definition is only about running automated tests in the continuous integration, delivery, deployment pipelines. I started asking around to see how other people define “continuous testing”, and some people shared that definition – only the automated suites in CI/CD.

A book on “continuous testing” which I had not gotten around to reading also has this definition, and it references the Wikipedia page which says the same thing.

Automated tests in CI are not new!

Continuous integration has been around for decades. My waterfall team back in the early 90s did CI, complete with automated regression tests and automated deployments (to 20+ Unix environments!) We didn’t call the automated regression tests “continuous testing”. We called them automated regression tests. It puzzles me that people would apply the term “continuous testing” to these automated test suites, which teams have been using since long before the term “continuous delivery” appeared.

Going to the source

I learned about continuous delivery by reading the terrific book of the same name by Jez Humble and David Farley. If you read it, you will find it is a book about testing. It goes into much detail about many different kinds of testing activities – not only automated regression tests!

So I went to Jez’s website to see how he defines “continuous testing” . Please click that link and read it for yourself. To quote Jez,

Our goal is to run many different types of tests—both manual and automated—continually throughout the delivery process.

That is followed by a picture of Brian Marick’s agile testing quadrants, illustrating a range of tests that are business-facing, technology-facing, that guide development and that critique the product. In Jez’s quadrants model, the tests that critique the product include manual tests.

Jez goes on to explain the concept of the deployment pipeline and the importance of getting quick feedback from automated acceptance tests that are in the pipeline. But he does not limit “continuous testing” to only those tests.

Here’s the latest version of the agile testing quadrants model that Janet Gregory and I use. We’ve added examples of testing activities we can do today thanks to newer technology and tools, such as monitoring and observability. Those are also part of continuous testing.

Agile Testing Quadrants from Agile Testing Condensed: A Brief Introduction by Janet Gregory and Lisa Crispin

Does the terminology matter?

I don’t like to be too pedantic about terms, but we need to be able to communicate with each other. If my idea of continuous delivery was limited to only the pipeline and did not include the concept of breaking features into small slices and deploying small changes frequently, that would not match what most practitioners I know consider to be “continuous delivery”. It would be hard to have a conversation about it with two very different definitions.

Many people use the term “automated regression checks” rather than “automation regression tests” because these tests are checking that requirements are met and that established behavior continues to work as expected. They reserve the term “testing” for activities that are done by humans. Janet suggested that maybe we can refer to the automated regression test suites in CI / CD pipelines as “continuous checking”, and reserve the term “continuous testing” to include those checks PLUS all the testing activities that happen around Dan’s infinite testing loop.

How can we resolve this terminology dilemma? Feel free to comment here. There’s also a discussion topic for this on the Ministry of Testing Club .

Meanwhile, I shall see if there are even more definitions of continuous testing I hadn’t heard yet!

Path to Production – Test Environments as a Product

By Rob Hornby

It could be easier, we could have teams for each application we run, that those teams treat their production and non-production application environments the same. That they have fully automated full stack continuous deployment pipelines and the software is so well designed that there is no dependency between each application owner. Teams deliver code by the hour to production and use advance deployment techniques such as blue-green deployment coupled with highly effective monitoring, alerting and automated rollback to identify and resolve issues in production within minutes.

We do have “agile” application teams working within a cloud environment, they have growing autonomy as they build independent features and function. That’s awesome. The problem is that we have not been working that way since we started building technology. We have a sprawling on-premise (we own two data centres) environment where Non-Production environments equates for 60%+ of the overall infrastructure. This is made up of dependent applications running together in an environment.

We have followed traditional programme structure over the last 10 years, with multi-year investments and waterfall based delivery. These have sought to reduce risk through separation rather than integration. This has resulted in multiple non-production environments. Our applications (100’s) have been built tightly coupled and need multiple other applications to work (Commerce systems through to legacy mainframe systems all meshed together), increasing the size of the basic/core environment. These programmes have not invested in automation, leaving a challenging sprawling estate to manage.

Path to Production (Variety of routes followed by teams ensuring lack of consistency in versioning in each environment)

This has created a number of problems. The estate is costly to maintain. There is hidden cost in each application change or infrastructure upgrade. Our people are linked to projects and programmes, when they finish, the applications in non-production fall in to disrepair, they are costly and time consuming to get working again when the next team comes along. A continued desire for separation results in projects hunting down that unused application/environment that has nobody else working in that space. The sprawling nature and lack of ownership results in a small band of “Environment Management” who work to bridge the communication and knowledge gaps across this sprawling estate. These people work for Projects and Programmes and through “best endeavours” get Delivery teams, Operations and Infrastructure teams to stitch the environment back together each time and retain the illusion these environments are good to test in. We effectively have 10 mini-lives beating away with no real Service Management to deal with them.

That long introduction was needed just to give me time to breathe and reflect. The organisation is changing for the better, which is great. We are re-organising and realising the mistakes of the past. We are now building most new systems in a cloud environment, are focused on Micro-service design principles and have teams that own and run the service (well almost). These are small individual successes which highlight the organisation is changing. Our online teams have gone from 10 deployments a year to over 1000 in just over 18 months. It should be a factor of 10 greater by next year. However there is a long way to go till we are fully working in this way, maybe we will never get there. This is where Environment as a Product comes in. How do we change our existing non-production environment landscape for the better.

We have started with two key initiatives. Initiative 1, Let’s get support to introduce an “enabling constraint” through removing the environments that least support delivery. They are the most difficult to maintain, are off the beaten path to production and the most costly to re-enable them. If the project or programme don’t have them they can’t use them. We know they are costly to build which is part of the reason they have been maintained. With the remaining environments we are going to create a paved road of sorts. One of the key things is to not construct the path to production based on your current organisational/delivery construct. We have been very waterfall in nature and have therefore not made change often to production.

This results in us having multiple hops to production for a single change based on the type of testing to be undertaken at each stage. Development (unit, system), Integration, Release (end to end), Performance (performance, operational), Pre-Production (incident and support) and Production. You introduce delay just through deploying to each of these environments and fixing the issues that come up as a result. That’s before you think of data, types of testing, handoff implications. So initially let’s get down to one of each type to support all teams, projects and programmes. Let’s draw it up so it looks like some are optional. Development, Integration, Pre-production, Production (even this is too many).

This was difficult to get going, convince people that we might be able to go faster through having less environments, that there might be more contention initially, however that might force us to look at the organisational shape and how we design software. Whether the full extent of the idea has sunk in, is yet to be seen, however the cost avoidance articulation of maintenance running in to millions of pounds a year was hard to ignore and gave us the initial green light.

Initiative 2, is to consolidate the team we have and build a paved road (common environments, tools and approach). We have taken everyone responsible for non-production “Environment-Release Management” and centralised them in to a single Product team. This wasn’t easy. You have to convince the leadership team that they want to pay for this as a separate service, that the cost is neutral, value adding even and that in doing so you can look at how the team works and make efficiency, as well as improving the path to production. What this doesn’t do is centralise the technical people responsible (not owning) for those environments. These remain fragmented across Operations, Infrastructure and Delivery teams. While the number of different technologies, teams and ways of working make this impractical at a small team level.

It does allow you to put a very clear Service Management approach in place for Non-Production and with leadership support allow you to be more robust in enforcing a better approach. You can start to work with individual teams to support and help them approach their delivery differently, focusing on automation, repeatable process and decoupling of change. The aim is that teams do this themselves and that the Environment-Release team can identify teams that need support. We are also backed by organisational , which have been established to give teams direction in how they should deliver.

Centralising the team has also allowed us to look at how we approach managing the environments. A single Product team with longer term goals, where the value is what we do to the environments, not the end customer outcome that the Project or Programme was focused on, has a very different priority and outlook. What was several ways of working can now be focused in to a single way of working. We have started by making sense of what people were doing daily. This has been done through several value stream mapping workshops but also to get everyone’s work up on a board and try to make sense of the work we do. Get a feel for reactive vs planned work. Upon doing this we can start to build a very clear backlog of priorities for our own ways of working, how we track issues, do we manage problems, how do teams onboard on to our path to production, can we pre-empt issues, what do we do with data, looks at our skills and most importantly clear communication.

Value is now a clear driver, as is ownership. We can now mandate changes to production are part of non-production thinking. We can also start to look at cost of delay and overall costs of supporting teams and drive our own value outcomes. Lead time is also a key measure, whether that that is to measure how quickly teams move through our environments, how long it takes us to respond to incident, downtime within environments etc… We can also help to start to measure teams frequency of deployment/release and to show organisational pace of change. Traditional service management is very much focused on controls but here we want to make it much more about value outcomes and enabling flow within teams.

We also have one foot firmly in the future. Over time the need for this team will be less. Greater ownership, more end to end owning teams, “DevOps” will all drive less need for a Environment-Release Product team. Our life span is based on the value we can show to generate. However we need to develop our skills in to other areas, whether that be Platform management within our Cloud platform or in to more “DevOps” skill sets which can work within teams to help them deliver. For now we are focused on our Product and value. It would be interesting to hear how other companies and people manage non-production environments, especially where there has been ongoing development of a variety of approaches for many years.

Rob Hornby is a Product Engineer working for John Lewis based in London. He has undertaken a number of roles, over the last 20 years, within Testing, Release, Platform and Environments.

Outside of work, Rob is a keen runner, father of an amazing young girl, Sofia and works on a number of local community events.

Open Sourcing Your Mindset

By Brendan Connolly

Asdevelopment has shifted from waterfall to agile and now DevOps, at each stageyou see developers taking on more responsibility and accountability.

Waterfallwas isolationist, building silos around different groups in softwaredevelopment. Developers owned the act of coding and in the stereotypicaldeveloper way, had little to no interest what happened before or after theycoded. Other roles on teams in many ways acted as insulators to keep externalforces away from developers.

Agilewas developers pulling that insulation away, shifting themselves left andtaking more ownership of the product creation process. Developers found theyrequired a real time feedback loop to build the right things and successfullyreact to change.

DevOpsis essentially the same process on the right side of the spectrum. Developmenttakes more ownership of deployment, monitoring and supporting the applicationshealth. It’s just too slow and cumbersome to react to the requirements ofmaintaining applications by filtering feedback through separate teams at scale.

Thebenefits of integration outweigh the cost of the additional responsibility.

Code Continuous

Tocope with the additional responsibilities, developers have turned to theirtrusty friend code .

Withagile we see rise of tooling developed for ContinuousIntegration to alleviate the pain and reduce the risk in iterativedevelopment.

InDevOps we see the rise of code and tools supporting Continuous Deployment . Even when many teams already had productinstallers, or automated deployments what was missing was the integration ofthat information.

Ifdevelopers are owning more of the product as well as release and deployments,then testing naturally gets swept up in the process. In fact we see thispattern emerging with the concept of ContinuousTesting . Code is being written, tools created to help solve the issue ofintegrating test execution and reporting into teams release pipeline. Thefeedback loop is extended to provide rapid information to make it easier andfaster for development to be responsive to change.

Continuous Testing

Automatedtesting is a loaded term. In developer circles it almost exclusively means unitand possibly integration tests. In QA it almost almost exclusively means end toend (e2e) tests, and is often synonymous with Selenium .

Onteams where both sets of tests exist, their worlds may seldom interact.Developers may be aware of the e2e tests but they probably aren’t writing them and even less likely running them.

*Illustration from “Continuous testing in DevOps”, Dan Ashby, https://danashby.co.uk/2016/10/19/continuous-testing-in-devops/*

ContinuousTesting has nothing to do with the testing that most QA team members are mostfamiliar with, exploratory testing. The purpose of the code and tooling thatcomprises a pipeline isn’t for there to be some manual intervention requiredpart of the way through it. Putting exploratory testing somewhere in the laterstages for the pipeline is akin to pointing a fire hose at someone and askingthem to direct the flow to the next stage.

Ifthe speed of DevOps wasn’t enough, the presence of continuous testing only puts morepressure on testers, simply because more attention is being drawn to testing.

Automationengineers and product engineers are being drawn closer together. The importanceand relevance of end to end automated tests is becoming more of a reality forproduct developers as they become run regularly as a post deploy verificationtests.

Whilecode is being thrown at testing with newfound passion, exploratory testers needto be prepared for their time in the spotlight. If automated tests are runningall the time what tests are you doing? What makes them special? Why are theyrequired before we can deploy?

What do you do different than Selenium?

Beforeyou dismiss that question, pause and consider what all selenium and the testsit drives truly provide.

Seleniumis cross browser, scalable, schedulable, repeatable, and can provide logs andscreenshots of its tests. Can you compete with that?

Flaky,slow, and wrong sometimes, you say? And you and all other humans aren’t?

Conversationsthat try to explain using people instead of machines for testing software canend up justifying them under the guise of thereare some tests that aren’t good candidates for automation .

Don’t settle for executing tests that don’t have enough ROIto automate.

Thistype of conversation relies on the belief that testing is comprised to twoelements: test case creation and test execution . Perhaps at its most baselevel testing is just that, the value proposition of the role is in the nuancethat a skilled tester can provide to teams through those elements.

Testersdo more than a fill a gap in automated test coverage. The problem is manytesters express this nuance solely through actions.

Developersproduce code, a measurable artifact from their effort. The generation of thatartifact is often backed by computer science, a formalized area of study.Artifacts of effort are reassuring, people like to see the fruits of theirlabors. Tangible results can be measured and quantified, they might even helpestimate future effort.

A tester’s value tends to be recognized at the same rate as relationships develop. Testers get characterized as folk heroes rather than members of a skilled profession. There’s no trade, education, or craft just individuals saving the day periodically.

The Intuition trap

Evenwhen teams do value testing, people are usually primarily concerned with theprogression or regression of testing efforts. Testers comments in stand-ups caneasily turn into a user story status change report:

Passed this ticket, working this ticket, noblockers

Ifyou aren’t taking the opportunity to discuss your actual actions in stand ups,then you may fall prey to the intuition trap. When you talk about your workthis way, sometimes you end up actually working this way.

Open Source Your Mindset

It’sfun being a hero, but there’s no skill in being Superman or Superwoman (I was going to say Wonder Woman, but she actually trained and I think everyonerecognized that. So if you are the teams Wonder Woman then kudos, you haveserious skills but I digress).

Whentesting goes smoothly the result is nothing .Testing isn’t required for things to go right, it just might help prevent athing going wrong.

Opensourcing your mindset means teaching others another way to look at and interact with their code, or the product.This cuts both ways though, it means being willing to accept a pull request ofoutside perspective. Openly integrating new ideas into your testers mindset .

Replace Execution with Communication

. Like acting or athletics there might be a script or game plan to start with but there is a lot going on behind the scenes, adapting to circumstance and being thought up on the fly. It’s intuition driven.

Whenactions, practices and motivations are not being verbalized and expressed tothe team and stakeholders it puts testing on the level of superstitions andtoken time sacrifices to the quality gods.

Testersneed to pry open this experience, expose the internals driving motivations.Breaking out of that shell grants visibility into a testers actions andmotivations to provide a context for team members to attribute value.

Replace Isolation with Collaboration

Itcan be uncomfortable or difficult to open up and share what goes through your testing mindset .

Theadditional ownership developers have taken in DevOps has changed relationshipdynamics. What was at one point a contentious us vs. them relationship is muchmore open. There just isn’t the time to have as much formal process drivencommunication. Success at scale requires collaboration, pairing and mobbingmore.

When with developers, it’s an opportunity to test at their desk, against their local changes. There are risks testing here, the developers local setup is definitely different than production but so are the testing and staging environments. Let go of the taboo of it works on my machine and consider what can be tested here and what needs to be covered on a different stage. Take the opportunity to work together when the code is the freshest in the developers mind and where they are most comfortable.

It’s not only about working with developers. Mob testing with UX and Product alongside developers is a way to draw the entire team together. When the team is together seeing how the application is working it bypasses all the communication gates. There is no need for a separate follow-up conversations to loop someone in, that person can see and feel how things are working by being a part of the group experience.

Replace Tedium with Stratagem

Teamsare more open to testing and sharing in quality than ever. Testers have beenbuilding quality mindsets for years. Developers are building tooling but teamsare looking for guidance and mentorship in Quality .

Testing isn’t just button pressing, by showing that through communication and working together we elevate our status. Draw people together, into the quality circle, and enable the team to be more comfortable and aware of testing. We can set working agreements, talk about overarching strategy for how testing throughout the development cycle builds quality in. We can embrace automated testing as a team activity where unit, integration and end to end coverage is part of a teams strategy for mitigating risk. We can forgo complete control of testing and opt into being a voice of authority and expertise in quality.

Brendan Connolly is an experienced Software Tester, Automator, Speaker, and Blogger. Currently he is a Senior Quality Engineer at Procore Technologies in Santa Barbara, California. He maintains a testing blog @ http://www.brendanconnolly.net

Testers, DevOps, and Enabling Digital Transformation

By Gerie Owen

To be successful in today’s businessenvironment, virtually all organizations must plan for digital transformation,not only to thrive but to survive. Often, when organizations embark on digitaltransformation initiatives, they turn to DevOps to increase business value andimprove customer experience.

Testers, as champions of the customer andguardians of their organizations’ digital reputations, are uniquely positionedto enable digital transformation. Weenable digital transformation by focusing testing on business risk and customervalue as well as transforming our test processes.

DevOps enablesDigital Transformation. The synergy is clear: digital transformation and the principlesof DevOps are closely aligned. Specifically,continuous integration and continuous delivery enable innovation in operations,streamline processes and eliminate bottlenecks.

Continuoustesting plays a key role in the continuous delivery pipeline by enabling bothhigh quality and high velocity and ensuring customer happiness. Let’s examinehow continuous delivery enables digital transformations by streamliningprocesses and focusing on value to the customers and how testers can enable andchampion this process.

DevOps Principles Aligned with Digital Transformation

Digital transformationcan be defined in many ways; however, it is a fundamental shift in the way anorganization approaches its business and operational processes by usingtechnology to increase focus on and improve customer experience. Most digital transformations are focused oncustomer experience, operational processes and business models. DevOps principles described in the ThreeWays — systems thinking, amplifying feedback loops and creating a culturalof experimentation and continual learning — can be the basis for creating the majorchanges that are required for digital transformation.

Systemsthinking or focusing on the performance of an entire process is the mosteffective approach to transforming business models and operationalprocesses. If changes are made tofundamental operations in siloed initiatives, they will likely fail. Amplifying feedback loops is critical todigital transformation so that corrections can be made early and often. And creating a culture of continuousexperimentation and learning encourages innovation without fear of consequencesif a new business, process or technological approach fails.

TestersEnabling Digital Transformation by Transforming Test Processes

Testers enable digital transformation not onlyby championing the customer, but also by transforming their testpractices. Transformational testpractices include implementing the practices of continuous testing, including“shift left”, test optimization, as well as the use of service virtualizationand advanced analytics, and “shift right”, i.e., testing in production.

Test Automation Pyramid, from More Agile Testing by Janet Gregory and Lisa Crispin

Continuoustesting is an approach to managing risk by focusing not only on eliminatingtesting bottlenecks but more importantly, increasing the effectiveness of ourtest processes. The test pyramid is a“shift left” pattern whereby most of the automated test cases are concentratedin the unit, API, component and integration layers with fewer automated testcases used to test the GUI layer.

Testersas Champions of the Customer

As testers,meeting the challenges of our role in digital transformation requires acoordinated approach made up of several components. Customers expect a seamless experience acrossall channels; however, they do not necessarily expect to perform the same actionson every channel. Therefore, a digitaltransformation test strategy will require using customer behavior analytics tounderstand customer’s browsing, purchasing and other transactions acrosschannels in order to test efficiently and effectively.

As champions ofthe customers, we must focus on testing the customer experience. This means we must learn to use customeranalytics, personas and customer journeys and combine the results of those withexploratory testing.

Testers and DevOps can be the keys to successful digitaltransformation through by focusing on technological transformation based onbusiness value and customer happiness. Testers, this is your opportunity to goabove and beyond and contribute to your organizations in innovative,customer-centric ways.

Gerie Owen is a Testing Strategist and Evangelist. She is a Certified Scrum Master, Conference Presenter and Author on technology and testing topics. She enjoys mentoring new QA Leads and brings a cohesive team approach to testing. Gerie is the author of many articles on technology including Agile and DevOps topics. She recently developed a curriculum for DevOps 101 training. Gerie chooses her presentation topics based on her experiences in technology, what she has learned from them and what she would like to do to improve them. Gerie can be reached through her website, www.gerieowen.com , her blog, Testing in the Trenches, is at https://testinggirl.wordpress.com/ and @GerieOwen on Twitter and on LinkedIn .