Testers, DevOps, and Enabling Digital Transformation

By Gerie Owen

To be successful in today’s businessenvironment, virtually all organizations must plan for digital transformation,not only to thrive but to survive. Often, when organizations embark on digitaltransformation initiatives, they turn to DevOps to increase business value andimprove customer experience.

Testers, as champions of the customer andguardians of their organizations’ digital reputations, are uniquely positionedto enable digital transformation. Weenable digital transformation by focusing testing on business risk and customervalue as well as transforming our test processes.

DevOps enablesDigital Transformation. The synergy is clear: digital transformation and the principlesof DevOps are closely aligned. Specifically,continuous integration and continuous delivery enable innovation in operations,streamline processes and eliminate bottlenecks.

Continuoustesting plays a key role in the continuous delivery pipeline by enabling bothhigh quality and high velocity and ensuring customer happiness. Let’s examinehow continuous delivery enables digital transformations by streamliningprocesses and focusing on value to the customers and how testers can enable andchampion this process.

DevOps Principles Aligned with Digital Transformation

Digital transformationcan be defined in many ways; however, it is a fundamental shift in the way anorganization approaches its business and operational processes by usingtechnology to increase focus on and improve customer experience. Most digital transformations are focused oncustomer experience, operational processes and business models. DevOps principles described in the ThreeWays — systems thinking, amplifying feedback loops and creating a culturalof experimentation and continual learning — can be the basis for creating the majorchanges that are required for digital transformation.

Systemsthinking or focusing on the performance of an entire process is the mosteffective approach to transforming business models and operationalprocesses. If changes are made tofundamental operations in siloed initiatives, they will likely fail. Amplifying feedback loops is critical todigital transformation so that corrections can be made early and often. And creating a culture of continuousexperimentation and learning encourages innovation without fear of consequencesif a new business, process or technological approach fails.

TestersEnabling Digital Transformation by Transforming Test Processes

Testers enable digital transformation not onlyby championing the customer, but also by transforming their testpractices. Transformational testpractices include implementing the practices of continuous testing, including“shift left”, test optimization, as well as the use of service virtualizationand advanced analytics, and “shift right”, i.e., testing in production.

Test Automation Pyramid, from More Agile Testing by Janet Gregory and Lisa Crispin

Continuoustesting is an approach to managing risk by focusing not only on eliminatingtesting bottlenecks but more importantly, increasing the effectiveness of ourtest processes. The test pyramid is a“shift left” pattern whereby most of the automated test cases are concentratedin the unit, API, component and integration layers with fewer automated testcases used to test the GUI layer.

Testersas Champions of the Customer

As testers,meeting the challenges of our role in digital transformation requires acoordinated approach made up of several components. Customers expect a seamless experience acrossall channels; however, they do not necessarily expect to perform the same actionson every channel. Therefore, a digitaltransformation test strategy will require using customer behavior analytics tounderstand customer’s browsing, purchasing and other transactions acrosschannels in order to test efficiently and effectively.

As champions ofthe customers, we must focus on testing the customer experience. This means we must learn to use customeranalytics, personas and customer journeys and combine the results of those withexploratory testing.

Testers and DevOps can be the keys to successful digitaltransformation through by focusing on technological transformation based onbusiness value and customer happiness. Testers, this is your opportunity to goabove and beyond and contribute to your organizations in innovative,customer-centric ways.

Gerie Owen is a Testing Strategist and Evangelist. She is a Certified Scrum Master, Conference Presenter and Author on technology and testing topics. She enjoys mentoring new QA Leads and brings a cohesive team approach to testing. Gerie is the author of many articles on technology including Agile and DevOps topics. She recently developed a curriculum for DevOps 101 training. Gerie chooses her presentation topics based on her experiences in technology, what she has learned from them and what she would like to do to improve them. Gerie can be reached through her website, www.gerieowen.com , her blog, Testing in the Trenches, is at https://testinggirl.wordpress.com/ and @GerieOwen on Twitter and on LinkedIn .

Chaos Engineering: Rethinking testing in the era of the Cloud and DevOps – by Peter Varhol

A lot hasbeen written about testing in the cloud over the past few years. Probably the biggest testing revelation hasbeen to “shift-left”, which means to perform more testing activities earlier inthe lifecycle.

I disagreethat shift-left is either new or radical. Shifting left is something that testers should have done well before thecloud, and while important, it’s not a panacea. Instead, I think testers need to shift right; that is, to test intoproduction. The first time I wrote aboutthis, about three years ago, I got serious pushback from testers who wereaghast at testing live software. Today,in the era of DevOps, there seems to be grudging acceptance that under certaincircumstances, it is okay to do some production testing.

It’s morethan okay; it’s essential. And I’d liketo take that several steps further, into a realm known as chaos engineering. Chaos engineering was a practice originally developedby Netflix to ensure that its movies would continue to stream in the event offailures in the application or cloud infrastructure. Streaming certainly occurs in real time, andthe loss of streaming features wreaks havoc upon both Netflix and its payingcustomers.

So Netflixstarted breaking things in production. This includes simulating DDoS attacks, bringing down servers, segments,and even entire data centers, or injecting faults into running applications. In many cases, these failures did bring downthose applications, so Netflix engineers built in redundancy and fixed somesingle points of failure.

You Should Be Practicing ChaosEngineering

Chaosengineering is about application resiliency; that is, how well it can withstandadverse circumstances. That’s arelatively small part of testing overall, but vital when viewed from thestandpoint of cloud computing. The goalis to ensure that anything that can go wrong in production is tested andevaluated, usually before application deployment but sometimes in productiontoo.

Thecomputing world is a different place than a decade ago. Most applications are Web-based, and/ordelivered from the cloud. They includemany third-party components, such as open source libraries, purchased code, andthird-party services such as advertisements and services. And they frequently change, as projectupdates add new features or address issues, sometimes even daily.

Add to thiscomplex collection of software the fact that network services remain relativelyfragile, especially in comparison to the internal data center. Data from dozens or hundreds of servers andmany users travels for thousands of miles, across an almost infinite number ofpossible routes, following the DNS address tables.

In a highlydistributed application, with code running on multiple data centers,virtualized systems, and data coming from perhaps dozens of sources, I defy anypre-production testing strategy to realistically test many failure scenarios.

Chaosengineering works from a defined engineering process to define the normalbehavior of a system, developing an experimental scenario (such as shuttingdown a server or breaking a network connection), then carrying out thatscenario and comparing the resulting behavior to the normal behavior. Has performance changed? Have we lost availability? Is the application accessible from differentparts of the world? Does the applicationstill work, but is lacking certain essential features?

Testing in Production – and TestingProduction!

Chaos engineering is vital in cloud computing and in DevOpspractices. Testing outside of productionsimply gives you no insight into the quality of the application when it counts. And remember that an important elementof traditional testing is to try to break the application. While we don’t want to deprive the enterpriseor users of the application services, a planned and careful application ofchaos can test the application while greatly limiting fallout if there is afailure.

Also, notice that chaos engineering means more than testingapplication quality. It also looks atthe entire production environment, including data center issues, operatingsystem instances, hardware, load balancing, network outages, and DNS. In my mind, this is an important expansion ofresponsibility for testers, and one they should embrace.

Testers are used to focusing on the application and its code in a vacuum. That approach is changing, but not fast enough. Testers need to look at complex interactions between code, services, systems, network, and provider. Future tests have to include the ability to take all of these factors into account, even breaking some of them through the practice of chaos engineering.

Peter Varhol is a well-known writer and speaker on software and technology topics, having authored dozens of articles and spoken at a number of industry conferences and webcasts. He has advanced degrees in computer science, applied mathematics, and psychology, and is Managing Director at Technology Strategy Research, consulting with companies on software development, testing, and machine learning. His past roles include technology journalist, software product manager, software developer, and university professor.

The QA in the DevOps world

By Theresa Neate

Where you see “DevOps” being practised but the QAs (or Testers) are still relegated to the “checking things corner” and not explicitly and proactively involved in providing input into both applications and infrastructure code, are we perhaps doing the very opposite of what DevOps was meant to be?

Beware the cargo cult

Most of us know the tale of how DevOps was conceived and I trust most of us have read The Phoenix Project (Kim et al, 2013) as one of our first DevOps books.

Therein we learned that DevOps is meant to be a collaborative and efficient partnership between all disciplines to achieve a common goal, which ultimately means: working software in a production system, delivering value to customers.

Where apparent ‘DevOps’ excludes other team members, like Security or QA, we have missed the point. DevOps is meant to be development teams working with Operations teams , not just individuals in developer and operations roles working together.

The QA role is considered to be a member of the Development team. The QA is therefore intrinsically a member of DevOps.

Where we find that QAs have no input into the Operations or Continuous Delivery conversations because they “are not developers” but we think we are doing DevOps, we are in fact practising a cargo cult — i.e. an imitation and approximation of the real thing because the real reasons of WHY have not been fully grasped.

If we are going to do DevOps, it behooves us to know WHY we do it, and not only HOW.

Waste

‘Agile’ and ‘DevOps’ fundamentally assist with reducing waste. (I generally prefer not to use these capitalised nouns as they have become overloaded , so use them here cautiously).

Project churn, conflict, friction, slow response, unreliable and unstable systems, needless expenses, lengthy handovers, defects, time delays, failed projects, manual overheads, double-handling and many similar issues, are all waste.

Every time a feature is double-handled (such as the churn when a bug is discovered later), we have incurred waste. Feedback should be sought earlier rather than later so that one may course-correct or remedy issues as early as possible and prevent waste.

QA is feedback

QAs do not provide assurance; they help provide and define analysis and feedback.

Monitoring is a form of feedback of the system’s behaviour.

Just like monitoring is feedback, so is testing. Whether the testing is manual exploratory testing or automated checks, it is feedback.

Testing tells you about components and relationships; monitoring tells you about the system.

To minimise and reduce waste, you want to receive that feedback as early as possible, and that the feedback occurs in meaningful levels and depths in your system.

Early feedback

QA (testing and monitoring) should be done early and continuously. Receiving early feedback enables one to act on that information and improve or course-correct as early as possible.

Your QAs can and should be involved in these conversations – and if they cannot, then as their manager or peer the onus is on you to empower them to become involved.

The days of QAs telling you if you’re OK to ‘go live’, are numbered. The whole team should know that, based on the continuous feedback they have been receiving.

Lean QA

The reduction of waste (rework, defects, friction) by holistically testing the whole system, early and continuously (and into production), including measuring what really matters & incrementally improving thereupon, and using your humans intelligently in doing so, is what I call “Lean QA”.

And what I consider essential to DevOps.

DevTestOps: It’s alive, but what can testers do to make it alive AND kicking into overdrive!

By Toyer Mamojee

The journey of the ‘typical tester’ has been a pretty bumpy ride over the past few years, more so recently with the massive uptake of DevOps. Apart from the many new and intimidating terms thrown our way, we are now required to do anything from digging in to business case details to monitoring how applications are performing in Production. It’s become a world which is very different to the one a tester would have experienced around the year 2005 or so.This world has incorporated testing to be at the heart of it, and not just testing but full on quality! So its only natural for the tester to play a massive role here.

But what can we do?

As mentioned, the tester should be at the very heart of every activity happening in the software delivery process. Below I highlight some of the areas where testers can directly influence and should influence if the process is to go from just being alive, to becoming Alive and Kicking!

Help Sculpt a Team Culture (Team player)

When one speaks about DevOps, technical aspects are usually at the front of the conversation whereas the softer aspects like Team culture and communication are very rarely mentioned.

Everyone loves quality products, testers love quality. A good healthy, dedicated team can help make better quality products. So naturally a tester should be one of the key individuals to drive a dedicated team. Testers should keep their critical thinking quality as their most prized asset by asking the right question at key moments during team discussions/meetings and interactions. This gets the team thinking and hopefully to a point where this starts to become second nature to all team members. Quality should be a driving force of every team member in everything they do, this should be a non-negotiable principle.

Another key area around team culture is by re-iterating the culture of ‘safe to fail’ and the ability to recover quickly. Without these important concepts inherent in a team the ability to deliver faster might prove to be a huge challenge.

Promote Visibility and Transparency (Seeing is believing)

I cannot stress how key this aspect is to a solid DevTestOps model. Gone are the days of working in isolation. A tester should look at some of the key artefacts that traditionally would fall within his/her space and how these can be made visible and transparent to the entire team.

Some of these include Test coverage visibility, defect statuses per sprint and release, test automation coverage, test automation scripts (functional and non-functional). One can make these visible in the form of dashboards or even repositories for test checklist or automation scripts.

At any point in time any team member should be able to give you the current status of your testing per sprint and/or release. After all if it is a difficult and an intensive process to decide on whether you are ready to deploy to production, then you have to question what you as the tester can do to change this.

Increase your skillset (Up your game)

Learn to use tools that help you work better — Learn to use tools that help you contribute more (Photo by on

A modern tester needs a modern armoury. Without it, I battle to see how the transition to the DevTestOps model can be easily made.There is so much to mention under this area, up-skilling should span these key areas:

Automation tools

From backend to front end focused automation tools, a tester should not only know these tools but know when and where to use them. Tools learned should include those which at least cover API testing, UI Testing, Performance/Load Testing, Security and Databases.

Context switching between these tools and types of testing is a rather intimidating thought for those not usually exposed to modern ways of software delivery. My advice here is to take baby steps in learning these tools in order to not just learn something for the sake of it, but rather for the cause of true DevTestOps.

Technical Tools

Technical type tools that equip a tester to contribute effectively in a DevOps environment would once again span the entire spectrum of the application solution. Tools like Postman & Swagger for API testing, knowing your way around version control applications (GitHub, GitLab etc), SQL and NoSql interfaces (like DBKoda, Sql Server etc), CI/CD apps (TravisCI, GoCD, Jenkins etc) , Logging applications (GrayLog, ElasticSearch,ElkStack etc) and Monitoring tools (NewRelic, Dynatrace, Instana etc).

Contribute to your CI/CD (its in the pipeline)

Testers have a crucial part to play in defining the ideal pipeline for your applications deployment. Being part of discussions around composition of the pipeline, ideal duration of pipelines, pre and post deployment tests as part of the pipeline.
A point to consider even further is where do your automation tests fit in these pipelines? Should it be part of it?
CI/CD tools also promote to the point on visibility and transparency, if constructed properly the entire team should know exactly at what stage the deployment is at, what tests were run and the overall status of the deployments.

Monitor it (What’s trending?)

To know what your application is doing both in PROD and Test environments, contributes to how you approach your testing. Findings and trends even contribute to what scenarios you end up testing and has massive input regarding determining what load you put your system under during Performance testing as an example.
Traditionally testers would steer clear of any tool or application that falls outside the scope of testing, but in a fast paced delivery world, monitoring tools can be a testers best friend. A tester’s system or application knowledge should also push the tester to not just use existing monitoring metrics but to also determine what to include as part of monitoring.

In summary

The evolution into DevTestOps has meant that the tester now plays a much more involved role in the entire delivery process. The day to day tasks of a tester spans between a balance of soft skills, analytical, technical and even an investigative mindset sometimes seamlessly crossing between these tasks without even knowing. By following some of the pointers above the initial ground work to successful DevTestOps is well underway. Go on Testers, get DevTestOps Alive and Kicking in full swing!

Toyer Mamoojee is a ‘die-hard’ passionate Agile Automation Test Analyst with almost 16 years testing experience across many industries. From Waterfall to Agile approaches, Commercial to Open Source Automation tools and Manual to Automation Testing he has been through the wars of Software Delivery. He is the co-founder of Cape Town Automation User group and co-organizer of Cape Town Testing Meetup back in his home country of South Africa. In his spare time he loves exploring the great outdoors, taking on the world playing online video games , major football fan, adopting the title of technological husband and playing super dad to 2 pre-schoolers!

Toyer Mamoojee @tottiLfc