A place to share ideas, techniques, resources about all the testing activities needed to feel confident about delivering value to our customers frequently and sustainably.
We’re adding webinars to our Testing in DevOps community learning opportunities! Our first one is a panel discussion with awesome leading practitioners , , , and I will also join! Many thanks to
for hosting the webinar (they host this site as well).
So what will we talk about? As many of our guest posts here have pointed out, DevOps is about
the whole team
– including operations specialists – working together to deliver value to customers frequently, without drama or stress. It’s that continual loop: together we identify the most valuable capability to delivery next.
In the webinar, we’ll talk about how we build and test it, test and build it, together. We have created a pipeline to deploy it to production, together. We’ll probably get into topics such as monitoring and observability to learn how customers use the product and respond to problems, and get ideas for the next valuable feature slice to build. We will answer your questions!
For more details and to register, please go to my
(I am too lazy to copy it all over here!)
We welcome your questions in advance, just tweet them to @! Or, simply ask on the day in the Zoom chat. We look forward to an insightful and fun discussion!
The different pages of this website have links to so many helpful articles, podcasts, videos, webinars, books and more. Here are three that you might want to prioritize to read or listen to first, to build your understanding of devOps, continuous delivery, testing in production, and what makes software delivery teams successful.
Charity Majors on testing in production
Does the phrase “testing in production” scare you? Historically, it was an irresponsible practice – throw our changes into production and let the customers tell us what’s wrong. Today, it is a leading practice. In her recent article “
I test in prod
“, says it’s a superpower!
Most interesting man in the world
Charity points out that we test in production all the time. The key is to do it safely. It takes an investment in tooling and training. This is especially important with the complicated distributed apps we support today. We need to test more than the code – we need to test the whole system, including the environment, the infrastructure, and the customers.
There’s so much to learn in this one article, from Charity’s experiences and insights. She breaks down the three aspects needed to start testing in production in a useful and beneficial way: technical, cultural and managerial. Plus, she works in so many classic memes to hilarious effect! Give it a read and start overcoming the fear of what will happen with each new change released to production.
Theresa Neate: Stop the whisper campaign that devOps means no testers needed!
We’ve been fortunate to have a
guest post
here from , one of the leading testing practitioner who is also a leader in devOps practices. She gives us a preview of her
TestBash Australia
talk in this
Ministry of Testing podcast
with Michael Lang.
Theresa starts with one of the best explanations of devOps and continuous delivery that I have heard. If you are not yet familiar with these terms, listen to the first few minutes of this podcast.
One of the most important messages Theresa delivers here is that we need to stop the “whisper campaign” that devOps means “no testers needed” and “no testers welcome here”. We know we testers can add huge value by participating in devOps practices and helping our teams succeed with continuous delivery. it’s up to us to use our natural curiosity, learn important new skills, and show what we can do.
Photo from Christina Morillo on Pexels.com
This podcast is less than 30 minutes long, and it’s going to be the best half hour you’ve invested in your career in awhile.
I wish I could go to TestBash Australia and participate in Theresa’s workshop “Hands-on Intro to AWS and a Taste of DevOps, for Testers”!
Accelerate: State of DevOps 2019
This led by is building on 6 years of data from a wide group of respondents. The executive summary alone is a gold mine of information about what practices and capabilities correlate with team success.
From the
The report includes insights into testing practices. For example, though previous years found that test automation has a significant impact on continuous delivery (CD), this year they found that automated testing positively impacts continuous integration (CI).
The ability to reproduce and fix failures, gather feedback from tests, improve test quality and iterate test runs quickly also ties into automated testing.
The report also focuses on disaster recovery testing. One finding is that organizations who do disaster recovery testing and create and implement action items based on what they learn from it are more likely to be in the elite performing group.
The report delves into the “Four Key Metrics” of software development and delivery, one of which is culture – too often overlooked by organizations. To improve, organizations need to start with the foundations, and adopt a continuous improvement mindset.
Read the report and help your own team be one of the many teams who are continuing to improve their ability to deploy small changes to production frequently, while enjoying their work.
We’ve embarked on work to move off our legacy monolithic Oracle Commerce platform and are creating a
microservice
-based architecture hosted in . As part of this we have created a Platform (
John Lewis
Digital Platform, JLDP) which enables teams to get started quickly through providing key capabilities that are needed by all teams e.g. Google Cloud environment, code repository, path to production, telemetry. We call this our “
”. We started out on this journey as it was observed that we were not delivering business value quickly enough and technology was believed to be a key impediment to doing this.
One of the key initiatives we undertook as part of building the platform was being able to measure the progress of teams working on the platform through their approach to
Continuous Delivery
. Previously we had lacked this information and could not make data-driven decisions about our service/application health. We wanted to avoid this and build it in to the platform from the start.
We started capturing much of the data through spreadsheets based on data visible in Google’s logs. We’ve iterated on this and have developed an application which we call the “Service Catalogue”, that brings together the various metrics we’ve collected on health of a Service (one or more microservices to achieve a particular outcome), along with key information on each service/application e.g. Search.
Lead Time to Value
Our initial focus was to show that access to the Platform was no longer a constraint in getting a service live to customer. We wanted to capture some information that proved this was no longer the constraint. We were able to quickly show our
service creation lead time
took hours not months as it had been previously.
We then looked at capturing the lead time of the first service deployment to production, we called this our
onboarding lead time
. For the platform this evidenced that teams had managed to navigate our paved road and deployed at least a “hello world” to production. This confirmation also reduced the platforms reactive work load and was a key message to teams when onboarding them, get to production early.
The final addition was to measure of
lead time to first value
. This was our final measure of success and confirmed the team had made their service live to customers.
JLDP v2 Services – Lead to value
Deployment Indicators
Once a team had deployed and were live to customer, we wanted to continue to understand their continuous delivery on the platform. We set out to capture more information:
deployment intervals
,
deployment lead time
and
deployment throughput.
We have again used systems data to capture this information, looking at git commit history through to container deployment to derive the data. We are looking to observe teams’ approach to agile principles and lean methodology through their continuous delivery practice.
The
deployment interval
captured is the time between deployments to our production cluster. We want to see a low interval and consistency which suggests a team is then deploying frequently and continuing to work on the Service.
The
deployment lead time
is then how quickly a code change gets through the pipeline to production. This tells us how long it takes a team to get a change live and again looks for consistency and a short time from commit to production. This tells us that teams can get change through to production quickly through their pipeline and their ways of working.
The
deployment lead time
is then how quickly a code change gets through the pipeline to production. This tells us how long it takes a team to get a change live and again looks for consistency and a short time from commit to production. This tells us that teams can get change through to production quickly through their pipeline and their ways of working.
Delivery Indicators (Interval Time, Lead Time)
Deployment throughput
then helps us confirm overall deployment volumes across the platform helping us to evidence our overall deployment trends within teams and services. We have the data for this but have yet to fully integrate this in to our Service Catalogue, though we can present it on a on-demand basis.
Live to Customer — Monthly Service Deployments
What did we learn?
These data points were then critical for our initial conversations with teams which we called platform research. We used the data to talk to teams that were taking time to get to their first customer and understand what might be constraints in either the platform or the wider delivery process. We are also trying to build a product not just a platform and so the capabilities we had will be informed through the customers (service teams) running on our platform.
We learnt quickly that we’d started to unblock some key constraints.
The access to the platform has reduced the provision of infrastructure to teams from months to hours.
Our overall deployment throughput was increasing substantially from 10’s of deployments a years to 1000’s.
Total deployment throughout the yearContinuous Delivery patterns (Lead Time: Blue, Interval Time: Red) – so much to read in these graphs
Through further ongoing research we learnt that there were common service constraints
Teams were struggling with the security process which was designed for large waterfall projects and hadn’t caught up with a more agile, microservice-based architecture.
Teams had too much work in progress. We had teams which were requesting multiple services without getting their first service live to customer.
Teams had external dependencies, usually around integrations outside the platform ingesting data from other external systems.
Teams were attempting to get to feature parity before going live to customer.
Teams were still attempting to undertake end-to-end functional or performance testing with external dependencies.
Change freeze still plays a part in reducing consistency and increasing batch size.
These were all things we could now start initiatives on to improve lead time. For example, we have created a Platform Security team to help engagement with teams in to our current security process with a goal of changing and developing a pattern more consistent with our delivery approach.
It also helped us inform the direction for the platform. We increment the version of our platform quarterly (although deploy daily). Each increment we aim to delivery new capabilities such as improved service creation, monitoring and logging capabilities and resilience, all focused on building a product that helps the teams running on our platform go faster or improves their stability.
What Next?
So far we have focused on measuring our continuous delivery indicators to show throughput to production. Operability is the next key focus. Teams are now starting to support their services, and the traditional operational structures which supported many of the operability aspects are now transferring their knowledge on processes to teams.
Continue to develop the platform with operability as a focal point. We have already provided teams with indicators of health e.g.
based on Operability patterns outlined in .
We want to combine this with alerting focused around and
.
We will look to expand our Service Catalogue to include
deployment stability indicators. Deployment Failure Rate
(number of failed production deployments and duration between),
Recovery time from failure
(time taken to recover the service after a failure).
The aim is to combine our continuous delivery and operability metrics together in one place. The future will then be tying this data together with business data to truly show how deployment frequency relates to value. The platform and being able to evolve it as a product will continue based on our ability to evidence and communicate the value it can add.
Rob Hornby is a Product Engineer working for John Lewis based in London. He has undertaken a number of roles, over the last 20 years, within Testing, Release, Platform and Environments.
Rob is a keen believer in being able to treat all things as Products and be able to evidence the value generated by that team. Ideally everything would be delivered by small teams, delivering continuously and iterating on value. However not all teams start at this point, not all teams are the same or born equal. This is where the challenge and enjoyment comes from.
Outside of work, Rob is a keen runner, father of an amazing young girl, Sofia and works on a number of local community events.
