Keeping Up with Compliance in a DevOps World

By Areti Panou

About 5years ago I was working in a small product that was doing (almost) everythingthe DevOps way. We had the works: code and tests versioned in git, trunk baseddevelopment, building the code once (more or less), feature toggles, dailydeployments, the ability to fix issues fast, monitoring and alerting of ourproduction system.

But therewas one tiny problem.

We stillneeded to adhere to the company’s Quality Management System (QMS) whichincluded compliance to ISO 9001 certification as well as other regulatoryrequirements. While development was adapting to the new world in a fast pace,the same could not be said for the company’s processes at that time.

So hereare some things we did to address the situation.

Identify the “must haves”

Yes, the QMS consisted of a huge list of requirements but our common sense told us that they didn’t all have the same weight and importance. Minor performance issues didn’t seem as serious as upholding security and licensing requirements that could prompt legal ramifications. With the help of our company’s QMS owners, we understood the severity of each requirement and created 2 lists: One with all the requirements that had to be fulfilled with every deployment and one with the ones that had to be fulfilled for the release of new functionality. For example we were scanning for any OWASP Top Ten vulnerabilities before each deployment but running and documenting performance tests on a new feature only before its release to customers. Respecting the requirements on these 2 lists meant that our product and the company were safe from any wrongdoing.

Customise to your product’s needs

Once weknew what was important for the company, we needed to implement therequirements’ checks in a manner, that they indeed improved the quality of ourproduct. Instead of blindly using a recommended tool, we checked if there werealso other options that would better support our technology stack. We even cameup with our own tools , as wedid for example for complying with the traceability requirements. When weneeded to only take care of priority 1 issues for breaking a certainrequirement, sometimes we would decide to not allow priority 2 issues getthrough either. We evaluated which of the non-essential requirements would be beneficialfor us and added their checking to our to-do list as well. Our goal was to takecare of the regulations, not because the company imposed it on us, but insteadbecause we saw true value in them.

Add requirements’ checks in the pipeline

What naturallycame next, was to automate as many of the requirements’ checks as possible andinclude them in our deployment pipeline. The priority was to have the onesnecessary for deployment running right away once something was pushed by thedevelopers to the main trunk, while requirements’ checks needed for releasing newfunctionality run daily. Our aim was to get feedback on the status of corporaterequirements fast, since any violation could either block us from deliveringsoftware or take a great amount of time to resolve as QMS representativesneeded to get involved. Feeling confident about compliance gave us more time tofocus on additional activities that would further improve the quality of ourproduct.

Share with the company

The lastaction we took was to keep an open channel of communication with the QMSowners. We showed them our implementation of requirements’ checks and got theirfeedback whether they were enough to ensure compliance. We adopted theirsuggestions and equally they understood and took note of what our pain pointswere. What I found to be the most surprising thing, is that they were respondingpretty positively to our proposals and they were eager to learn more about howthings could be even better. Because the truth of the matter was that for along time the QMS has been used as the bogeyman to do things right, so nobodywas eager to talk to its owners, out of fear they were doing something wrong.

Of coursenot everything was ideal and sometimes we needed to debate about significantpoints, for example regarding the separation of responsibilities. How candevelopers test their own code? But once we discussed about code reviews orpair programming and the multiple layers of testing taking place before anycode ended up in production, their concerns were at least alleviated.

In the spring of 2018 some of theleading figures in the DevOps community wrote a love letter to auditors, titled “Dear Auditor” , anattempt to apologise for leaving them behind and not adequately explaining howa product can be compliant in this fast-paced world. The letter includes a where audit concerns are listed along with methods to address them.

Looking back now I realise that we treated compliance requirements like any other Agile project: identified the essentials, refined them, implemented them and asked for feedback from our stakeholders, the QMS owners. And from their side they saw the need to adapt to the times and facilitate us by revisiting policies and processes. In the end, the biggest lesson we learned was that only through an honest conversation between development and internal auditors were we able to accomplish what we both wanted: deliver quality software to our customers fast.

About Areti:

Amathematician by vocation, a software tester by profession, Areti Panou is nowworking as a cloud quality coach. Her job is to help development teams withinSAP come up with test and quality strategies that best fit their needs.Before that, she was the sole tester in one of the first products at SAP, toput a full Continuous Delivery approach into action and the first one in SAPhistory to daily deliver to its customers.

She blogs about questions that occupy her mindregarding software development, testing, and quality and shares random thoughtson Twitter as .

Anticipating Problems with Infrastructure as Code in DevOps

By Nilanjan Bhattacharya

There are different interpretations of theterm, ‘DevOps’. However, most peoplewill endorse the idea – ‘DevOps also focuses on the organization change tosupport great collaboration between the many functions involved’.

Despite the focus on collaboration andorganization change, a critical part of DevOps is the enabling technology. There are different aspects of thetechnology. Continuous deliveryautomates the entire process of delivery, including building the software,validating the build and progressing through other steps in the form of apipeline. A much more elaborate part ofthe technology is Infrastructure as code (IaC). Infrastructure as code was pioneered by Amazon Web Services (AWS). Thereare newer entrants such as Microsoft Azure and Google Cloud.

The main concept of an IaC service, likeAWS, is to access infrastructure such as servers, databases and networks, asvirtual resources. It’s critical to beable to create and access the infrastructure programmatically . You can then treat the scripts which interactwith the infrastructure as software and use software development principles, todevelop and manage your code/infrastructure, such as source code control andcontinuous delivery (of infrastructure). IaC provides the ability to scale up and scale down components at will,greatly increasing the economy while meeting customer/market demands.

DevOps isoften introduced with examples from Netflix and Facebook. However, the software industry is vast. It includes both software product companieslike Netflix, but also includes enterprise software development in banks,telecom, government. It also includesservice providers. Given the universalavailability of cloud-based infrastructure and the accessibility due to lowcost and the pay-as-you-go model, the size of teams adopting IaC can varywidely. The two-person team creating amobile app, or the government department creating a health portal, is verydifferent from a team in Netflix. Thechallenge for all teams is preventing problems when deploying IaC as part ofDevOps.

How can you anticipate problems?

One way to anticipate problems is to askquestions.

A database as a service, like AWS RDS, isvery appealing both from a technical and business point of view. However, what happens when there is downtime,even for a few seconds, due to routine maintenance? For non-critical applications it may notmatter (you may still want to find out if the application is affected). For critical applications, you may want tocreate a highly available solution.

If you are working on an existingapplication, how do you replace the single instance of the database with ahighly available solution? Are there anymaintenance windows? What is the natureof the business? Is the business seasonal?

Here are some questions we might ask:

For a database as a service, what happens whenthere is downtime, due to maintenance?
Does the business have a maintenance window forreplacing the database?
What is the nature of the business? Can they afford downtime?
Is the business seasonal? Are there peak times when they can’t afforddowntime?

You could generalize the first question as,‘For any cloud-based service, what happens when there is downtime?’ You could also ask, ‘For any cloud-based service,what happens when you need to upgrade the version?’

Here is a question for you to practice (my response given below):

What happens when you need to upgrade thelanguage version (e.g., node.js) of a function as a service?

As an aside: Is this a problem with a cloud-basedservice/Infrastructure as code? Wouldthis have not been a problem if you controlled the server/environment, i.e., with physical servers?

How do you ask questions?

You want to ask questions which are open ended . You could ask, ‘What cancause the storage to become full on an EC2 instance?’ or ‘What could cause theinstance to become unresponsive?’ Theanswer will depend on your application and specific circumstances. (You will probably set up alerts for diskutilization reaching a threshold. However, alerts don’t replace the thinkingabout what might go wrong.)

You want to take asking open endedquestions a step further. What you arereally doing is looking for questionsthat you hadn’t thought about .

The idea of asking questions might seemvery unstructured. You can structure your thinking by using anapproach to generate ideas. One suchapproach is using the mnemonic, SFDiPOT, to structure your questions. SFDiPOTstands for structure, functions, data, interfaces, platform, operations,time. You could use S-structure to thinkabout the different components of the system such as the database, lambdas and instances and how they mightpose a risk. In the case of IaC, itisn’t a physical product, but an entire platform.

You can use another mnemonic CIDTESTD(Customers, Information, Developer relations, Test Team, Equipment & Tools,Schedules, Test Items, Deliverables) to generate ideas. If you are working with an enterpriseapplication like Adobe AEM, do your DevOps engineers have enough knowledge ofthe product? If not, does that representa risk? (We used the letter T, which stands for (Test) Team, to think aboutrisks related to the nature of the team).

When you ask questions, you want to holdyourself and your team accountable andincorporate feedback. Retrospectives andpost-mortems can identify questions that weren’t asked. You also want to identify how you could have generated ideas which would have led tothose questions.

Here is what I would ask when thinkingabout upgrades to the environment (e.g. node.js) for a function as a service,like AWS Lambda:

When we look at the Lambda code a few years fromnow, will it make sense to a new person/team?
Do we have automated tests for the larger functions?
Are there functions used for infrastructure, asopposed to, being used by the application?
Do we have any automated tests for the Lambdafunctions?
Are there differences in the quality of codewritten by the infrastructure team as opposed to the application team?
Will we be able to roll back the functions afterupgrade, in case there are problems?
Is there a development environment for theLambda functions, i.e., does the dev environment include the Lambda functionsespecially the ones used for Infrastructure?
Will the development and production environmentsbe kept in sync (for the next few years)?
Can we track differences in the development andproduction environment?

Here are a few more general questions youcould think about when working with infrastructure as code:

What can cause a component to not respond?
When can an instance not respond?
When can a component run out of capacity, e.g.,storage?
Can you upgrade a component to a highercapacity?
Are higher capacity components always availablein the region that you are working?
Does the region that you are working in, impactyour available options? Does that impactyour choices for increasing capacity in the future?

Beyond questions

There are other activities which cangenerate ideas on risk, for example, stress testing a component or creating arealistic customer environment, including real time data. These activities can also be used to generatequestions. Asking questions is part ofthe much broader activity of critical thinking .

For a start, I recommend beginning with asking questions in a work situation(see example on functions as a service above). You should accompany that by using retrospectives to evaluate questionsthat weren’t asked and identifying the ideas or activities which could have ledto those questions.

The idea of asking questions related to IaCcan form a foundation for thinking about your model for anticipatingproblems/risk in DevOps. It may alsolead the team to make inroads into critical thinking.

Notes

Thedescription of the core of DevOps is from Wikipedia: https://en.wikipedia.org/wiki/DevOps
SFDiPOT and CIDTESTD are part of the Heuristic Test StrategyModel created by James Bach ( )
The idea ofusing questions to identify risk has been used earlier, e.g., ‘a test case is aquestion that you ask of the program’, in ‘What is a Good Test Case?, StareastConference, May 2003,

Nilanjan Bhattacharya

Nilanjan works as a DevOps Delivery coordinator for AC3 in Sydney. AC3 is a multi-cloud service provider and Australia’s leading AWS Premier Consulting Partner and AWS Managed Services Provider.

Nilanjan has a background in software testing and project management. He has worked with a variety of software teams, including software product teams. His experience in analyzing customer support issues and working with successful software products with a multi-year lifecycle, provides a unique perspective on software quality and defects.

Measuring Our Continuous Delivery Progress

by Rob Hornby

We’ve embarked on work to move off our legacy monolithic Oracle Commerce platform and are creating a microservice -based architecture hosted in . As part of this we have created a Platform ( John Lewis Digital Platform, JLDP) which enables teams to get started quickly through providing key capabilities that are needed by all teams e.g. Google Cloud environment, code repository, path to production, telemetry. We call this our “ ”. We started out on this journey as it was observed that we were not delivering business value quickly enough and technology was believed to be a key impediment to doing this.

One of the key initiatives we undertook as part of building the platform was being able to measure the progress of teams working on the platform through their approach to Continuous Delivery . Previously we had lacked this information and could not make data-driven decisions about our service/application health. We wanted to avoid this and build it in to the platform from the start.

We started capturing much of the data through spreadsheets based on data visible in Google’s logs. We’ve iterated on this and have developed an application which we call the “Service Catalogue”, that brings together the various metrics we’ve collected on health of a Service (one or more microservices to achieve a particular outcome), along with key information on each service/application e.g. Search.

Lead Time to Value

Our initial focus was to show that access to the Platform was no longer a constraint in getting a service live to customer. We wanted to capture some information that proved this was no longer the constraint. We were able to quickly show our service creation lead time took hours not months as it had been previously.

We then looked at capturing the lead time of the first service deployment to production, we called this our onboarding lead time . For the platform this evidenced that teams had managed to navigate our paved road and deployed at least a “hello world” to production. This confirmation also reduced the platforms reactive work load and was a key message to teams when onboarding them, get to production early.

The final addition was to measure of lead time to first value . This was our final measure of success and confirmed the team had made their service live to customers.

JLDP v2 Services – Lead to value

Deployment Indicators

Once a team had deployed and were live to customer, we wanted to continue to understand their continuous delivery on the platform. We set out to capture more information: deployment intervals , deployment lead time and deployment throughput. We have again used systems data to capture this information, looking at git commit history through to container deployment to derive the data. We are looking to observe teams’ approach to agile principles and lean methodology through their continuous delivery practice.

The deployment interval captured is the time between deployments to our production cluster. We want to see a low interval and consistency which suggests a team is then deploying frequently and continuing to work on the Service.

The deployment lead time is then how quickly a code change gets through the pipeline to production. This tells us how long it takes a team to get a change live and again looks for consistency and a short time from commit to production. This tells us that teams can get change through to production quickly through their pipeline and their ways of working.

Delivery Indicators (Interval Time, Lead Time)

Deployment throughput then helps us confirm overall deployment volumes across the platform helping us to evidence our overall deployment trends within teams and services. We have the data for this but have yet to fully integrate this in to our Service Catalogue, though we can present it on a on-demand basis.

Live to Customer — Monthly Service Deployments

What did we learn?

These data points were then critical for our initial conversations with teams which we called platform research. We used the data to talk to teams that were taking time to get to their first customer and understand what might be constraints in either the platform or the wider delivery process. We are also trying to build a product not just a platform and so the capabilities we had will be informed through the customers (service teams) running on our platform.

We learnt quickly that we’d started to unblock some key constraints.

The access to the platform has reduced the provision of infrastructure to teams from months to hours.
Our overall deployment throughput was increasing substantially from 10’s of deployments a years to 1000’s.

Total Deployments Throughout the Year — Total deployment throughout the year

Continuous Delivery patterns (Lead Time: Blue, Interval Time: Red) - so much to read into these graphs. — Continuous Delivery patterns (Lead Time: Blue, Interval Time: Red) – so much to read in these graphs

Through further ongoing research we learnt that there were common service constraints

Teams were struggling with the security process which was designed for large waterfall projects and hadn’t caught up with a more agile, microservice-based architecture.
Teams had too much work in progress. We had teams which were requesting multiple services without getting their first service live to customer.
Teams had external dependencies, usually around integrations outside the platform ingesting data from other external systems.
Teams were attempting to get to feature parity before going live to customer.
Teams were still attempting to undertake end-to-end functional or performance testing with external dependencies.
Change freeze still plays a part in reducing consistency and increasing batch size.

These were all things we could now start initiatives on to improve lead time. For example, we have created a Platform Security team to help engagement with teams in to our current security process with a goal of changing and developing a pattern more consistent with our delivery approach.

It also helped us inform the direction for the platform. We increment the version of our platform quarterly (although deploy daily). Each increment we aim to delivery new capabilities such as improved service creation, monitoring and logging capabilities and resilience, all focused on building a product that helps the teams running on our platform go faster or improves their stability.

What Next?

So far we have focused on measuring our continuous delivery indicators to show throughput to production. Operability is the next key focus. Teams are now starting to support their services, and the traditional operational structures which supported many of the operability aspects are now transferring their knowledge on processes to teams.

Continue to develop the platform with operability as a focal point. We have already provided teams with indicators of health e.g. based on Operability patterns outlined in .
We want to combine this with alerting focused around and .
We will look to expand our Service Catalogue to include deployment stability indicators. Deployment Failure Rate (number of failed production deployments and duration between), Recovery time from failure (time taken to recover the service after a failure).

The aim is to combine our continuous delivery and operability metrics together in one place. The future will then be tying this data together with business data to truly show how deployment frequency relates to value. The platform and being able to evolve it as a product will continue based on our ability to evidence and communicate the value it can add.

Reference:

For more information, we recommend reading Measuring Continuous Delivery by Steve Smith ().

Rob Hornby is a Product Engineer working for John Lewis based in London. He has undertaken a number of roles, over the last 20 years, within Testing, Release, Platform and Environments.

Rob is a keen believer in being able to treat all things as Products and be able to evidence the value generated by that team. Ideally everything would be delivered by small teams, delivering continuously and iterating on value. However not all teams start at this point, not all teams are the same or born equal. This is where the challenge and enjoyment comes from.

Outside of work, Rob is a keen runner, father of an amazing young girl, Sofia and works on a number of local community events.

Similarities between SRE and testing: Alert/automation suite maintenance

By Abby Bangser

I want to take a few minutes to share with youhow I have grown to once again realise how common so many of our challenges arein tech. This time the commonality is between the world of test automation andoperational alerting. Before I dive into these details though, let me quicklyexplain my career journey which led me here.

From the beginning of 2012 to the middle of2018 I worked as a test consultant in organisations ranging from small non-profitsthrough to large multinational financial institutions. During this time I sawthe scale, financing, and cultures change fairly drastically, however thechallenges were often quite similar when it came down to the team level ofdelivering software. Most notably, over these years I realised that being aspecialist in quality often meant my biggest contribution was in how we delivercode through Continuous Integration and Delivery processes rather than just how I test things. This led me to working with many DevOps and infrastructure relatedteams and activities and eventually on to joining MOO as a Test Engineer ontheir Platform Engineering team.

Introduction to our platformengineering team

At MOO the Platform Engineering team isresponsible for the infrastructure of both production and pre-productionenvironments as well as the running of internal tooling like source control andbacklog management tooling. As you may expect this includes being on call forthese systems 24 hours a day, 7 days a week. From what I can see around theindustry our team is not particularly unique in how we handle this.

We have a rota which makes sure we share theresponsibility and our team generates both warning and critical alerts to makesure we respond if issues occur. These alerts provide us huge opportunity forspeeding up our Mean Time To Recovery (MTTR) by decreasing our time todiscovery. However when I joined the team, they were also generating a lot ofstress which is why one of my first tasks was to review our alerts for valueand how to remove that stress.

Tackling our alerting challenges

Our alert stress was generated for a few majorreasons. First of all, some of our alerts were things that our MOO team memberscould do nothing about. For example, if our Google authentication is failingthere is an impact to our internal users which could trigger an alert to our oncall engineer, however there is nothing we can do but wait until Google sortsout their incident. Another issue with our alerts was critical alerts (meaningsomeone is woken up in the middle of the night) that resolved themselves withinminutes. Often these were issues with machine health in AWS which has its ownself-healing mechanisms and our thresholds were just too low. Finally, we hadsome alerts that made people wonder if they were worthwhile as they had nevertriggered once in over a year of tracking.

When I joined the team there was a ticket onthe backlog to “Improve team alert fatigue”. This was intentionally a prettyvague topic and was meant to be an analysis placeholder as there wasn’t justone solution to all these concerns with our alerts. Alert fatigue can bedescribed as “exposed to a large number of frequent alarms and consequentlybecomes desensitized to them. Desensitization can lead to longer response timesor to missing important alarms.” and our team was definitely showing the earlysymptoms. After spending some time with each team member we decided thatdefinition of done for this ticket would be to:

Identify the shape andrequirements of a “good” alert
Review our alerts against this definition
Set up future work to bring ourcurrent alerts in line with our desired state

Achieving a working solution byputting it to the test

To achieve these goals I went about puttingtogether a straw man definition of what a warning and a critical alert wouldneed to have. Then scheduled reviews of all of our alerts against thesedefinitions to let them be hardened and evolved by our real context. This was a(very) long process but by grouping alarms together, sharing the load acrossall team members in small groups and doing no more than 30 minute meetings wewere able to get through the review in less than a month and with only alimited amount of meeting fatigue!

In the end these became our team’s workingdefinitions:

All critical alerts should have…
…an immediate and quantifiable end user impact which must require a specific human interaction to rectify
…and should be paged on immediately

Whereas
All warning alerts should have…
…no short term action required, but do have proven critical impact to the system if left unmanaged
…and should be paged on after 72 hours in a persistent state

We of course dove into each of the detailslike how to define “short term” but seeing this high level framework canhopefully demonstrate how we also uncovered the need to discuss the informationprovided by an alert (if it requires human interaction, shouldn’t that actionbe clear from the get go?) and also how we handle different types of services(grafana which just displays data and is used most heavily in office hours maynot need to be paged on Sunday morning).

Outcomes and learnings from theprocess

By the end of the review we identified somealerts that could be removed, some alerts which should be retired throughretiring the systems they were attached to, and many alerts which could usesome sprucing up. In all these cases we created story cards with the reviewcontext and have been regularly feeding them into our workload since then.

As we are now about four months on, I thought it may be a good time to retrospect on the experience and the value of the work done. My biggest takeaway is that this experience closely mirrored my experience with test automation suite reviews.

In particular, stepping away from a singlestory card or a single feature helped our team realise the bigger picture ofalerts and come up with a framework to keep our alerts in order. Nothing wecame up with was novel to our team, it is just that we had not been able to sitdown and agree while simultaneously being able to apply or theories againstreal world scenarios. The alert review is what really helped cement agreementover our alert framework through discussion and debate about possible outliers.

Another similarity to test automation is howthe team cared deeply about quality alerts, but given the intermittent pain ofpoor alerts and lack of customer facing impact, it was hard to spend the timeto improve them. To be honest, this continues to be a challenge as we try andstrike a balance between interrupt driven work (incidents or alerts on a givenday), feature delivery, and long term maintenance. That being said, having thethoughtfully created and thorough documentation of fixes in places has meantwhen a “mildly annoying” alert goes off again we can often pluck that card from the backlog and prioritise it quickly.

Finally, the test community doesn’t seem tohave as well known a term for red build fatigue as the operations community hasfor alert fatigue, but trust me they are the same thing! Our team started torely on their instincts and experience to decide which warning alerts actually mattered and more often thannot just wanted to remove the culprit alerts because it was the only way theysaw any chance of improving the situation.

I am not suggesting we are in an ideal placenow, but by introducing some practices for bringing the pain forward likeescalating all warnings after 72 hours, it has increased our on call engineerand product owners focus on alert quality for the sake of team sanity.

Abby Bangser is a software tester with a keen interest in working on products where fellow engineers are the users. Abby brings the techniques of analysing and testing customer facing products to tools like delivery pipelines and logging so as to generate clearer feedback and greater value. Currently Abby is a Test Engineer on the Platform Engineering team at MOO which supports the shared infrastructure and tooling needs of the organisation.