Anticipating Problems with Infrastructure as Code in DevOps

By Nilanjan Bhattacharya

There are different interpretations of theterm, ‘DevOps’. However, most peoplewill endorse the idea – ‘DevOps also focuses on the organization change tosupport great collaboration between the many functions involved’.

Despite the focus on collaboration andorganization change, a critical part of DevOps is the enabling technology. There are different aspects of thetechnology. Continuous deliveryautomates the entire process of delivery, including building the software,validating the build and progressing through other steps in the form of apipeline. A much more elaborate part ofthe technology is Infrastructure as code (IaC). Infrastructure as code was pioneered by Amazon Web Services (AWS). Thereare newer entrants such as Microsoft Azure and Google Cloud.

The main concept of an IaC service, likeAWS, is to access infrastructure such as servers, databases and networks, asvirtual resources. It’s critical to beable to create and access the infrastructure programmatically . You can then treat the scripts which interactwith the infrastructure as software and use software development principles, todevelop and manage your code/infrastructure, such as source code control andcontinuous delivery (of infrastructure). IaC provides the ability to scale up and scale down components at will,greatly increasing the economy while meeting customer/market demands.

DevOps isoften introduced with examples from Netflix and Facebook. However, the software industry is vast. It includes both software product companieslike Netflix, but also includes enterprise software development in banks,telecom, government. It also includesservice providers. Given the universalavailability of cloud-based infrastructure and the accessibility due to lowcost and the pay-as-you-go model, the size of teams adopting IaC can varywidely. The two-person team creating amobile app, or the government department creating a health portal, is verydifferent from a team in Netflix. Thechallenge for all teams is preventing problems when deploying IaC as part ofDevOps.

How can you anticipate problems?

One way to anticipate problems is to askquestions.

A database as a service, like AWS RDS, isvery appealing both from a technical and business point of view. However, what happens when there is downtime,even for a few seconds, due to routine maintenance? For non-critical applications it may notmatter (you may still want to find out if the application is affected). For critical applications, you may want tocreate a highly available solution.

If you are working on an existingapplication, how do you replace the single instance of the database with ahighly available solution? Are there anymaintenance windows? What is the natureof the business? Is the business seasonal?

Here are some questions we might ask:

For a database as a service, what happens whenthere is downtime, due to maintenance?
Does the business have a maintenance window forreplacing the database?
What is the nature of the business? Can they afford downtime?
Is the business seasonal? Are there peak times when they can’t afforddowntime?

You could generalize the first question as,‘For any cloud-based service, what happens when there is downtime?’ You could also ask, ‘For any cloud-based service,what happens when you need to upgrade the version?’

Here is a question for you to practice (my response given below):

What happens when you need to upgrade thelanguage version (e.g., node.js) of a function as a service?

As an aside: Is this a problem with a cloud-basedservice/Infrastructure as code? Wouldthis have not been a problem if you controlled the server/environment, i.e., with physical servers?

How do you ask questions?

You want to ask questions which are open ended . You could ask, ‘What cancause the storage to become full on an EC2 instance?’ or ‘What could cause theinstance to become unresponsive?’ Theanswer will depend on your application and specific circumstances. (You will probably set up alerts for diskutilization reaching a threshold. However, alerts don’t replace the thinkingabout what might go wrong.)

You want to take asking open endedquestions a step further. What you arereally doing is looking for questionsthat you hadn’t thought about .

The idea of asking questions might seemvery unstructured. You can structure your thinking by using anapproach to generate ideas. One suchapproach is using the mnemonic, SFDiPOT, to structure your questions. SFDiPOTstands for structure, functions, data, interfaces, platform, operations,time. You could use S-structure to thinkabout the different components of the system such as the database, lambdas and instances and how they mightpose a risk. In the case of IaC, itisn’t a physical product, but an entire platform.

You can use another mnemonic CIDTESTD(Customers, Information, Developer relations, Test Team, Equipment & Tools,Schedules, Test Items, Deliverables) to generate ideas. If you are working with an enterpriseapplication like Adobe AEM, do your DevOps engineers have enough knowledge ofthe product? If not, does that representa risk? (We used the letter T, which stands for (Test) Team, to think aboutrisks related to the nature of the team).

When you ask questions, you want to holdyourself and your team accountable andincorporate feedback. Retrospectives andpost-mortems can identify questions that weren’t asked. You also want to identify how you could have generated ideas which would have led tothose questions.

Here is what I would ask when thinkingabout upgrades to the environment (e.g. node.js) for a function as a service,like AWS Lambda:

When we look at the Lambda code a few years fromnow, will it make sense to a new person/team?
Do we have automated tests for the larger functions?
Are there functions used for infrastructure, asopposed to, being used by the application?
Do we have any automated tests for the Lambdafunctions?
Are there differences in the quality of codewritten by the infrastructure team as opposed to the application team?
Will we be able to roll back the functions afterupgrade, in case there are problems?
Is there a development environment for theLambda functions, i.e., does the dev environment include the Lambda functionsespecially the ones used for Infrastructure?
Will the development and production environmentsbe kept in sync (for the next few years)?
Can we track differences in the development andproduction environment?

Here are a few more general questions youcould think about when working with infrastructure as code:

What can cause a component to not respond?
When can an instance not respond?
When can a component run out of capacity, e.g.,storage?
Can you upgrade a component to a highercapacity?
Are higher capacity components always availablein the region that you are working?
Does the region that you are working in, impactyour available options? Does that impactyour choices for increasing capacity in the future?

Beyond questions

There are other activities which cangenerate ideas on risk, for example, stress testing a component or creating arealistic customer environment, including real time data. These activities can also be used to generatequestions. Asking questions is part ofthe much broader activity of critical thinking .

For a start, I recommend beginning with asking questions in a work situation(see example on functions as a service above). You should accompany that by using retrospectives to evaluate questionsthat weren’t asked and identifying the ideas or activities which could have ledto those questions.

The idea of asking questions related to IaCcan form a foundation for thinking about your model for anticipatingproblems/risk in DevOps. It may alsolead the team to make inroads into critical thinking.

Notes

Thedescription of the core of DevOps is from Wikipedia: https://en.wikipedia.org/wiki/DevOps
SFDiPOT and CIDTESTD are part of the Heuristic Test StrategyModel created by James Bach ( )
The idea ofusing questions to identify risk has been used earlier, e.g., ‘a test case is aquestion that you ask of the program’, in ‘What is a Good Test Case?, StareastConference, May 2003,

Nilanjan Bhattacharya

Nilanjan works as a DevOps Delivery coordinator for AC3 in Sydney. AC3 is a multi-cloud service provider and Australia’s leading AWS Premier Consulting Partner and AWS Managed Services Provider.

Nilanjan has a background in software testing and project management. He has worked with a variety of software teams, including software product teams. His experience in analyzing customer support issues and working with successful software products with a multi-year lifecycle, provides a unique perspective on software quality and defects.

Measuring Our Continuous Delivery Progress

by Rob Hornby

We’ve embarked on work to move off our legacy monolithic Oracle Commerce platform and are creating a microservice -based architecture hosted in . As part of this we have created a Platform ( John Lewis Digital Platform, JLDP) which enables teams to get started quickly through providing key capabilities that are needed by all teams e.g. Google Cloud environment, code repository, path to production, telemetry. We call this our “ ”. We started out on this journey as it was observed that we were not delivering business value quickly enough and technology was believed to be a key impediment to doing this.

One of the key initiatives we undertook as part of building the platform was being able to measure the progress of teams working on the platform through their approach to Continuous Delivery . Previously we had lacked this information and could not make data-driven decisions about our service/application health. We wanted to avoid this and build it in to the platform from the start.

We started capturing much of the data through spreadsheets based on data visible in Google’s logs. We’ve iterated on this and have developed an application which we call the “Service Catalogue”, that brings together the various metrics we’ve collected on health of a Service (one or more microservices to achieve a particular outcome), along with key information on each service/application e.g. Search.

Lead Time to Value

Our initial focus was to show that access to the Platform was no longer a constraint in getting a service live to customer. We wanted to capture some information that proved this was no longer the constraint. We were able to quickly show our service creation lead time took hours not months as it had been previously.

We then looked at capturing the lead time of the first service deployment to production, we called this our onboarding lead time . For the platform this evidenced that teams had managed to navigate our paved road and deployed at least a “hello world” to production. This confirmation also reduced the platforms reactive work load and was a key message to teams when onboarding them, get to production early.

The final addition was to measure of lead time to first value . This was our final measure of success and confirmed the team had made their service live to customers.

JLDP v2 Services – Lead to value

Deployment Indicators

Once a team had deployed and were live to customer, we wanted to continue to understand their continuous delivery on the platform. We set out to capture more information: deployment intervals , deployment lead time and deployment throughput. We have again used systems data to capture this information, looking at git commit history through to container deployment to derive the data. We are looking to observe teams’ approach to agile principles and lean methodology through their continuous delivery practice.

The deployment interval captured is the time between deployments to our production cluster. We want to see a low interval and consistency which suggests a team is then deploying frequently and continuing to work on the Service.

The deployment lead time is then how quickly a code change gets through the pipeline to production. This tells us how long it takes a team to get a change live and again looks for consistency and a short time from commit to production. This tells us that teams can get change through to production quickly through their pipeline and their ways of working.

Delivery Indicators (Interval Time, Lead Time)

Deployment throughput then helps us confirm overall deployment volumes across the platform helping us to evidence our overall deployment trends within teams and services. We have the data for this but have yet to fully integrate this in to our Service Catalogue, though we can present it on a on-demand basis.

Live to Customer — Monthly Service Deployments

What did we learn?

These data points were then critical for our initial conversations with teams which we called platform research. We used the data to talk to teams that were taking time to get to their first customer and understand what might be constraints in either the platform or the wider delivery process. We are also trying to build a product not just a platform and so the capabilities we had will be informed through the customers (service teams) running on our platform.

We learnt quickly that we’d started to unblock some key constraints.

The access to the platform has reduced the provision of infrastructure to teams from months to hours.
Our overall deployment throughput was increasing substantially from 10’s of deployments a years to 1000’s.

Total Deployments Throughout the Year — Total deployment throughout the year

Continuous Delivery patterns (Lead Time: Blue, Interval Time: Red) - so much to read into these graphs. — Continuous Delivery patterns (Lead Time: Blue, Interval Time: Red) – so much to read in these graphs

Through further ongoing research we learnt that there were common service constraints

Teams were struggling with the security process which was designed for large waterfall projects and hadn’t caught up with a more agile, microservice-based architecture.
Teams had too much work in progress. We had teams which were requesting multiple services without getting their first service live to customer.
Teams had external dependencies, usually around integrations outside the platform ingesting data from other external systems.
Teams were attempting to get to feature parity before going live to customer.
Teams were still attempting to undertake end-to-end functional or performance testing with external dependencies.
Change freeze still plays a part in reducing consistency and increasing batch size.

These were all things we could now start initiatives on to improve lead time. For example, we have created a Platform Security team to help engagement with teams in to our current security process with a goal of changing and developing a pattern more consistent with our delivery approach.

It also helped us inform the direction for the platform. We increment the version of our platform quarterly (although deploy daily). Each increment we aim to delivery new capabilities such as improved service creation, monitoring and logging capabilities and resilience, all focused on building a product that helps the teams running on our platform go faster or improves their stability.

What Next?

So far we have focused on measuring our continuous delivery indicators to show throughput to production. Operability is the next key focus. Teams are now starting to support their services, and the traditional operational structures which supported many of the operability aspects are now transferring their knowledge on processes to teams.

Continue to develop the platform with operability as a focal point. We have already provided teams with indicators of health e.g. based on Operability patterns outlined in .
We want to combine this with alerting focused around and .
We will look to expand our Service Catalogue to include deployment stability indicators. Deployment Failure Rate (number of failed production deployments and duration between), Recovery time from failure (time taken to recover the service after a failure).

The aim is to combine our continuous delivery and operability metrics together in one place. The future will then be tying this data together with business data to truly show how deployment frequency relates to value. The platform and being able to evolve it as a product will continue based on our ability to evidence and communicate the value it can add.

Reference:

For more information, we recommend reading Measuring Continuous Delivery by Steve Smith ().

Rob Hornby is a Product Engineer working for John Lewis based in London. He has undertaken a number of roles, over the last 20 years, within Testing, Release, Platform and Environments.

Rob is a keen believer in being able to treat all things as Products and be able to evidence the value generated by that team. Ideally everything would be delivered by small teams, delivering continuously and iterating on value. However not all teams start at this point, not all teams are the same or born equal. This is where the challenge and enjoyment comes from.

Outside of work, Rob is a keen runner, father of an amazing young girl, Sofia and works on a number of local community events.

Applications are more than just code

By Matt Long

I was at TestBash Brighton recently, and a common theme kept coming up:

“You can be technical without knowing how to code!”

, TestBash organiser, even did a 99 second talk on the topic. He listed various skills – git, debugging, changing the microwave clock. If you have these skills as a tester, there are plenty of uses for these skills. If you have an agile or DevOps culture – where roles are less set in stone, and teams swarm on problems – you’ll get loads of opportunities to use them.

Well, apart from the microwave clock.

Learning the ecosystem

A yak dominates an ecosystem

It’s important to realise most developer skills aren’t coding. If you’ve tried to learn coding, you’ll know lots of other skills can get in the way. You start writing a simple application and lots of good practice pops up. Your developer friend says you should set up a git repository. Then instead of installing dependencies manually, it’s better to set up a dependency manager. Then you learn how to compile and run the code. The list goes on. This can be very daunting, and quickly turns into yak-shaving .

Learning doesn’t stop with developer skills

This seemingly never ending list of things to learn doesn’t stop with a developer’s skillset. Run a production-grade environment, and SRE/Ops skills will get in the way. Ops ensure that users can use your environments. They do all kinds of things. Backups. User and authentication control. Familiarity with the Linux CLI to debug issues. Log monitoring and analysis. Failover processes like blue/green. Containerisation like Docker , or container orchestration like Kubernetes . Your workplace may differ in the specifics, but production environments have lots of additional complex functionality. Complex functionality you can’t replicate by running the code on your laptop, or even sometimes in a test environment.

Broad skills are valuable too

Your broad skills are valuable

There are specialists in these subjects. Nobody is expecting testers to be experts with all of these skills. Much like testers don’t have to have all the coding skills of developers. That said, knowing about these things will help us understand our application. It will help understand facets of our application otherwise overlooked. If testing is about understanding the quality of our application, then we need to look beyond the code. As far as your application is concerned, code is the tip of the iceberg.

Let’s take an example: logging in to an application. Your users are not only expecting that they can login successfully. They’re expecting that the database stores their user credentials. They’re expecting that their data is backed up in the event of disaster. They’re expecting that they can still login if lots of other people are. They’re expecting their confidential data is securely stored and transported.

Much of that functionality is not done in code by developers. It is often the responsibility of ops, or sysadmins, or database admins. They configure databases and networks and servers that ensure all these things happen.

Ops opportunities for testers

Before DevOps, it was difficult for testers to have influence upon this. Much was siloed away from us. We test the code, then let them deal with that stuff. Out of sight, out of mind.

Cross-functional teams include Ops specialists

But now, Ops specialists are being moved into our teams, or working closely with our teams (hopefully not as a permanent ‘DevOps team’!). We see the work they do, and it needs testing too. That’s because it’s part of the application. It’s part of what users expect. We don’t just deliver code, we deliver a service.

Pairing — Pair with an ops expert to assess risks

So how can testers start helping? To test code, we didn’t need to be developers. But learning related skills helped us empathise and co-operate. It helped us ask the right questions. So try pairing with an ops expert on your team and work out what risks might arise. Or doing some shaping of some infrastructure tickets, and thinking about who the users are (with pipelines, they’re often developers!). Or try writing some automation scripts to ensure that associated problems get caught before production. Or go to a devops meetup/conference. Learn about bad practice and security/infrastructure smells. Start treating ops like developers – to be assisted and questioned.

Because our application is more than just code. It’s all the processes, and the infrastructure too. That stuff needs testing too.

Matt Long is a QA & devops consultant for OpenCredo , an open source software consultancy based in London.

He has worked as a QA consultant for many years, and built test automation frameworks in half a dozen programming languages. After working on many cloud infrastructure projects, he’s become particularly interested in the intersection of modern cloud infrastructure and testing.

BuryTheHammer.com ,

Similarities between SRE and testing: Alert/automation suite maintenance

By Abby Bangser

I want to take a few minutes to share with youhow I have grown to once again realise how common so many of our challenges arein tech. This time the commonality is between the world of test automation andoperational alerting. Before I dive into these details though, let me quicklyexplain my career journey which led me here.

From the beginning of 2012 to the middle of2018 I worked as a test consultant in organisations ranging from small non-profitsthrough to large multinational financial institutions. During this time I sawthe scale, financing, and cultures change fairly drastically, however thechallenges were often quite similar when it came down to the team level ofdelivering software. Most notably, over these years I realised that being aspecialist in quality often meant my biggest contribution was in how we delivercode through Continuous Integration and Delivery processes rather than just how I test things. This led me to working with many DevOps and infrastructure relatedteams and activities and eventually on to joining MOO as a Test Engineer ontheir Platform Engineering team.

Introduction to our platformengineering team

At MOO the Platform Engineering team isresponsible for the infrastructure of both production and pre-productionenvironments as well as the running of internal tooling like source control andbacklog management tooling. As you may expect this includes being on call forthese systems 24 hours a day, 7 days a week. From what I can see around theindustry our team is not particularly unique in how we handle this.

We have a rota which makes sure we share theresponsibility and our team generates both warning and critical alerts to makesure we respond if issues occur. These alerts provide us huge opportunity forspeeding up our Mean Time To Recovery (MTTR) by decreasing our time todiscovery. However when I joined the team, they were also generating a lot ofstress which is why one of my first tasks was to review our alerts for valueand how to remove that stress.

Tackling our alerting challenges

Our alert stress was generated for a few majorreasons. First of all, some of our alerts were things that our MOO team memberscould do nothing about. For example, if our Google authentication is failingthere is an impact to our internal users which could trigger an alert to our oncall engineer, however there is nothing we can do but wait until Google sortsout their incident. Another issue with our alerts was critical alerts (meaningsomeone is woken up in the middle of the night) that resolved themselves withinminutes. Often these were issues with machine health in AWS which has its ownself-healing mechanisms and our thresholds were just too low. Finally, we hadsome alerts that made people wonder if they were worthwhile as they had nevertriggered once in over a year of tracking.

When I joined the team there was a ticket onthe backlog to “Improve team alert fatigue”. This was intentionally a prettyvague topic and was meant to be an analysis placeholder as there wasn’t justone solution to all these concerns with our alerts. Alert fatigue can bedescribed as “exposed to a large number of frequent alarms and consequentlybecomes desensitized to them. Desensitization can lead to longer response timesor to missing important alarms.” and our team was definitely showing the earlysymptoms. After spending some time with each team member we decided thatdefinition of done for this ticket would be to:

Identify the shape andrequirements of a “good” alert
Review our alerts against this definition
Set up future work to bring ourcurrent alerts in line with our desired state

Achieving a working solution byputting it to the test

To achieve these goals I went about puttingtogether a straw man definition of what a warning and a critical alert wouldneed to have. Then scheduled reviews of all of our alerts against thesedefinitions to let them be hardened and evolved by our real context. This was a(very) long process but by grouping alarms together, sharing the load acrossall team members in small groups and doing no more than 30 minute meetings wewere able to get through the review in less than a month and with only alimited amount of meeting fatigue!

In the end these became our team’s workingdefinitions:

All critical alerts should have…
…an immediate and quantifiable end user impact which must require a specific human interaction to rectify
…and should be paged on immediately

Whereas
All warning alerts should have…
…no short term action required, but do have proven critical impact to the system if left unmanaged
…and should be paged on after 72 hours in a persistent state

We of course dove into each of the detailslike how to define “short term” but seeing this high level framework canhopefully demonstrate how we also uncovered the need to discuss the informationprovided by an alert (if it requires human interaction, shouldn’t that actionbe clear from the get go?) and also how we handle different types of services(grafana which just displays data and is used most heavily in office hours maynot need to be paged on Sunday morning).

Outcomes and learnings from theprocess

By the end of the review we identified somealerts that could be removed, some alerts which should be retired throughretiring the systems they were attached to, and many alerts which could usesome sprucing up. In all these cases we created story cards with the reviewcontext and have been regularly feeding them into our workload since then.

As we are now about four months on, I thought it may be a good time to retrospect on the experience and the value of the work done. My biggest takeaway is that this experience closely mirrored my experience with test automation suite reviews.

In particular, stepping away from a singlestory card or a single feature helped our team realise the bigger picture ofalerts and come up with a framework to keep our alerts in order. Nothing wecame up with was novel to our team, it is just that we had not been able to sitdown and agree while simultaneously being able to apply or theories againstreal world scenarios. The alert review is what really helped cement agreementover our alert framework through discussion and debate about possible outliers.

Another similarity to test automation is howthe team cared deeply about quality alerts, but given the intermittent pain ofpoor alerts and lack of customer facing impact, it was hard to spend the timeto improve them. To be honest, this continues to be a challenge as we try andstrike a balance between interrupt driven work (incidents or alerts on a givenday), feature delivery, and long term maintenance. That being said, having thethoughtfully created and thorough documentation of fixes in places has meantwhen a “mildly annoying” alert goes off again we can often pluck that card from the backlog and prioritise it quickly.

Finally, the test community doesn’t seem tohave as well known a term for red build fatigue as the operations community hasfor alert fatigue, but trust me they are the same thing! Our team started torely on their instincts and experience to decide which warning alerts actually mattered and more often thannot just wanted to remove the culprit alerts because it was the only way theysaw any chance of improving the situation.

I am not suggesting we are in an ideal placenow, but by introducing some practices for bringing the pain forward likeescalating all warnings after 72 hours, it has increased our on call engineerand product owners focus on alert quality for the sake of team sanity.

Abby Bangser is a software tester with a keen interest in working on products where fellow engineers are the users. Abby brings the techniques of analysing and testing customer facing products to tools like delivery pipelines and logging so as to generate clearer feedback and greater value. Currently Abby is a Test Engineer on the Platform Engineering team at MOO which supports the shared infrastructure and tooling needs of the organisation.