Anticipating Problems with Infrastructure as Code in DevOps

By Nilanjan Bhattacharya

There are different interpretations of theterm, ‘DevOps’. However, most peoplewill endorse the idea – ‘DevOps also focuses on the organization change tosupport great collaboration between the many functions involved’.

Despite the focus on collaboration andorganization change, a critical part of DevOps is the enabling technology. There are different aspects of thetechnology. Continuous deliveryautomates the entire process of delivery, including building the software,validating the build and progressing through other steps in the form of apipeline. A much more elaborate part ofthe technology is Infrastructure as code (IaC). Infrastructure as code was pioneered by Amazon Web Services (AWS). Thereare newer entrants such as Microsoft Azure and Google Cloud.

The main concept of an IaC service, likeAWS, is to access infrastructure such as servers, databases and networks, asvirtual resources. It’s critical to beable to create and access the infrastructure programmatically . You can then treat the scripts which interactwith the infrastructure as software and use software development principles, todevelop and manage your code/infrastructure, such as source code control andcontinuous delivery (of infrastructure). IaC provides the ability to scale up and scale down components at will,greatly increasing the economy while meeting customer/market demands.

DevOps isoften introduced with examples from Netflix and Facebook. However, the software industry is vast. It includes both software product companieslike Netflix, but also includes enterprise software development in banks,telecom, government. It also includesservice providers. Given the universalavailability of cloud-based infrastructure and the accessibility due to lowcost and the pay-as-you-go model, the size of teams adopting IaC can varywidely. The two-person team creating amobile app, or the government department creating a health portal, is verydifferent from a team in Netflix. Thechallenge for all teams is preventing problems when deploying IaC as part ofDevOps.

How can you anticipate problems?

One way to anticipate problems is to askquestions.

A database as a service, like AWS RDS, isvery appealing both from a technical and business point of view. However, what happens when there is downtime,even for a few seconds, due to routine maintenance? For non-critical applications it may notmatter (you may still want to find out if the application is affected). For critical applications, you may want tocreate a highly available solution.

If you are working on an existingapplication, how do you replace the single instance of the database with ahighly available solution? Are there anymaintenance windows? What is the natureof the business? Is the business seasonal?

Here are some questions we might ask:

For a database as a service, what happens whenthere is downtime, due to maintenance?
Does the business have a maintenance window forreplacing the database?
What is the nature of the business? Can they afford downtime?
Is the business seasonal? Are there peak times when they can’t afforddowntime?

You could generalize the first question as,‘For any cloud-based service, what happens when there is downtime?’ You could also ask, ‘For any cloud-based service,what happens when you need to upgrade the version?’

Here is a question for you to practice (my response given below):

What happens when you need to upgrade thelanguage version (e.g., node.js) of a function as a service?

As an aside: Is this a problem with a cloud-basedservice/Infrastructure as code? Wouldthis have not been a problem if you controlled the server/environment, i.e., with physical servers?

How do you ask questions?

You want to ask questions which are open ended . You could ask, ‘What cancause the storage to become full on an EC2 instance?’ or ‘What could cause theinstance to become unresponsive?’ Theanswer will depend on your application and specific circumstances. (You will probably set up alerts for diskutilization reaching a threshold. However, alerts don’t replace the thinkingabout what might go wrong.)

You want to take asking open endedquestions a step further. What you arereally doing is looking for questionsthat you hadn’t thought about .

The idea of asking questions might seemvery unstructured. You can structure your thinking by using anapproach to generate ideas. One suchapproach is using the mnemonic, SFDiPOT, to structure your questions. SFDiPOTstands for structure, functions, data, interfaces, platform, operations,time. You could use S-structure to thinkabout the different components of the system such as the database, lambdas and instances and how they mightpose a risk. In the case of IaC, itisn’t a physical product, but an entire platform.

You can use another mnemonic CIDTESTD(Customers, Information, Developer relations, Test Team, Equipment & Tools,Schedules, Test Items, Deliverables) to generate ideas. If you are working with an enterpriseapplication like Adobe AEM, do your DevOps engineers have enough knowledge ofthe product? If not, does that representa risk? (We used the letter T, which stands for (Test) Team, to think aboutrisks related to the nature of the team).

When you ask questions, you want to holdyourself and your team accountable andincorporate feedback. Retrospectives andpost-mortems can identify questions that weren’t asked. You also want to identify how you could have generated ideas which would have led tothose questions.

Here is what I would ask when thinkingabout upgrades to the environment (e.g. node.js) for a function as a service,like AWS Lambda:

When we look at the Lambda code a few years fromnow, will it make sense to a new person/team?
Do we have automated tests for the larger functions?
Are there functions used for infrastructure, asopposed to, being used by the application?
Do we have any automated tests for the Lambdafunctions?
Are there differences in the quality of codewritten by the infrastructure team as opposed to the application team?
Will we be able to roll back the functions afterupgrade, in case there are problems?
Is there a development environment for theLambda functions, i.e., does the dev environment include the Lambda functionsespecially the ones used for Infrastructure?
Will the development and production environmentsbe kept in sync (for the next few years)?
Can we track differences in the development andproduction environment?

Here are a few more general questions youcould think about when working with infrastructure as code:

What can cause a component to not respond?
When can an instance not respond?
When can a component run out of capacity, e.g.,storage?
Can you upgrade a component to a highercapacity?
Are higher capacity components always availablein the region that you are working?
Does the region that you are working in, impactyour available options? Does that impactyour choices for increasing capacity in the future?

Beyond questions

There are other activities which cangenerate ideas on risk, for example, stress testing a component or creating arealistic customer environment, including real time data. These activities can also be used to generatequestions. Asking questions is part ofthe much broader activity of critical thinking .

For a start, I recommend beginning with asking questions in a work situation(see example on functions as a service above). You should accompany that by using retrospectives to evaluate questionsthat weren’t asked and identifying the ideas or activities which could have ledto those questions.

The idea of asking questions related to IaCcan form a foundation for thinking about your model for anticipatingproblems/risk in DevOps. It may alsolead the team to make inroads into critical thinking.

Notes

Thedescription of the core of DevOps is from Wikipedia: https://en.wikipedia.org/wiki/DevOps
SFDiPOT and CIDTESTD are part of the Heuristic Test StrategyModel created by James Bach ( )
The idea ofusing questions to identify risk has been used earlier, e.g., ‘a test case is aquestion that you ask of the program’, in ‘What is a Good Test Case?, StareastConference, May 2003,

Nilanjan Bhattacharya

Nilanjan works as a DevOps Delivery coordinator for AC3 in Sydney. AC3 is a multi-cloud service provider and Australia’s leading AWS Premier Consulting Partner and AWS Managed Services Provider.

Nilanjan has a background in software testing and project management. He has worked with a variety of software teams, including software product teams. His experience in analyzing customer support issues and working with successful software products with a multi-year lifecycle, provides a unique perspective on software quality and defects.

Path to Production – Test Environments as a Product

By Rob Hornby

It could be easier, we could have teams for each application we run, that those teams treat their production and non-production application environments the same. That they have fully automated full stack continuous deployment pipelines and the software is so well designed that there is no dependency between each application owner. Teams deliver code by the hour to production and use advance deployment techniques such as blue-green deployment coupled with highly effective monitoring, alerting and automated rollback to identify and resolve issues in production within minutes.

We do have “agile” application teams working within a cloud environment, they have growing autonomy as they build independent features and function. That’s awesome. The problem is that we have not been working that way since we started building technology. We have a sprawling on-premise (we own two data centres) environment where Non-Production environments equates for 60%+ of the overall infrastructure. This is made up of dependent applications running together in an environment.

We have followed traditional programme structure over the last 10 years, with multi-year investments and waterfall based delivery. These have sought to reduce risk through separation rather than integration. This has resulted in multiple non-production environments. Our applications (100’s) have been built tightly coupled and need multiple other applications to work (Commerce systems through to legacy mainframe systems all meshed together), increasing the size of the basic/core environment. These programmes have not invested in automation, leaving a challenging sprawling estate to manage.

Path to Production (Variety of routes followed by teams ensuring lack of consistency in versioning in each environment)

This has created a number of problems. The estate is costly to maintain. There is hidden cost in each application change or infrastructure upgrade. Our people are linked to projects and programmes, when they finish, the applications in non-production fall in to disrepair, they are costly and time consuming to get working again when the next team comes along. A continued desire for separation results in projects hunting down that unused application/environment that has nobody else working in that space. The sprawling nature and lack of ownership results in a small band of “Environment Management” who work to bridge the communication and knowledge gaps across this sprawling estate. These people work for Projects and Programmes and through “best endeavours” get Delivery teams, Operations and Infrastructure teams to stitch the environment back together each time and retain the illusion these environments are good to test in. We effectively have 10 mini-lives beating away with no real Service Management to deal with them.

That long introduction was needed just to give me time to breathe and reflect. The organisation is changing for the better, which is great. We are re-organising and realising the mistakes of the past. We are now building most new systems in a cloud environment, are focused on Micro-service design principles and have teams that own and run the service (well almost). These are small individual successes which highlight the organisation is changing. Our online teams have gone from 10 deployments a year to over 1000 in just over 18 months. It should be a factor of 10 greater by next year. However there is a long way to go till we are fully working in this way, maybe we will never get there. This is where Environment as a Product comes in. How do we change our existing non-production environment landscape for the better.

We have started with two key initiatives. Initiative 1, Let’s get support to introduce an “enabling constraint” through removing the environments that least support delivery. They are the most difficult to maintain, are off the beaten path to production and the most costly to re-enable them. If the project or programme don’t have them they can’t use them. We know they are costly to build which is part of the reason they have been maintained. With the remaining environments we are going to create a paved road of sorts. One of the key things is to not construct the path to production based on your current organisational/delivery construct. We have been very waterfall in nature and have therefore not made change often to production.

This results in us having multiple hops to production for a single change based on the type of testing to be undertaken at each stage. Development (unit, system), Integration, Release (end to end), Performance (performance, operational), Pre-Production (incident and support) and Production. You introduce delay just through deploying to each of these environments and fixing the issues that come up as a result. That’s before you think of data, types of testing, handoff implications. So initially let’s get down to one of each type to support all teams, projects and programmes. Let’s draw it up so it looks like some are optional. Development, Integration, Pre-production, Production (even this is too many).

This was difficult to get going, convince people that we might be able to go faster through having less environments, that there might be more contention initially, however that might force us to look at the organisational shape and how we design software. Whether the full extent of the idea has sunk in, is yet to be seen, however the cost avoidance articulation of maintenance running in to millions of pounds a year was hard to ignore and gave us the initial green light.

Initiative 2, is to consolidate the team we have and build a paved road (common environments, tools and approach). We have taken everyone responsible for non-production “Environment-Release Management” and centralised them in to a single Product team. This wasn’t easy. You have to convince the leadership team that they want to pay for this as a separate service, that the cost is neutral, value adding even and that in doing so you can look at how the team works and make efficiency, as well as improving the path to production. What this doesn’t do is centralise the technical people responsible (not owning) for those environments. These remain fragmented across Operations, Infrastructure and Delivery teams. While the number of different technologies, teams and ways of working make this impractical at a small team level.

It does allow you to put a very clear Service Management approach in place for Non-Production and with leadership support allow you to be more robust in enforcing a better approach. You can start to work with individual teams to support and help them approach their delivery differently, focusing on automation, repeatable process and decoupling of change. The aim is that teams do this themselves and that the Environment-Release team can identify teams that need support. We are also backed by organisational , which have been established to give teams direction in how they should deliver.

Centralising the team has also allowed us to look at how we approach managing the environments. A single Product team with longer term goals, where the value is what we do to the environments, not the end customer outcome that the Project or Programme was focused on, has a very different priority and outlook. What was several ways of working can now be focused in to a single way of working. We have started by making sense of what people were doing daily. This has been done through several value stream mapping workshops but also to get everyone’s work up on a board and try to make sense of the work we do. Get a feel for reactive vs planned work. Upon doing this we can start to build a very clear backlog of priorities for our own ways of working, how we track issues, do we manage problems, how do teams onboard on to our path to production, can we pre-empt issues, what do we do with data, looks at our skills and most importantly clear communication.

Value is now a clear driver, as is ownership. We can now mandate changes to production are part of non-production thinking. We can also start to look at cost of delay and overall costs of supporting teams and drive our own value outcomes. Lead time is also a key measure, whether that that is to measure how quickly teams move through our environments, how long it takes us to respond to incident, downtime within environments etc… We can also help to start to measure teams frequency of deployment/release and to show organisational pace of change. Traditional service management is very much focused on controls but here we want to make it much more about value outcomes and enabling flow within teams.

We also have one foot firmly in the future. Over time the need for this team will be less. Greater ownership, more end to end owning teams, “DevOps” will all drive less need for a Environment-Release Product team. Our life span is based on the value we can show to generate. However we need to develop our skills in to other areas, whether that be Platform management within our Cloud platform or in to more “DevOps” skill sets which can work within teams to help them deliver. For now we are focused on our Product and value. It would be interesting to hear how other companies and people manage non-production environments, especially where there has been ongoing development of a variety of approaches for many years.

Rob Hornby is a Product Engineer working for John Lewis based in London. He has undertaken a number of roles, over the last 20 years, within Testing, Release, Platform and Environments.

Rob is a keen believer in being able to treat all things as Products and be able to evidence the value generated by that team. Ideally everything would be delivered by small teams, delivering continuously and iterating on value. However not all teams start at this point, not all teams are the same or born equal. This is where the challenge and enjoyment comes from.

Outside of work, Rob is a keen runner, father of an amazing young girl, Sofia and works on a number of local community events.