By Nilanjan Bhattacharya

There are different interpretations of theterm, ‘DevOps’.  However, most peoplewill endorse the idea – ‘DevOps also focuses on the organization change tosupport great collaboration between the many functions involved’. 

Despite the focus on collaboration andorganization change, a critical part of DevOps is the enabling technology.  There are different aspects of thetechnology.  Continuous deliveryautomates the entire process of delivery, including building the software,validating the build and progressing through other steps in the form of apipeline.  A much more elaborate part ofthe technology is Infrastructure as code (IaC). Infrastructure as code was pioneered by Amazon Web Services (AWS). Thereare newer entrants such as Microsoft Azure and Google Cloud.

The main concept of an IaC service, likeAWS, is to access infrastructure such as servers, databases and networks, asvirtual resources.  It’s critical to beable to create and access the infrastructure programmatically .  You can then treat the scripts which interactwith the infrastructure as software and use software development principles, todevelop and manage your code/infrastructure, such as source code control andcontinuous delivery (of infrastructure). IaC provides the ability to scale up and scale down components at will,greatly increasing the economy while meeting customer/market demands. 

DevOps isoften introduced with examples from Netflix and Facebook.  However, the software industry is vast.  It includes both software product companieslike Netflix, but also includes enterprise software development in banks,telecom, government.  It also includesservice providers.  Given the universalavailability of cloud-based infrastructure and the accessibility due to lowcost and the pay-as-you-go model, the size of teams adopting IaC can varywidely.  The two-person team creating amobile app, or the government department creating a health portal, is verydifferent from a team in Netflix.  Thechallenge for all teams is preventing problems when deploying IaC as part ofDevOps. 

How can you anticipate problems?

One way to anticipate problems is to askquestions. 

A database as a service, like AWS RDS, isvery appealing both from a technical and business point of view.  However, what happens when there is downtime,even for a few seconds, due to routine maintenance?  For non-critical applications it may notmatter (you may still want to find out if the application is affected).  For critical applications, you may want tocreate a highly available solution. 

If you are working on an existingapplication, how do you replace the single instance of the database with ahighly available solution?  Are there anymaintenance windows?  What is the natureof the business? Is the business seasonal?

Here are some questions we might ask:

• For a database as a service, what happens whenthere is downtime, due to maintenance?
• Does the business have a maintenance window forreplacing the database?
• What is the nature of the business?  Can they afford downtime?
• Is the business seasonal?  Are there peak times when they can’t afforddowntime?

You could generalize the first question as,‘For any cloud-based service, what happens when there is downtime?’  You could also ask, ‘For any cloud-based service,what happens when you need to upgrade the version?’

Here is a question for you to practice (my response given below):

What happens when you need to upgrade thelanguage version (e.g., node.js) of a function as a service?

As an aside: Is this a problem with a cloud-basedservice/Infrastructure as code?  Wouldthis have not been a problem if you controlled the server/environment, i.e., with physical servers?

How do you ask questions?

You want to ask questions which are open ended . You could ask, ‘What cancause the storage to become full on an EC2 instance?’ or ‘What could cause theinstance to become unresponsive?’  Theanswer will depend on your application and specific circumstances.  (You will probably set up alerts for diskutilization reaching a threshold. However, alerts don’t replace the thinkingabout what might go wrong.)

You want to take asking open endedquestions a step further.  What you arereally doing is looking for questionsthat you hadn’t thought about . 

The idea of asking questions might seemvery unstructured.  You can structure your thinking by using anapproach to generate ideas.  One suchapproach is using the mnemonic, SFDiPOT, to structure your questions.  SFDiPOTstands for structure, functions, data, interfaces, platform, operations,time.  You could use S-structure to thinkabout the different components of the system such as the database, lambdas and instances and how they mightpose a risk.  In the case of IaC, itisn’t a physical product, but an entire platform.

You can use another mnemonic CIDTESTD(Customers, Information, Developer relations, Test Team, Equipment & Tools,Schedules, Test Items, Deliverables) to generate ideas.  If you are working with an enterpriseapplication like Adobe AEM, do your DevOps engineers have enough knowledge ofthe product?  If not, does that representa risk? (We used the letter T, which stands for (Test) Team, to think aboutrisks related to the nature of the team). 

When you ask questions, you want to holdyourself and your team accountable andincorporate feedback.  Retrospectives andpost-mortems can identify questions that weren’t asked.  You also want to identify how you could have generated ideas which would have led tothose questions.

Here is what I would ask when thinkingabout upgrades to the environment (e.g. node.js) for a function as a service,like AWS Lambda:

• When we look at the Lambda code a few years fromnow, will it make sense to a new person/team?
• Do we have automated tests for the larger functions?
• Are there functions used for infrastructure, asopposed to, being used by the application?
• Do we have any automated tests for the Lambdafunctions?
• Are there differences in the quality of codewritten by the infrastructure team as opposed to the application team?
• Will we be able to roll back the functions afterupgrade, in case there are problems?
• Is there a development environment for theLambda functions, i.e., does the dev environment include the Lambda functionsespecially the ones used for Infrastructure?
• Will the development and production environmentsbe kept in sync (for the next few years)?
• Can we track differences in the development andproduction environment?

Here are a few more general questions youcould think about when working with infrastructure as code:

• What can cause a component to not respond?
• When can an instance not respond?
• When can a component run out of capacity, e.g.,storage?
• Can you upgrade a component to a highercapacity?
• Are higher capacity components always availablein the region that you are working?
• Does the region that you are working in, impactyour available options?  Does that impactyour choices for increasing capacity in the future?

Beyond questions

There are other activities which cangenerate ideas on risk, for example, stress testing a component or creating arealistic customer environment, including real time data.  These activities can also be used to generatequestions.  Asking questions is part ofthe much broader activity of critical thinking . 

For a start, I recommend beginning with asking questions in a work situation(see example on functions as a service above). You should accompany that by using retrospectives to evaluate questionsthat weren’t asked and identifying the ideas or activities which could have ledto those questions.

The idea of asking questions related to IaCcan form a foundation for thinking about your model for anticipatingproblems/risk in DevOps.  It may alsolead the team to make inroads into critical thinking. 

Notes

1. Thedescription of the core of DevOps is from Wikipedia: https://en.wikipedia.org/wiki/DevOps
2. SFDiPOT and CIDTESTD are part of the Heuristic Test StrategyModel created by James Bach ( )
3. The idea ofusing questions to identify risk has been used earlier, e.g., ‘a test case is aquestion that you ask of the program’, in ‘What is a Good Test Case?, StareastConference, May 2003,

Nilanjan Bhattacharya

Nilanjan works as a DevOps Delivery coordinator for AC3 in Sydney.   AC3 is a multi-cloud service provider and Australia’s leading AWS Premier Consulting Partner and AWS Managed Services Provider. 

Nilanjan has a background in software testing and project management.  He has worked with a variety of software teams, including software product teams.  His experience in analyzing customer support issues and working with successful software products with a multi-year lifecycle, provides a unique perspective on software quality and defects.