The ROI of Agile + Automation + Continuous Delivery + SRE

Software Engineering: The systematic application of scientific and technological knowledge, methods, and experience to the design, implementation, testing, and documentation of software. The Bureau of Labor - Statistics IEEE Systems and software engineering – Vocabulary

Introduction

One of the most frequent topics in software companies is about the advantages of Automation and the set of methodologies related to Automation. Continuous Delivery covers a set of concepts and methodologies focused on terminating waterfall approaches, overcrowded teams with non-technical profiles and on providing the best context for Agile. The iterative nature of real Agile Software Development Life Cycles and the continuous deployment of small pieces of software to the final stage are natural companions. On the other hand, Cloud infrastructure, Infrastructure as Code and DevOps methodologies have resulted into a new practice called SRE that break unnecessary silos by involving software developers in the support of software products.

Agile, Automation, Continuous Delivery and SRE are the natural application of Engineering principles to the making of Software products.

The Software industry, however, seems to be stuck in bad practices. Why are software projects in IT Consultancy companies overpopulated with such a bunch of people that do not write a single line of code and without technical responsibilities? How does it affect to the competitiveness of companies and the provision of the required level of Quality?

Continuous Delivery and Automation practices applied to Quality seem to be a possible solution. But, many questions are still in the air...

• Are actually these practices and methodologies profitable while providing a level of confidence on the final product?
• What is exactly the Return of Investment (ROI) of these methodologies? How do they impact on the cost of development and maintenance?

Other topics about the changes in the organization culture, training and skills are also important and we'll handle them in another post. We'll focus for now in the ROI aspects and why the Software Engineering point of view can provide us with the correct information about what is our best option for our software companies.

In The Beginning: A Little Bit of History of Software Engineering

It's usually said that Software Development is not a part of Engineering. After decades of Consultancy Companies trying to remove Engineering from their processes to maximize their profit despite the lower quality and life of the resulting products many people start thinking like them. Like Dr Goebbels said once, "Repeat a lie many times and it will become a truth in time".

The Dark Ages of Software Engineering, claimed by Consultancy managers and fanboys saying that in the beginning no Engineering was present in Software Development, simply never existed. Software Engineering was a term coined by Margaret Hamilton in the 1960s!! when she was trying to make compatible the development of software for the Apollo project with the ultra-strict engineering standards in the aerospace sector.

You'll not remember the matrix printers and the listings of code on paper rolls..

So, the myth about a naive and "happy" time when software development was not subject to any rule or quality standard is a hoax, something that never existed. It is something that happened much later. But when?

The answer comes from the era of the expansion and diversification of the software industry, specially in the 80s and 90s. The Consultancy Model was born as a way to provide IT services and resources and making easy money from it. It saw a rapid and vast expansion because many companies were looking for a quick adaptation to the digital world. The Software Consultancy model has been therefore always based on selling time and resources on one hand and selling the manufacturing of a specific service for a given period of time, usually accompanied by maintenance, evolution and so on. The companies following this model were the ideal vendors for companies that did not have a software department, public administration etc.

The main features of the IT Consultancy model are also its main flaws:

• They sell time and resources. So, the most people is involved, the most profitable is a project.
• Quality is out of the equation. They are only interested on the final balance of results. This means, they do not give a shit about the Quality status of the produced software because they also get a lot of money from maintenance, live issues and 24/7 manual service. Again, the most people, the most profit.
• Manual testing is a constant. And there is a good reason for this: they do not give actually a shit for real, repeatable deep testing. QA departments are populated of not skilled not technical people limited to pretend they are users. The result is terrible and one more time, the most people involved, the better.
• Overcrowded projects with an incredibly percentage of bureaucracy, including testers, managers of all types, etc. Yes, the most people involved, etc etc.

Definitely, the IT Consultancy Model is not related to Software Engineering at all because a very early business decision: Engineering is not required, Quality is not required. We have to compete and we have to make as much money as possible. Therefore, it does not give any role to technology, Quality methodologies nor engineering in general because the only target is to provide provide human services that eventually are related with some type of running code, getting a lot of money from their customers on the way.

This model is really extended in the software industry and it is terribly bad for product-oriented companies, start-ups, Fintechs and software departments. Why?

In these types of companies this model is not applicable at all (and yet, it is the most extended). These companies manufacture a product, a given piece of software and all the engineering context that makes possible to keep accessible, up and running software components in a scalable and responsive way. The product has to have the minimum cost as possible considering the making and the maintenance in time as well as to provide the maximum benefit, increasing the prestige of the producer. It is the so-called Product Model ans usually considered as the opposite the Consultancy model.

The real question here is, why companies that clearly are not Software Consultants adopt the Consultancy Model? Traditional mindsets, source of the IT staff, lack of skills and technical direction and ignorance are the usual causes.

In the Product model the presence of overpopulated projects and not technical departments does not make any sense,

• What are the main features of the Product Model?
• We do not sell people, we sell software.
• We cannot afford a legion of not technical people. This is terribly expensive and inefficient. The teams are composed by the technical team at a 90%.
• Bureaucracy does not make any sense.

To reduce to maximum the percentage of not technical team we need the right methodologies and procedures. Automation and Continuous Delivery (some people call it Continuous Deployment) offer to us the best combination of advantages and Return of Investment. We are going to demonstrate this thesis.

The Consultancy Model is not maintainable any longer. The lack of Quality, the competitors from India and South America with ultra-cheap workers, the naiveness of most of the solutions, etc made a difficult situation for IT Consultancy companies from Europe and North America . But the worst came later.

The appearance of the giant Cloud providers has been a mortal event for the IT Consultancy model. Cloud giants provide more and more services exclusively based on their infrastructure. The number, versatility and quality of these services have increased with no stops. They have started to occupy the consultancy sector, making it smaller. It will go to worst in the near future. Cloud providers do want ALL the sector, not only a piece (a big piece by the way). So, it is expectable that the consultancy industry will get less and less room until it eventually disappears.

In this case, given a model that is clearly obsolete, expensive and not maintainable, which are our options? What are our possibilities for a successful transition?

What is Automation + Continuous Delivery + SRE?

We'll make this short, easy and straightforward. Many details will not be taken in consideration because that is not the target of this post.

Automation. We are talking here about automatic procedures, composed by automatic phases with automatic triggers. Automatic processes provide always clear traces and reports. Other automatic actions are depending of the result of other automatic processes.

Special mention to automatic tests, covering all possible cases:

• Build Time Tests (Unit and Integration). They are part of the build targets and therefore are always automatic. What processes are completely automated?
• Run Time tests. This type includes tests against software components already built, up and running in environments that are clones of Production. Functional Tests (we test functional requirements, features, E2E), non Functional Tests (load, stress), Security (e.g. penetration tests) are included in this category.
• Detection of commits to main repository
• Static Code Analysis
• Deployments
• Injection of configurations
• Orchestration of the different stages

Yes, we are talking about Continuous Delivery Pipelines. It is because Automation and Continuous Delivery are tightly coupled and we work seamlessly with both concepts.
Continuous Delivery is based on Pipelines, automated sequences of automated processes that bring built software to Production through a chain of successive tests, following High Quality procedures without any manual intervention.

We have included in this blog several articles about SRE, the third element of this combination. It is a complex subject. But we understand SRE as a set of practices specially fit to Cloud Native IaC and Continuous Delivery based on Automation. The Kubernetes ecosystem makes a lot of sense when considering the possibility of spawning infrastructure rapidly emulating Production conditions. Besides, the development team works also with SRE practices, merging software development and looking after the software components until reaching Production, making the chain of responsibility along the pipeline and avoiding silos that separates developers from operations: they are now the same people.

ROI Calculation

I'll include here a good report about this topic that covers many aspects of modern delivery based on Automation, Continuous Delivery and most recent practices in DevOps. This report for Zend by Ido Benmoshe includes some key concepts and ways for a most accurate calculation of the application of these practices in Software Product Oriented companies. I strongly recommend to read the document. Anyway, we'll try to summarize here the main concepts.

The traditional ROI formula divides the net gains from an investment, by the cost of the investment.

ROI % = (Gain from investment - Cost of investment) x 100 / Cost of investment

But this is only one of the basic key indicators. Other PKIs are considered in the paper to estimate the real benefits of automation and Continuous Delivery.

• Cost Savings in operations (project budget)
• Cost Savings in maintenance
• Recovery Time
• GTM: Gains on Time to Market for changes and new products
• GHC: Gains on Headcount impact
• GQL: Gains on Quality Level
• GFX: Gains on Flexibility
• Productivity
• Soft Incomes

There are different indicators in the paper, considering Time to Market, general Headcount, Quality and Flexibility.

Soft variables in the ROI are considered as well. It is not something to be despised given the situation of the job market, specially if you want to offer appealing positions to attract the best talent as possible.

Well, it's true. Continuous Delivery + Automation + SRE are much faster in the resolution of software issues, allowing a much easier finding process, research and deployment of hot-fixes.

Sample Cases

The Consultancy Model: Traditional Waterfall Project

The main features of this model are:

• They pretend they've adopted Agile. They lie. Their procedures, practices and task management are completely waterfall. But yet, they think are Agile.
• Manual testers occupy a huge portion of the budget. They block delivery and perform error-prone user cases. It is expensive and most importantly, they do not ensure the complete perfection of the product. Besides, manual testing causes a lack of confidence on the technical team, provoking the rebound effect given the developers think any possible defect will be reported by QA and if it is not found it is not his problem any longer.
• Management is not Agile and tries to occupy all the space as possible. I've seen several project managers testing products. They want to control every meeting, every feedback, etc. This micro-management anti-pattern is a toxic bad practice inherited from years of legacy, expensive and inefficient projects and products.
• We have here traditional environments (QAT, UAT...). They are not replicas of Production and do not provide the right context to detect issues. The development team has not access, the communication with Traditional IT is slow, bureaucratic and inefficient.
• A bunch of developers try to write code with only a partial coverage of unit and integration tests, covering only the build-time phase. It is not strange they work with monoliths or macro-services (just a few of them).

The worst part of the Consultancy Model is its projection in time:

• Manual testers will be needed FOREVER (or at least until the product is dead).
• Traditional IT does not participate on the maintenance but only to be there (bureaucratic) and to make things slower with restrictions and boundaries (silos) without any counterpart.
• The developers are tied to the project/product FOREVER, trying to fix issues found by the manual testers but most of times live issues reported by the customer.

The result of the Software Consultancy project is, a very high cost of development and a ultra high cost of maintenance. Too big size of projects, poor or null re-usability (projects are real silos), authentic monoliths from the software and management point of view. The very expensive operational cost is mitigated with poorly experienced developers and outsourcing, resulting into general reduction of Quality and an exaggerated ratio of repetitive tasks.

So, this is the preferred model in Software Consultancy companies because the customer pays for everything.

The Product Model: Agile + Continuous Delivery + Automation

On the other side these cases:

• It keeps a limited role for management (purely for administration) while they can perform a good role as Agile coaches if needed.
• Architecture is fully involved in engineering and infrastructure. No technical team members without writing code.
• Traditional IT has been replaced by the SRE squad (please read this and this), composed by the engineering team and some core specialists. The goal is, engineering to provide all levels of technical support.
• SRE practices and Cloud Infrastructure allow to enable up and running images of a system in a very short time. The Continuous Delivery Pipelines use these raised images of the Production system to perform the different types of automated tests. So, Continuous Delivery pipelines use Stages, temporary images of the Production environment Stages are short-life (only up and running during the duration of the automated tests) and not reachable by humans. No more old-fashion expensive and not reliable QAT, UAT, etc.
• Development is now Engineering. This means they not only write code. They are responsible for the delivery and Quality of the software components to Production. No delegation of responsibility is possible in this approach.

In the Product Model engineers write all the automated tests for the Continuous Delivery pipelines. These automated tests are the physical implementation of the requirements collected in the Business Analysis phase (I do not include Business Analyst in this scope). So, the concept of "broken tests" is something from the past due to misconceptions about methodologies. The automated tests (running against deployed and fully functional software in identical conditions like in Production) are the incarnation of the requirements and therefore, they tell the code what is needed to do.

From the operational cost point of view the savings are huge:

• No manual testers are needed during the development nor maintenance.
• Limited Agile Coaching is much cheaper than traditional micro-management
• Business Analysis coupled to the production of business processes and test scenarios for automation has the same cost as traditional waterfall analysis and is much more profitable as it makes easier the automation of processes
• Cost of the making of automated tests are now part of the development cost and it has to be considered on estimations. As I said above, automated tests are hugely profitable as their repeatability and determinism ensures a good ROI.

What about the evolution of the product??

Small inter-disciplinary squads will take care of evolution with very specific and well defined objectives. Squads are short life by nature, with very specific purpose.

And as I said, no manual testers are needed any longer.

So, what are the conclusions from the points above?

1. The cost of making a product is much lower. Taking for granted a good capture of requirements by Business Analysts (you can read this post and this post about this important topic) the increased cost of the creation of automated tests is compensated by the removal of QAs and the re-sizing of Management areas.
2. Once the product is in Production, the ROI increases exponentially as the automated tests are repeated in time with no extra cost (only infrastructure). No manual testers are needed.
3. The ratio efficiency/quality of service/cost of Traditional IT is a disaster. In the Case 2 model SRE offers the optimal efficiency in support quality and speed according to the agreed SLA.
4. The evolution of the product takes advantage of the model, needing just a squad (3-5 members) to develop the changes and extend the automated Quality procedure that will increase the existing ROI.

After a product life of 5 years the operational cost for making, maintenance and support can reach the 33% of the waterfall model.

 

Train, Learn, Create a Platform... Change for adaptation!

The Product Model cannot be based on specific frameworks or re-invent the wheel scenario. The Platform approach is really useful for this model as it provides the proper scenario for a normalized application of automation methodologies.

The revolution of Linux containers and the Kubernetes ecosystem makes a lot of sense when considering how adapted and useful can be an Infrastructure based on Kubernetes clusters and Infrastructure as Code for a Software Platform based on Automation and Continuous Delivery.

The above model for a Platform is only one out of multiple possible options. It is good as a transitional model, the support and focus of a multi-tenant/product API providing central services and federating security and personal data management policies to accelerate the making of client multi-platform apps. Everything made following Automation + Continuous Delivery + SRE.

Training is mandatory and practice makes perfection. It requires time but it is worthy as the ROI is huge.

Conclusions about our God Idea: Return of Investment for Real Engineering

The importance of Software Engineering is not a matter of professional pride or corporatism. It is related to nature of Engineering in any field, the search for the continuous optimization of techniques, materials and methodologies, looking the optimal result with the minimum effort. The application of Engineering principles to software products is mandatory if we want to improve the quality, robustness and life of our products. The application of Automation, Continuous Delivery and SRE is Engineering applied to the production of Software. It is the right way if we want to be competitive regarding cost and quality.

We have tried to show here how old-fashioned projects based on the Consultancy Model are more the product of organizational malfunctions, company bureaucracy, ignorance and mistaken practices rather than the best possible solution. The old approach is expensive, inefficient and not maintainable without a huge cost of money and resources.

The Product Model based on Agile + Continuous Delivery + Automation + SRE is probably far from perfect. It requires skills, a new mindset and a lot of confidence on ourselves. Normally Software companies are really bad on these three factors. However, this approach improves dramatically the cost and ROI of software projects while makes maintainable the existence and improvement of software products in time. Besides, it is dramatically needed by Product-Oriented Software companies. Resistance to change has to be addressed with training and assumption of the targets to achieve at all directive levels.

The creation of APIs and Platforms based on this model makes much easier the integration of new engineers and improves the cost and quality of new products.

The adoption of the SRE practice is absolutely needed to support Automation, Continuous Delivery and the same existence of software products as profitable activities.

Thanks!