A Critical Look at Agile Software Development

Since the Agile Alliance formed and met at The Lodge at Snowbird Ski Resort in February of 2001 to explore better ways of developing software, an apparent love affair has developed with agile practices in the Software Engineering Community. Yet it is particularly strange that there is little in the way of critical analysis and evaluation of agile methods and whether the net result of the adoption of agile brings those adopters any measurable benefit.  Could agile be little more than a shell game and sleight of hand? Regardless, we are concerned that the lack of serious critical scrutiny of agile methods may be contributing to far greater challenges among Software Engineering organizations throughout the world than they started with.  To be certain, there has been to date a dearth of critical academic scrutiny of Agile or the multitude of self-proclaimed Agile Methodologies and the practice of Agile within the profession. Such a critical perspective is necessary.  An objective perspective is needed.  The authors do not intend to simply take a contrary perspective to those of the various proponents of Agile.  Instead, we wish to provide a critical assessment of the Agile Software movement and report a purely objective assessment of Agile Software Engineering in concept and in practice.

1. Introduction

The Software Engineering world is rather in love with the concept of Agile Development.  We cite and reference Dingsøyr, et al.[1] who published a paper trumpeting A Decade of Agile Methodologies, though the intent of this paper is to laud the success of Agile Methodologies versus conducting critical research as to the advantages and disadvantages of Agile Methodologies.  There is a large community of Software Engineers who see Agile as a magic bullet that will solve every problem under the sun.

We will proceed on the Converse assumption.  First, we will examine the Agile Manifesto itself and look for fallacies or pitfalls in the original document.  Second, we examine the most prevalent approaches claiming to be agile and examine the strengths and weaknesses of each approach and attempt to distill whether these agile methods are producing measurable results that differ meaningfully from the results teams achieved using more conventional methods.  Finally, we conclude our critical analysis of Agile Software Development and summarize our findings.

2. The Agile Manifesto – Strengths and weaknesses

Clearly, the team that produced the Agile Manifesto vetted their concepts in a rather formal manner.  They attempted to provide a set of principles -- in the Aristotelian sense -- that Software Developers would employ and that would improve the way that they worked.  Each member of the group assembled to draft this document brought with them a set of experiences and a vision of what they believed constituted best practices in Software Engineering.  In the end, they published a document not unlike the United States Constitution or the Communist Manifesto.  Clearly, the Agile Alliance was keen on starting a revolution against traditional methods.   What is not clear is whether Agile in practice actually solves any real problems or if it results in a superior outcome.  Working software is not a guarantee of any material results.  There is an awful lot of working software in the world that never gets used or which is used incorrectly thus negating the overall benefit of the software.  The Agile Alliance missed the point with the focus on working software.  It is far more important that the software be intuitive, adaptable and maintainable over the full product lifecycle.

One post on “agilemanifesto.org” reports the history of the team, self-named “The Agile Alliance” that drafted the Agile Manifesto on February 11^th-13^th of 2001, and states, “…a bigger gathering of organizational anarchists would be hard to find, so what emerged from this meeting was symbolic – a Manifesto for Agile Software Development – signed by all participants”  The result has been somewhat striking in that there is scarcely a software engineering team throughout the United States that does not aspire to be an Agile Team.  It is not absolutely clear why Software Engineering Teams are striving to evolve their methods from conventional techniques toward agile methods.  Nor is there any empirical evidence that Agile methods improve software engineering in a tangible manner such as reduced overall cost.  On the contrary, it would appear by the enormous size of a typical agile software engineering organization that Agile has accelerated more than the software development process, it appears that Agile is vastly superior to conventional methodologies in using up whatever financial resources are available. It is also unclear how agile techniques are evolving the role of the Software Engineer within the profession and whether the outcome is actually desirable.

3. Strengths of the Agile Manifesto

The Agile Manifesto is a brief document stating that the authors were uncovering better ways of developing software by doing it and by helping others to do it. Clearly the 4 core values that they focused on are intriguing.  We will explore these first:

1)      Valuing individuals and interactions over processes and tools

2)      Valuing working software over comprehensive documentation

3)      Valuing Customer collaboration over contract negotiations

4)      Valuing the ability to respond to change over following a plan

Objectively, there doesn’t appear to be anything particularly subversive about these modest principles.  Clearly there is notable good in valuing the individuals and their interactions over the processes and tools.  To wit, some organizations, especially large organizations had, at that time, become so overwhelmingly stagnated by process and governance that it became prohibitively difficult to get any real software developed at all.  We applaud the work of the signatories and admit that the emphasis on process that was the norm in the 25 years prior to the writing of the Agile Manifesto had reached a threshold of being stifling and often impeded the development of software products. 

Valuing working software over comprehensive documentation sounds pretty good, but we must pause to assess the implications, namely: Is this a good principle?  We have doubts which we will examine in the next section. 

Valuing Customer Collaboration over Contract Negotiations also sounds good insofar as we feel better about Collaborating with a Customer versus engaging in Contract Negotiations with that Customer, but at this too requires further analysis.

We also agree that it is important in many situations to be able to deviate from a plan in order to respond to changes that were not anticipated at the outset of a software project.  Such changes are ultimately guaranteed to occur in all but the most trivial software engineering endeavors.

Thus, on the surface the Agile Manifesto appears to be a rather intriguing document with generally good intentions.

4. Pitfalls of the Agile Manifesto

There are several fallacies and pitfalls associated with Agile Development in practice today. We trace these pitfalls to central omissions in the Manifesto:

1)       Agile methods turn over much of the systems analysis, design and technical architecture to software engineers/developers who may not have the proper skill set to analyze, design or architect such a solution. More often than not these individuals are sequestered from the users of the software and even the key stakeholders who are on point for the system requirements thus there is rarely any means to achieving effective requirements analysis and reaching a clear understanding of the solution domain.  The Agile Manifesto does not discuss the vitally important concern of systems architecture. 

•        Agile practitioners tend to take the quintessential square peg of solution requirements analysis and stuff it into the round hole of a 2-week sprint.  For the most trivial of information technology problems where the teams have access to the system owners and key stakeholders, the time-frame may be acceptable, but for more interesting problems, it is not. Examples where two-week sprints work in opposition to the business need include

1. A quoting system for insurance products, which demands extensive information gathering, and which is governed by complex actuarial tables and regulations is a far-fetched goal for a 2-week investigation.
2. The solution requirements for the guidance and anti-collision systems for a self-driving automobile.
3. The analysis of a system that uses human genome mapping to aid in the development of medications for human diseases also should not be packaged into even a series of 2-week iterations – this is an entirely unreasonable expectation.

• Agile also tends to bracket engineer learning cycles into these two-week intervals.   In reality, some learning takes more time than 2 weeks.
• Agile is not an architectural discipline even though it clearly requires tremendous architectural discipline to implement the requisite continuous integration processes which are at the core of most agile practice and embodied in the statement by the signatories, “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”
• In fact, Agile practitioners have widely learned that Agile methods in practice violate the first and over-riding principle of the Agile Manifesto, namely: that of valuing “Individuals and interactions over processes and tools.” If we are to be honest with ourselves, Agile techniques the world over are emphasizing processes and tools even more than conventional project philosophies and methodologies.  An important maxim is suggested: When adopting agile, avoid the adoption of complex processes and tooling like the plague!
• Too, it is foolish to believe the second principle of the agile Manifesto: that of valuing “Working software over comprehensive documentation” Teams that forego comprehensive documentation are doomed to a vicious cycle of software artifacts that need to be rewritten far too soon after they are delivered, as they will have failed to capture important knowledge relating to the modules that were developed.  Moreover, Testing, even exhaustive testing is not a replacement for comprehensive system documentation, nor is it a substitute for a thoughtful and carefully considered systems architecture that takes into account the functional and non-functional requirements of the solution.  Poorly documented products are almost always used incorrectly!

In the best-case scenario, Agile practices place solution architecture responsibilities are in the hands of a software architect who is secluded from the software engineers, and the software engineers must blindly trust that the analysis of the solution requirements was performed correctly.  Assuming so, they can proceed to build a solution that fits the need. Otherwise, the developers are routinely foisted into the role of back-slope architects cleaning up the mess made by the Software architect whose work is not vetted.  This is most often the crux of the problem:

1) Companies frequently do not employ software architects and do not consider the importance of preparing a sustainable software architecture

2) Companies place the responsibility for defining the software architecture in the hands of individuals who are either poorly suited to the role or who lack access to the sponsors and key stakeholders of the solution.

3) Companies are relying increasingly on Software engineers on agile teams to back in to a solution architecture that will satisfy the needs of the business and we see this as fraught with disaster.

5. Dominant Agile Methods

There are several dominant agile approaches that have been widely adopted within Software Engineering teams over the past 16 years:

• eXtreme Programming (XP)
• Scrum
• Lean Software Development
• Crystal Methodologies
• Feature-Driven Development

            We will examine each of these methodologies in detail in the following sections.

Extreme Programming

    This methodology is predominantly focused on pair programming; however, eXtreme Programming is also underscored by a formal methodology that emphasizes ceremony and checkpoints referred to as feedback loops common to most traditional software processes.  The process diagram for Extreme Programming (below) seems to be quite busy, and seems to add pair programming solely as an afterthought

Scrum

Scrum and its derivatives which include SAFe Agile (an acronym for Structured Agile Framework for Enterprise) are among the most process- and ceremony-heavy of the agile methods.  While, most Scrum teams disdain comprehensive documentation, they employ a broad complement of tooling including tools for continuous integration, test driven development tooling and with increasing frequency they choose to implement convoluted release management and branching strategies which would seem to be entirely contrary to the first principle of the Agile Manifesto, it makes us wonder if Scrum methods are even considered “Agile” by the members of the Agile Alliance.

Lean Software Development

The Lean Software Development concept was developed by Mary Poppendieck and Tom Poppendieck. Their approach presents traditional lean principles as typically found in manufacturing, in a modified form, as well as a set of 22 tools and compares the tools to agile practices. The Poppendiecks' involvement in the Agile software development community, including talks at several Agile conferences has resulted in such concepts being more widely accepted within the Agile community.  Clearly the Poppendiecks are interested in marketing their methodology more than they are interested working software or any of the other principles of the Agile Manifesto.  It seems most process heavy, and therefore, contrary to the first principle of The Agile Manifesto to require the employment of 22 tools to achieve agile practices. Lean Software development would be intriguing if not for these 22 tools needed to employ the methodology. And the authors have not observed any teams actually adopting this methodology in practice 

Crystal Methodology

The Crystal Methodology is an encompassing collection of eight different project lifecycles or methods; however each reflects 7 common characteristics:

1.   Frequent delivery

2.   Reflective improvement

3.   Close or osmotic communication

4.   Personal safety

5.   Focus

6.   Easy access to expert users

7.   Technical environment with automated tests, configuration management, and frequent integration

The focus on frequent delivery and frequent integration in Crystal Methodology is a concern as we have observed that all approaches that focus on Continuous Delivery tend to suffer from the inevitable complexity that Continuous Delivery implies. Notwithstanding, the emphasis that Crystal Methodology places on reflective improvement and personal safety are admirable.  Nevertheless, we have observed in practice that when Personal Safety and open communication are valued, there is the occasion for highly negative influences to monopolize the time of the team thus contributing to bottlenecks and reduced team throughput and performance. Nevertheless the value placed on easy access to expert users is a mitigating factor that is likely to further advance projects employing Crystal Methodology.  We have employed technical environments with automated tests and configuration management and frequent integration and release cycles.  What we conclude is that each software product must be architected with these activities in mind and when software is architected to support such an environment and such processes and is founded in stable architectural patterns, there is no better approach to building quality software 

Feature Driven Development

After examining the remaining approaches to agile we finally arrive at Feature-Driven Development.  FDD is founded in Peter Coad’s concept of Software Features, where a feature similar to a use case in the context of the Rational Unified Process (RUP).  The major activities of FDD include:

1)      Develop an Overall Model

2)      Build a features List

3)      Plan by feature

4)      Design by feature

5)      Build by feature

Each of these steps or activities produces a defined deliverable.  Most important the Feature List is the product of step 2 and is a governing document for the entire methodology.  FDD unlike the other methodologies reviewed here actually incorporates a formal Requirements Analysis process (termed Requirements Envisioning) as part of its Iteration 0: Envisioning, which consists of Initial Requirements Envisioning and Initial Architectural Envisioning the team then transitions to Iteration Modeling followed by Model Storming and Test-Driven Development, which comprise the parts of an Interation 1: Development activity.  Though FDD also has substantially more process and ceremony than indicated by the first principle of the Agile Manifesto, it is clear that the concept of FDD provides a more transparent process than the remaining dominant Agile approaches.  Moreover, it appears that FDD is rooted in the Coad and Yourdon methodology of Object-Oriented Design and incorporates practices that we view as similar to Rapid Application Development with FDD’s focus on Requirements Envisioning (reminiscent of JRD) and architectural modeling as well as joint design activities (reminiscent of JAD). The emphasis on Test-Driven Development (TDD) is excellent, but in the end analysis FDD, while a self-proclaimed agile methodology is difficult to classify as agile, due to its notable departures from the first principle of the Agile Manifesto.  Still the approach is compelling due to the fact that it appears that the knowledge gained over the course of developing the software is documented, if only informally, thus the knowledge is not lost which seems a cornerstone deficiency in the other methods which we studied. 

The purpose of this critical perspective of Agile Software Development is not to say that there are no benefits and nothing to be learned from agile software practices and methods.  Rather, we must be realistic in terms of what we expect from the adoption of agile methods.  There are some good ideas in the agile manifesto, but there are also ideas that are intrinsically in conflict with the reality of business and software engineering principles.  Good engineering principles should not in practice be shunned solely due to what amounts to religious fervor.  There are no magic bullets in Software engineering and as scientists, we should take exception when our leaders parade through the main thoroughfare of our profession without their knickers, asserting that Agile Software Development is the most beautiful paradigm they’ve ever seen.

In short, before adopting Agile, it is vital to frame the issue carefully and ask, “What problem are we attempting to solve by changing our fundamental approach to how we develop software within our organization?  The authors have observed some organizations that employ the ceremonial aspects of agile – such as the daily scrum -- while keeping much of their traditional waterfall process intact.  The end result is the ability to more closely manage the time-sensitive topics within a project while providing a higher level of communication within the team.  The authors are also familiar with certain CTOs who are strongly opposed to Agile methods in any form as they assert that they have been burned by Agile and will not accept such notions within their organization.  One team in working under a particular CTO was already practicing a lean approach to software engineering, which was not, strictly-speaking, Agile.  The team simply reported that they were practicing lean software development and truthfully testified that it was not “Agile”. The reality of this team's culture was as follows:

1. They did not lack documentation of the knowledge gained throughout the development effort
2. They were not over-burdened with either process, tools or ceremony
3. They worked together harmoniously
4. They had tremendous knowledge of the domain in which their products were targeted
5. They understood their mission and purpose with great clarity

As a result, this team consistently delivered outstanding software and exceeded expectations time and time again.  In the end analysis isn’t this the reality that the Agile Alliance was really targeting?

Leaders and leading researchers in the Software Engineering discipline must strive to keep in front of the trends in the profession and caution practitioners from dealings with snake-oil purveyers and hair tonic salespeople.  The tremendous flocking to agile that we observe throughout the profession should be a cause for alarm.  There has been no legitimate vetting of Agile practices and no metrics to help practitioners to make an informed decision whether Agile offers and tangible benefits in terms of cost or time to market.  Without such data points there is no way for software engineering teams to make any informed decisions about the advantages or disadvantages of adopting Agile methods.

---

1Dingsøyr, Nerur, Balijepally and Moe -   A decade of agile methodologies: Towards explaining agile software development, Journal of Systems and Software V85, Issue 6, June 2012 Pages 213 – 221 (http://www.sciencedirect.com/science/article/pii/S0164121212000532)