Succeeding with Agile.

Abstract

Agile has been all the rage for a decade.  Some organizations have been more successful than others in adopting agile methods.  Agile has been marketed very heavily both by software developers and by the members of the agile alliance; therefore, it is important  to separate the wheat from the chaff. To be certain, agile is not a silver bullet.  Merely, adopting agile ceremonies will not ensure that an organization will reap benefits from adoption of agile practices  Simply stated: an organization cannot expect a stellar transformation in software engineering outcomes by proclaiming "We are an agile shop."  Nor will the goal be achieved by overlaying scrum ceremony on top of a broken software engineering approach.

To place agile in the proper context, we raise four germane questions:

1)Why should an organization adopt agile?
2) How can agile benefit an organization?
3) What decisions can aid in successful agile adoption and ensure that an organization reaps the  intended benefits of agile?
4) What software products are good candidates for agile?


Why Agile? How can Agile benefit an organization?

These are the two most important questions when contemplating agile, and they are clearly reciprocal to one another.  An organization should adopt agile if it is very firmly committed and resolved to long-term improvement in software engineering cost controls and overall improvement of software quality  This is precisely how agile can benefit an organization; however, Agile does not come with this guarantee.  Agile must be adopted with pragmatic expectations and a willingness not only to disrupt traditional conventions and practices, but to embrace the foundational changes that are necessary for a successful agile practice.

What decisions can aid in successful agile adoption and ensure that an organization reaps the  intended benefits of agile?

There are many changes that are necessary and proscribed for any organization adopting agile.  There are no shortcuts here, all of the proscribed tooling and practices are essential to getting agile right. All of these proscribed elements are necessary and, in fact, crucial to employing agile successfully, and thereby achieving the outcomes that underpins the question "Why Agile?"  These prerequisites for agile success are as follows:

1. An organization must possess the willingness to part with brittle or failed software architectures that do not adapt well to Agile practice.  In short Agile will not work with any old software and there is no way to evolve a brittle or unwieldy architecture through iterative development to arrive at a viable agile architecture.  Consider the following realities:
• One of the most fundamental paradigm shifts that is vital for Agile software development is test-ability and test-driven development.  Software that is not designed for test-ability is not well-suited for agile teams. 
• Experience shows us that organizations who try to superimpose agile on software that is not designed to be testable are prone to fail at agile adoption.  
• Similarly, certain software architectures fundamentally lend themselves to agile development where as others do not.  
• An organization must be pragmatic about whether it's underlying software architecture is sufficiently agile and understand that an unwieldy architecture will invariably lead to a failure to succeed with agile.  A governing maxim: software that is failing without agile practices will almost certainly yield the same results in an agile regime.
• Agile doesn't just happen because it's proclaimed.  It takes a great amount of discipline at all levels within an organization to successfully employ agile practices.  Furthermore, there are fixed costs for successfully managing agile development which cannot be overlooked.  The organization needs to have a reliable and efficient platform and tooling to manage it's agile deliverables.  This goes far beyond SCM. products such as JIRA,HP ALM and similar ticket management technologies are prerequisites for a successful agile implementation, simply because there must be transparency and line-of-site visibility of every in-progress activity solution at any point in time for agile to work.  Tooling alone does not achieve the discipline.  There is a need to devote time and resources to triage status, activities and iterations on a burn-down list.

• Agile is essentially a gateway drug for Continuous Integration.

• In fact you cannot realistically implement agile development methods without having a comprehensive set of tooling for build and release automation that are needed to adhere to the agile cadence.
• SCM is a given, Build Automation with a tool such as Jenkins is also essential and automated deployments are needed.  These in turn are commonly gateway drugs for tools such as Artefactory and a variety of other software products that are commonly found within agile teams.

• Agile is also a gateway drug for DevOps and most agile teams cannot succeed without a strong DevOps team supporting the work. 

• DevOps greases the skids for Agile development teams and effectively clears the path for innovation by the Development Team by validating key platform decisions and verifying that the technologies that are proposed in a given Epic will function reliably within the release environment.  DevOops also serves as the gate-keepers of the environments so that each environment within the release cycle can be certified for use before developers introduce newly released sofware in the environment that may be faulty and may have side-effects that destabilize the entire release environment.

In short, it is important for decision-makers to avoid getting sold on the hype of agile.  It is also important to firmly grapple with the realities of agile and the importance of agile planning and deliberate and measured implementation of agile.  Most of all it is imperative that decision-makers not allow themselves to make the mistake of trying to superimpose agile on a brittle and failed software architecture, nor can it be expected to transform legacy code in even a few short years, much less the few short months that management would like to see results.

What's Next in Software Engineering?

In recent years, I have become disenchanted with the java community and also most software engineering practiced in businesses throughout the country.  The most commonly practiced approaches in software engineering have produced little more than prohibitively expensive approaches to developing software and approaches that far too often result in unpredictable and unexpected outcomes, including systems integration mismatches, software that is not readily adaptable to the constantly changing information security landscape and software that simply doesn’t work as desired.  These problems are compounded such that increasingly, software is delivered that simply doesn’t work and fails to meet the solution requirements as defined in any manner that resembles the reason that the software project was undertaken in the first place.  Furthermore modern software solutions are delivered with utterly convoluted user interfaces and such utter lack of documentation that no one really can tell you how the software is designed to function.  The end result is a prohibitively costly model of doing business.  This paper looks at a variety of the causes contributing to this new software engineering crisis and poses the question: “What is next in Software Engineering?”

Frankly speaking, the Java ecosystem has become needlessly cumbersome and developers spend absurd amounts of time working around the bloated technology that is Java with Spring Framework plus 100 or so alphabet soup libraries that are needed to make java useful for solving real-world problems then add to that 7 or more flavors of JavaScript libraries needed to build Web UIs, plus an assortment of scaffolding tools, build and test frameworks and more alphabet soup needed to make javascript useful.  All of this rather than creating software solutions that address real needs.  Call me dense or foolhardy, but when too much technology is getting in the way of building solutions that actually accelerate the construction of reliable software, it causes me to question the decision-makers and others who influenced the technology choices in the first place.

Perhaps I'm too critical of the Java Community.  Perhaps at the end of the day it's really the fault of Systems Architects who are far too wrapped up in jargon, buzzwords and bleeding edge frameworks and cutting edge technology to see the forest through the trees.

What attracted me to Software Engineering in the first place was the ability of engineers to build functional, full-featured software solutions by employing a programming language in a very elegant manner.

Once upon a time I was especially attracted to the C programming language due to the elegance of the language, it's syntax and the various constructs that were present in the language and supported by C compilers.

As a professional developer, I also experienced some of the limits of the C programming language in the real world.  In time, Java caught my attention as it added the object-oriented constructs that were missing from the conventional C programming language without introducing a lot of bloat or inelegant features to the language, which I perceived as a failing of C++.

I was also attracted to functional programming languages such as LISP and PROLOG, and I found the list-processing abilities within LISP particularly inspiring so much so that to this day, I find myself building and/or employing parsers that are very similar to techniques that are found in LISP.  I also find myself building semantic constructs such as those found in PROLOG, and use these to craft solutions for complex domains with rich semantic-oriented knowledge.  But of course I typically try to reuse what is there in the stack and the libraries, but Java has lost much of it's luster for me.

I have been more impressed with the .NET stack than the Java stack.  Perhaps .NET benefits from the fact that it has mostly been a closed-source platform until recent years, whatever the case, it is far more elegant in my opinion than it's SUN-come-Oracle counterpart technology.  But .NET is showing some signs of bloat too.

My mind is always looking for or building short-cuts that enable highly maintainable software that exhibits high reuse and high-reliability.  My philosophy: "Why build a solution with 1 million lines of code if it can be done in 7500 lines?  Not only is the 7500 line solution going to be less brittle and have fewer points of failure, but it will be orders of magnitude more maintainable.  I was particularly pleased when command line compilers gave way to GUI-based Development Environments.  In my opinion, if I was building slick GUIs for my applications, why wasn't I working within a development environment that came with a slick GUI?  In my assessment Software Engineering as a science has failed to lead the profession in the  proper direction.  We've lost sight of the principle that making the construction of rich software more accessible to all of the best and brightest will inspire them to build software with greater functionality, flexibility, portability, maintainability, reuse and security.  That is not to say that this doesn't happen today, but I've been frequently disappointed in the cumbersome techniques that are employed in building software products and solutions -- within both the largest companies in the world as well as the smallest companies.  Why  would anyone deliberately place so much red tape in the way of building useful software?  Perhaps when building for the DOD or NASA, it is unavoidable, but for most software built today, it seems that the programming languages and the tooling that is used gets in the way.

Is the software crisis any better than it was 30 years ago? Arguably it is not.  Perhaps there are a greater handful of successful projects, but these are almost assuredly offset by a greater number of failed projects.  Just because we have first-hand evidence of software from Google, Amazon.com, Microsoft, Facebook, and countless other technology companies that is used every day, with some measure of reliability and functionality doesn't mean that there aren't thousands of other products that died within the software engineering organization.

So, too, I've learned that Agile is little more than a gateway drug.  As anyone who has transitioned to Agile understands, to do agile properly requires the procurement of a vast soup of tools that are needed to properly administer Agile teams and also tools that are needed to achieve continuous integration (CI), which after all is the premise of most Agile methods, Don't get me wrong, I appreciate the idea that iterative development is at the heart of building working software   But CI tooling today requires far too much cumbersome work, and even more ingredients in its soup to employ properly.  It seems that a leap is needed to achieve CI more seamlessly.

Perhaps in time that is what will evolve, but i don't imagine that happening through a community process or any of the approaches that have brought the state of the art of Software Engineering or Java development to the place that we are at today.  To improve upon software development methods, we need to think about software engineering and Continuous Integration differently.  I firmly believe that this is precisely what is next in software engineering.  By thinking about the complexity of modern software engineering methods differently, we will find methods and technologies that enable software engineers to do more engineering in shorter amounts of time and there is great goodness in such a vision.  I am confident that this vision will soon be a reality thanks to the work of like-minded software engineers, researchers and innovators.