Featured Post

Thursday, July 12, 2018

What's Next in Software Engineering? A Story of Disenchantment

In recent years, I have become rather disenchanted with the java community.  Frankly speaking, the Java ecosystem has become needlessly cumbersome and developers spend absurd amounts of time working around the bloated technology that is Java with Spring Framework plus 100 or so alphabet soup libraries that are needed to make java useful for solving real-world problems then add to that 7 or more flavors of JavaScript libraries needed to build Web UIs, plus an assortment of scaffolding tools, build and test frameworks and more alphabet soup needed to make javascript useful.  All of this rather than creating software solutions that address real needs.  Call me dense or foolhardy, but when too much technology is getting in the way of building solutions that actually accelerate the construction of reliable software, it causes me to question the decision-makers and others who influenced the technology choice.

Perhaps I'm too critical of the Java Community.  Perhaps at the end of the day it's really the fault of Systems Architects who are far too wrapped up in jargon, buzzwords and new technology to see the forest through the trees.

What attracted me to Software Engineering in the first place was the ability of engineers to build functional, full-featured software solutions by employing a programming language in a very elegant manner.

Once upon a time was especially attracted to the C programming language due to the elegance of the language, it's syntax and the various constructs that were present in the language and supported by the compilers.

As a professional developer, I explored some of the  limits of the C programming language in the real world.  In time, Java caught my attention as it added the object-oriented constructs that were missing from the conventional C programming language without introducing a lot of bloat or inelegant features to the language.

I was also attracted to functional programming languages such as LISP and PROLOG, and I especially found the list-processing abilities within LISP particularly inspiring so much so that to this day, I find myself building parsers that are very similar to techniques that discovered in LISP.  I also find myself building semantic constructs such as those found in PROLOG, and use these to craft solutions for complex domains with rich semantic-oriented knowledge.  But of course I typically try to reuse what is there in the stack and the libraries, but Java has lost much of it's luster for me.

I have been more impressed with the .NET stack than the Java stack.  Perhaps .NET benefits from the fact that it has mostly been a closed-source platform until recent years, whatever the case, it is far more elegant in my opinion than it's SUN-come-Oracle counterpart technology.  But .NET is showing some signs of bloat too.

My mind is always looking for or building short-cuts that enable highly maintainable software that exhibits high reuse and high-reliability..  My philosophy: "Why build a solution with 1 million lines of code if it can be done in 7500 lines?  Not only is the 7500 line solution going to be less brittle and have fewer points of failure, but it will be orders of magnitude more maintainable.  I was particularly pleased when command line compilers gave way to GUI-based Development Environments.  In my opinion, if I was building slick GUIs for my applications, why wasn't I working within a development environment that came with a slick GUI?  In my assessment Software Engineering as a science has failed to lead the profession in the  proper direction.  We've lost sight of the principle that making the construction of rich software more accessible to all of the best and brightest will inspire them to build software with greater functionality, flexibility, portability, maintainability, reuse and security.  That is not to say that this doesn't happen today, but I've been frequently disappointed in the cumbersome techniques that are employed in building software products and solutions -- within both the largest companies in the world as well as the smallest companies.  Why  would anyone deliberately place so much red tape in the way of building useful software?  Perhaps when building for the DOD or NASA, it is unavoidable, but for most software built today, it seems that the programming languages and the tooling that is used gets in the way.

Is the software crisis any better than it was 30 years ago? Arguably it is not.  Perhaps there are a greater handful of successful projects, but these are almost assuredly offset by a greater number of failed projects.  Just because we have first-hand evidence of software from Google, Amazon.com, Microsoft, Facebook, and countless other technology companies that is used every day  with great reliability and functionality doesn't mean that there aren't thousands of other products that died within the software engineering organization.

So, too, I've learned that Agile is little more than a gateway drug.  As anyone who has transitioned to Agile understands, to do agile properly requires the procurement of an alphabet soup of tools that are needed to properly administer Agile teams and also tools that are needed to achieve continuous integration (CI), which after all is the premise of most Agile methods, Don't get me wrong, I appreciate the idea that iterative development is at the heart of building working software   But CI tooling today requires far to much elbow grease, and more alphabet soup to employ properly.  It seems that a leap is needed to achieve CI more seamlessly.  Perhaps in time that is what will evolve, but i don't imagine that happening through a community process or any of the approaches that have brought the state of the art of Software Engineering or Java development to the place that we are at today.  To improve upon software development methods we need to think about software engineering and Continuous Integration differently.  I dearly believe that this is precisely what is next in software engineering.  By thinking about the complexity of modern software engineering methods differently, we will find methods and technologies that enable software engineers to do more engineering in shorter amounts of time and there is great goodness in such a vision.  I am confident that this vision will soon be a reality thanks to the work of like-minded software engineers, researchers and innovators.

Wednesday, November 1, 2017

Is a Lack of Diversity Equivalent to Racism?

If you listen to Liberals and Conservatives you get a strikingly different characterization of our country.  Liberals want you to think that the country has never been more divided and that a divisive narrative is playing out in this country that threatens to doom our democracy; whereas Conservatives will have you think that the Liberal narrative is little more than sour grapes over the fact that Hillary Clinton lost the presidential election.
I have a theory which is different from the message that we hear from the President, from the Republican leadreship or from the Democratic leadership.  Let's take a look at the makeup of the leadership of the two parties.
Democrats have been preaching diversity for over 40 years.  The topmost leadership of the Democratic party are people who primarily come from New York and California, Michigan and Illinois.  These are states  where every day you are confronted with  diverse people and diverse mindsets every day, especially within the most populous cities in those states.  In Los Angeles, San Francisco, New York City, Chicago and Detroit, you are exposed to people of all races on a daily basis.  You are literally surrounded by every color of the Rainbow Coalition constantly.  Acceptance of diversity is not a question, it is an imperative.  You either accept diversity or you are out of step with  your community in a most profound manner.  However, even for liberals in Middle America, the daily exposure to a divers population is less common. you may hold old fashioned attitudes and not be out of step with your community.
Democrats and liberals in general would have you believe that failure to accept every modality of diversity including every category within the LGBTQO continuum is racist and intolerant.  Perhaps this is true and perhaps the liberal leadership is correct, but is it not equally intolerant to call out everyone in the country who is not awake to the LGBTQO continuum as racists. Is it evil to be unenlightened?  I think not!  Rather, I think that it is wrong to label middle Americans as racists and anti-diversity solely because they have not been exposed to diversity.  This would be akin to labeling all sea life as alien simply because we cannot survive in a salt water ecosystem!  A fish is no more evil simply because it is a fish.  No more is a farmer in Iowa or Wyoming a racist simply because they do not have the same concept of diversity as someone from San Francisco or Chicago.  They may not be tolerant or enlightened, but they are also not racists.
Liberals do not accept that there is a difference between lack of tolerance or lack of accepting diversity and racism. It seems to me a rather intolerant perspective to classify everyone who doesn't think the way you do as intolerant, opposed to diversity and racist.  Clearly not all who do not think the same as the liberal establishment are not intolerant racists.  Moreover, isn't it hypocritical to label people who don't think the way that you do? Isn't that a lack of diversity itself?

Saturday, October 21, 2017

An unexpected encounter with Bharpinder

Graduating from an accredited University Software Engineering Program is quite the achievement.  It’s even more exciting if you are recruited by TNGT Software with a offer of a salary that may actually provide the means to allow you to pay off the 6-figure student loans that you incurred during your 4-year program at the University of Texas at Austin and off you go to Mountain View, California, where they don’t talk the way that you do and they don’t dress the way that you do.  In a matter of days PETA has a group of volunteers following you around running background checks on you that TNGT hadn’t even considered:

Are your motorcycle chaps real leather? (was a cow harmed in the manufacture of your safety equipment?) Do you use environmentally friendly hair gel in the morning? And finally, “Why is it that you appear to live at TGNT’s headquarters? – Oh! That’s right you really don’t make enough money to pay your student loan payments and live in the most expensive region in the Country.  Instead of rocketing to the top of the organization, you find yourself toiling for 10 to 14 hours daily, including weekends just so that Bharpinder Nagasalamandanam doesn’t out-toil you with his own effort, which is underscored by mounting a goal of 18 hours per day plus weekends, and you quickly are reassigned to a team of under-performers who are tasked with reverse engineering a 20 year old application that was converted from Microsoft Visual C++ to Java using an automated code conversion utility.  Your initial assessment of this new assignment is flawless:

  1. The code needs to be rewritten from scratch. There is nothing that can be salvaged.  You have read this code and clearly it was code-generated by another program and though you do not have access to the original source code you recognize the entire asset is complete and utter garbage.
  2. You’ve looked around for any accurate and credible documentation of this module, but you have been given 3 answers when you ask:

  • We have a cryptic, hard-copy document provided by the vendor of the original C++-developed solution, but it’s in JoAnne Stevenson’s desk, and she will not share it with anyone.

  • There is an equally cryptic appendix that describes more business rules that are referenced in a document maintained by a 3rd-party company, but their version reflects thousands of changes to the specification that have been implemented in the ensuing 15 years after the original software was written.

  • The appendix has an even more cryptic reference to another document that hasn’t been in circulation for over 10 years, and another not-so-cryptic reference to a web site that contains additional code/value pairs that you will need evaluate within your module, but notably many of code/value references are completely missing from the industry standards organization’s website, or any other legitimate resource. 

Your team is practicing a combination of eXtreme Programming, Scrum and Kanban, but no one has a name for this Agile potpourri, and it’s not clear which practices of the Company’s agile variant that you are using is responsible for, governing which activities in the method, but Mr. Salamandanam seems comfortable and seems to know what needs to be done. He’s willing to help you, but it seems that part of the agreement will be that you will apprentice under him for 6 months, and only if you figure out the secret to deciphering the agile ceremony du jour will you be allowed to actually work on Bharpinder’s team going forward. In a matter of 4 days, the team concludes that you are rather dense, and you are re-assigned to build unit tests for a module that was developed based on User Stories that were written by a Stanford Undergraduate software Engineering Student who has dropped out of College and is no longer involved in the internship program with TNGT. 

From software engineer you quickly descend the corporate ladder to settle comfortably as a Test Developer reporting to the QA Team with an after-hours role as Bharpinder’s gopher.  After three weeks, you realize a necessity to pronounce that you not, in fact, gay - that everyone has misinterpreted your status on the LGBTQO continuum for which you have been written up by Human Resources, in which they cite your failure to uphold TNGT’s commitment to diversity which is a core tenet of the new Employee Guidelines.  

You are placed on a 2 week probationary term with no pay, though you will be allowed to purchase food from TNGT’s world-famous cafeteria which offers cuisines from all over the world -- Sadly, there is nothing on the menu that you would eat back home in Austin, TX. By the start of your third week, you sadly deliver your letter of resignation to HR admitting that you are simply not a very good Software Engineer, and that you’ve taken a new job at Yahoo at double the pay as a writer for Yahoo Sports.  Finally you are able to enjoy your two favorite things in life.  Computing and College sports!  Not to worry about Bharpinder.  He’s found a politically correct apprentice to work under him.

Sunday, September 24, 2017

Working for nothing

It is the year 2030.  Inflation as measured by the US government remains at an all-time low of 0.65%.  Unemployment is remains low at just 3% nationally. Interest rates remain below 3.5% for 12 years running, following a second recession and another round of quantitative easing by the FED. People of every race, creed, color, gender(all 5 of them) and national origin now work for large corporations for free -- just for the sake of needing healthcare insurance..  Almalgamated Healthcare is the largest corporation on the planet with over 500 Trillion USD in cash reserves -- enough to pay of the US National debt (currently sitting at $100 Trillion. 5 times over.  Amalgamated (AMC for Short) is now diversified into government consulting projects, and literally runs the US Government.  No one doubts the power of AMC.  They have both sustained and torpedoed 3 presidential administrations, including Trump (2 terms) Michelle Obama (2 terms) and in a surprising twist, Hillary Clinton won the 2028 election at the age of 80! After only one year in office, AMC lead a successful campaign to have her impeached, alleging Corporate Fraud within the Clinton Foundation, Colluding with Russia in the fixing of the 2028 elections, and Money laundering.

Saturday, August 12, 2017

The Sweet Life

It is the year 2085, and statistics prove that there are more dead people on Facebook than living.  That's partly because hardly anyone joins Facebook anymore, now that AT&T provides unlimited Social Vid for only $800 per month with purchase of an IPhone Mark 90.  The last movie theater closed about 5 years ago.  Pa took me to a movie show once. I think that we saw Planet of the Apes 3000.  I still can't understand why anyone went to the movie shows --  $75 per ticket, $60 for a soda and 3 years in prison if you sneaked in candy from outside the theater.  You can see all of the latest movies in VR on YouTube courtesy of AT&T on the stunning Perfect Definition (tm) display of the IPhone Mark 90.with Retina-Vision.Besides, there hasn't been a great movie in decades according to Pa, and I think he's right.  Cool kids don't watch movies or play video games like Pa did when he was my age.  The Cool kids spend their summers on the Coast of Spain and drive there in their Chevy Camaro hovercrafts.  I've been there every summer since I was 9.  It will be six years in a row this summer, and I'm still not clear why my school advisor insists that I need 3 years of foreign language classes to get into college.  Heck I'm fluent in Espanol.  Mom and Dad don't spend much time at home now that they each have their own personal time machines  They're always going back in time to play the stock market with Grandpa and Grammy. Shoot!  We've inherited billions and there's more rolling in every day..  Yeah, it's a pretty sweet life, but someone's gotta do it and it may as well be me.

Tuesday, February 28, 2017

Commentary on President Trump's address to Congress

There was a time in our country that it didn't matter what the president spoke about, those in the party that opposed the president conducted themselves with dignity and demonstrated respect for the President.  The democratic party is terribly disappointing.

Personally, I have a problem when the Democratic congress-people are giving the thumbs-down to the president on national TV and feel that they are somehow justified in doing so, when the president clearly articulated that he was seeking unity and cooperation and reaching across the aisle.

Watching Elizabeth Warren poo-pooing the president is utterly disgusting. Shameful even.  It's also rude and unacceptable for the democrats to rush out of the building so quickly after the speech.  The democrats are clearly taking a hard-line position, based solely on allegation that the President is a meglomaniac, a tyrant, a dictator and a racist.  The democrats are claiming that they are worried about the country, because such a man as Donald Trump is in a position of so much power -- Get over it.  The democrats lost the election just as they did in 2000! Life goes on -- lose with dignity!

The democrats claim that they are the guardians of democracy and that they are inclusive and forward-looking, even non-partisan.  They claim to be the party of looking forward, not back.  How so? The democratic response claims simply that the President's cabinet is just a group of billionaires who don't care about the common folk.  From that they defend that social defiance is acceptable, because it somehow makes the president is horrible.  Ha!  As Ronald Reagan always said, "There you go again."  I hated Reagan with a passion. I still feel that he was a horrible president and not my president, but the Democrats are committed to doing everything within their power to oppose the President without a tangible platform of their own.  The claim is that he is a dangerous man who has declared war on immigrants.

What I hear is partisan bickering by the Democrats -- baseless claims that President Trump is eroding the credibility of our nation.  I have seen fear-mongering and divisiveness the likes of which I never imagined that I would see in the USA. The Democratic response is that our representatives are out of control, because they (The Republicans) fail to understand that they work for us (the people).  I don't feel that the nay-saying response by the Democrats has any solid ground.  The speech was a positive speech and the President clearly reached across the aisle.  The Democrats refuse to offer any specific policy proposals, but then they have the audacity to claim that the President's proposals are empty.

The bad behavior and stonewalling by liberals throughout the country is simply not acceptable. History may prove me wrong, but from my perspective the Democratic party is utterly out of touch with Americans and is inciting riots an division among people all across the country.  I have observed friends and family members severing ties over politics and it saddens me.  More troubling is that the Democratic party is the party inciting this divisiveness with rhetoric and propaganda beyond belief.

The actions of liberals, specifically the deliberate action to stoke the fires of fear is unacceptable.  I was a registered Democrat for more than 25 years.  I campaigned for Bill Clinton in 1992 and again in 1996, but the Democratic party has been so firmly planted in gridlock for so many years it's difficult to take the Democrats seriously.

I'm not hear to tout the virtues of the president.  I simply would encourage people to give the man and his policies a chance.  The Democrats have rejected President Trump out-of-pocket.  I listened to the President's address objectively and I heard a man who was preaching inclusion and I hear liberals preaching only divisiveness. Shameful!  Let's hear your comments on Thought-Rising.

Friday, December 23, 2016

A Critical Look at Agile Software Development

Since the Agile Alliance formed and met at The Lodge at Snowbird Ski Resort in February of 2001 to explore better ways of developing software, an apparent love affair has developed with agile practices in the Software Engineering Community. Yet it is particularly strange that there is little in the way of critical analysis and evaluation of agile methods and whether the net result of the adoption of agile brings those adopters any measurable benefit.  Could agile be little more than a shell game and sleight of hand? Regardless, we are concerned that the lack of serious critical scrutiny of agile methods may be contributing to far greater challenges among Software Engineering organizations throughout the world than they started with.  To be certain, there has been to date a dearth of critical academic scrutiny of Agile or the multitude of self-proclaimed Agile Methodologies and the practice of Agile within the profession. Such a critical perspective is necessary.  An objective perspective is needed.  The authors do not intend to simply take a contrary perspective to those of the various proponents of Agile.  Instead, we wish to provide a critical assessment of the Agile Software movement and report a purely objective assessment of Agile Software Engineering in concept and in practice.

1. Introduction

The Software Engineering world is rather in love with the concept of Agile Development.  We cite and reference Dingsøyr, et al.[1] who published a paper trumpeting A Decade of Agile Methodologies, though the intent of this paper is to laud the success of Agile Methodologies versus conducting critical research as to the advantages and disadvantages of Agile Methodologies.  There is a large community of Software Engineers who see Agile as a magic bullet that will solve every problem under the sun.

We will proceed on the Converse assumption.  First, we will examine the Agile Manifesto itself and look for fallacies or pitfalls in the original document.  Second, we examine the most prevalent approaches claiming to be agile and examine the strengths and weaknesses of each approach and attempt to distill whether these agile methods are producing measurable results that differ meaningfully from the results teams achieved using more conventional methods.  Finally, we conclude our critical analysis of Agile Software Development and summarize our findings.

2. The Agile Manifesto – Strengths and weaknesses

Clearly, the team that produced the Agile Manifesto vetted their concepts in a rather formal manner.  They attempted to provide a set of principles -- in the Aristotelian sense -- that Software Developers would employ and that would improve the way that they worked.  Each member of the group assembled to draft this document brought with them a set of experiences and a vision of what they believed constituted best practices in Software Engineering.  In the end, they published a document not unlike the United States Constitution or the Communist Manifesto.  Clearly, the Agile Alliance was keen on starting a revolution against traditional methods.   What is not clear is whether Agile in practice actually solves any real problems or if it results in a superior outcome.  Working software is not a guarantee of any material results.  There is an awful lot of working software in the world that never gets used or which is used incorrectly thus negating the overall benefit of the software.  The Agile Alliance missed the point with the focus on working software.  It is far more important that the software be intuitive, adaptable and maintainable over the full product lifecycle.

One post on “agilemanifesto.org” reports the history of the team, self-named “The Agile Alliance” that drafted the Agile Manifesto on February 11th-13th of 2001, and states, “…a bigger gathering of organizational anarchists would be hard to find, so what emerged from this meeting was symbolic – a Manifesto for Agile Software Development – signed by all participants”  The result has been somewhat striking in that there is scarcely a software engineering team throughout the United States that does not aspire to be an Agile Team.  It is not absolutely clear why Software Engineering Teams are striving to evolve their methods from conventional techniques toward agile methods.  Nor is there any empirical evidence that Agile methods improve software engineering in a tangible manner such as reduced overall cost.  On the contrary, it would appear by the enormous size of a typical agile software engineering organization that Agile has accelerated more than the software development process, it appears that Agile is vastly superior to conventional methodologies in using up whatever financial resources are available. It is also unclear how agile techniques are evolving the role of the Software Engineer within the profession and whether the outcome is actually desirable.

3. Strengths of the Agile Manifesto

The Agile Manifesto is a brief document stating that the authors were uncovering better ways of developing software by doing it and by helping others to do it. Clearly the 4 core values that they focused on are intriguing.  We will explore these first:

1)      Valuing individuals and interactions over processes and tools
2)      Valuing working software over comprehensive documentation
3)      Valuing Customer collaboration over contract negotiations
4)      Valuing the ability to respond to change over following a plan

Objectively, there doesn’t appear to be anything particularly subversive about these modest principles.  Clearly there is notable good in valuing the individuals and their interactions over the processes and tools.  To wit, some organizations, especially large organizations had, at that time, become so overwhelmingly stagnated by process and governance that it became prohibitively difficult to get any real software developed at all.  We applaud the work of the signatories and admit that the emphasis on process that was the norm in the 25 years prior to the writing of the Agile Manifesto had reached a threshold of being stifling and often impeded the development of software products. 

Valuing working software over comprehensive documentation sounds pretty good, but we must pause to assess the implications, namely: Is this a good principle?  We have doubts which we will examine in the next section. 

Valuing Customer Collaboration over Contract Negotiations also sounds good insofar as we feel better about Collaborating with a Customer versus engaging in Contract Negotiations with that Customer, but at this too requires further analysis.

We also agree that it is important in many situations to be able to deviate from a plan in order to respond to changes that were not anticipated at the outset of a software project.  Such changes are ultimately guaranteed to occur in all but the most trivial software engineering endeavors.

Thus, on the surface the Agile Manifesto appears to be a rather intriguing document with generally good intentions.

4. Pitfalls of the Agile Manifesto

There are several fallacies and pitfalls associated with Agile Development in practice today. We trace these pitfalls to central omissions in the Manifesto:

1)       Agile methods turn over much of the systems analysis, design and technical architecture to software engineers/developers who may not have the proper skill set to analyze, design or architect such a solution. More often than not these individuals are sequestered from the users of the software and even the key stakeholders who are on point for the system requirements thus there is rarely any means to achieving effective requirements analysis and reaching a clear understanding of the solution domain.  The Agile Manifesto does not discuss the vitally important concern of systems architecture. 

  •        Agile practitioners tend to take the quintessential square peg of solution requirements analysis and stuff it into the round hole of a 2-week sprint.  For the most trivial of information technology problems where the teams have access to the system owners and key stakeholders, the time-frame may be acceptable, but for more interesting problems, it is not. Examples where two-week sprints work in opposition to the business need include

    1. A quoting system for insurance products, which demands extensive information gathering, and which is governed by complex actuarial tables and regulations is a far-fetched goal for a 2-week investigation.
    2. The solution requirements for the guidance and anti-collision systems for a self-driving automobile.
    3. The analysis of a system that uses human genome mapping to aid in the development of medications for human diseases also should not be packaged into even a series of 2-week iterations – this is an entirely unreasonable expectation.
  • Agile also tends to bracket engineer learning cycles into these two-week intervals.   In reality, some learning takes more time than 2 weeks.
  • Agile is not an architectural discipline even though it clearly requires tremendous architectural discipline to implement the requisite continuous integration processes which are at the core of most agile practice and embodied in the statement by the signatories, “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”
  • In fact, Agile practitioners have widely learned that Agile methods in practice violate the first and over-riding principle of the Agile Manifesto, namely: that of valuing Individuals and interactions over processes and tools.” If we are to be honest with ourselves, Agile techniques the world over are emphasizing processes and tools even more than conventional project philosophies and methodologies.  An important maxim is suggested: When adopting agile, avoid the adoption of complex processes and tooling like the plague!
  • Too, it is foolish to believe the second principle of the agile Manifesto: that of valuing “Working software over comprehensive documentation” Teams that forego comprehensive documentation are doomed to a vicious cycle of software artifacts that need to be rewritten far too soon after they are delivered, as they will have failed to capture important knowledge relating to the modules that were developed.  Moreover, Testing, even exhaustive testing is not a replacement for comprehensive system documentation, nor is it a substitute for a thoughtful and carefully considered systems architecture that takes into account the functional and non-functional requirements of the solution.  Poorly documented products are almost always used incorrectly!

In the best-case scenario, Agile practices place solution architecture responsibilities are in the hands of a software architect who is secluded from the software engineers, and the software engineers must blindly trust that the analysis of the solution requirements was performed correctly.  Assuming so, they can proceed to build a solution that fits the need. Otherwise, the developers are routinely foisted into the role of back-slope architects cleaning up the mess made by the Software architect whose work is not vetted.  This is most often the crux of the problem:

1) Companies frequently do not employ software architects and do not consider the importance of preparing a sustainable software architecture
2) Companies place the responsibility for defining the software architecture in the hands of individuals who are either poorly suited to the role or who lack access to the sponsors and key stakeholders of the solution.
3) Companies are relying increasingly on Software engineers on agile teams to back in to a solution architecture that will satisfy the needs of the business and we see this as fraught with disaster.

5. Dominant Agile Methods

There are several dominant agile approaches that have been widely adopted within Software Engineering teams over the past 16 years:

  • eXtreme Programming (XP)
  • Scrum
  • Lean Software Development
  • Crystal Methodologies
  • Feature-Driven Development

            We will examine each of these methodologies in detail in the following sections.

Extreme Programming

    This methodology is predominantly focused on pair programming; however, eXtreme Programming is also underscored by a formal methodology that emphasizes ceremony and checkpoints referred to as feedback loops common to most traditional software processes.  The process diagram for Extreme Programming (below) seems to be quite busy, and seems to add pair programming solely as an afterthought


Scrum and its derivatives which include SAFe Agile (an acronym for Structured Agile Framework for Enterprise) are among the most process- and ceremony-heavy of the agile methods.  While, most Scrum teams disdain comprehensive documentation, they employ a broad complement of tooling including tools for continuous integration, test driven development tooling and with increasing frequency they choose to implement convoluted release management and branching strategies which would seem to be entirely contrary to the first principle of the Agile Manifesto, it makes us wonder if Scrum methods are even considered “Agile” by the members of the Agile Alliance.

Lean Software Development

The Lean Software Development concept was developed by Mary Poppendieck and Tom Poppendieck. Their approach presents traditional lean principles as typically found in manufacturing, in a modified form, as well as a set of 22 tools and compares the tools to agile practices. The Poppendiecks' involvement in the Agile software development community, including talks at several Agile conferences has resulted in such concepts being more widely accepted within the Agile community.  Clearly the Poppendiecks are interested in marketing their methodology more than they are interested working software or any of the other principles of the Agile Manifesto.  It seems most process heavy, and therefore, contrary to the first principle of The Agile Manifesto to require the employment of 22 tools to achieve agile practices. Lean Software development would be intriguing if not for these 22 tools needed to employ the methodology. And the authors have not observed any teams actually adopting this methodology in practice 

Crystal Methodology

The Crystal Methodology is an encompassing collection of eight different project lifecycles or methods; however each reflects 7 common characteristics:
1.   Frequent delivery
2.   Reflective improvement
3.   Close or osmotic communication
4.   Personal safety
5.   Focus
6.   Easy access to expert users
7.   Technical environment with automated tests, configuration management, and frequent integration
The focus on frequent delivery and frequent integration in Crystal Methodology is a concern as we have observed that all approaches that focus on Continuous Delivery tend to suffer from the inevitable complexity that Continuous Delivery implies. Notwithstanding, the emphasis that Crystal Methodology places on reflective improvement and personal safety are admirable.  Nevertheless, we have observed in practice that when Personal Safety and open communication are valued, there is the occasion for highly negative influences to monopolize the time of the team thus contributing to bottlenecks and reduced team throughput and performance. Nevertheless the value placed on easy access to expert users is a mitigating factor that is likely to further advance projects employing Crystal Methodology.  We have employed technical environments with automated tests and configuration management and frequent integration and release cycles.  What we conclude is that each software product must be architected with these activities in mind and when software is architected to support such an environment and such processes and is founded in stable architectural patterns, there is no better approach to building quality software 

Feature Driven Development

After examining the remaining approaches to agile we finally arrive at Feature-Driven Development.  FDD is founded in Peter Coad’s concept of Software Features, where a feature similar to a use case in the context of the Rational Unified Process (RUP).  The major activities of FDD include:

1)      Develop an Overall Model
2)      Build a features List
3)      Plan by feature
4)      Design by feature
5)      Build by feature

Each of these steps or activities produces a defined deliverable.  Most important the Feature List is the product of step 2 and is a governing document for the entire methodology.  FDD unlike the other methodologies reviewed here actually incorporates a formal Requirements Analysis process (termed Requirements Envisioning) as part of its Iteration 0: Envisioning, which consists of Initial Requirements Envisioning and Initial Architectural Envisioning the team then transitions to Iteration Modeling followed by Model Storming and Test-Driven Development, which comprise the parts of an Interation 1: Development activity.  Though FDD also has substantially more process and ceremony than indicated by the first principle of the Agile Manifesto, it is clear that the concept of FDD provides a more transparent process than the remaining dominant Agile approaches.  Moreover, it appears that FDD is rooted in the Coad and Yourdon methodology of Object-Oriented Design and incorporates practices that we view as similar to Rapid Application Development with FDD’s focus on Requirements Envisioning (reminiscent of JRD) and architectural modeling as well as joint design activities (reminiscent of JAD). The emphasis on Test-Driven Development (TDD) is excellent, but in the end analysis FDD, while a self-proclaimed agile methodology is difficult to classify as agile, due to its notable departures from the first principle of the Agile Manifesto.  Still the approach is compelling due to the fact that it appears that the knowledge gained over the course of developing the software is documented, if only informally, thus the knowledge is not lost which seems a cornerstone deficiency in the other methods which we studied. 

The purpose of this critical perspective of Agile Software Development is not to say that there are no benefits and nothing to be learned from agile software practices and methods.  Rather, we must be realistic in terms of what we expect from the adoption of agile methods.  There are some good ideas in the agile manifesto, but there are also ideas that are intrinsically in conflict with the reality of business and software engineering principles.  Good engineering principles should not in practice be shunned solely due to what amounts to religious fervor.  There are no magic bullets in Software engineering and as scientists, we should take exception when our leaders parade through the main thoroughfare of our profession without their knickers, asserting that Agile Software Development is the most beautiful paradigm they’ve ever seen.

In short, before adopting Agile, it is vital to frame the issue carefully and ask, “What problem are we attempting to solve by changing our fundamental approach to how we develop software within our organization?  The authors have observed some organizations that employ the ceremonial aspects of agile – such as the daily scrum -- while keeping much of their traditional waterfall process intact.  The end result is the ability to more closely manage the time-sensitive topics within a project while providing a higher level of communication within the team.  The authors are also familiar with certain CTOs who are strongly opposed to Agile methods in any form as they assert that they have been burned by Agile and will not accept such notions within their organization.  One team in working under a particular CTO was already practicing a lean approach to software engineering, which was not, strictly-speaking, Agile.  The team simply reported that they were practicing lean software development and truthfully testified that it was not “Agile”. The reality of this team's culture was as follows:
  1. They did not lack documentation of the knowledge gained throughout the development effort
  2. They were not over-burdened with either process, tools or ceremony
  3. They worked together harmoniously
  4. They had tremendous knowledge of the domain in which their products were targeted
  5. They understood their mission and purpose with great clarity

As a result, this team consistently delivered outstanding software and exceeded expectations time and time again.  In the end analysis isn’t this the reality that the Agile Alliance was really targeting?

Leaders and leading researchers in the Software Engineering discipline must strive to keep in front of the trends in the profession and caution practitioners from dealings with snake-oil purveyers and hair tonic salespeople.  The tremendous flocking to agile that we observe throughout the profession should be a cause for alarm.  There has been no legitimate vetting of Agile practices and no metrics to help practitioners to make an informed decision whether Agile offers and tangible benefits in terms of cost or time to market.  Without such data points there is no way for software engineering teams to make any informed decisions about the advantages or disadvantages of adopting Agile methods.

1Dingsøyr, Nerur, Balijepally and Moe -   A decade of agile methodologies: Towards explaining agile software development, Journal of Systems and Software V85, Issue 6, June 2012 Pages 213 – 221 (http://www.sciencedirect.com/science/article/pii/S0164121212000532)