Succeeding with Agile.

Abstract

Agile has been all the rage for a decade.  Some organizations have been more successful than others in adopting agile methods.  Agile has been marketed very heavily both by software developers and by the members of the agile alliance; therefore, it is important  to separate the wheat from the chaff. To be certain, agile is not a silver bullet.  Merely, adopting agile ceremonies will not ensure that an organization will reap benefits from adoption of agile practices  Simply stated: an organization cannot expect a stellar transformation in software engineering outcomes by proclaiming "We are an agile shop."  Nor will the goal be achieved by overlaying scrum ceremony on top of a broken software engineering approach.

To place agile in the proper context, we raise four germane questions:

1)Why should an organization adopt agile?
2) How can agile benefit an organization?
3) What decisions can aid in successful agile adoption and ensure that an organization reaps the  intended benefits of agile?
4) What software products are good candidates for agile?


Why Agile? How can Agile benefit an organization?

These are the two most important questions when contemplating agile, and they are clearly reciprocal to one another.  An organization should adopt agile if it is very firmly committed and resolved to long-term improvement in software engineering cost controls and overall improvement of software quality  This is precisely how agile can benefit an organization; however, Agile does not come with this guarantee.  Agile must be adopted with pragmatic expectations and a willingness not only to disrupt traditional conventions and practices, but to embrace the foundational changes that are necessary for a successful agile practice.

What decisions can aid in successful agile adoption and ensure that an organization reaps the  intended benefits of agile?

There are many changes that are necessary and proscribed for any organization adopting agile.  There are no shortcuts here, all of the proscribed tooling and practices are essential to getting agile right. All of these proscribed elements are necessary and, in fact, crucial to employing agile successfully, and thereby achieving the outcomes that underpins the question "Why Agile?"  These prerequisites for agile success are as follows:

1. An organization must possess the willingness to part with brittle or failed software architectures that do not adapt well to Agile practice.  In short Agile will not work with any old software and there is no way to evolve a brittle or unwieldy architecture through iterative development to arrive at a viable agile architecture.  Consider the following realities:
• One of the most fundamental paradigm shifts that is vital for Agile software development is test-ability and test-driven development.  Software that is not designed for test-ability is not well-suited for agile teams. 
• Experience shows us that organizations who try to superimpose agile on software that is not designed to be testable are prone to fail at agile adoption.  
• Similarly, certain software architectures fundamentally lend themselves to agile development where as others do not.  
• An organization must be pragmatic about whether it's underlying software architecture is sufficiently agile and understand that an unwieldy architecture will invariably lead to a failure to succeed with agile.  A governing maxim: software that is failing without agile practices will almost certainly yield the same results in an agile regime.
• Agile doesn't just happen because it's proclaimed.  It takes a great amount of discipline at all levels within an organization to successfully employ agile practices.  Furthermore, there are fixed costs for successfully managing agile development which cannot be overlooked.  The organization needs to have a reliable and efficient platform and tooling to manage it's agile deliverables.  This goes far beyond SCM. products such as JIRA,HP ALM and similar ticket management technologies are prerequisites for a successful agile implementation, simply because there must be transparency and line-of-site visibility of every in-progress activity solution at any point in time for agile to work.  Tooling alone does not achieve the discipline.  There is a need to devote time and resources to triage status, activities and iterations on a burn-down list.

• Agile is essentially a gateway drug for Continuous Integration.

• In fact you cannot realistically implement agile development methods without having a comprehensive set of tooling for build and release automation that are needed to adhere to the agile cadence.
• SCM is a given, Build Automation with a tool such as Jenkins is also essential and automated deployments are needed.  These in turn are commonly gateway drugs for tools such as Artefactory and a variety of other software products that are commonly found within agile teams.

• Agile is also a gateway drug for DevOps and most agile teams cannot succeed without a strong DevOps team supporting the work. 

• DevOps greases the skids for Agile development teams and effectively clears the path for innovation by the Development Team by validating key platform decisions and verifying that the technologies that are proposed in a given Epic will function reliably within the release environment.  DevOops also serves as the gate-keepers of the environments so that each environment within the release cycle can be certified for use before developers introduce newly released sofware in the environment that may be faulty and may have side-effects that destabilize the entire release environment.

In short, it is important for decision-makers to avoid getting sold on the hype of agile.  It is also important to firmly grapple with the realities of agile and the importance of agile planning and deliberate and measured implementation of agile.  Most of all it is imperative that decision-makers not allow themselves to make the mistake of trying to superimpose agile on a brittle and failed software architecture, nor can it be expected to transform legacy code in even a few short years, much less the few short months that management would like to see results.

What's Next in Software Engineering?

In recent years, I have become disenchanted with the java community and also most software engineering practiced in businesses throughout the country.  The most commonly practiced approaches in software engineering have produced little more than prohibitively expensive approaches to developing software and approaches that far too often result in unpredictable and unexpected outcomes, including systems integration mismatches, software that is not readily adaptable to the constantly changing information security landscape and software that simply doesn’t work as desired.  These problems are compounded such that increasingly, software is delivered that simply doesn’t work and fails to meet the solution requirements as defined in any manner that resembles the reason that the software project was undertaken in the first place.  Furthermore modern software solutions are delivered with utterly convoluted user interfaces and such utter lack of documentation that no one really can tell you how the software is designed to function.  The end result is a prohibitively costly model of doing business.  This paper looks at a variety of the causes contributing to this new software engineering crisis and poses the question: “What is next in Software Engineering?”

Frankly speaking, the Java ecosystem has become needlessly cumbersome and developers spend absurd amounts of time working around the bloated technology that is Java with Spring Framework plus 100 or so alphabet soup libraries that are needed to make java useful for solving real-world problems then add to that 7 or more flavors of JavaScript libraries needed to build Web UIs, plus an assortment of scaffolding tools, build and test frameworks and more alphabet soup needed to make javascript useful.  All of this rather than creating software solutions that address real needs.  Call me dense or foolhardy, but when too much technology is getting in the way of building solutions that actually accelerate the construction of reliable software, it causes me to question the decision-makers and others who influenced the technology choices in the first place.

Perhaps I'm too critical of the Java Community.  Perhaps at the end of the day it's really the fault of Systems Architects who are far too wrapped up in jargon, buzzwords and bleeding edge frameworks and cutting edge technology to see the forest through the trees.

What attracted me to Software Engineering in the first place was the ability of engineers to build functional, full-featured software solutions by employing a programming language in a very elegant manner.

Once upon a time I was especially attracted to the C programming language due to the elegance of the language, it's syntax and the various constructs that were present in the language and supported by C compilers.

As a professional developer, I also experienced some of the limits of the C programming language in the real world.  In time, Java caught my attention as it added the object-oriented constructs that were missing from the conventional C programming language without introducing a lot of bloat or inelegant features to the language, which I perceived as a failing of C++.

I was also attracted to functional programming languages such as LISP and PROLOG, and I found the list-processing abilities within LISP particularly inspiring so much so that to this day, I find myself building and/or employing parsers that are very similar to techniques that are found in LISP.  I also find myself building semantic constructs such as those found in PROLOG, and use these to craft solutions for complex domains with rich semantic-oriented knowledge.  But of course I typically try to reuse what is there in the stack and the libraries, but Java has lost much of it's luster for me.

I have been more impressed with the .NET stack than the Java stack.  Perhaps .NET benefits from the fact that it has mostly been a closed-source platform until recent years, whatever the case, it is far more elegant in my opinion than it's SUN-come-Oracle counterpart technology.  But .NET is showing some signs of bloat too.

My mind is always looking for or building short-cuts that enable highly maintainable software that exhibits high reuse and high-reliability.  My philosophy: "Why build a solution with 1 million lines of code if it can be done in 7500 lines?  Not only is the 7500 line solution going to be less brittle and have fewer points of failure, but it will be orders of magnitude more maintainable.  I was particularly pleased when command line compilers gave way to GUI-based Development Environments.  In my opinion, if I was building slick GUIs for my applications, why wasn't I working within a development environment that came with a slick GUI?  In my assessment Software Engineering as a science has failed to lead the profession in the  proper direction.  We've lost sight of the principle that making the construction of rich software more accessible to all of the best and brightest will inspire them to build software with greater functionality, flexibility, portability, maintainability, reuse and security.  That is not to say that this doesn't happen today, but I've been frequently disappointed in the cumbersome techniques that are employed in building software products and solutions -- within both the largest companies in the world as well as the smallest companies.  Why  would anyone deliberately place so much red tape in the way of building useful software?  Perhaps when building for the DOD or NASA, it is unavoidable, but for most software built today, it seems that the programming languages and the tooling that is used gets in the way.

Is the software crisis any better than it was 30 years ago? Arguably it is not.  Perhaps there are a greater handful of successful projects, but these are almost assuredly offset by a greater number of failed projects.  Just because we have first-hand evidence of software from Google, Amazon.com, Microsoft, Facebook, and countless other technology companies that is used every day, with some measure of reliability and functionality doesn't mean that there aren't thousands of other products that died within the software engineering organization.

So, too, I've learned that Agile is little more than a gateway drug.  As anyone who has transitioned to Agile understands, to do agile properly requires the procurement of a vast soup of tools that are needed to properly administer Agile teams and also tools that are needed to achieve continuous integration (CI), which after all is the premise of most Agile methods, Don't get me wrong, I appreciate the idea that iterative development is at the heart of building working software   But CI tooling today requires far too much cumbersome work, and even more ingredients in its soup to employ properly.  It seems that a leap is needed to achieve CI more seamlessly.

Perhaps in time that is what will evolve, but i don't imagine that happening through a community process or any of the approaches that have brought the state of the art of Software Engineering or Java development to the place that we are at today.  To improve upon software development methods, we need to think about software engineering and Continuous Integration differently.  I firmly believe that this is precisely what is next in software engineering.  By thinking about the complexity of modern software engineering methods differently, we will find methods and technologies that enable software engineers to do more engineering in shorter amounts of time and there is great goodness in such a vision.  I am confident that this vision will soon be a reality thanks to the work of like-minded software engineers, researchers and innovators.

Is a Lack of Diversity Equivalent to Racism?

If you listen to Liberals and Conservatives you get a strikingly different characterization of our country.  Liberals want you to think that the country has never been more divided and that a divisive narrative is playing out in this country that threatens to doom our democracy; whereas Conservatives will have you think that the Liberal narrative is little more than sour grapes over the fact that Hillary Clinton lost the

2016 presidential election.

I have a theory which is different from the message that we hear from the President, from the Republican leadreship or from the Democratic leadership.  Let's take a look at the makeup of the leadership of the two parties.

Democrats have been preaching diversity for over 40 years.  The topmost leadership of the Democratic party are people who primarily come from New York and California, Michigan and Illinois.  These are states  where every day you are confronted with  diverse people and diverse mindsets, especially within the most populous cities in those states.  In Los Angeles, San Francisco, New York City, Chicago and Detroit, you are exposed to people of all races on a daily basis.  You are literally surrounded by every color of the Rainbow Coalition constantly.  Acceptance of diversity is not a question, it is an imperative.  You either accept diversity or you are out of step with  your community in a most profound manner.  However, even for liberals in Middle America, the daily exposure to a diverse population is less common. you may hold old fashioned attitudes and not be out of step with your community.

Democrats and liberals, in genera,l would have you believe that failure to accept every modality of diversity including every category within the LGBTQO+ continuum is racist and intolerant.  Perhaps this is true and perhaps the liberal leadership is correct, but is it not equally intolerant to call out everyone in the country who is not awake to the LGBTQO+ continuum as racists. Is it evil to be unenlightened?  I think not!  Rather, I think that it is wrong to label middle Americans as racists and anti-diversity solely because they have not been exposed to diversity.  This would be akin to labeling all sea life as alien simply because we cannot survive in a salt water ecosystem!  A fish is no more evil simply because it is a fish.  No more is a farmer in Iowa or a rancher in Wyoming a racist simply because they do not have the same concept of diversity as someone from San Francisco or Chicago.  They may not be tolerant or enlightened, but they are also not racists.

Liberals do not accept that there is a difference between lack of tolerance or lack of accepting diversity.  All are lumped into the efficient catchphrase "racist". It seems to me a rather intolerant perspective to classify everyone who doesn't think the way you do as intolerant, opposed to diversity and racist.  Clearly not all who do not think the same as the liberal establishment are not intolerant racists.  Moreover, isn't it hypocritical to label people who don't think the way that you do? Isn't that a lack of diversity itself?

An unexpected encounter with Bharpinder

Was a cow harmed in the manufacture of your safety equipment? 

Do you use environmentally friendly hair gel in the morning? 

And finally, “Why is it that you appear to live at TNGT’s headquarters? – Oh! That’s right, you really don’t make enough money to pay your student loan payments and live in the most expensive region of the Country.  

Instead of rocketing to the top of the organization, you find yourself toiling for 10 to 14 hours daily, including weekends just so that Bharpinder Nagasalamandanam doesn’t out-toil you with his own effort, which is underscored by mounting a goal of 18 hours per day plus weekends, and you quickly are reassigned to a team of under-performers who are tasked with reverse engineering a 20 year old application that was converted from Microsoft Visual C++ to Java using an automated code conversion utility.  Your initial assessment of this new assignment is flawless:

• The code needs to be rewritten from scratch. There is nothing that can be salvaged.  You have read this code and clearly it was code-generated by another program and though you do not have access to the original source code you recognize the entire asset is complete and utter garbage.

• You’ve looked around for any accurate and credible documentation of this module, but you have been given 3 answers when you ask:

• We have a cryptic, hard-copy document provided by the vendor of the original C++-developed solution, but it’s in JoAnne Stevenson’s desk, and she will not share it with anyone.

• There is an equally cryptic appendix that describes more business rules that are referenced in a document maintained by a 3^rd-party company, but their version reflects thousands of changes to the specification that have been implemented in the ensuing 15 years after the original software was written.

• The appendix has an even more cryptic reference to another document that hasn’t been in circulation for over 10 years, and another not-so-cryptic reference to a web site that contains additional code/value pairs that you will need to evaluate within your module, but notably many of the code/value references are completely missing from the industry standards organization’s website, or any other legitimate reference.

Your team is practicing a combination of eXtreme Programming, Scrum and Kanban, but no one has a name for this Agile potpourri, and it’s not clear which practices of the Company’s agile variant that you are using is responsible for governing which activities in this "agile" method, but Bharpinder seems comfortable and seems to know what needs to be done. 

In fact, he’s willing to help you, but it seems that part of the agreement will be that you will apprentice under him for 6 months, and only if you figure out the secret to deciphering the agile ceremony du jour will you be allowed to actually work on Bharpinder’s team going forward. 

In a matter of 4 days, the team concludes that you are rather dense, and you are re-assigned to build unit tests for a module that was developed based on User Stories that were written by a Stanford Undergraduate Software Engineering Student who has dropped out of College and is no longer involved in the internship program with TNGT. 

From software engineer you quickly descend the corporate ladder to settle comfortably as a Test Developer reporting to the QA Team with an after-hours role as Bharpinder’s gopher.  

After three weeks, you realize a necessity to pronounce that you not, in fact, gay, and that everyone has misinterpreted your status on the LGBTQO+ continuum. For this violation of inclusivity, you have been written up by Human Resources, in which they cite your failure to uphold TNGT’s commitment to diversity and inclusion which is a core tenet of the new Employee Guidelines.

You are placed on a 2 week probationary term with no pay, though you will be allowed to purchase food from TNGT’s world-famous cafeteria which offers cuisines from all over the world, but sadly, there is nothing on the menu that you would eat back home in Austin, TX. 

By the start of your third week, you gladly deliver your letter of resignation to HR admitting that you are simply not a very good Software Engineer, and that you’ve taken a new job at Yahoo at double the pay as a writer for Yahoo Sports.  

Finally you are able to enjoy your two favorite things in life: Computing and College sports!  Not to worry about Bharpinder; he’s found a politically correct apprentice to work under him.

Working for nothing

It is the year 2030.  Inflation as measured by the US government remains at an all-time low of 0.65%. Unemployment is remains low at just 3% nationally. Interest rates remain below 3.5% for 12 years running, following a second recession and another round of quantitative easing by the FED. People of every race, creed, color, gender(all 15 of them) and national origin now work for large corporations for free -- just for the sake of needing healthcare insurance.  

At this time, Amalgamated Healthcare Corporation is the largest corporation on the planet with over 500 Trillion USD in cash reserves -- enough to pay off the US National debt (currently sitting at $100 Trillion. 5 times over.  Amalgamated (AHC for Short) is now diversified into government consulting projects, and literally runs the US Government.  No one doubts the power of AHC.  They have both sustained and torpedoed 3 presidential administrations, including Trump (2 terms) Michelle Obama (2 terms) and in a surprising twist, Hillary Clinton won the 2028 election at the age of 80! After only one year in office, AHC lead a successful campaign to have her impeached, alleging Corporate Fraud within the Clinton Foundation, Colluding with Russia in the fixing of the 2028 elections, and Money laundering.

The Sweet Life

It is the year 2085, and statistics prove that there are more dead people on Facebook than living. That's partly because hardly anyone joins Facebook anymore, now that AT&T provides unlimited Social Vid for only $800 per month with purchase of an iPhone Mark 90.  The last movie theater closed about 5 years ago.  

Pa took me to a movie show once. We saw Planet of the Apes 3000.  I still can't understand why anyone went to movie shows --  $75 per ticket, $60 for a soda and 3 years in prison if you were caught sneaking in candy from outside the theater.  Besides, you can see all of the latest movies in VR on YouTube courtesy of AT&T on the stunning Perfect Definition (tm) display of the iPhone Mark 90 with Retina-Vision (tm). Besides, there hasn't been a great movie in decades according to Pa, and I think he's right.  

Cool kids don't watch movies or play video games like Pa did when he was my age.  The Cool kids spend their summers on the Coast of Spain and drive around in their Chevy Camaro Hovercrafts.  I've been there every summer since I was 9.  It will be six years in a row this summer, and I'm still not clear why my school advisor insists that I need 3 years of foreign language classes to get into college. Heck I'm fluent in Espanol.

Mom and Dad don't spend much time at home now that they each have their own personal time machines  They're always going back in time to play the stock market with Grandpa and Grammy. We've inherited billions and there's more rolling in every day.  Yeah, it's a pretty sweet life, but someone's gotta do it and it may as well be me.

Commentary on President Trump's address to Congress

There was a time in our country that it didn't matter what the president spoke about, those in the party that opposed the president conducted themselves with dignity and demonstrated respect for the President.  The democratic party is terribly disappointing.

Personally, I have a problem when the Democratic congress-people are giving the thumbs-down to the president on national TV and feel that they are somehow justified in doing so, when the president clearly articulated that he was seeking unity and cooperation and reaching across the aisle.

Watching Elizabeth Warren poo-pooing the president is utterly disgusting. Shameful even.  It's also rude and unacceptable for the democrats to rush out of the building so quickly after the speech.  The democrats are clearly taking a hard-line position, based solely on allegation that the President is a meglomaniac, a tyrant, a dictator and a racist.  The democrats are claiming that they are worried about the country, because such a man as Donald Trump is in a position of so much power -- Get over it.  The democrats lost the election just as they did in 2000! Life goes on -- lose with dignity!

The democrats claim that they are the guardians of democracy and that they are inclusive and forward-looking, even non-partisan.  They claim to be the party of looking forward, not back.  How so? The democratic response claims simply that the President's cabinet is just a group of billionaires who don't care about the common folk.  From that they defend that social defiance is acceptable, because it somehow makes the president is horrible.  Ha!  As Ronald Reagan always said, "There you go again."  I hated Reagan with a passion. I still feel that he was a horrible president and not my president, but the Democrats are committed to doing everything within their power to oppose the President without a tangible platform of their own.  The claim is that he is a dangerous man who has declared war on immigrants.

What I hear is partisan bickering by the Democrats -- baseless claims that President Trump is eroding the credibility of our nation.  I have seen fear-mongering and divisiveness the likes of which I never imagined that I would see in the USA. The Democratic response is that our representatives are out of control, because they (The Republicans) fail to understand that they work for us (the people).  I don't feel that the nay-saying response by the Democrats has any solid ground.  The speech was a positive speech and the President clearly reached across the aisle.  The Democrats refuse to offer any specific policy proposals, but then they have the audacity to claim that the President's proposals are empty.

The bad behavior and stonewalling by liberals throughout the country is simply not acceptable. History may prove me wrong, but from my perspective the Democratic party is utterly out of touch with Americans and is inciting riots an division among people all across the country.  I have observed friends and family members severing ties over politics and it saddens me.  More troubling is that the Democratic party is the party inciting this divisiveness with rhetoric and propaganda beyond belief.

The actions of liberals, specifically the deliberate action to stoke the fires of fear is unacceptable.  I was a registered Democrat for more than 25 years.  I campaigned for Bill Clinton in 1992 and again in 1996, but the Democratic party has been so firmly planted in gridlock for so many years it's difficult to take the Democrats seriously.

I'm not hear to tout the virtues of the president.  I simply would encourage people to give the man and his policies a chance.  The Democrats have rejected President Trump out-of-pocket.  I listened to the President's address objectively and I heard a man who was preaching inclusion and I hear liberals preaching only divisiveness. Shameful!  Let's hear your comments on Thought-Rising.