Breaking Software Wholesale I

An advertisement has been airing on Television where the central message is that the self-driving software in Tesla vehicles is the worst software the engineer speaking has ever seen.  This prompts the question: What is the best software that this individual has ever seen?

To be certain, the state of software engineering is in tremendous disarray, in part because the science of software engineering has not been owned by scientists in Academia for over 40 years!  The science of Software Engineering is now owned by practitioners.  Not well-meaning practitioners, but practiners who are interested in their own fame and fortune.  This foments a "Not Invented Here" mentality where each contributor to a software solution is driven to simplify a highly complex mess and to do so in the absolute minimum of time.  In short, the ship of nearly every Software Application is on a collision course with failure

Enter Agile practices.  Now we take the software that is already on a collision course with disarrary and we force our highly skilled and highly paid engineers to do everything all at the same time, simply because the members of  the Agile Alliance said it was a better way of building software.  

A number of challenges emerge: 

1)  Agile is the antithesis of inclusivity.  It teaches engineers a notion of survival of the fittest, rather than the intended team-work model which is most successful in any engineering endeavor.

2) It is nearly impossible to Agilify legacy software and commercial software.  

• Agile methods work best when there is 100% code coverage and the team understands the inner workings of every facet of the software product.  
• 100% code coverage is exceedingly difficult and costly to achieve, but it is, nevertheless far easier to achieve if you own every line of code is owned by a software engineer who understands it's purpose at a profound and deep level of technical understanding.
• Frew commercial software vendors embody this deep level of ownership of  their code and it is unlikely that a software vendor would consent to deliver a 100% coverage test suite with their software (Although enterprises procuring commercial software ought to demand this deliverable contractually).
• Legacy software suffers from a lack of test coverage by definition.  Since it was not built using a test-first methodology, it is going to be difficult to build out the required test suites for any sizeable application at a reasonable cost.  Hence, it is common to see organizations concede that 50% or 70% code coverage is better than 0%.

"Fail Fast," shout the Agilists, but take caution about these words.  Managers are not so tolerant of failure.  It is not in their nature to praise the team's ability to fail fast.  Moreover, "Fail Fast" quickly morphs into "Fail at Velocity," and there you have it a perfect recipe for failure -- Breaking Software Wholesale!

The situation gets far worse when most companies throughout the country are frantically trying to migrate to Cloud-based solutions.

Experience shows that many more software projects are failing under an Agile Regime than failed under Waterfall methods, and Waterfall methodology was intended to keep the project moving forward until it ultimately achieve success, yet we know that around the turn of the century that software projects were failing at an unacceptable rate.  Today the numbers have barely changed, and if anything, software failures are trending in the wrong direction. 

On Unity

While many are working to rally support amongst those in their 1/2 of the country, there are many of us who would like to rally the support of the entire country, united for the common good, based on universal truths, without platitudes and tropes about existential threats and the annihilation of the planet.  

We are optimists who have faith in the resilience and innovative spirit of humankind. We don't want our cities and states to be Californiacated, by faceless and soulless elites who value only power and control.  

Nor do we want to be lectured by a Cadre of government talking heads who spew word-salad, simply because they believe that the public are too stupid to handle the truth, and because the truth includes words of legitimate science, which would be offensive to one special interest or another. 

We care about our one nation, indivisible, with liberty and justice for all. And you may choose whether to say "under God" or simply pause at that part.  Wouldn't that be far better than filing a petition with the U.S. Supreme Court to officially and legally rework the pledge of allegiance so as not to offend atheists, and instead of marching devout followers of God to re-education camps to learn the new gospel of nihilism, but God-damn-it, I can sure be pedantic!

Is climate change a cataclismic event?  I cannot say as I have only a cursory knowledge of the science.  However, I am not discouraged and I do not believe that the planet will self-destruct by 2030 if we don't achieve a Zero-carbon footprint within the year.  I believe that scientists are more formidable than the doom and gloom that we hear from certain climate alarmists.  In fact Bill Gates is already working with researchers to seed the atmosphere with material that will absorb the Carbon in the upper atmosphere and this could very legitimately reverse the damage to the ozone layer, or it could once and for all destroy the planet.  No one knows.  But I believe in the undying innovation of the human spirit.  We will find a way to heal our planet without killing the vast number of people inhabiting our earth.

The value of capitalism

Modern Democratic Socialists decry the notion of capitalism.  For example, Alexandria Ocasio Cortez has famously declared, "Capitalism hasn't always been here, and it won't always be here."  Effectively, she has promounced that Capitalism is in the throws of death.  Others within the Democratic Socialist ranks decry that they don't want to exchange their time for money.  A logical question: who does? However, the asking of the question is not acknowledgement that Capitalism is dying. Rather, it is awareness that at all times, people would honestly conclude that they don't want to exchange their time for a wage or a salary.  This does not mean that Capitalism is dying.  It is merely a different question.

It has been said that there is dignity in work.  Without endeavor, what is a human being and what do you leave as your legacy after your days on earth, if there is no work?

Democratic Socialists have over the past several decades attempted to substitute equity for equality. Having tremendouse exposure to the notion of equity in our legal system, I must conclude that the concept is entirely meaningless.  There is no empirical way of measuring equity.  How can anyone know if the outcome achieved is equitable.

There is popular notion that states: What gets measured gets improved.  Capitalism is merely a mechanism in which good are exchanged for currency or for other goods.  The accumulation of currency or capital, is merely a measure of outcome.

If you achieve an outcome that cannot be measured, how do you know that you have achieved anything at all.

I will assert that equity is a preposterous and moronic  of outcome.  There is no objective measure of outcome if your measure is some abstract notion of equity or outcome.  Capitalism provides a much greater measure of outcome and equity.  Capital can be measured; therefore, outcomes in a capitalist society can be measured.  Since outcomes can be measured - not just notionally, but intrinsically using accepted economic formulas, the ability to improve the outcome can similarly be improved by manipulating the inputs and optimizing the equations.  This is the manner in which economists analyze the economy within a city, town, county, state or nation.  We've built national and global standards based on capitalism and financial measures of GDP, Profit and Loss and even currency exchange rates.  These standards have worked for centuries.  Throwing out capitalism is a dangerous notion, particularly if some poorly defined notion of "equity" is the only alternative.

It occurs to me that this is precisely why Socialist countries have failed miserably throughout history.  Living standards in Socialist countries are abhorrent.  Morover, Capitalisim provides an objective measure of the standard of living in a country.  If we jettison Capitalism, how do we measure standard of living? It seems a frivolous notion.

Moreover, Capitalism allows for the meaasurement of wealth.  By measuring the aggregate wealth of a nation, you have a strong measure of the standard of living of that nation.  Perhaps some people fall below the median while others are measurably above the median standard, but at least it is an objective measurement.  As such, we can improve the measurement, and construct policies which endeavor to raise the aggregate level of wealth in the country.  In addition, we can employ such measures of aggregate wealth and median income to determine who needs a leg up by an objective measure.  

In conclusion, I find that Democratic Socialism is a joke and needs to be eradicated from the social discourse.  it provides no suitable replacement for objective measurement, thus no way to quantify who we need to pull up.  Democratic Socialism effectively replaces the useful measures of wealth with nothing... The Emporer's new clothes.

CO2 and Climate Change

According to The Earth Institute, CO2 is a greenhouse gas as it traps heat that would otherwise escape the earths atmosphere.  In contrast, Oxygen and Nitrogen gases do not behave the same way.

This prompts more questions than The Earth Institute would like to think about:

1) Don't humans as well as cattle emit CO2 when exhaling?

Of course we do?  So, it would seem, this means that the only way to save the planet is to wipe out a large number of humans!

Or is it?  

As we all learned in grade school. We live in a most wonderful ecosystem, in which Humans breath Oxygen and exhale Carbon Dioxide (CO2).  Our partners in the ecosystem are plants.  Plants are our equalizers.  They consume Carbon Dioxide and expel Oxygen.  

Unfortunately, the cycle has been damaged by enormous deforestation across the globe.  So, there is a simple fix to our CO2 problem:  Plant more trees!!!

Thoughts on the Art of Abstraction in Software Engineering

My esteemed colleague, Dr. Mohamed Fayad has stressed the importance of abstractions in Software Engineering.   I firmly agree with Dr. Fayaad that the Art of Abstraction is a key area of concern in the science and practice of Software Engineering.  This concern is not well addressed in any widely adopted software engineering methodology.

What I have learned over the course of my career is that for any given computing problem there is a right-sized abstraction, which substantially simplifies the design and the code required to realize the solution.  

For example, in one of my first software engineering jobs, the PhD. who owned the firm asked me to write a program using recursion to address the requirements of an interface protocol to automate a piece of factory floor equipment.  Initially I spent time to design a solution that employed backward chaining recursion, but unfortunately the interface protocol in question was far more elegantly implemented with forward-chaining iteration rather than recursion.  Could I have built a solution with Recursion?  Absolutely!  I'm a talented engineer, and I would have made it work, but at what additional cost of design, development, and maintenance over the life of the software?

The challenge was that the solution utilizing recursion was vastly more convoluted and brittle than the natural solution that utilized a simple forward looping structure.  My analysis convinced me that this was inherent to the nature of the interface specification. 

Imagine me as a young software engineer attempting to explain to my new boss that his approach was far more costly and risky than the alternative that I envisioned.  Would he be upset and perhaps fire me on the spot?  The risk was tangible and worrisome.  But at the end of the day, Courage is the Key to life itself.  as a software engineer, you must be confident in your skills and be willing to defend your work, even against very hostile opposition.  My boss listened intently to my findings and recommendations.  It was clear that there was some degree of disappointment and frustration on his face as I explained why recursion was a poor abstraction for this particular design; however, he couldn't argue with my reasoning and he concluded our conversation, by saying, "Dave, you have made a convincing case.  Please proceed as you have recommended.  I was a valuable member of that team for several years.

A software engineer must  be open-minded, and deeply understand the inherent nature of a given solution that you are trying to build with a computer program.  It is helpful to think about how the problem might be solved using a functional programming language versus a declarative programming language. or even using a rules engine or an AI platform.  Most envisioned solutions broadcast a natural design along with natural abstractions that are the most simple and sensible for the domain, as well as the underlying  requirements of a solution (both the functional and non-functional requirements).  This is the inherent nature of virtually any software engineering exercise.  

In the converse, we may contemplate why do software projects fail?  

Failures are often attributable to time and money pressures, which tend to short-shift design cycles and tend to restrict efforts to analyze the correct abstraction for the solution.  Thus engineers fail to conceptualize a proper abstraction.  because we lack the tools and methods to arrive at an exceptional abstraction to a given software engineering problem.

More often than not it's because the wrong abstraction is used - perhaps due to the biases of the engineers on the team which lead the solution away from the natural and implicit abstractions that yield a superior solution.  In many cases, it is a side-effect of the software engineering methodology that the team employs.  Fundamentally, these failures result because our profession lacks predictable and reusable tools for abstracting a solution and measuring the goodness of fit of the abstraction to the domain and the requirements.

As engineers, we dance around these topics (time, money, resources and scope) because the first three are always constrained whereas the 4th is always expected to be maximized.

Perhaps this critique of the software engineering domain is unfair, after all, software has permeated virtually every facet of our lives and we might conclude that the solutions that make our lives so much easier are proof that software projects do not fail.

I argue that this assessment is a complacent argument The plethora of software that permeates our daily lives is not proof that software projects are not failing at a very high rate.  Rather it is a testament of the size of the software engineering profession that we have such a critical mass of software engineering professionals in our world who are producing software at a phenomenal pace.  This doesnt mean that the vast majority of solutions are successful.  It just means that there is a critical mass of successful projects and we don't see the failures since they don't have commercial value, and therefore are not visible from an everyday perspective.

I have observed many hundreds of solutions in my career that simply needed to be re-engineered, because they were the artifacts of failed software engineering products.  Furthermore, I have studied dozens of solutions that were so artificially complex and cumbersome that they are simply not viable.  Yet every company in the same industry or domain uses these products because a prominent software vendor is marketing the product.  This does not mean that the project was successful, per se.  It mearly means that the vendor has a marketecture that meets a market demand; however, it is common that failures of the underlying technology are frequent and repeatable.  

Why, exactly, does this happen?  In the end analysis, it is most often due to needless complexity of the abstractions used in modeling the solution.  A complex abstraction will invariably lead to a complex design and a complex code-base.

Perhaps at no time was this more obvious to me than when building a commissioning engine.  Initially, our CTO suggested that we didn't need any fancy tools to build the solution.  He further suggested that a p-code solution (a poor-man's rules engine would do the trick) our team was skeptical about this as we would first need to construct the p-code interpreter and then apply it to our problem domain.  Therefore, we decided to invest in a rules engine from a respected product vendor.  At first we learned that many features of the rules engine technology did not work as advertised.  This caused us to throw out many assumptions at the start of the project.  

A second challenge was that the vendor's consultant to our team was not fully expert in the rules engine technology that the vendor sold us and he heavily struggled in identifying a proper abstraction to meet our requirements.  So, the original solution was rather cumbersome and difficult to maintain.  The company had invested $1.6 million to develop the solution, but regrettably, it didn't work especially well.  

Less than a year into the project I persuaded management to pursue an overhaul of the solution to make the software easier to maintain.  This iteration of our software was far more maintainable, but still required a team of 4 engineers to support the technology.  The cost of sustaining the software was still too high, as was the failure rate of the software.  

We realized that no one possesssed a deep understing of the  the features of the rules engine platform well enough to further simplify the solution.  Several production failures of the software prompted the team to invest in formal training for the engineers on the team, but no one returned from the training with any viable solutions to our woes.  

Finally, I went to the training with my mind completely focused on abstracting our solution to be even more elegant.  I quickly learned about features of our development platform that could further simplify our model and reduce the support costs by 50% and reduce the defects by more than 90%.  I shared my  recommendations and showed that the developmnent could be completed in just 3 weeks time.

Ultimately our solution became recognized as a gold-standard solution in the industry and we realized a positive cash return on investment for the solution within 2 years.

When the "Whole" is less than the sum of it's parts

Agile practitioners and advocates are prone to a fallacy that the "Whole" is equal to the sum of it's parts.  While this principle should be true in most cases, it is not always true.  

For example, lets consider two major categories of transactions in a system  Category A and Category B:

 

Category A: Domestic Purchase/Shipment		Category B: International Purchase/Shipment
Taxable?	Yes	Taxable?	Yes
Requirements	State Sales Taxes apply bsased on the location where the product is shipped. Some states have no sales tax (e.g.:  0%)	Requirements	Value Added Tax Import Tax and Tariffs as per current regulations.

 

A subsystem that orchestrates the shipment, and also the application of Taxes and Tariffs may not work correctly in  particularly in cases, when Tariffs may be assigned by a foreign government on short notice.  This is a rather simple example, but it demonstrates a form of complexity that may surface in a Scrum Team.  

If the work of the overall orchestration of the Taxation and Tariff assessment is split out among different Scrum teams, the results could be unpredictable.  Taxation rates vary from state to state and country to country.  The proper division and ownership of duties is important here.  Perhaps there is an expert on the team who understands US State sales taxes, and another expert who primarily works with international sales taxes and tariffs, but no one who understand both. In addition, there is a great need for coordination with the DevOps team who will be responsible for the build and deployment of the solution.  

Consider that certain product is no longer available for shipment from the United States to customers in the United States due to a structural change in off-shoring of the product's manufacturing.  As a result the products will ship from China to the USA; however, China has recently had new Tariffs and sanctions imposed on its imports into the USA, due to its failure to adhere to a trade agreement.  In this scenario, we have import shipments from a foreign nation to the US.  This breaks the division of labor into a less-than-ideal outcome.  As the shipments will be taxed based on the destination state, plus the shipment will also have a tarrif, which is not normally the case for imports into the USA. The Scrum Team may decide to break this piece out into a module of its own; because, over time, things may change and the product again may be sourced domestically.

Where the difficulty arises for the overall Agile team is when the teams are not clear about the division of responsibilities and overlap exists or the work is divided up in a hurry due to pressing enterprise obligations.  As a result team-members anxious to help with the effort may needlessly duplicate efforts or fail to provide the coverage for a feature that is required.  The end result is defect that are unanticipated.  Who is responsible for pruning the citrus tree of code artifacts and ensuring that this doesn't happen within agile teams?  Perhaps the scrum master owns this responsibility, or does she?  If agile teams are self-organizing, then they may not receive this coverage from the scrum master.  Moreover a question is prompted.  When was the last time that you observed a living breathing scrum-master in the wild?  They are an interesting, but rare species, and apparently nocturnal creatures, because they are rarely observed in the daytime.

There are vast examples of agile teams dropping the ball as in the scenario that I featured above.  The question is two-fold:

1) Are we so arrogant as to believe that agile is the end-all be-all of software engineering practice?

2) What is missing from Agile that seems to produce these unfortunate outcomes?

I have found specific software engineering discipline to be nearly absent in agile teams.  Many teams simply struggle to adopt the ceremonies required for agile with the level of discipline that is needed.  DevOps teams pushed to the hilt with a barrage of deployment tickets simply jury-rig the deployments just-in time to meet the insane schedules that they deal with on a daily basis.  This results in further concerns such as deployment of artifacts that were never intended to be deployed, simply because they have worked before, or adding unnecessary and volatile steps to a release as a stop-gap, but introducing production failures as a side effect, If you think that these are funny examples, I've got a million of them.

Perhaps instead of riffing on the unfortunate souls who have been hit and run by agile practice we need to shed the arrogance of agile and get back to some modicum of software engineering.

First principles of software engineering are a good place to start.  

1.1) There is a root cause for every effect or outcome.

1.2) Garbage in yields garbage out.

By analogy, we can assert the following Theorems as well:

2.1) There's no such thing as a free lunch.

2.2) Agile methods require software that is crafted to satisfy agile expectations

       2.2.1) Testable

                  2.2.1.1) Has a very high percentage of automated integration tests.

                   2.2.1.2) Code coverage through unit tests is also equally high.

                   2.2.1.3)  Every test is correlated to functionality specified in user stories.

       2.2.2) Well understood

                  2.2.2.1) There are no significant gaps in the user stories, unit tests or integration test suites.

                  2.2.2.2) The application or solution is well understood.

                  2.2.2.3) Knowledge of the solution is substantial and not limited to the minds of a small cadre of experts.

                  2.2.2.4) Experts are expected to share their knowledge with the team, and this knowledge is shared freely

Regrettably, these principles are not widely adopted in software engineering organizations, nor is knowledge shared broadly.  In fact, critical knowledge is often owned by a small fraction of the team and there are large portions of a system that are so complex that no one could ever explain how it works as it does or why it works in such a way.

In practice leading software engineers understand that there is an elegant solution for every software application and there are fully elegant techniques for reducing software complexity.  Agile teams rarely reach the maturity required to realize how to mitigate software complexity, because everyone is so focused on sprints and deadlines.

This prompts many questions.  The first is this notion of deadlines.  Weren't deadlines supposed to be a thing of the past in the agile world?  We were supposed to be killing the notion of artificial deadlines with the transition to agile, but it ultimately turns out that there are more deadlines than ever.  Daily deadlines, weekly deadlines, sprint deadlines, story deadlines and epic deadlines.  But in the agile ceremonies, no one is really managing to these deadlines, certainly not the scrum masters that eschew deadlines, and think in terms of nothing more than the current sprint and the next sprint.

 

KARMA!!!

 I worry that I might have inadvertently been responsible for the wholly self-righteous model of the left as a result of my time as a progressive activist at UCSD in the 1980s.  Imagine if you will that the totality of liberal strategies and methods was authored by a sex-and-drug-crazed twenty-something college student in the 1980s.

I was merely hoping to get laid by some loose liberal babes and get buzzed, and accidentally created Modern Democratic Socialism... Crap!

Words of Wisdom

If you died tonight, your employer will advertise to fill your job by the end of the month, but your loved ones, friends and family will miss you forever. 

Don't ever get too busy making a living, that you forget to work on making a life.