Featured Post

Thursday, May 5, 2022

Thoughts on the Art of Abstraction in Software Engineering

My esteemed colleague, Dr. Mohamed Fayad has stressed the importance of abstractions in Software Engineering.   I firmly agree with Dr. Fayaad that the Art of Abstraction is a key area of concern in the science and practice of Software Engineering.  This concern is not well addressed in any widely adopted software engineering methodology.

What I have learned over the course of my career is that for any given computing problem there is a right-sized abstraction, which substantially simplifies the design and the code required to realize the solution.  

For example, in one of my first software engineering jobs, the PhD. who owned the firm asked me to write a program using recursion to address the requirements of an interface protocol to automate a piece of factory floor equipment.  Initially I spent time to design a solution that employed backward chaining recursion, but unfortunately the interface protocol in question was far more elegantly implemented with forward-chaining iteration rather than recursion.  Could I have built a solution with Recursion?  Absolutely!  I'm a talented engineer, and I would have made it work, but at what additional cost of design, development, and maintenance over the life of the software?

The challenge was that the solution utilizing recursion was vastly more convoluted and brittle than the natural solution that utilized a simple forward looping structure.  My analysis convinced me that this was inherent to the nature of the interface specification. 

Imagine me as a young software engineer attempting to explain to my new boss that his approach was far more costly and risky than the alternative that I envisioned.  Would he be upset and perhaps fire me on the spot?  The risk was tangible and worrisome.  But at the end of the day, Courage is the Key to life itself.  as a software engineer, you must be confident in your skills and be willing to defend your work, even against very hostile opposition.  My boss listened intently to my findings and recommendations.  It was clear that there was some degree of disappointment and frustration on his face as I explained why recursion was a poor abstraction for this particular design; however, he couldn't argue with my reasoning and he concluded our conversation, by saying, "Dave, you have made a convincing case.  Please proceed as you have recommended.  I was a valuable member of that team for several years.

A software engineer must  be open-minded, and deeply understand the inherent nature of a given solution that you are trying to build with a computer program.  It is helpful to think about how the problem might be solved using a functional programming language versus a declarative programming language. or even using a rules engine or an AI platform.  Most envisioned solutions broadcast a natural design along with natural abstractions that are the most simple and sensible for the domain, as well as the underlying  requirements of a solution (both the functional and non-functional requirements).  This is the inherent nature of virtually any software engineering exercise.  

In the converse, we may contemplate why do software projects fail?  

Failures are often attributable to time and money pressures, which tend to short-shift design cycles and tend to restrict efforts to analyze the correct abstraction for the solution.  Thus engineers fail to conceptualize a proper abstraction.  because we lack the tools and methods to arrive at an exceptional abstraction to a given software engineering problem.

More often than not it's because the wrong abstraction is used - perhaps due to the biases of the engineers on the team which lead the solution away from the natural and implicit abstractions that yield a superior solution.  In many cases, it is a side-effect of the software engineering methodology that the team employs.  Fundamentally, these failures result because our profession lacks predictable and reusable tools for abstracting a solution and measuring the goodness of fit of the abstraction to the domain and the requirements.

As engineers, we dance around these topics (time, money, resources and scope) because the first three are always constrained whereas the 4th is always expected to be maximized.

Perhaps this critique of the software engineering domain is unfair, after all, software has permeated virtually every facet of our lives and we might conclude that the solutions that make our lives so much easier are proof that software projects do not fail.

I argue that this assessment is a complacent argument The plethora of software that permeates our daily lives is not proof that software projects are not failing at a very high rate.  Rather it is a testament of the size of the software engineering profession that we have such a critical mass of software engineering professionals in our world who are producing software at a phenomenal pace.  This doesnt mean that the vast majority of solutions are successful.  It just means that there is a critical mass of successful projects and we don't see the failures since they don't have commercial value, and therefore are not visible from an everyday perspective.

I have observed many hundreds of solutions in my career that simply needed to be re-engineered, because they were the artifacts of failed software engineering products.  Furthermore, I have studied dozens of solutions that were so artificially complex and cumbersome that they are simply not viable.  Yet every company in the same industry or domain uses these products because a prominent software vendor is marketing the product.  This does not mean that the project was successful, per se.  It mearly means that the vendor has a marketecture that meets a market demand; however, it is common that failures of the underlying technology are frequent and repeatable.  

Why, exactly, does this happen?  In the end analysis, it is most often due to needless complexity of the abstractions used in modeling the solution.  A complex abstraction will invariably lead to a complex design and a complex code-base.

Perhaps at no time was this more obvious to me than when building a commissioning engine.  Initially, our CTO suggested that we didn't need any fancy tools to build the solution.  He further suggested that a p-code solution (a poor-man's rules engine would do the trick) our team was skeptical about this as we would first need to construct the p-code interpreter and then apply it to our problem domain.  Therefore, we decided to invest in a rules engine from a respected product vendor.  At first we learned that many features of the rules engine technology did not work as advertised.  This caused us to throw out many assumptions at the start of the project.  

A second challenge was that the vendor's consultant to our team was not fully expert in the rules engine technology that the vendor sold us and he heavily struggled in identifying a proper abstraction to meet our requirements.  So, the original solution was rather cumbersome and difficult to maintain.  The company had invested $1.6 million to develop the solution, but regrettably, it didn't work especially well.  

Less than a year into the project I persuaded management to pursue an overhaul of the solution to make the software easier to maintain.  This iteration of our software was far more maintainable, but still required a team of 4 engineers to support the technology.  The cost of sustaining the software was still too high, as was the failure rate of the software.  

We realized that no one possesssed a deep understing of the  the features of the rules engine platform well enough to further simplify the solution.  Several production failures of the software prompted the team to invest in formal training for the engineers on the team, but no one returned from the training with any viable solutions to our woes.  

Finally, I went to the training with my mind completely focused on abstracting our solution to be even more elegant.  I quickly learned about features of our development platform that could further simplify our model and reduce the support costs by 50% and reduce the defects by more than 90%.  I shared my  recommendations and showed that the developmnent could be completed in just 3 weeks time.

Ultimately our solution became recognized as a gold-standard solution in the industry and we realized a positive cash return on investment for the solution within 2 years.


No comments:

Post a Comment