What's Next in Software Engineering?

In recent years, I have become disenchanted with the java community and also most software engineering practiced in businesses throughout the country.  The most commonly practiced approaches in software engineering have produced little more than prohibitively expensive approaches to developing software and approaches that far too often result in unpredictable and unexpected outcomes, including systems integration mismatches, software that is not readily adaptable to the constantly changing information security landscape and software that simply doesn’t work as desired.  These problems are compounded such that increasingly, software is delivered that simply doesn’t work and fails to meet the solution requirements as defined in any manner that resembles the reason that the software project was undertaken in the first place.  Furthermore modern software solutions are delivered with utterly convoluted user interfaces and such utter lack of documentation that no one really can tell you how the software is designed to function.  The end result is a prohibitively costly model of doing business.  This paper looks at a variety of the causes contributing to this new software engineering crisis and poses the question: “What is next in Software Engineering?”

Frankly speaking, the Java ecosystem has become needlessly cumbersome and developers spend absurd amounts of time working around the bloated technology that is Java with Spring Framework plus 100 or so alphabet soup libraries that are needed to make java useful for solving real-world problems then add to that 7 or more flavors of JavaScript libraries needed to build Web UIs, plus an assortment of scaffolding tools, build and test frameworks and more alphabet soup needed to make javascript useful.  All of this rather than creating software solutions that address real needs.  Call me dense or foolhardy, but when too much technology is getting in the way of building solutions that actually accelerate the construction of reliable software, it causes me to question the decision-makers and others who influenced the technology choices in the first place.

Perhaps I'm too critical of the Java Community.  Perhaps at the end of the day it's really the fault of Systems Architects who are far too wrapped up in jargon, buzzwords and bleeding edge frameworks and cutting edge technology to see the forest through the trees.

What attracted me to Software Engineering in the first place was the ability of engineers to build functional, full-featured software solutions by employing a programming language in a very elegant manner.

Once upon a time I was especially attracted to the C programming language due to the elegance of the language, it's syntax and the various constructs that were present in the language and supported by C compilers.

As a professional developer, I also experienced some of the limits of the C programming language in the real world.  In time, Java caught my attention as it added the object-oriented constructs that were missing from the conventional C programming language without introducing a lot of bloat or inelegant features to the language, which I perceived as a failing of C++.

I was also attracted to functional programming languages such as LISP and PROLOG, and I found the list-processing abilities within LISP particularly inspiring so much so that to this day, I find myself building and/or employing parsers that are very similar to techniques that are found in LISP.  I also find myself building semantic constructs such as those found in PROLOG, and use these to craft solutions for complex domains with rich semantic-oriented knowledge.  But of course I typically try to reuse what is there in the stack and the libraries, but Java has lost much of it's luster for me.

I have been more impressed with the .NET stack than the Java stack.  Perhaps .NET benefits from the fact that it has mostly been a closed-source platform until recent years, whatever the case, it is far more elegant in my opinion than it's SUN-come-Oracle counterpart technology.  But .NET is showing some signs of bloat too.

My mind is always looking for or building short-cuts that enable highly maintainable software that exhibits high reuse and high-reliability.  My philosophy: "Why build a solution with 1 million lines of code if it can be done in 7500 lines?  Not only is the 7500 line solution going to be less brittle and have fewer points of failure, but it will be orders of magnitude more maintainable.  I was particularly pleased when command line compilers gave way to GUI-based Development Environments.  In my opinion, if I was building slick GUIs for my applications, why wasn't I working within a development environment that came with a slick GUI?  In my assessment Software Engineering as a science has failed to lead the profession in the  proper direction.  We've lost sight of the principle that making the construction of rich software more accessible to all of the best and brightest will inspire them to build software with greater functionality, flexibility, portability, maintainability, reuse and security.  That is not to say that this doesn't happen today, but I've been frequently disappointed in the cumbersome techniques that are employed in building software products and solutions -- within both the largest companies in the world as well as the smallest companies.  Why  would anyone deliberately place so much red tape in the way of building useful software?  Perhaps when building for the DOD or NASA, it is unavoidable, but for most software built today, it seems that the programming languages and the tooling that is used gets in the way.

Is the software crisis any better than it was 30 years ago? Arguably it is not.  Perhaps there are a greater handful of successful projects, but these are almost assuredly offset by a greater number of failed projects.  Just because we have first-hand evidence of software from Google, Amazon.com, Microsoft, Facebook, and countless other technology companies that is used every day, with some measure of reliability and functionality doesn't mean that there aren't thousands of other products that died within the software engineering organization.

So, too, I've learned that Agile is little more than a gateway drug.  As anyone who has transitioned to Agile understands, to do agile properly requires the procurement of a vast soup of tools that are needed to properly administer Agile teams and also tools that are needed to achieve continuous integration (CI), which after all is the premise of most Agile methods, Don't get me wrong, I appreciate the idea that iterative development is at the heart of building working software   But CI tooling today requires far too much cumbersome work, and even more ingredients in its soup to employ properly.  It seems that a leap is needed to achieve CI more seamlessly.

Perhaps in time that is what will evolve, but i don't imagine that happening through a community process or any of the approaches that have brought the state of the art of Software Engineering or Java development to the place that we are at today.  To improve upon software development methods, we need to think about software engineering and Continuous Integration differently.  I firmly believe that this is precisely what is next in software engineering.  By thinking about the complexity of modern software engineering methods differently, we will find methods and technologies that enable software engineers to do more engineering in shorter amounts of time and there is great goodness in such a vision.  I am confident that this vision will soon be a reality thanks to the work of like-minded software engineers, researchers and innovators.