When computer scientist Edsger Dijkstra famously remarked that “Testing can be used to show the presence of bugs, but never to show their absence,” he made it painfully evident that the all-too-common view of software quality assurance as primarily a testing activity was woefully inadequate. And yet, the software community has never really settled on an agreed vision of software quality. This was noted in a provocative editorial  in IEEE Software Magazine last year that pondered the question “Can we really achieve software quality?”.

As a basis for discussion, the definition was offered in the editorial that “ … assuring software quality requires that the implemented software achieves its desired behaviour without any unintended consequences.” A look at the trends in the functionality being offered by software in recent years reveals just how difficult it has become to reach that goal.

Let’s begin with the phrase “ … without any unintended consequences.” The editorial brings up one important trend in software technology, which is software-intensive mission-critical applications, such as self-driving cars. It is noted that certification generally enters the picture here, whereby frequent updates in the field (e.g., for cybersecurity) bring with them issues of re-certification (to avoid those unintended consequences). Yet, this remains a largely unsolved problem. In fact, the Software Engineering Research Roadmap proposed in a report  issued by the Software Engineering Institute listed “Assuring Continuously Evolving Software Systems” as one of its Advanced Development Paradigms to be addressed by researchers over the next 15-year timespan.

Now let’s take the phrase “ … its desired behaviour …” Here, too, trends in software technology are leading to functionalities whose “desired behaviour” is more and more difficult to pin down, both in terms of description and verification. To a great extent, the rise of artificial intelligence, particularly in the form of machine learning, has contributed to this problem. It is becoming more and more the case that we are no longer sure what an AI-assisted program is supposed to exhibit in terms of behaviour. Even when we are able to specify its desired behaviour, and even when we are able to verify that the desired behaviour has been achieved, we are often unable to determine why it was achieved, because of the opacity of the machine-learning algorithms. This may be acceptable for a language translation application, but rightfully makes us nervous in a self-driving car. Not surprisingly, that same Software Engineering Research Roadmap listed “Engineering AI-Enabled Software Systems” as one of its Advanced Architectural Paradigms to address over the next 15 years.

Muddying the waters even further are considerations such as speed of development and delivery and achievement of business goals. Agile practices and their offspring such as DevOps were born to address the thorny problem of delivering high-quality software within restrictions of business objectives. But it was also these agile practices that helped to promote the notion of software that is “good enough” for its business purposes, not “gold-plated”. The notion of “good enough” may be found everywhere in software development. For example, the risk-based processes of mission-critical standards embed the notion that software is “good enough” if it reduces risk to an agreed tolerable level – not perfection.

And yet, the IEEE Software editorial argues that perhaps this mindset has been taken too far; that maybe we can achieve software quality without the concessions to “good enough” that we have become used to making in our work, despite the advances in software technology in recent years.

Can it be done? What are your thoughts on the challenges in software quality we are facing?

Let us know what you think here

References:

Ipek Ozkaya, “Can we really achieve software quality?”, IEEE Software ( Volume: 38, Issue: 3, May-June 2021)

A. Carleton et al., “Architecting the Future of Software Engineering: A National Agenda for Software Engineering Research & Development”, Software Engineering Institute, November 2021