TITLE: Thoughts on the future of software architecture RESPONSE: (Terry Bollinger), 29 Nov 94 Hi folks, I did not personally attend this workshop (sigh), but I really liked what I saw in Ed McCamey's excellent summary of the June 1994 Architectures Workshop that was held at USC. This is important stuff, I think. Please note also that I've taken, er, some _liberties_ in expanding some parts of Ed's talk. 'Tis very dangerous to let someone who is working on a book on the same general subject area (reuse-intensive software processes) act as "secretary" for this kind of presentation... Cheers, Terry P.S. -- Disclaimer: I am not medically or legally responsible for any heart attacks that may occur purely by coincidence just as someone is reading that I am an officer in a local SPIN organization... :) v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^ Software Architecting: Do YOU Have the Right Stuff? ^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v By Terry Bollinger, ASEE / DFW-SPIN Secretary, 1994-11-29 [pre-editing version -- please pardon any typos] In a powerful and thought-provoking presentation to the ASEE, Ed McCamey of E-Systems described a possible future of the software industry that is vastly different from the way we now develop software. It is a future of _software architecting_, and it holds the promise of radical changes not only in the process by which software is developed, but also in the roles that we as software developers will need to play to become active participants in it. Ed McCamey leads the Corporate Software Technology Workshop at E-Systems, and for many years he has played an active industry role in understanding and analyzing advanced software and quality methods and techniques. The vision he described was developed collectively at a ground-breaking Software Architectures Workshop in which Ed participated in June 1994 at the University of Southern California (USC) in Los Angeles. This vision has some teeth, as it is being supported by a diverse range of companies, government agencies, industry leaders, and universities whose interest in software architecting and reuse-intensive development has been kindled by reports from around the world of impressive cost and schedule benefits from early pilot projects. Significant Gains Companies that have publically reported software productivity gains in the range of 1.5 to 2 times include HP, AT&T, GTE, IBM, TRW, and JPL. AT&T has also reported a huge 10 times reduction in the cost of operations-phase software support (maintenance) as a result of using software architecture methods. Indeed, the gains reported have been large enough to make some companies reluctant to describe all of the details of their ongoing reuse and software architecture programs. For example, representatives of one of the above companies become very quiet when asked how reuse-based development has affected their middle-management policies, because they see these such process-level changes as crucial to obtaining the full competitive advantage that they see in reuse-based architecture approaches. The large early gains from pilot projects have made such companies realize that the upper range of productivity and cost benefits are almost certainly yet to be realized, and thus represent a significant opportunity for gaining cost and productivity advantages over competitors in fast-moving high technology marketplaces. Participants at the the USC workshop also included such noted industry and academic leaders in as Barry Boehm, Ed Reckton, and Mary Shaw. By combining their knowledge and insight with the growing base of encouraging results from industry, participants at the workshop were able to form a collective vision of where, why, and how the new discipline of software architecting needs to progress over the next decade. It is also a vision that includes more than just software, as it has become painfully obvious that software will be at the very core of new high technology systems as design complexity that once resided in hardware is increasingly being moved into software for reasons of cost, flexibility, and adaptability. As the infamous case of a new Denver airport that has yet to open due to the vastly underestimated complexity of its baggage handling software, the process of specifying and designing large new high-technology systems needs a discipline like software architecting at the very core its early exploration of what can and cannot be done with existing software technology. Without it, new systems run a grave risk of specifying combinations of realistic hardware components with software components that no one has any idea how to implement. Software Architecting So what is "software architecting," and how does it differ from ordinary software design? Why have projects that used it reported such remarkable cost and productivity benefits. The heart of software architecting and its surprising productivity benefits is learning how to build by reusing past work, and with future reuse in mind. That is, a software architect does not simply assume that every new project is "from scratch" as is so often done in the waterfall model, but instead views a large, well-identified, and modular body of existing industry and proprietary software as the building blocks with which to specify creation of new, more powerful software designs. This is very much in line with what a conventional architect do, since no building architect in her or his right mind would specify a new design with the starting assumption that the only available construction materials would be mud, rock, and metal ores. The fundamental concept of software architecting is simply that this same idea of "building on the past" also holds true for software, especially as new software systems become increasingly complex. This is very different from the older perspective that if you can simply specify what needs to be done in excruciating detail, implementation will necessarily follow. Software architecting recognizes the limitations of having to deal with what is available, and makes this recognition into an integral part of the process of synthesizing new systems. Just as a building architect cannot simply "assume" a structural support of infinite strength, a software architect cannot simply "assume" implementation of an impossibly fast and flexible database component unless she or he can define a plausible path for reaching it in terms of existing frameworks and components. Wild and fanciful extrapolations of software abilities thus are not permitted, and expectations must be restricted to what can be accomplished within a limited amount of development time. However, far from being an highly cumbersome restriction, this idea of using what has already been done in the past can actually represent a creative opportunity for an architect. The highest level of software design becomes a challenge in how to combine many different pieces and potentials into a single well-orchestrated design, just as building architects must combine many functional elements into new, more powerful designs. In fact, USC notes that so far its best software architects have been not from an engineering background, but from creative backgrounds emphasizing music and art. Such backgrounds apparently indicate individuals who are better at holding many different opportunities and concepts in their minds at one time, and then combining them in new and unexpected ways to create a new design. As Ed emphasized in his talk, it is a set of skill that many of us currently in software development simply may not have, given the heavy emphasis on non-creative engineering and elaboration techniques that most universities have emphasized for selecting and training software engineers. Building for Reuse The second aspect of software architecting is that is must also include a powerful undercurrent of "building for the future" to ensure the creation of a sufficiently rich substrate of components for software architects to use. The difficulty here is that the current model of start-from-scratch software development tends to be severely short sighted in the way it builds a new software component. Like a toy model that was constructed by gluing it directly to sticks and rods that nailed into a table, the typical start-from- scratch software product is so thoroughly stuck in its initial development environment that it has very little hope of ever being reused anywhere else. In the software marketplace of the future, this critically important second tier of the overall architecting process would be done by personnel skilled in generating, composing, and integrating systems based on the "blueprints" that would provided by software architectures. This second tier would be responsible not only for the actual conversion of software blueprints into specific, executable systems, but also for ensuring that the best possible use is made of existing reusable components, and that potentially valuable new software constructions will be "captured" of new components for possible use in the software architecting process. This second tier of personnel skilled in generating, composing, and integrating software components is the one that shares the most resemblance to our current process of software development, but it differs significantly in its very strong emphasis on both with and for intensive reuse of software components. Building for reuse is one of the techniques that has helped produce some of the impressive early successes seen in software architecting. If a product is designed for reuse from the beginning, it often costs only modestly more than a "once only" version of the same thing. The cost and productivity gains from reusing such a generalized (product line) architecture even one additional time thus can be highly substantial when compared to the older approach of developing the two systems more-or-less independently. Also, the gains of 1.5 to 2 already reported by many companies for first-time reuse of their product line architectures. The benefits of reuse continue to grow as the number of projects reusing the code increases, so that a well- designed architecture with many users can lead to whopping big gains in the cost of creating and supporting such systems. Such benefits are even more pronounced during maintenance, when ensuring that the shared parts of the systems are preserved as a single code base can keep support costs down to a fraction of what they would be if separate groups had to maintain each set of application source code separately. Automating Ourselves Out Of A Job The largest tier of application builders in the future that was foreseen by the USC conference is taken up not by software developers at all, but by end users. It is anticipated that a software marketplace that pursues the software architecting approach aggressively will produce as one of its most important products a wide range of "application generators" which, like the 4GLs and spreadsheets that have already made major inroads into business software development, will permit most users to express their needs in terms that are so close to their application areas that no special programming expertise will be required. Only when their requirements become sufficiently new or complex that an off-the-shelf tool cannot obviously meet it would such users go to the much costlier alternative of calling in a software component or integrator specialist to configure a new tool. And just as there are few electronics users indeed who would think about casually calling up Intel and asking them to design a new chip because their Pentium "wasn't adequate for my application," calls to software architects would not normally be part of the process the end user would go though to create new software. Instead, the software architects would be called in only when a need had been firmly established for a type of software never before seen by software developers. Such a scenario obviously implies some rather drastic changes in how software is marketed and the roles used to support it. _Ad hoc_ software development in particular would be pretty much automated out of existence, replaced by an extensive variety of advanced application generators that would for the most part be supported by invisible personnel whom the customer would never see -- or for that matter, want to see! Software architecting thus is a real since simply the process of recognizing the need and ability for the software community as a whole to begin in earnest the process of "automating ourselves out of a job" in much the same way that we have automated so many other now obsolete jobs out of existence in past decades. What will be left is instead a creative core of developers devoted to exploring genuinely new software challenges, and to supporting the suite of tools that have made most forms of end-user level software development obsolete. The Architecting Process A key feature of a high quality software architecture is that it represents not just one software application, but an entire family of closely related applications that need to be readily accessible (easy to build) in future projects. This generality is needed not only for long-term support and extension (maintenance) of the first application, but also to gain the many cost benefits of reusing the architecture in other future applications. Thus an architecture is not so much a design for a single application as it is a design for an entire _product line_ of closely related future applications. This is again one of the distinctive aspects of software architecting when compared to traditional software development, when more often than not the emphasis has been to develop an overall design whose purpose was limited by the needs of a single application, often without even much concern for how that application might change over time. Although people such as Parnas have foreseen the need for this kind of generality in the past, this kind of "up front" emphasis on product line generality has not been consistently practiced by the software industry as a whole. In terms of the job role of a software architect, this need to anticipate future application needs as well as current ones again emphasizes the need for both creativity and the ability to balance many possibly conflicting needs within a single framework. In the initial stages of developing a product line, a software architect will produce a "blueprint" that specifies the desired product in terms of major components and interfaces between those components. Ideally, she or he would be assisted by tools that would help present the full range of software components and earlier architectures that are available, and by tools to help prove out the overall concept at a high-level interface level. This idea of "middleware" for helping to specify and validate new software architectures is an area that is being actively pursued for some object- oriented languages, and it has already produced some examples of such tools. By characterizing major interfaces and components at a high level, new generations of middleware design tools would facilitate the architect in making trade-offs and in selecting viable combinations. Working at the interface level is also critical to the generality of the resulting sets of components, since such generality is usually not possible unless the communications between systems components are themselves sufficiently general to permit many types of future changes, extensions, and re-arrangements. Risk analysis is a crucial component of the initial architecting process, since errors or omissions at this point can have a critical impact on the later success of the process. Simply going to a reuse-intensive model obviously helps risk analysis greatly by providing a variety of components with well-known performance and reliability characteristics, something that is flatly not possible with pure start-from-scratch software. Instead, risk analysis in software architecting can focus more on the process-level issues of how and where a particular selection of interfaces and components may later produce unforeseen implementation, reliability, or performance problems that could make the architecture difficult or even impossible to implement. Since these are the very issues most likely to cause unexpected process-level failure of a project late in its development (e.g., a sudden realization late in integration that two major subsystems have incompatible interfaces that will require several staff months of effort to reconcile), this ability to provide a higher level of process-oriented risk analysis thus is likely to be another major benefit of the software architecting approach. As Ed emphasized in his talk, most of the truly critical decisions in most current software projects are made on the first day, often with very little regard or understanding for what the consequences of those decisions may be down the line. Careful risk analysis as part of the software architecting process will go a long way towards keeping those bad decisions at bay, and ensuring that the rest of the project really will go as planned. Selecting a Paradigm A key issue in the initial definition of an architecture is selecting what kind of a software process paradigm to choose. Ed quoted several examples of such styles from a yet-to-be-published book by David Garlan and Mary Shaw entitled "An Introduction to Software Architecture" (available in draft form through Carnegie-Mellon Press). Examples of these general frameworks for software architectures included Unix-style pipes and filters, data abstraction and object-oriented, event-based implicit invocation, layered system, repositories, and table driven interpreters (including rule-based expert systems). Ed emphasized that no one of these styles is appropriate for all applications, and thus that the current tendency to focus solely on one paradigm such as object-oriented may prove to be short-sighted. As one example, the very powerful pipe architecture of Unix is for the most part incompatible with object-oriented styles, and also has a quite different range of problems (mostly data intensive ones) on which it works best. In all probability, the frameworks which the software architects of the near future will use will extend beyond these basic examples, and will encompass complex multi-system combinations of that use more than one of the basic models. One especially important example where such composite models may prove useful is in the burgeoning community of distributed network-based software, where interactions at the user, network, and individual system levels may all require different basic frameworks for effective composition and easy analysis. Implementing the Blueprint Once a viable component-and-interfaces blueprint has been created and validated by the software architect, the next step if for the blueprint to be implemented by the next tier of personnel skilled in reuse-intensive generation, composition, and integration. The result is an _implemented architecture_, in which the recommendations of the software architect have been converted into a highly specific, executable collection of software and other products necessary to support the product line. A fully implemented architecture is simply a carefully structured, highly generic collection of modules, templates, and code generators that can be used to rapidly and efficiently implement new, customized application of the overall product line design. Ideally, collection should support not only the generation of an executable "instance" (specific example) of the product line), but also generation of all the other associated products such as documentation and test cases that may be needed to make the resulting product instance fully accessible to its end user. It is in this generality that is the key difference between an ordinary "once-only" software product and a fully implemented software architecture. Whereas a once-only software product usually has an internal structure that does little to support reuse or even significant modification, let alone reuse, a well-designed software architecture contains a degree of generality that is proportional to how many times it is likely to be changed or re-implemented in the future. Implemented architectures may vary in their degree of generality from being little more than conventional collections of software modules designed for a high degree of flexibility in how they can be linked and used, to being 4GL-like "system generators" that require little more than a description of how the new application varies from past ones to create a new software product that is complete with documentation and tests. In between these two extremes lie a broad range of designs and techniques that share a reliance on extensive reuse of earlier modules and associated work products. Choosing the degree of generality for an architecture should be based on how many instances of that product line are likely to be needed, and how flexible the instantiation process will need to be. Techniques that rely on a large amount of manual coding are less powerful and more costly, but may be more appropriate when only a few instances of the product line are expected. The most highly automated techniques such as full application generators with special-purpose, application-specific languages should be used only when a very wide range of instances of the architecture is expected, such as when the architecture represents a powerful end-user tool. So What's Ahead? Ed has provided us with a fascinating preview of some important trends of which everyone working in software needs to be aware. In particular, the impressive early successes of reuse-intensive software architecting are likely to cause a significant increase in interest in such methods in the near future. If such successes continue (as seems likely given the large cost-savings opportunities of such methods over start-from-scratch), then the competitive pressures from companies who develop software faster and at a lower cost with architecture-based approaches is likely to cause other groups to "join the bandwagon" -- if in fact this process has not already begun, given the existing strong interest in reuse-intensive architecture approaches by many leading-edge technology companies. And although foresee all the consequences of such a trend, it certainly does seem to portend a major shift in the use of software development resources and the roles we all will play. Thus the questions we may all need to asking ourselves are: _Do_ I have the right stuff? And if I don't, what should I be doing to get there?