Introduction

Domain-driven design (E. Evans, Domain-Driven Design, 2004), extreme programming (XP) (K. Beck, Extreme Programming Explained, 2004), Scrum (K. Schwaber, Agile Project Management with Scrum, 2004) , service-oriented architecture (SOA) (T. Erl, SOA: Principles of Service Design, 2007) and DevOps (G. Kim et al., The Pheonix Project, 2018) are most widely accepted agile development practices.

Some may think SOA isn’t one of agile development practices. But it should be noted that without SOA, meaning that the firn has monolithic applications, it is difficult for developnent teams to release upgrades frequenty.  In web-scale firms like Amazon and Netflix, a DevOps team takes responsibility for a small number of SOA services (often in the form of microservices).  The teams can upgrade and release the microservices several times a day because the services are small and mutually independent.  SOA is indeed a prerequisite for successful agile developnent.

Today these agile development practices are combined with design thinking (T. Brown, Change by Design, 2009), Lean Startup (E. Ries, The Lean Startup, 2011), DevOps (J. Kim, et al., The Phoenix Project, 2013), microservice architecture (MSA) (S. Newman, Building Microservices, 2015), BizDevOps practices, etc.  All these practices have a commonality. They all try to create the maximum value for the users of the software with minimal time and cost, and do so by releasing working software increments frequently to users, learn from the users’ feedback and adapt the software to their needs. So these all can also be regarded agile development practices.

The essential characteristics of agile development include value proposition, continuous and Lean delivery, validated learning, and software engineering competency.  If your version of agile development lacks any of these essential components, you have misunderstood what agile development is about.  A successful adoption of agile development requires a mechanism that allows you to nail down the specific value proposition to be delivered by your software to the users, a mechanism of Lean production and continuous delivery of zero-bug software increments in a fast release cycle, a mechanism to learn from customer responses, and strong competencies in software engineering capabilities that make these three mechanisms not just possible, but globally competitive.

Reading this page will make you rethink about agile development as a business strategy and as a business process that help you survive and succeed through digital transformations.  It will let you understand that software talent management—an HR management process—is fundamentally important for agile development, and so are the product management process in software product businesses, and the service engineering and service engagement processes in professional IT service businesses.  Agile development happens in the context of a business, and the business processes surrounding the software development should have been properly reengineered to be supportive of agile development.  It will be shown in this page that many practices recommended in agile development are indeed very natural consequences of ordinary process reengineering efforts that have been observed in manufacturing and service industries since 1980s.  Agile development practices adopted common business process reengineering (BPR) patterns such as concurrent engineering, empowerment, job enrichment, flattened organization, Lean production, upstream shift, self-service, etc., as will be explained later in this page.

Comprehensive understanding of agile development will help you set up comprehensive initiatives to adopt and improve your agile development practices successfully.  One of the most important things to learn from this page is to understand what knowledge, skills, practices, techniques and tools software project team members need to be trained in and become capable of, in order to be successful in performing an agile development project.  Depending on the current skill maturity level of your software development organization, you might better develop a long-term road map plan and a strategic organizational transformation plan to adopt state-of-the-art agile development step-by-step.  It should be noted that no one can skip the maturity level. 

Please visit this page from time to time to see the progress in writing, and send me your comments.

Agile Manifesto Revisited

The Manifesto for Agile Software Development (K. Beck, et al., Manifesto for Agile Software Development, 2001. http://agilemanifesto.org/) valued

individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan.

No one can deny that Scrum has been the most widely adopted practice of agile development. The State of Agile survey by CollabNet/VersionOne in 2017 (https://explore.versionone.com/state-of-agile/versionone-12th-annual-state-of-agile-report) reported that 56% of respondents used Scrum as their agile practice (with the second most used being a hybrid of multiple practices at 14%). It also reported that 88%, 67%, 46% and 35% of respondents did sprint planning, release planning, product roadmapping, and agile portfolio planning, respectively; and that the percentage of respondents who used such tools as agile project management tool, bug tracker, automated build tool, unit test tool, continuous integration tool, requirement management tool, release/deployment automation tool ranged between 74% and 44%.

We can see that many people who actually have done agile development followed a very rigorous process such as Scrum. They employed many tools for automating some human activities in software development. They made many plans from long-term plans such as project portfolio plans and product roadmaps, mid-term plans such as release plans to short-term plans such as sprint plans. Process, tools and planning are so common ways how people work when they try to collaborate to achieve certain goals in an efficient manner. Therefore, people developing software in teams would also seek the benefit of well-honed processes, adaptable plans that all team members share, and advanced software tools that can improve productivity and quality.

Documentation isn’t always bad either. It’s basically converting implicit knowledge to explicit knowledge so that the knowledge can be stored, searched, classified, shared, reused, continuously improved, taught, and so on. Application programming interface (API) is a sort of documentation, for which most use the language of SOAP or REST web service today. SOAP and REST API documents made service-oriented architecture (SOA) the prevailing architecture style since the mid-2000s, which in turn led us into cloud computing, microservices, DevOps and further into the current era of digital businesses.

Agile manifesto really does not address the essence of agile development that all agile projects should be characterized by. It suggests a direction of change of common practices of software development that were becoming outdated in 2001. Jim Highsmith, for example, said in 2001 on behalf of Agile Alliance (http://agilemanifesto.org/history.html):

The Agile movement is not anti-methodology, in fact, many of us want to restore credibility to the word methodology. We want to restore a balance. We embrace modeling, but not in order to file some diagram in a dusty corporate repository. We embrace documentation, but not hundreds of pages of never-maintained and rarely-used tomes. We plan, but recognize the limits of planning in a turbulent environment.

Principles behind the Agile Manifesto

Agile development has become a buzzword for modern, effective way of software development. When I talk about the essence of agile development, it really means the essential things that software developers should know and do today. Now, if we look at the Principles behind the Agile Manifesto (http://agilemanifesto.org/), there are more clues about the essence of agile development.

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. … Working software is the primary measure of progress. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

A keyword I see here is value. We should develop software that provides value to the user. In other words, software should have a value proposition; viz., relieve pains associated with incumbent solutions, and/or create unseen gains to the user (http://www.businessmodelgeneration.com/canvas/vpc). The software that provides discriminant value proposition to a customer will help the customer gain competitive advantage over its competitors who don’t use the software or use other vendors’ software.

How do software developers know the software they plan to build has distinctive value to the target customer? This uncertainty requires the developers to pay for the information they want, i.e. the true information about the set of features that have distinctive values to the customer. This information is best obtained when the customer is given a working software that the developers built based on their hypothesis about the valuable features for the customer. The customer’s feedback from actually using the working software is the best source of information to learn valuable features. This is what Eric Ries called validated learning in his book on Lean Startup in 2011. The Lean Startup method of creating new software goes through a Build-Measure-Learn cycle—a fast feedback loop where a product increment is delivered to the customer frequently to get feedback to learn desirable features.

Design Thinking is widely adopted by software development organizations these days because it helps them run the Build-Measure-Learn cycle effectively. Design Thinking suggests a creative design process that goes through Empathize, Define, Ideate, Prototype and Test (Tim Brown, Change by Design, 2009). Empathize with customers to learn what they really want consciously and unconsciously; make a hypothetical definition of the customer problem to solve; ideate the software solution, perhaps producing user stories, innovative business processes, intuitive user experiences, etc.; then build and release the minimum viable product (MVP) (Eric Ries, The Lean Startup, 2011) many times a day, in order to test the hypotheses and learn from the customer feedback.

Allowing the user to actually experience the use of a solution is the best way to learn real (sometimes hidden) needs of the user. Therefore, prototyping is used in many fields of engineering. In software engineering, prototyping is relatively more productive than in hardware engineering because software can be changed more easily and therefore prototypes can be salvaged into the shippable product. In software engineering, prototyping has been encouraged since as early as mid-1970s. (F. Brooks, The Mythical Man-Month, 1975) It’s thus not a surprise at all that Agile Manifesto promoted continuous delivery, which is to produce prototypes incrementally based on user feedback in a very fast cycle, maintaining a constant pace indefinitely.

Another important keyword from the Principles behind the Agile Manifesto is, therefore, continuous delivery. In Wikipedia, continuous delivery (CD) is defined as a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. It aims at building, testing, and releasing software with greater speed and frequency.

Continuous delivery is one of the hardest parts of agile development. Do your developers have the competencies to release zero-bug software increments several tens of times a day, or at least once a month? Do you know what competencies are required for your developers to be able to do that? If your answers are ‘no’ to these questions, then stop and first check your readiness for adopting agile development! This page helps you know how to get ready for agile development, and succeed in your journey of improving software development productivity and quality by becoming better agile.

Now let’s review the rest of the Principles behind the Agile Manifesto quoted in the following:

Business people and developers must work together daily throughout the project. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. … Continuous attention to technical excellence and good design enhances agility. Simplicity--the art of maximizing the amount of work not done--is essential. The best architectures, requirements, and designs emerge from self-organizing teams. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Why must business people and developers work together daily throughout the project? Continuous delivery is possible if developers can build software fast without errors. What makes continuous delivery impossible is the software bug. If bugs are not cleaned up continuously, they tend to accumulate and you can’t release the software knowing that there are an unknown number of bugs in the software. Among all kinds of errors, requirement errors are the most costly to fix. Leffingwell and Widrig estimated that requirement errors contributed the majority—often 70 percent or more—of the rework costs (D. Leffingwell and D. Widrig, Managing Software Requirements, Addison-Wesley, 1999). Since rework typically consumes 30–50% of a typical project cost (B. Boehm and P. Papaccio, Understanding and Controlling Software Costs, IEEE Transactions on Software Engineering, 14:10, Oct. 1988, pp. 1462 - 1477), it follows that requirement errors can easily consume 25%–40% of the total project cost! The best way to avoid requirement errors is to have business people and business analysts (or product managers) work together to specify requirements and validate them—first as designed in models and then as implemented in shippable product increments.

Often the requirement engineering role and the solution design and implementation role are separated in a project team. If business analysts or product managers don’t implement the solution for themselves, but have a separate team of developers do it (as is promoted in the Scrum practice), business people, i.e. customers and users, business analysts or product managers, and developers (in its narrow sense) should work together daily throughout the project.

But should they work together through face-to-face conversation? How many agile projects are run that way? Is it desirable to do so? Many development teams often use cloud services for distributed software development, distributed source code management, distributed project management, real-time communication, social collaboration, and video conferencing instead of meeting face-to-face. More and more components of a software solution are not home-grown, but are sourced from external services or third-party software systems, and assembled together via APIs. You can’t invite all those developers of external sources for face-to-face meeting. You need to read documents, exchange emails and interact with them using social media, web conferencing, open or closed communities, etc. It almost sounds nonsense today to say that software project team members should work in the same location and collaborate face-to-face.

What would have been an underlying principle that required face-to-face collaboration? I think that the underlying fundamental principle is the Lean principle. The Lean process tries to maximize the customer value with a minimum waste in production (Lean Enterprise Institute, What is Lean? 2018. https://www.lean.org/WhatsLean/). If there is a gap between what business people want and what developers think they want, it will cause waste of development efforts. The gap is often introduced by errors in written requirement specifications (structured or unstructured). It will help reduce that waste if business people and developers (in the broad sense) meet frequently (online or offline), work together on requirements specification, and validate them together using frequently released working systems.

One of the principles behind Agile Manifesto, working software is the primary measure of progress, means that the team should minimize producing non-value- adding, work-in-process (WIP) which may include market requirements documentation (MRD), product requirements documentation (PRD), analysis models, design models, and so forth. But the Principles behind the Agile Manifesto also include: Continuous attention to technical excellence and good design enhances agility. ... The best architectures, requirements, and designs emerge from self-organizing teams. It encourages producing work-in-process such as architecture, requirements and design.

Yes, it is all about the balance, not black or white. We don’t want to produce non-value-adding documentation, but want to produce just-enough documentation, just in time, pulled by the working system to be released next. The Lean principle is also expressed as simplicity—the art of maximizing the amount of work not done—in the Principles behind the Agile Manifesto.

Lastly, another important keyword in the Principles behind the Agile Manifesto is technical excellence and good design. A closely related keyword is self-organizing teams. Effective agile development is possible only if the team members collectively have excellent software engineering capabilities including domain knowledge, analysis and design modeling skills, and mastery of state-of-the-art implementation technologies relevant for the project. The more multitasking team members are capable of, the faster the agile development can be, because there are less hand-offs between the members which tend to introduce delays and rework. This is why Scrum insists on having only three roles in a project—product manager, developer and Scrum Master. Developers are responsible for understanding and elaborating requirement specifications provided by the product owner, designing and refactoring architecture and detailed design, coding, testing, build and deployment—pretty much all kinds of activities required in software development. 

Essential Elements of Agile Development

To summarize, we have extracted a few essential elements, hence recommendable patterns, of agile development in this section from reviewing the Manifesto for Agile Software Development. They are:

Value Proposition: The whole purpose of agile development is to create distinctive value for the user.

Validated Learning: The best way to overcome the uncertainty about what functionalities are valuable to the user is to build a working software and have the user experience the software.

Continuous, Lean Delivery: We want to shorten the feedback cycle (i.e., release cycle) because the shorter the cycle, the less the waste according to the Lean principle.

Software Engineering Competency: People with certified competencies in knowledge, skills and tools necessary for the project must form the development team. Value proposition is the goal, while validated learning and continuous, lean delivery are the methods to achieve the goal, and software engineering competency is the essential resource for carrying out the methods successfully.

Agile Development as a Business Process

A software system is a subsystem of the business system, i.e. enterprise, for which it is used. Therefore, software engineering in an enterprise is one of its business processes that strive to achieve business strategies and goals. (June Sung Park, Software Engineering in the Context of Business Systems, in I. Jacobson and H. Lawson (ed.) Software Engineering in the Systems Context, College Publications: London, 2015) Hence many business management disciplines have been applied to software engineering. Systems approach, process improvement, balanced scorecard (BSC), total quality management (TQM), six sigma, project portfolio management, activity-based costing (ABC), cost of quality (CoQ), business agility, Kanban, Design Thinking are some of the management disciplines that have influenced software engineering methods. Here a method (or methodology) means to encompass a process, the practices applied to process activities, and the tools used in applying the practices.

In a software business, in particular, a software system is a product to sell. When a software system is developed for the mass market for profit like in Microsoft and Google, the software engineering process is a subprocess of the product management process. On the other hand, when a software system is developed for a specific customer for profit like in Accenture and Capgemini, the software engineering process is a subprocess of the service engagement process.

In the digital economy, every company (manufacturing companies in particular) has to be a software company as Jeff Immelt, former CEO of GE, said. (McKinsey & Company, GE’s Jeff Immelt on Digitizing in the Industrial Space, Oct. 2015. https://www.mckinsey.com/business-functions/organization/our-insights/ges-jeff-immelt-on-digitizing-in-the-industrial-space) Therefore, software engineering is one of the most important core business processes in any company in the digital era. Agile development as a prevailing software engineering method is surely one of the most important core business processes today.

Michael Cusumano, in his book titled The Business of Software (M. A. Cusumano, The Business of Software, Free Press, New York, 2004), wrote that

If the software business were like other businesses, there would be no need for this book. But software is not like other businesses.

The uniqueness of software business stems from the peculiar nature of software—intangible, free from reproduction costs, continually changeable, more complex than perhaps any other human construct, requiring team-oriented, intellect-intensive endeavors, and tolerate no mismatch among the interfaces of system components. (F. Brooks, The Mythical Man-Month, Addison-Wesley, 1975) It is thus important to understand successful patterns of modern business management in general, and successful patterns of software business management in particular, to be able to understand the essence of agile development as a core business process.

Business Management and Agile Development

Business strategy determines the long-term goals of an enterprise, the courses of action and the allocation of resources necessary for carrying out these goals. Michael Porter identified three principles underlying strategy: creating a unique and valuable position, making trade-offs by choosing what not to do, and creating fit by aligning company activities with one another to support the chosen strategy. (M. Porter, What is Strategy? Harvard Business Review, November 1996) A strategic plan determines target customers, a portfolio of products, value propositions of the products for the customers, internal operational processes, resources for running the processes, and the ecosystem consisting of competitors, complementors, suppliers and sales channels, at a specified time in the future.

A business process is a set of interacting activities undertaken in an enterprise to produce a specific product or service for particular customers. A business process has goals which should be in alignment with strategic goals of the enterprise. Information systems support execution of business processes and should enable the processes to achieve their goals. A company continually adapts its business strategy to changes in its business environments such as technology, market and competition environments. As the business strategy is changed, business processes should be changed accordingly. As business processes change, the supporting information systems must be changed as well. This chain of changes in environment, strategy, processes and information systems is called business-IT alignment.

Michael Hammer suggested that information systems, if they are to significantly improve business performances, should not be designed to automate existing processes, but to radically reengineer business processes. (M. Hammer, Reengineering Work: Don’t Automate, Obliterate, Harvard Business Review, July 1990. https://hbr.org/1990/07/reengineering-work-dont-automate-obliterate) Business Week in 1993 reported a dramatic increase in U.S. industrial productivity owing to the reengineering of business processes based on information technology. (H. Gleckman, et al., The Technology Payoff: A Sweeping Reorganization of Work Itself Is Boosting Productivity, Business Week, June 14, 1993, pp. 57-68)

The information engineering methodology widely used for designing and developing information systems in 1990s incorporated information strategy planning (ISP) and business process reengineering (BPR). (C. Finkelstein, An Introduction to Information Engineering: From Strategic Planning to Information Systems, Addison-Wesley, Sydney, 1989; J. Martin, Information Engineering, Prentice-Hall, Englewood-Cliffs, NJ, 1989) ISP analyzed the existing information systems landscape in an enterprise, and determined a number of projects for new or enhanced information systems that are required to achieve business strategies. BPR then was used to radically redesign business processes to be implemented in each project.

Enterprises also started applying object-oriented paradigm to information systems development in 1990s. Object-oriented software engineering evolved into component-based software engineering to increase software reuse. (June Sung Park, A New Revolutionary Paradigm of Software Development for Mainstream Business Operations, International Journal of Technology Management 20:3/4, 2000, pp. 272-286; June Sung Park, Component-Based Development of e-Business Solutions Using COOL:Plex, CA-World Conference, Computer Associate, Orlando, FL, July 8-12, 2001)  By the end of 1990s object-oriented and component-based software engineering were packaged into the Rational Unified Process (RUP) which promoted use case-driven, component-based, iterative development. (I. Jacobson, G. Booch and J. Rumbaugh, The Unified Software Development Process, Addison-Wesley, 1999)

Michael Hammer and Steven Stanton argued that companies which made the leap from BPR to business process management (BPM) reaped enormous benefits from transforming themselves into true process enterprises. (M. Hammer and S. Stanton, How Process Enterprises Really Work, Harvard Business Review, November-December 1999. https://hbr.org/1999/11/how-process-enterprises-really-work) BPM is a management discipline that pursues enterprise-wide optimization of end-to-end processes to improve corporate performance. It requires continuous monitoring of process performances, identification of the opportunities for improving processes, and frequent optimization or reengineering of selected processes.

Many companies, including U.S. Federal Government, adopted the discipline of enterprise architecture (EA) since early 2000s. EA applies principles and practices of systems architecture to guide an enterprise through changes in business processes, business information, information systems and technical infrastructure necessary to achieve its business strategy. BPM and EA are closely related. BPM with its current process models of the enterprise provides the business architecture view for EA. IT-enabled BPR projects can be triggered as a result of process monitoring and mining in BPM, and governed and guided by architectural considerations and targets provided by EA. (C. T. Jensen, O. Cline and M. Owen, Combining Business Process Management and Enterprise Architecture for Better Business Outcome, IBM, 2011)

From mid-2000s service-oriented architecture (SOA) has been increasingly adopted by companies to design information systems. Information systems in SOA are composed of web services—self-contained units of software providing functionality as services to other software via a standard Internet protocol. (June Sung Park, Web Services, Keynote Speech, Edge U.S. 2002 Conference, Computer Associate, Dallas, TX, 2002) SOA allows reuse of standardized core business services when the company transforms its business and application architectures in response to changes in its environment and strategy. Reusable and composable common services constitute the digital foundation for business transformation and innovation.

SOA supports BPM in such a way that business processes, newly created or redesigned, can be quickly implemented by composing services. The services can be obtained from an existing service registry, legacy applications wrapped by service interfaces, or external sources of services provided by COTS packages or SaaS. SOA promotes separation of the layers of business processes, services and software components. Services virtualize software components and provide APIs to business processes. This layered architecture allows separation of concerns among business process design, application design and software acquisition. BPM and SOA, when applied enterprise-wide, separate business architecture, application architecture, and technical architecture from each other. Consequently, combination of BPM, SOA and EA increases the agility of transforming EA in response to changes in business strategies. This agility of business-IT alignment is one of the most important agenda of today’s enterprises. (C. T. Jensen, et al., Leveraging SOA, BPM and EA for Strategic Business and IT Alignment, IBM, 2008. http://public.dhe.ibm.com/software/dw/wes/bpmjournal/0812_jensen/SOA_BPM_EA.pdf)

Ross and her colleagues in the Center for Information Systems Research in MIT found, from a study of over 200 companies, four different patterns of EA which represent different maturity levels of EA. (J. W. Ross, P. Weill and D. Robertson, Enterprise Architecture as Strategy: Creating a Foundation for Business Execution, Harvard Business School Press, Boston, 2006) These 4 levels are: (1) locally optimal business solutions (enabled by BPR), (2) enterprise-wide technology standards, (3) standardized enterprise processes and data (enabled by BPM), and (4) standard interfaces and business componentization (enabled by SOA). 12%, 48%, 34% and 6% of the surveyed companies were in the EA maturity level of 1, 2, 3 and 4, respectively, at the time.

To summarize, an enterprise as a business system has strategic goals. Strategic goals are often achieved through business process redesign. Enterprise architecture ensures that business strategy, business processes and information systems are aligned despite their frequent changes. Service-oriented architecture of information systems facilitates the business-IT alignment by breaking down information systems into discrete services and composing these services for quick implementation of business processes. Agile development tries to achieve agility of the enterprise, i.e., fast adaptation of the enterprise to changes in its technology, market and competition environments. Fast adaptation to environmental changes requires a fast chain of changes from business strategy, to business processes, to information systems. This chain of changes should be made in each iteration of the agile development, if the company explores opportunities to create a new business model based on emerging technologies, as most companies are doing now for digital transformations. Agile development is not just about how to develop a working software incrementally, but is about how to make a fast, successful adaptation of the business to changes in its environments.

More to come ...