Systems Engineering Strategies for Uncertainty and Complexity


Definitions: Uncertainty comprises both external and internal elements. External uncertainty includes changing markets, operating environments, priorities, business processes, and threats as well as emerging requirements/expectations, competitors (including users), and technologies. Internal uncertainties include program/project execution as well as design, implementation, and performance challenges [1].

Complexity can be characterized as the interactions and interdependencies among people, organizations, technologies, tools, techniques, procedures, and economics that cause patterns to emerge that transcend the goals of any one group. Complex interactions can result in resilience and robustness but also in cascading failures [2, 3].

Keywords: adaptability, agility, complex systems, complexity, ecosystem, emergent behavior, fitness, flows, interactions, interdependency, robustness, selection, uncertainty, variety

MITRE SE Roles and Expectations: MITRE systems engineers (SEs) are expected to understand the nature and sources of uncertainty, lack of effective control [4], and complexity [5] in their environment and then select and apply appropriate strategies for managing or mitigating their effects.

Introduction

The complexity we are seeing in the enterprises and systems that MITRE helps engineer requires a spectrum of systems engineering techniques. When a system is bounded with relatively static, well-understood requirements, the classical methods of systems engineering are applicable and powerful. At the other end of the spectrum, when systems are networked and each is individually reacting to technology and mission changes, the environment for any given system becomes essentially unpredictable.

The metaphor of the watchmaker and gardener is sometimes used to describe the differences between engineering in the two types of environments [6]. Classical systems engineering is like watchmaking. Its processes, techniques, and tools are applicable to difficult problems that are essentially deterministic or reductionist. Like gardening, engineering for the enterprise draws on the fundamental principles of evolution, ecology, and adaptation. It uses techniques to increase the likelihood of desirable or favorable outcomes in complex environments that are characterized by uncertainty and that may change in unpredictable ways. Engineering for the enterprise is not a replacement for classical systems engineering. Increasingly, both disciplines must be used in combination to achieve success.

This article begins with a discussion of ecosystems and includes a large number of footnotes and references for the interested reader. This may strike some as an odd place to start. But, in many ways, the point of view required to understand ecology is analogous to that needed to comprehend the complex environment in which MITRE SEs find themselves today. In fact, several of the emerging best practices and lessons learned discussed later in this article are inspired by ecology or evolutionary biology. Last, the best practices and lessons learned are organized around important conundrums, needs, or issues that SEs face in complex environments.

Because engineering in complex environments is an emerging and rapidly changing field, MITRE SEs and others are developing its processes, techniques, and tools as they execute their program responsibilities. As a result, in many cases, there is an inherent uncertainty about the right wisdom to recommend. But pointing the issues out has value even if we don't yet know exactly the right wisdom to convey about solving them. When it has been possible to suggest at least potential approaches to dealing with a problem, the article does so.

Background

People exist within an ecosystem. We have a sense of what that means and understand the natural world around us as elements in an ecosystem. Our technical systems exist within ecosystems as well. We need to unwrap what it means to be systems engineers within an ecosystem and thus understand the nature and sources of uncertainty, lack of control, and complexity in our environment [7].

Most people have a keen sense of what a natural ecosystem consists of and how it morphs over time in response to changes of (and to) its constituent members. Ecosystems are dynamic. Their aggregate state is arrived at through various interactions among the elements present (some connected directly, others indirectly through some transitive path) and how the elements push and pull all the time along many dimensions. Any apparent stability is a dynamic stability—when one of the interacting elements is altered, the stability point of the aggregate changes. And the changes ripple through the connected pieces—sometimes rather rapidly and unexpectedly—until another dynamic stability point is found.

Ecosystems are distributed and also have no leader; no one's "in charge." This is nothing that needs to be "fixed"—in fact, it can't be fixed [8].

All the systems we work on at MITRE have always existed in this type of ecosystem and have been subject to this type of push-and-pull. But three things have changed:

  1. In the past, we used people as the "impedance matching" element between the artificial elements (traditionally, fully formed, and conceived systems); now artificial elements are connecting to other artificial elements (machine to machine).
  2. We now accept (and demand) wide potential interconnections among the artificial elements (composition on demand [9]).
  3. We are now expected to perform engineering at larger scopes and scales (enterprise engineering).

We now find our systems to be primary elements of the ecosystems in which they reside, rather than augmentations to the primary elements (i.e., the people using them), and we must factor that into our requirements, analyses, and designs [10]. They must be able to respond to changes in the context they find themselves within, rather than relying on people to be the elements that change in response to context changes (i.e., the environment).

Also, this environment is changing at rapid and unpredictable rates and in places we didn't necessarily predict. And, the technology itself is changing at unprecedented rates. Thus, we are finding that agility is most desired. The systems themselves must be agile; not just the users of the systems [11, 12]. Most important, isolation (or attempted isolation) doesn't work.

Having made the argument for variety and interaction, it is important to add the guiding factor: selection. Arbitrary and random change merely leads to chaos. However, the environment guides or channels change by selecting the winners and the losers among those present. Those chosen are said to be "more fit" for the environment. Thus fitness, and its measurement, might be something to pursue [13].

Given multiple interdependencies, rippling change, an unknown (and possibly unknowable) future, and selection among choices, clearly, we can expect uncertainty. Therefore, agility is a top need. But agility of what?

  • Agility of the aggregate: "Systems" and "systems of systems" are nothing more than collections of technical elements, people, and processes that perform useful work. It is in the aggregate where this useful work is realized. To be agile, the aggregates must be able to adapt (and possibly, to be assembled) near the point of need and in a time frame that allows the potential solution (the aggregate) to be applied to the need in a timely way.
  • Agility of the elements: Each element within an aggregate must itself be able to evolve. The rate of its evolution—or its ability to change rapidly with the emergence of a new need—may well define its value to the enterprise. Thus, adaptability becomes a strong design aspect.

It is within this environment, and with these challenges, that MITRE SEs are expected to perform. They must have this understanding and mindset. It is within this mindset that we can see users as arbiters of "fitness" and the greater stakeholder community as the environment [14].

Government Interest and Use

The government has a direct interest in seeing that systems built are agile and composable in order to meet the changing ecosystem in which our government customers live. Examples of capabilities that MITRE has built this way are in the SEG article Special Considerations for Conditions of Uncertainty: Prototyping and Experimentation. Being agile and composable satisfies the ability to change quickly as conditions, technologies, missions, and procedures change. It also suggests that we may be able to achieve more (re)usability and thus more effectively manage cost. Uncertainty becomes less of a problem if agility is possible. It allows rapid reaction to current conditions rather than prediction of future conditions followed by subsequent reaction/change. Best practices and lessons learned fall along these lines [15, 16, 17, 18, 19].

Best Practices and Lessons Learned

Technology

Build options into designs. Given an unknown future, MITRE SEs are expected to consider and recommend the value of building options into designs [20]. They are expected to envision possible system or enterprise extensions in advance, the likelihood of whether and when they would be needed, and the cost of extending the design versus creating a replacement.

Partition design by both functionality and time differences of change. Traditional design tends to partition primarily by function. However, partitioning also by rate of change allows you to isolate elements that change quickly (or might change quickly) from those elements that are more stable and will change slowly [21].

Encapsulate change. A basic tenet of design that has weathered the test of time is to isolate things that change. Key to this is the use of interfaces as a method to isolate the partitions from each other, yet allow interaction.

Carefully choose "bow ties" [22]. In the design, identify and codify the key decoupling points that divide designs into coherent layers. These points should be small in number and ruthlessly enforced. It is the essence of workable architectures. A small number of connection/decoupling points of very low variety (i.e., goal of one) allows high variety on each side of these strategic points. The key decoupling points should use well-known and popular protocols and methods to ensure they have "legs."

Build an enterprise element while building a local system. Understanding your offering to the enterprise:

  • What does it do (the single thing that provides value to the enterprise)?
  • How do others interact with it?
  • Where/how is it available?
  • How do others find it?

Refactor for the enterprise. Once the enterprise (i.e., consumers outside of those originally anticipated by the program originators) discovers and uses local elements, refactoring their appearance and presentation to the enterprise is likely warranted. This could mean:

  • Splitting a system into two or more, which allows each part to change at its own rate or permits access and interaction to only a piece of the original whole.
  • Substituting one element for another, which allows a new element to perform a role previously provided by another. This allows evolution and change and is the fundamental idea behind interface implementer substitution.
  • Augmenting a system with new elements, which may allow new roles for the system.
  • Inverting element dependencies to alter business/political considerations. Consider the different political/business dynamics resulting from using a design pattern such as subclass/inheritance versus containment/delegation.

These actions have been argued to be design primitives [23].

Flows [24] and their emergence. Information flows are the essence of command and control systems. In the past, we often used defined flows within our designs to decide which system elements needed to connect to realize a system's behavior. To achieve agility, however, we need to create designs that allow technical elements to join and leave existing flows dynamically and that enable the creation of new flows.

Structure and Organization

MITRE SEs are expected to consider, recommend, and apply systems engineering strategies such as early prototyping, exploratory integration testbeds, field trials, and experiments to support early and continuous discovery activities in situations in which the required behavior of the deployed system(s) is difficult to predict.

  • Development networks: Mimic the real world as much as possible.
    • Providing vetted access to online software services that are also found in the fielded system allows third parties to learn about and use aspects of the system of record that they would otherwise need to guess at.
    • Third-party developers who use the resources available on the development network will require less integration, hand-holding, and rework, thus speeding fielding and holding costs under control.
  • Developmental spirals: Because the future is difficult to predict, using spirals (smaller scope, shorter duration) to sharpen the focus on future requirements lowers uncertainty and risk.
  • Modeling and simulation: People are poor at predicting patterns formed from the interactions of elements (e.g., rules, computing artifacts). The only way that we can fairly, and without introducing additional bias, elicit patterns (other than the choices and assumptions that go into a model, which should be explicit) is to use modeling and simulation to explore the interactions (be they operational, technical, or systemic).

Piloting integration strategies. MITRE SEs are expected to consider, recommend, and implement integration strategy pilots to explore terminology, operational patterns, technology, and desired features when interoperating systems cross multiple seams and lack a history of effectively working together.

Using "technical intimacy"—from casual relations to deep commitment—we are most likely to use (and depend on) an external element when it:

  • Already exists.
  • Is available.
  • Is likely to remain available.
  • Is understandable.
  • Makes small demands on our resources.
  • Requires few unique agreements.
  • Appears to be part of the environment.

Replaceability vs. reusability. Focus on designs that offer the ability to replace elements with other (similar) elements as experience is gained with a system and/or as requirements change, rather than seeking or designing elements that purport to include all future needs. We can start with small sets of known functionality and then grow it. This lowers risk greatly.

Partnerships build trust [25]. Forming partnerships among both consumers and producers of services builds trust. Activity taking place on a development network can provide pointers to potential partnering opportunities that may not have been obvious.

Business and Economic: [26, 27, 28]

Reduce uncertainty [29]. MITRE SEs are expected to understand the elements that may drive uncertainty in the tasks they're supporting. Uncertainty may come from requirements and/or technologies, and MITRE SEs must help customers understand this environment and help mitigate the uncertainty. See Table 1.

Table 1. Strategies Based on Uncertainty

Project Characterization

Strategy

Stable requirements and technologies

Plan ahead and then execute the plan.

Dominated by new or emerging technologies

"Portfolio of small bets."

Dominated by evolving requirements

"Staged commitments."

All of the above

Hybrid

Reduce uncertainty in cost estimation. MITRE SEs are expected to understand the principles underlying good cost estimation and be able to recommend and implement techniques to mitigate cost uncertainty, including developing design alternatives as bases for cost.

Two "truths" are in conflict:

  1. We need to know what to build before building it.
  2. Things always change.

Thus, the idea that requirements must be known before building is desired, but the requirements themselves may be changing. So, if things always change, knowing what to build may be fuzzy. But "what to build" needs to be known to estimate well.

If it's fuzzy, tighten it up, either in time or scope. Can we define what will be done this year? This month? This week? Find a time slice where this is clear, outcomes are definite, and the method to achieve them is known. Where things become fuzzy, this may well be a point where there's a logical branching of possibilities and a perfect opportunity for "Real Options" [30] to be developed. This is good for interfaces in which details can be deferred.

With respect to estimation:

  • The smaller it is, the easier it is to estimate.
  • The simpler it is, the easier it is to estimate.
  • The more mature the technology is, the easier it is to estimate.
  • The more that is supplied by others, the less needs to be done (i.e., the smaller it is).

For our estimation methodology, we have many choices. They all share one key characteristic: none is able to satisfy all. This goes from agile and lean techniques [31], which measure team velocity delivering story points, to function points, and the classic SLOC (source line of code) counts. Be very wary of whatever technique is chosen. Don't automatically accept it—always seek supporting and refuting evidence on the estimates during execution.

Establishing baselines. The baselines should be appropriate for the estimation methodology and the development measurement methods. For example, "done done" in agile methods should be ruthlessly watched. This fits well with defining earned value milestones (EVM) [32]. A potential benefit of EVM is that it demands a crisp definition of a milestone and provides early hints when the cost and schedule assumptions are being violated. This may provide a tool for knowing when to abandon one option and pick another.

A hidden problem with service-oriented approaches [33]. Ironically, although service-oriented approaches offer the potential agility and composability desired, the manner in which we contract with developers may erect barriers to realizing the benefits. Consider the situation in which a program offers a service that delivers some of its information bundled in a collection. Suppose further that many outside the originally planned users and stakeholders discover this and find it useful. Under these circumstances, we might expect the use of the service to be greatly beyond the planned use. However, there may be transaction densities that exceed the design limits and degrade the performance. Whose problem is this, and how is it mitigated?

Contract types. Consider using contract structures for which the default behavior is not continuation. We might do this using an indefinite delivery/indefinite quantity contract with a series of tasks that build on one another, yet where each has a clear ending.

Consider using "supplier" models in contrast with "developer" models. Payout is based on use.

Working the uncertainty and complexity of MITRE's customer environment has many challenges. The need to manage these challenges has become more prevalent in the tasks MITRE takes on as we continue to work the strategic and hard problems of our sponsors and their enterprises. The practices listed can help work this critical area—as MITRE staff gain more experience, these practices will evolve as well in our uncertain and complex MITRE world.

References and Resources

  1. As characterized in Stevens, R., September 24, 2009, "Adapting Venture Capital Concepts to Enterprise System Acquisitions."
  2. Dorner, D., 1996, The Logic of Failure, Basic Books.
  3. Mitchell, M., 2009, Complexity: A Guided Tour, Oxford.
  4. "Lack of control" includes many conditions and situations, but the most general sense of its use here is the inability to set the desired state, or vector, of an element under our authority, and which we are expected to exercise control over. The "traditional" approach to ensuring control is isolation of a system—both in development and in use. With more interconnected and interdependent elements and systems, this presumption (and technique) is violated.
  5. "Complex" has become a term of art in engineering and science, and its meaning is slightly different than how it is used in the vernacular. At the risk of oversimplifying, "complicated" means difficult to understand, whereas "complex" means stable collections and patterns arising (or emerging) from simple interactions among component pieces. See almost any of the references for more description.
  6. Metzger, L. M., April 27, 2009, "MITRE Systems Engineering: So What's It All About?"
  7. Bar-Yam, Y., 2004, Making Things Work: Solving Complex Problems in a Complex World, Knowledge Press.
  8. Johnson, S., 2001, Emergence: The Connected Lives of Ants, Brains, Cities, and Software, Scribner.
  9. Also see the SEG article Composable Capabilities on Demand in the Engineering Information-Intensive Enterprises topic.
  10. DeRosa, J. K., et al., 2008,"The Requirements Conundrum in Complex Systems Engineering," ICSSEA 2008, Paris.
  11. Watts, D., 2003, Six Degrees: The Science of a Connected Age, Norton.
  12. Barabasi, A-L., 2002, Linked: The New Science of Networks, Perseus.
  13. Perhaps only the recognition of fitness as a process is sufficient, and the understanding and management of choices and "choice-spaces" is where we can make our engineering processes reflect fitness. If we were to be able to understand and quantify fitness, it might give us a predictive tool that is currently absent.
  14. Many efforts attempt to quantify fitness in engineering systems. In our own community, there are attempts to define measures of effectiveness focusing on operational metrics.
  15. Norman, D. O., and B. E. White, 2008,"Asks the Chief Engineer: "So What Do I Go Do?!," SysCon 2008, IEEE International Systems Conference.
  16. Norman, D. O., and M. Kuras, 2006, Engineering Complex Systems, in Complex Engineered Systems, A. A. Minai, D. Braha, and Y. Bar-Yam (eds.), New York, N.Y., Springer, Ch. 10.
  17. Norman, D. O., 2009, "Engineering Complex Systems: Challenges in the Theory and Practice," in Organizing for a Complex World, CSIS Press, Ch. 7.
  18. Friedman, T. L., 2006, The World Is Flat, New York, N.Y., Farrar, Straus, and Giroux.
  19. Ackoff, R. L., and F. Emery, 2006, On Purposeful Systems, Transaction.
  20. Options provide variety. And, variety is absolutely necessary to promote evolution. This may seem counterintuitive to those who view variety as mere redundancies. It must be recognized that variety (also diversity) requires selection to lead toward effective evolution. Variety is explained nicely by R. Ashby in "Law of Requisite Variety" in his book Introduction to Cybernetics, 1956, Chapman and Hall, London. Currently out of print, it can be found .
  21. Think about automobiles. If we didn't allow for the removal and replacement of the wheels/tires, we would need to wait for a different (redesigned) vehicle to operate the vehicle in a different environment—like loose sand rather than asphalt. By recognizing that the wheels/tires can be changed more quickly than the vehicle can be replaced, we allow change at that point, and the evolution of the whole can occur more rapidly.
  22. See the SEG articles Architectures Federation and Design Patterns in the  Engineering Information-Intensive Enterprises topic.
  23. Baldwin, C., and K. Clark, 2000, Design Rules: The Power of Modularity, Vol. 1, Cambridge, Mass., MIT Press.
  24. Holland, J., 1995, Hidden Order: How Adaptation Builds Complexity, Perseus.
  25. Moore, J. F., 1996, The Death of Competition: Leadership & Strategy in the Age of Business Ecosystems, Harper Business.
  26. Beinhocker, E., 2006, The Origin of Wealth: Evolution, Complexity, and the Radical Remaking of Economics, Boston, Mass., HBS Press.
  27. Wheatley, M. J., 1999, Leadership and the New Science: Discovering Order in a Chaotic World, B. Koehler.
  28. Also see the SEG topic Acquisition Program Planning in Acquisition Systems Engineering.
  29. Stevens, R., September 24, 2009, "Acquisition Strategies to Address Uncertainty: Acquisition Research Findings," from her MITRE-Sponsored Research "Enterprise Systems Acquisition Using Venture Capital Concepts."
  30. Miller, L. T. and C. S. Park, 2002, "," Engineering Economist, Vol. 47, Issue 2.
  31. Shore, J., and S. Warden, 2008, The Art of Agile Development, O'Reilly.
  32. Fleming, Q., and J. Koppelman, 2005, Earned Value Project Management, 3rd Ed., Project Management Institute.
  33. Martin, J., 1995, The Great Transition: Using the Seven Disciplines of Enterprise Engineering to Align People, Technology, and Strategy, Macon.

Publications

Download the SEG

MITRE's Systems Engineering Guide

Download for EPUB
Download for Amazon Kindle
Download a PDF

Questions?
Contact the SEG Team