How to Conduct Process and Product Reviews Across Boundaries


Definition: Process and Product Quality Assurance are activities that provide staff and management with objective insight into processes and associated work products [1]. A Quality Management Process assures that products, services, and implementations of life-cycle processes meet organization quality objectives and achieve customer satisfaction [2].

Keywords: noncompliance, objective evaluation, process, process description, process review, product review, quality, quality assurance, quality management, work products

MITRE SE Roles & Expectations: MITRE systems engineers (SEs) conduct process and product reviews across boundaries in the government systems acquisition and/or operational organizations. In this role, they assist both government and contractor organizations to document quality processes and work product specifications. To ensure adherence to documented processes and work product specifications, MITRE SEs review government and contractor quality processes and products and contractor quality assurance programs; prioritize quality process improvement opportunities and corrective actions; report to key decision makers the results of process and product reviews; and elevate high-priority corrective actions [3].

Background

When projects pay attention to the quality of their products and the processes that produce them, they have a better chance of succeeding. When there is a government-to-contractor relationship established through a contract, MITRE's role needs to sharpen because the focus of our analysis and guidance then has two sides involved. We need to think through such situations carefully to give customers our best advice. To do that, MITRE SEs should look to a well-recognized common process framework like ISO/IEC 15288 [2] and established process improvement models such as the Capability Maturity Model Integrated® (CMMI) [1] to ground our analysis and guidance and to get both sides on a firm process foundation.

Both the government and contractors need a common framework to improve communication and cooperation among the parties that create, utilize, and manage modern systems so they can work in an integrated, coherent fashion. ISO/IEC 15288 is an international standard that provides this framework and covers the life cycle of human-made systems. This life cycle spans a system from idea conception to system retirement. ISO/IEC 15288 provides the processes for acquiring and supplying systems and for assessing and improving the life-cycle processes.

ISO/IEC 15288 defines outcomes that should result from the successful implementation of a Quality Management Process:

  • Organization quality management policies and procedures are defined.
  • Organization quality objectives are defined.
  • Accountability and authority for quality management are defined.
  • The status of customer satisfaction is monitored.
  • Appropriate action is taken when quality objectives are not achieved.

Furthermore, the standard defines certain activities and tasks an implementation is expected to follow:

  • Plan quality management, which includes:
    • Establishing quality management policies, standards, and procedures.
    • Establishing organization quality management objectives based on business strategy for customer satisfaction.
    • Defining responsibilities and authority for implementation of quality management.
  • Assess quality management, which consists of:
    • Assessing and reporting customer satisfaction.
    • Conducting periodic reviews of project quality plans.
    • Monitoring the status of quality improvements on products and services.
  • Perform quality management corrective action, which consists of:
    • Planning corrective actions when quality management goals are not achieved.
    • Implementing corrective actions and communicating results through the organization.

Again, both government and contractor processes should each perform, in some fashion, these types of activities in their respective domains. The government has an additional responsibility: to have insight into and oversight of the contractor's quality management process to ensure it is in place, it is performing, and defects are identified and removed as a matter of daily practice. Otherwise, the government could be on the receiving end of poor quality products that impact its commitments to its customers.

Both the government and contractors need to improve their processes. CMMI was developed by a group of experts from industry, government, and the Software Engineering Institute (SEI) at Carnegie Mellon University. It is a process improvement approach to software engineering and organizational development that provides organizations with the essential elements for effective process improvement to meet business goals. It can be used to guide process improvement across a project, a division, or an entire organization.

CMMI's Process and Product Quality Assurance Process Area supports the delivery of high-quality products and services by providing project staff and managers at all levels with appropriate visibility into, and feedback on, processes and associated work products throughout the life of the project. It establishes expectations that a project's (or an organization's) quality assurance process will objectively evaluate processes and work products so that when non-conformance issues are identified, tracked, and communicated, resolution is ensured. For every process, CMMI further establishes generic expectations, which include, for example, the establishment of policy, plans, and monitoring. Having a common process improvement reference model gets both sides to think about process improvement in the same way, establishes a common framework and language, and promotes cooperation at any troublesome touch points between the two.

For related Systems Engineering Guide information, see the other articles in this Quality Assurance and Measurement topic and those in the topic areas Continuous Process Improvement, Contractor Evaluation, and MITRE FFRDC Independent Assessments.

Best Practices and Lessons Learned

Yes...Quality Is Important for Government Work, Too. Strive for the establishment of an independent, properly positioned quality management function on the government side. It needs to be positioned at a high enough level to have senior leadership's attention and not be bullied by project managers and mid-level management. To expect quality assurance only on the contractor side is not good enough.

Use Standards and Previous Efforts in Establishing Your Quality Capabilities. Don't start from scratch in developing an organization's quality management process. Use a standard like ISO/IEC 15288 for starters. Check with other government organizations and take a look at their quality office, and what policies, processes, and templates they use. Stand on the shoulders of previous efforts and apply them to your situation.

Set Quality Standards Up Front. Make sure there are organizational standards established for project work process and products. It's tough to check quality if you don't have a standard to check against. Look to IEEE or your government department or agency standards and tailor them for your organization. Check with the program's prime commercial contractor for examples, but remember the perspective from which those standards were built...from the supplier side, so they'll need to be adjusted.

Ensure Quality Expectations are Built into the Contracts/Task Orders. Build the expectation of quality into your contracts and/or task orders. If you don't state it, you're not likely to get it. Expect products from the contractor will be checked against the agreed upon standard, and defects will be identified and removed before the products are delivered. In the contract, make sure the government can periodically check defect records, process appraisal results, defect tracking databases, and process improvement plans to see if they're in place and actively being worked. Don't just expect quality to magically appear from the contractor. Many times it does not unless they know the sponsor cares enough to be checking.

Trust...but Verify. Once the contract arrangement is up and running, the contractor's quality management function is operational, and trust is beginning to solidify, the government may want to consider only periodic reviews of contractor processes on a sampling basis. Objective evidence of active reviews and defect resolution is key.

Have a Common Strategy for Improving the Process. On both the government and contractor sides, there should be active process improvement efforts in place that continuously look to mature the efficiency, effectiveness, and timeliness of their quality assurance activities. It is highly desirable that both sides agree on the same process improvement model, such as CMMI. Both sides should have a common vocabulary and improvement and appraisal methods, which promote effective communication and collaboration opportunities to help speed performance improvement initiatives.

Remember the Bottom Line. MITRE SEs should actively encourage and promote quality management processes and standards on both sides, properly positioned in their management structures, with a culture that encourages process improvement to ultimately result in higher quality, on-time, and useful systems.

References & Resources

  1. CMMI Product Team, , CMU/SEI-2006-TR-008.
  2. ISO/IEC 15288, —.
  3. "MITRE Systems Engineering (SE) Competency Model, Version 1," September 1, 2007, The MITRE Institute, p. 46.

Publications

Download the SEG

MITRE's Systems Engineering Guide

Download for EPUB
Download for Amazon Kindle
Download a PDF

Questions?
Contact the SEG Team