Defense Acquisitions:

DOD Needs to Develop Performance Criteria to Gauge Impact of Reform Act Changes and Address Workforce Issues

GAO-10-774, Jul 29, 2010

Additional Materials:

Contact:

Michael J. Sullivan
(202) 512-3000
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

In May 2009, Congress passed the Weapon Systems Acquisition Reform Act of 2009 (Reform Act). The Reform Act contains a number of systems engineering and developmental testing requirements that are aimed at helping weapon programs establish a solid foundation from the start of development. GAO was asked to examine (1) DOD's progress in implementing the systems engineering and developmental testing requirements, (2) views on the alignment of the offices of the Directors of Systems Engineering and Developmental Test and Evaluation, and (3) challenges in strengthening systems engineering and developmental testing activities. In conducting this work, GAO analyzed implementation status documentation and obtained opinions from current and former DOD systems engineering and testing officials on the placement of the two offices as well as improvement challenges.

DOD has implemented or is implementing the Reform Act requirements related to systems engineering and developmental testing. Several foundational steps have been completed. For example, new offices have been established, directors have been appointed for both offices, and the directors have issued a joint report that assesses their respective workforce capabilities and 42 major defense acquisition programs. Many other requirements that have been implemented will require sustained efforts by the directors' offices, such as approving systems engineering and developmental testing plans, as well as reviewing these efforts on specific weapon programs. DOD is studying the option of allowing the Director, Developmental Test and Evaluation, to serve concurrently as the Director of the Test Resource Management Center. The directors have not yet developed joint guidance for assessing and tracking acquisition program performance of systems engineering and developmental testing activities. It is unclear whether the guidance will include specific performance criteria that address long-standing problems and program risks, such as those related to concurrency of development and production activities and adequacy of program resources. Current and former systems engineering and developmental testing officials offered varying opinions on whether the new directors' offices should have been placed under the Director of Defense Research and Engineering organization--an organization that focuses primarily on developing and transitioning technologies to acquisition programs. The Director of Defense Research and Engineering believes aligning the offices under his organization helps address congressional and DOD desires to increase emphasis on and strengthen activities prior to the start of a new acquisition program. Most of the officials GAO spoke with believe the two offices should report directly to the Under Secretary for Acquisition, Technology and Logistics or otherwise be more closely aligned with acquisition programs because most of their activities are related to weapon programs. They also believe cultural barriers and staffing issues may limit the effectiveness of the two offices under the current organizational structure. Currently, DOD is not reporting to Congress on how successfully the directors are effecting program changes, making it difficult to determine if the current placement of the offices makes sense or if the Reform Act is having an impact. The military services face a number of challenges as they try to strengthen systems engineering and developmental testing activities on acquisition programs. Although the services believe they have enough staff to perform both of these activities, they have not been able to clearly identify the number of staff that are actually involved. The Director of Developmental Test and Evaluation does not believe the military services have enough testing personnel and is concerned that DOD does not have the capacity to train the large influx of contractors that are expected to be converted to government employees.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: In order to determine the effectiveness of the newly established offices, the Secretary of Defense should direct the Directors of Systems Engineering and Developmental Test and Evaluation to develop plans for addressing the training needs of the new hires and contractors who are expected to be converted to government personnel.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: Both the Deputy Assistant Secretary for Systems Engineering and Developmental Test and Evaluation sponsored Defense Acquisition University course development and class revisions in fiscal year 2010. The offices are identifying any skills gaps that may exist between the workforce's current capabilities and those needed to meet future mission requirements; and shape future workforce training, planning, and development. They will report on the status of their actions in the 2011 Joint Annual Report.

    Recommendation: In order to determine the effectiveness of the newly established offices, the Secretary of Defense should direct the Directors of Systems Engineering and Developmental Test and Evaluation to work with the services to determine the appropriate number of government personnel needed to perform the scope of systems engineering and developmental testing activities.

    Agency Affected: Department of Defense

    Status: Open

    Comments: The Deputy Assistant Secretary for Systems Engineering has been working with the services to track current and future projections for their workforce. They reported in their 2011 annual report that the current authorities are sufficient to support high-priority and critical positions but that military departments will need to prioritize their resources for the future. The Deputy Assistant Secretary for Developmental Test and Evaluation plans to work with the services to provide short and long-term strategies for developing an optimum balance of their workforce. An update of developmental test and evaluation workforce efforts will be reported in the 2012 annual report.

    Recommendation: In order to determine the effectiveness of the newly established offices, the Secretary of Defense should direct the Directors of Systems Engineering and Developmental Test and Evaluation to track the extent to which program offices are adopting systems engineering and developmental testing recommendations.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: The systems engineering office has begun to track the extent to which recommendations included in program support reviews are being adopted. According to the Deputy Assistant Secretary for Systems Engineering, an average of 86 percent of actionable recommendations included in 17 program support reviews conducted in fiscal year 2010 were accepted by program managers. The developmental testing office has begun to track the extent to which recommendations to programs are being adopted. According to the Deputy Assistant Secretary for Developmental Test and Evaluation office, an average of 50 percent of recommendations in fiscal year 2011 were adopted. The office will report updated information in their 2012 annual report.

    Recommendation: In order to determine the effectiveness of the newly established offices, the Secretary of Defense should direct the Directors of Systems Engineering and Developmental Test and Evaluation to ensure development and implementation of performance criteria for systems engineering plans and developmental test and evaluation master plans, such as those related to technology maturity, design stability, manufacturing readiness, concurrency of development and production activities, prototyping, and the adequacy of program resources.

    Agency Affected: Department of Defense

    Status: Open

    Comments: The Deputy Assistant Secretary for Systems Engineering issued guidance on the development of performance criteria in the new Systems Engineering Plan Outline Version 1.0, dated April 2011. Development and implementation of performance criteria must be documented in all plans requiring the Deputy Assistant Secretary for Systems Engineering's approval. The Director of Developmental Testing has identified an initial set of performance criteria that have been piloted on two programs. Based on results of the pilot, the office plans to update the metrics and begin using them throughout 2012 on programs that will be included in the 2012 annual report.

    Recommendation: In order to determine the effectiveness of the newly established offices, the Secretary of Defense should direct the Directors of Systems Engineering and Developmental Test and Evaluation to report to Congress on the status of these efforts in future joint annual reports required by the Reform Act.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: The 2011 Joint Annual Report included information on military service efforts to grow their respective SE and DT workforces and train new employees. The directors plan to help the services address problems in these areas. The Deputy Assistant Secretaries did not provide information on the extent to which programs are implementing their recommendations. DOD stated in their response to our report that they plan to track implementation status, but would not report that information formally to the Congress in the joint annual report. We have obtained information that confirms that the Deputy Assistant Secretary for Systems Engineering is tracking implementation status. The Deputy Assistant Secretary for Developmental Test and Evaluation is also tracking implementation status. They plan to continue to report on actions in the 2012 Joint Annual Report.

    Jun 11, 2013

    May 31, 2013

    May 29, 2013

    May 23, 2013

    May 17, 2013

    May 16, 2013

    May 15, 2013

    May 14, 2013

    May 13, 2013

    Looking for more? Browse all our products here