This is the accessible text file for GAO report number GAO-12-480R entitled 'Defense Management: Actions Needed to Evaluate the Impact of Efforts to Estimate Costs of Reports and Studies' which was released on May 10, 2012. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. GAO-12-480R: United States Government Accountability Office: Washington, DC 20548: May 10, 2012: The Honorable Carl Levin: Chairman: The Honorable John McCain: Ranking Member: Committee on Armed Services: United States Senate: The Honorable Howard P. "Buck" McKeon: Chairman: The Honorable Adam Smith: Ranking Member: Committee on Armed Services: House of Representatives: Subject: Defense Management: Actions Needed to Evaluate the Impact of Efforts to Estimate Costs of Reports and Studies: Citing long-term fiscal challenges affecting the federal government, in May 2010, the Secretary of Defense directed the Department of Defense (DOD) to undertake a departmentwide initiative to assess how the department is staffed, organized, and operated with the goal of reducing excess overhead costs and reinvesting these savings in sustaining DOD's current force structure and modernizing its weapons portfolio.[Footnote 1] The Secretary's initiative targeted both shorter- and longer-term improvements and set specific goals and targets for achieving cost savings and efficiencies. The initiative was organized along four tracks, each of which had a different focus (see enclosure I). The fourth track focused on specific areas where DOD could take immediate action to reduce inefficiencies and overhead, in particular, to reduce headquarters and support bureaucracies and to instill a culture of cost consciousness and restraint in the department. As part of the fourth track, the Secretary of Defense announced a number of specific initiatives, including actions intended to address the need to reduce or eliminate reporting requirements for DOD reports and studies. For example, in his August 9, 2010, speech announcing the overall efficiency initiative, the Secretary of Defense stated that the department is "awash in taskings for reports and studies" and directed several specific actions that, according to the press release accompanying the announcement, were intended to "combat the enormous amounts of taskings for reports and studies."[Footnote 2] In his remarks, the Secretary noted that there is little basis to determine whether the value gained is worth the considerable time and resources expended to generate reports and studies. With respect to specific actions, the Secretary directed DOD and its components to track the approximate cost of preparing DOD reports and studies and publish the cost on the front cover of each report or study. He also called for a comprehensive review of all oversight reports and use of the results to reduce the volume of reports and studies generated internally while engaging with Congress on ways to meet their needs while working together to reduce the number of reports. In September 2010, the Secretary of Defense tasked DOD's Cost Assessment and Program Evaluation (CAPE) office with implementing the Secretary's requirement to publish the cost of preparing DOD reports and studies. By November 2010, the CAPE office had developed a cost estimating tool and corresponding guidance on using the tool, which described the categories of costs to be estimated and identified the types of reports and studies for which costs were to be estimated. [Footnote 3] The following month, the Secretary of Defense issued a memorandum requiring the use of the tool beginning on February 1, 2011.[Footnote 4] In House Report Number 112-78, which accompanied a bill for the National Defense Authorization Act for Fiscal Year 2012,[Footnote 5] the House Committee on Armed Services commended DOD for implementing a process for collecting an estimate of resources required to generate internally and externally required reports and agreed that additional transparency would be useful for decision makers when determining the utility of various reporting requirements. The committee also observed that any tool used to collect costs was only as useful as the inputs received and directed GAO to conduct an assessment of DOD's methodology and tools for collecting cost data on both internally and externally required reports and to submit a report, including any recommendations needed to improve the data collection, transparency, and utility of the tool. Specifically, we evaluated DOD's approach to estimate the costs of selected reports and studies, including (1) whether DOD entities are capturing and presenting costs in a manner consistent with relevant cost estimating best practices; (2) the status of DOD's efforts to track whether organizations are developing cost estimates for all required types of reports and studies; and (3) whether DOD has evaluated the usefulness of its efforts to estimate the costs of reports and studies as one of the means for achieving the Secretary's intended purpose of increasing the transparency of costs, reducing or eliminating reporting requirements, and instilling a culture of cost consciousness across DOD. To determine whether the tool is being used to capture and present costs in a manner consistent with relevant cost estimating best practices, we reviewed the cost estimating tool and compared it to characteristics of cost estimating best practices identified in GAO's Cost Estimating and Assessment Guide[Footnote 6] and related managerial cost accounting guidance.[Footnote 7] GAO's Cost Estimating and Assessment Guide reflects best practices for developing and managing cost estimates for capital programs. Because DOD is estimating noncapital costs, we assessed DOD's tool used to generate cost estimates against the best practices most applicable to noncapital costs as well as against relevant managerial cost accounting standards and guidance, which we determined are consistent with the selected best practices. Specifically, we determined the extent to which the estimates resulting from use of the tool were "comprehensive," "well documented," and "accurate"--essential elements for high-quality, reliable cost estimates. We also interviewed DOD officials to corroborate our understanding of their use of the tool and its corresponding guidance. All data elements we assessed are sufficiently reliable for the purpose of assessing DOD's cost estimating tool against best practices. We had originally intended to review a representative sample of DOD reports and studies for which cost estimates had been generated and published. However, as described in this report, DOD does not currently have the ability to distinguish between real estimates and estimates that were developed while individuals were using the tool for training purposes. As a result, DOD could not provide us with a comprehensive list of all reports and studies for which costs had been estimated. However, DOD officials were able to provide us information on nine reports prepared for Congress for which cost estimates had been generated in 2011. We reviewed these nine reports and interviewed the key officials responsible for developing the cost estimates for them. As described in this report, in some cases, officials were not able to readily retrieve documentation; therefore, we were unable to fully evaluate the extent to which they consistently applied DOD's guidance for estimating costs. To describe the status of DOD's efforts to track whether organizations are developing cost estimates for all required types of reports and studies, we reviewed guidance and interviewed DOD officials to assess the nature of their tracking efforts. To determine whether DOD has evaluated the usefulness of its efforts to estimate the costs of reports and studies as one of the means for achieving the Secretary's intended purpose of increasing the transparency of costs, reducing or eliminating reporting requirements, and instilling a culture of cost consciousness across DOD, we reviewed DOD guidance. We also interviewed DOD officials responsible for developing and using the tool, including the individuals who generated the nine reports that we reviewed to determine any steps taken to evaluate the impact of the cost estimating effort, including whether they had obtained feedback from internal and external decision makers. We conducted this performance audit between October 2011 and May 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. More detailed information on our scope and methodology is provided in enclosure II. Results in Brief: DOD is estimating and publishing approximate costs for selected types of internally and externally required reports, but in some cases its approach is not fully consistent with relevant cost estimating best practices and cost accounting standards. Specifically, DOD entities have been directed to use the cost estimating tool to capture marginal costs of activities associated with completing a report or study that would not have been performed otherwise. These costs consist of certain manpower costs (such as the prorated salaries of military and civilian personnel based on the time they spent) and nonlabor costs (such as contract services, travel, or printing). In comparing DOD's approach to (1) GAO's Cost Estimating and Assessment Guide--which states that high-quality, reliable cost estimates should be comprehensive, well documented, and accurate--and (2) relevant accounting standards, we found the following. Comprehensive. GAO's cost guide states that cost estimates should include all costs, but allows flexibility for the estimates to exclude costs where information is limited as long as steps are taken to clearly define and document what costs are included or excluded. DOD's guidance on using the tool defines and documents decisions to include certain manpower and all nonlabor costs in its calculation. In addition, it documents the decision to exclude other manpower costs, such as those for health care and training expenses. It also provides broad examples of some types of activities to consider in estimating costs, but leaves it up to the discretion of the individuals generating the cost estimate to decide which types of activities to include or exclude. As a result, based on the nine reports we reviewed, we found inconsistencies in the types of activities individuals decided to include when developing their cost estimates. For example, we found that for six of the nine reports we reviewed, some individuals calculated the manpower costs for activities associated with coordinating the report through the final review while others did not. In addition, according to relevant accounting standards, appropriate procedures and practices should be established to enable, among other things, the interpretation and communication of cost information. However, when presenting the cost estimate on the front cover of a report or study, the language DOD uses does not provide information on what costs are included or excluded. Specifically, it does not indicate that the estimate reflects only certain marginal manpower and nonlabor costs. Without further explanation, this wording could be subject to misinterpretation such that recipients of the reports or studies may assume that all costs are included in the estimate when that is not the case. Well documented. GAO's Cost Estimating and Assessment Guide states that cost estimates should be easily traceable to source documents. While DOD's guidance on using the tool states that individuals should be prepared to explain how they developed the cost estimate, it does not include a requirement for individuals to retain source documentation. In practice, we found that of the nine reports we reviewed, individuals were able to easily retrieve documentation for the estimates prepared for three reports, but not for the other six. As a result, DOD may not have the information needed so that others can understand how a cost estimate was derived and readily replicate it. Accurate. GAO's Cost Estimating and Assessment Guide further states that cost estimates should have all cost inputs checked to verify that they are accurate. However, we found that DOD's guidance on using the tool has no requirement to independently verify the costs used to generate the cost estimate. In practice, the individual generating the cost estimate is responsible for ensuring its accuracy. As a result, without independent verification, DOD cannot ensure that cost estimates are accurate and reliable. DOD currently lacks a means to ensure that organizations are developing cost estimates for all required types of reports and studies, but is taking steps to enhance its ability to monitor the preparation of reports and studies to satisfy reporting requirements, including those for which a cost estimate is required to be generated. DOD's guidance on using the tool identified 10 specific types of reports or studies that require a cost estimate. The guidance states that the DOD component preparing a report or study is responsible for ensuring that a cost estimate, if required, is included. However, the guidance does not include any process or requirement to track whether organizations are developing required cost estimates. According to CAPE officials who developed the tool, they were not directed to ensure that cost estimates have been developed for all required reports and studies, and they added that the cost estimating tool was not designed to track reports and studies that require a cost estimate. These same officials noted that the magnitude of DOD's reporting requirements makes it challenging to identify the universe of these requirements and to track the completion of reports and studies to meet them, including whether cost estimates have been generated. In the past year, DOD has initiated efforts to improve its visibility over its internal and external reporting requirements. In March 2011, the Secretary of Defense issued a memorandum requiring the Assistant Secretary of Defense for Legislative Affairs to enhance that office's database capability and further automate the process of tracking external reports to Congress and required the Office of the Director of Administration and Management to track internal reports. The Office of Legislative Affairs enhanced its Congressional Hearings and Reporting Requirements Tracking System by adding a data field to record the estimated costs of reports or studies generated by DOD components. To gain greater visibility of internal reporting requirements, the Office of the Director of Administration and Management tasked Washington Headquarters Services with developing a repository for tracking internal reports. According to a Washington Headquarters Services official, the repository has a field to capture whether an internal report or study has a cost estimate. Currently, the repository is in development and has been populated with reports that date back 3 years. DOD has not evaluated and currently does not plan to evaluate the usefulness of its efforts to estimate the costs of selected reports and studies as one of the means for achieving the Secretary of Defense's intended purpose of increasing the transparency of costs, reducing or eliminating reporting requirements, and instilling a culture of cost consciousness across DOD. According to GAO's Standards for Internal Control in the Federal Government,[Footnote 8] activities such as assessments need to be established to monitor performance, and managers need to compare actual performance against targets and then analyze any significant differences so that appropriate action can be taken to ensure that organizational goals are met. However, under current guidance, no requirement exists for DOD to evaluate the impact of its cost estimating efforts, such as whether these efforts have prompted internal or external decision makers to consider cost as a factor in deciding whether to establish a reporting requirement for DOD. Without evaluating the impact of its efforts, including seeking the views of internal and external decision makers, DOD does not have the information it needs to assess whether the time and effort it is investing to develop cost estimates is having the desired effect of achieving greater transparency, reducing reporting requirements, and raising cost consciousness, as intended by the Secretary. We are making a recommendation to the Secretary of Defense to take steps to evaluate DOD's effort to estimate costs to determine whether that effort is having the desired effect of achieving greater transparency, reducing or eliminating reporting requirements, and raising cost awareness. Based on the results of this evaluation, if DOD plans to continue the effort to estimate costs for selected reports and studies, we recommend that the Secretary modify the current guidance or otherwise take steps to improve DOD's cost estimating approach. In commenting on a draft of our report, DOD partially concurred with our recommendations, except it disagreed with one of the steps we recommended to improve its cost estimating approach. Our comments on specific points made in DOD's letter begin on page 14 and DOD's comments are reprinted in enclosure V. The department also provided technical comments on our draft report, which we incorporated as appropriate. Background: In September 2010, the CAPE office, in conjunction with the military services, established the Cost Guidance Group to implement the Secretary of Defense's requirement to estimate the cost of preparing DOD reports and studies and publish that cost on the front cover. This group, consisting of representatives from the Office of the Director of Administration and Management, the Joint Staff, and the military departments, among others, developed the guidance and tool that the Secretary of Defense directed DOD and its components to begin using by February 1, 2011. Figure 1 provides a timeline of key events related to the CAPE office's implementation of the Secretary's requirement. Figure 1: Timeline of the Development and Implementation of DOD's Cost Estimating Tool: [Refer to PDF for image: timeline] May 2010: Secretary of Defense directs DOD to review how it is staffed, organized, and operated. August 9, 2010: Secretary of Defense gives a speech on the DOD efficiency initiatives, which include tracking and publishing the costs of reports and studies. August 16, 2010: Secretary of Defense issues a memorandum directing a series of fourth track initiatives designed to reduce duplication, overhead, excess, and instill a culture of cost consciousness. September 30, 2010: The Director of the Cost Assessment and Program Evaluation (CAPE) office establishes the Cost Guidance Group to develop guidance and tool to estimate costs. October 25, 2010: Scheduled start date for development of the cost estimating tool and website. November 2010: Cost estimating tool made available for beta testing and guidance on use of the tool released. December 1, 2010: Cost guidance and support tool are made available on CAPE website to all DOD employees. December 27, 2010: Secretary of Defense issues a memorandum mandating the use of the cost estimating tool for selected reports and studies by February 1, 2011. February 1, 2011: Cost estimating tool required to be used to generate cost estimates for selected reports and studies. Source: GAO analysis of DOD information. [End of figure] The tool, also referred to as a calculator, is web based and maintained on an unclassified DOD website (see enclosure III for a snapshot of the tool). DOD entities use the tool for training purposes as well as to generate cost estimates that will be published on reports and studies. The corresponding guidance for the tool is also maintained on the website and identifies the types of reports and studies that require a cost estimate, instructs individuals on how to access and utilize the tool, and identifies manpower and nonlabor costs that could be considered for inclusion in the cost estimate, among other things. The guidance and the tool are updated incrementally with both minor and major changes. Guidance on using the tool identifies 10 types of reports or studies that require a cost estimate. Specifically, a cost estimate is required if the report or study is: * to be provided to the Secretary of Defense, * to be provided to the Deputy Secretary of Defense, * to be provided to Congress, * to be included in the Defense Technical Information Center holdings, * executed by DOD contractors, * executed by federally funded research and development centers, * executed by a board created by DOD, * executed by a board created by law, * executed by a commission created by DOD, or: * executed by a commission created by law. In addition, although it is not required, guidance on using the tool recommends that a cost estimate be generated for all reports and studies. DOD Captures Certain Costs for Selected Reports and Studies, but Its Approach Does Not Fully Reflect Relevant Cost Estimating Best Practices and Cost Accounting Standards: DOD is estimating and publishing approximate costs for selected internally and externally required reports, but in some cases, its approach is not fully consistent with relevant cost estimating best practices and cost accounting standards. DOD Estimates and Publishes Certain Marginal Costs as Directed by the Secretary of Defense: DOD entities use the cost estimating tool to capture marginal costs-- costs for activities that would not otherwise have been performed had the report or study not been developed--associated with preparing and completing a report or study. Costs to be captured consist of (1) manpower costs, such as the prorated salaries based on time spent by military personnel and DOD civilians to develop and publish a report or study,[Footnote 9] and (2) nonlabor costs, such as costs for travel, contractor-related services, and printing. Once the estimate is generated, individuals include the dollar amount of the estimate on the front cover of the corresponding report or study. Specifically, the language on the front cover states that "Preparation of this report/study cost the Department of Defense a total of approximately $_______ in Fiscal Year _____" and includes the date that the cost estimate was generated and a unique reference identification number for the report or study. CAPE officials responsible for developing the tool stated that the Secretary of Defense approved this approach for estimating and publishing the costs of reports and studies and emphasized that the Secretary's intent was to develop an easy to use, nonburdensome approach for estimating costs. As a result, the tool was not designed with the level of precision of a more formal cost estimating process. DOD's Approach for Estimating Costs Is Not Fully Consistent with Relevant Best Practices and Cost Accounting Standards: We evaluated DOD's approach for estimating costs against some of the best practices contained in GAO's Cost Estimating and Assessment Guide, which states that high-quality and reliable cost estimates should be comprehensive, well documented, and accurate. In addition, we compared DOD's approach with Statement of Federal Financial Accounting Standards No. 4 and relevant portions of DOD's Financial Management Regulation, which state that appropriate procedures and practices should be established to enable, among other things, the interpretation and communication of cost information. We found that DOD's approach in using the tool to estimate costs is not fully consistent with relevant best practices and cost accounting standards, as discussed below. Comprehensive: While GAO's Cost Estimating and Assessment Guide states that cost estimates should include all costs, it allows flexibility for the estimates to exclude costs where information is limited.[Footnote 10] In such cases, the decisions on what costs are to be included or excluded should be clearly defined and documented in the ground rules and assumptions used to develop the cost estimate.[Footnote 11] Consistent with this guide, DOD's guidance on using the tool defines and documents decisions to include or exclude certain manpower costs and to include all nonlabor costs. For example, it identifies specific manpower costs that are excluded from the calculation, such as costs associated with health care and training. Additionally, the guidance provides broad examples of some types of activities to consider in estimating costs, but states that individuals should use their best judgment in determining what costs to include. In calculating manpower costs, the guidance indicates that if individuals developing a report or study collected data or took other actions that they would have done even if the report or study was not required, the costs associated with these activities should not be included in the cost estimate. For example, the guidance states activities that are part of an individual's normal job activities--such as attending meetings; coordinating reports, studies, or proposals; and completing security reviews of documents--should not be included in the manpower calculation. On the other hand, if a study or report required activities that would not have otherwise been undertaken, costs associated with these activities should be included in the estimate of manpower costs. The guidance also provides examples of nonlabor activities individuals could consider in developing the cost estimate, such as costs associated with contractors, printing, software or hardware purchases, lease or rental fees for space, and travel. Ultimately, the guidance leaves it up to the individuals generating the cost estimate to decide which types of activities to include or exclude in the cost estimate. As a result, in our review of nine reports that DOD prepared for Congress, we found inconsistencies in the types of activities individuals decided to include when developing their cost estimates (see enclosure IV for a description of these reports, which ranged in estimated costs from about $3,000 to over $2.3 million). For example, for six of the nine reports we reviewed, we found that individuals included in their manpower calculations the time they spent coordinating the report for review and approval and the time that their supervisors spent finalizing it. For the other reports we reviewed, costs for this activity were not included. One of the officials that decided to include time spent on coordinating the report in the cost estimate stated that DOD's guidance was unclear as to whether coordination was limited to the review and approval of a report. Therefore, given the significant amount time he spent coordinating the development of the report (e.g., time spent clarifying guidance or coordinating inputs), he decided to include those costs in the cost estimate. We also found instances where costs for nonlabor activities that DOD's guidance states individuals could consider in developing estimates were not included even though costs were incurred. Specifically, we found that for four of the reports we reviewed, officials told us that costs had been incurred for printing but were not included in the cost estimate. They stated that they did not include them because they believed their printing costs were so small that the level of effort needed to calculate those costs outweighed the value of including them in the estimate. In some cases, officials were not able to readily retrieve documentation; therefore, we were unable to fully evaluate the extent to which they consistently applied DOD's guidance for estimating costs. In addition, according to DOD's Financial Management Regulation and incorporated portions of Statement of Federal Financial Accounting Standards No. 4, appropriate procedures and practices should be established to enable, among other things, the interpretation and communication of cost information. While DOD has developed boilerplate language for use when presenting the cost estimate on the front cover of a report or study, the language does not provide information on what costs are included or excluded. Specifically, it does not indicate that the estimate reflects only certain marginal manpower and nonlabor costs. Rather, DOD includes a general statement that says "Preparation of this report/study cost the Department of Defense a total of approximately $_______ in Fiscal Year _____." Without further explanation, this wording could be subject to misinterpretation such that recipients of the reports or studies may assume that all costs are included in the estimate when that is not the case. Well Documented: GAO's Cost Estimating and Assessment Guide states that high-quality, reliable cost estimates should be easily traceable to source documents. While DOD's guidance on using the tool states that individuals should be prepared to explain how they developed a cost estimate, it does not include a requirement for individuals to retain source documentation. According to DOD officials responsible for the development of the tool, the tool is not a database and therefore was never intended to be used to capture references to supporting documents. When talking with officials responsible for developing cost estimates, we found that for three of the nine reports, they were able to easily retrieve the documentation they used to generate the cost estimate. For example, some individuals were able to provide us with e- mails containing data on the amount of time individuals spent working on the report or study. Officials responsible for three of the other six reports stated that they could not easily provide the information because they had moved on to other positions and no longer had access to their records. The remaining officials stated that with time, they might be able to find and provide the source documentation. Accurate: GAO's Cost Estimating and Assessment Guide further states that high- quality, reliable cost estimates should have all cost inputs independently checked to verify that they are accurate. The guidance on using the tool does not include any process or requirement for another individual to independently verify the manpower and nonlabor costs used to generate the cost estimate. In the absence of any guidance, CAPE and service officials responsible for administering the tool stated that they had not been assigned responsibility for validating or vetting inputs used to develop the cost estimate. According to these same officials, the individual generating the cost estimate is responsible for ensuring its accuracy. In some cases, individuals who were responsible for generating cost estimates for the nine reports we reviewed stated that they included the information used to develop the cost estimates when they submitted the reports to their supervisors for approval. In the absence of clearly defined parameters as to which activities should be covered, full disclosure of the scope of the estimates, the availability of source documentation, and a means to independently verify the estimates, DOD does not have the information it needs to readily replicate the estimates and assure itself that the estimates are consistent, accurate, and reliable. DOD Does Not Currently Have a Means to Ensure That Required Types of Reports and Studies Have Cost Estimates, but Has Efforts Under Way to Improve Its Visibility: DOD currently lacks a means to ensure that organizations are developing cost estimates for all required types of reports and studies, but is taking steps to enhance its ability to monitor the preparation of reports and studies to satisfy reporting requirements, including those for which a cost estimate is required to be generated. DOD's guidance on using the tool identified 10 specific types of reports or studies that require a cost estimate. The guidance also states that the DOD office preparing the report or study is responsible for ensuring that a cost estimate, if required, is included. However, the guidance does not include any process or requirement to track whether organizations are developing required cost estimates. According to CAPE officials who developed the tool, they were not directed to ensure that cost estimates have been developed for all required reports and studies. They added that the cost estimating tool was not designed to track reports and studies that require a cost estimate. For example, officials stated that while the tool stores information each time a user generates a cost estimate, they cannot distinguish between real estimates and those that were developed while individuals were using the tool for training purposes. Therefore, when we requested a list of all the reports and studies for which individuals had prepared cost estimates, they were unable to provide us such a list. CAPE officials also noted that the magnitude of DOD's reporting requirements makes it challenging to identify the universe of these requirements and to track the completion of reports and studies to meet them, including whether cost estimates have been generated. In the past year, DOD has taken steps to improve the department's visibility over its internal and external reporting requirements. For example, in March 2011, the Secretary of Defense issued a memorandum requiring the Assistant Secretary of Defense for Legislative Affairs to enhance that office's database capability and further automate the process of tracking external reports to Congress, and required the Office of the Director of Administration and Management to track internal reports.[Footnote 12] The Office of Legislative Affairs enhanced its Congressional Hearings and Reporting Requirements Tracking System[Footnote 13] by adding a data field to record the estimated costs of reports or studies generated by DOD components. To gain greater visibility of internal reporting requirements, the Office of the Director of Administration and Management tasked Washington Headquarters Services with developing a repository for tracking internal reports. According to a Washington Headquarters Services official, the repository has a field to capture whether an internal report or study has a cost estimate. Currently, the repository is in development and has been populated with reports that date back 3 years. However, officials stated that this only captures reporting requirements involving multiple DOD components and does not reflect all of DOD's internal reporting requirements. The official noted that Washington Headquarters Services is planning to send out a data call to all of DOD in an effort to capture the universe of internal reporting requirements. DOD Lacks a Plan to Evaluate the Impact of Its Efforts to Estimate Costs: DOD has not evaluated and currently does not plan to evaluate the usefulness of its efforts to estimate the costs of selected reports and studies as one of the means for achieving the Secretary of Defense's intended purpose of increasing the transparency of costs, reducing or eliminating reporting requirements, and instilling a culture of cost consciousness across DOD. According to GAO's Standards for Internal Control in the Federal Government, activities such as assessments need to be established to monitor performance and managers need to compare actual performance against targets and then analyze any significant differences so that appropriate action can be taken to ensure that organizational goals are met. However, under current guidance, no requirement exists for DOD to evaluate the impact of its cost estimating efforts, such as whether it has prompted internal or external decision makers to consider cost as a factor in deciding whether to establish a reporting requirement for DOD. In addition, CAPE officials responsible for developing and administering the tool stated they have not been otherwise directed to take steps nor have they taken steps to conduct any such evaluation, including reaching out to internal and external decision makers to obtain their views. CAPE officials stated that anecdotally they have heard some officials remark that they believe the tool has increased the transparency of costs and improved cost awareness. However, most of the DOD officials who generated cost estimates for the nine reports we reviewed stated that they had not sought or received any feedback from internal or external decision makers about the cost estimates published on the front covers of the reports or studies. Without evaluating the impact of its efforts, including seeking the views of internal and external decision makers, DOD does not have the information it needs to assess whether the time and effort it is investing to develop cost estimates is having the desired effect of achieving greater transparency, reducing reporting requirements, and raising cost consciousness, as intended by the Secretary. Conclusions: When the Secretary of Defense directed DOD to track the cost of preparing reports and studies, his intention was to generate an approximate cost that could raise awareness of the costs associated with the level of effort DOD expends in generating reports or studies. In response to the Secretary's directions, officials from the CAPE office, in conjunction with officials from other DOD offices, expended time and effort to develop an approach to estimate the cost of reports and studies. DOD components are also expending time and effort to use the tool to develop cost estimates for selected reports and studies. Because of this investment, it is important that DOD officials have an understanding of the impact publishing a cost estimate is having on internal and external decision makers. Further, there are actions that DOD can take to improve its approach so that it better aligns with key characteristics of cost estimating best practices--actions that would help DOD ensure that it has the information it needs to readily replicate the estimate and assure itself and the recipients of the reports or studies that the estimates are clearly defined and consistent, accurate, and reliable. Without first evaluating the impact its cost estimating approach is having, DOD officials will not be positioned to determine whether the resources invested in generating and publishing the approximate costs are having the desired effect or whether they should take steps to improve their cost estimating approach. Recommendations for Executive Action: To determine whether its effort to estimate costs is having the desired effect of achieving greater transparency, reducing or eliminating reporting requirements, and raising cost awareness, we recommend that the Secretary of Defense take steps to evaluate the effort, including reaching out to internal and external decision makers to obtain their views on how it has affected their decisions to establish internal and external reporting requirements and whether they have a common understanding of what types of costs are being reported. Based on the results of this evaluation, if DOD plans to continue the effort to estimate costs for selected reports and studies, we recommend that the Secretary modify the current guidance or otherwise take steps to improve DOD's cost estimating approach, including: * clarifying those activities that should be included in the estimate as they are incurred, including time spent at every level to review and approve the report or study; * requiring that the language on the front cover of reports and studies make clear that the estimate is based on certain marginal manpower and nonlabor costs; * requiring that source documentation used to develop the cost estimate is retained and easily accessible for review purposes; and: * establishing and implementing a verification process to provide reasonable assurance of consistency and completeness of cost inputs used to develop the cost estimate. Agency Comments and Our Evaluation: DOD provided written comments on a draft of this report. The full text of DOD's written comments is reprinted in enclosure V. The department also offered technical comments, which we incorporated as appropriate. Overall, DOD stated that it largely agreed with GAO's recommendation to evaluate the effectiveness of estimating and publishing the costs of DOD studies and reports and that the department remains committed to improving the existing cost estimating guidance, methods, and tools associated with this policy. However, DOD expressed concern that we had inaccurately characterized the objectives of its cost estimating policy. For example, DOD believed that the title of our review-- Department of Defense's (DOD) Reporting Cost Data Collection-- suggested that the department was collecting cost data on reports. DOD also stated that our draft report inaccurately stated that an objective of the current policy is to reduce or eliminate reporting requirements. According to DOD, the department's effort to reduce or eliminate reporting requirements is a separate endeavor entailing the formal submission of legislation to Congress, and is inappropriately linked in our draft report to the initiative to estimate and publish the costs of DOD studies and reports. We disagree with DOD's view that we have inappropriately linked the initiative to estimate and publish costs to the objective of eliminating and reducing reporting requirements. We recognize that DOD has undertaken a specific effort to review all reports stemming from both external and internal reporting requirements, and have clarified our report accordingly. However, in our review of various DOD documents, including prepared remarks made by the Secretary and a press release from his office, it is clear that the review cited by DOD was one of several actions, which included the effort to estimate and publish costs, that were intended to address the volume of reporting requirements. We, therefore, modified the report to provide additional contextual information from these sources to amplify the Secretary's intent. In the introductory paragraphs of the report, we now include information contained in the Secretary of Defense's August 9, 2010, speech announcing the efficiency initiative as well as the accompanying press release further describing his remarks. Specifically, we note the following: In his August 9, 2010 speech announcing the overall efficiency initiative, the Secretary of Defense stated that the department is "awash in taskings for reports and studies" and directed several specific actions that according to the press release accompanying the announcement, were intended to "combat the enormous amounts of taskings for reports and studies."[Footnote 14] In his remarks, the Secretary noted that there is little basis to determine whether the value gained is worth the considerable time and resources expended to generate reports and studies. With respect to specific actions, the Secretary directed DOD and its components to track the approximate cost of preparing DOD reports and studies and publish the cost on the front cover of each report or study. He also called for a comprehensive review of all oversight reports and use of the results to reduce the volume of reports and studies generated internally while engaging Congress on ways to meet needs while working together to reduce the number of reports. With regard to the title of our review, this title was used for administrative purposes, including its use in our letter to notify DOD of our work. We note that in several places throughout the draft report, we describe DOD's effort as a cost estimating approach rather than an effort to collect cost data. With regard to our recommendation that DOD evaluate its effort to estimate the costs of reports and studies, including reaching out to internal and external decision makers to obtain their views on how it has affected their decisions to establish internal and external reporting requirements and whether they have a common understanding of what types of costs are being reported, DOD partially concurred. DOD agreed that there could be value gained in evaluating its efforts but stated that any such evaluation must avoid linking the efforts of this policy to objectives that are unrelated to its original intent. Specifically, DOD stated that its policy was not designed to reduce or eliminate reporting requirements, which it characterized as a separate effort. In addition, DOD stated that its policy was not designed to apply cost estimating best practices from GAO's Cost Estimating and Assessment Guide. It noted that DOD's guidance and methodology for estimating costs differ from criteria used to estimate costs for other DOD decision making. It further stated that DOD would not support a change to the rigor associated with estimating the costs of DOD studies and reports unless it is guaranteed that the marginal value gained by increasing the level of accuracy outweighs the additional efforts required to produce more accurate cost estimates. With regard to engaging with internal and external decision makers to better understand the effectiveness of the current cost estimating policy, DOD stated that this process had already begun. Specifically, DOD noted that members of the department have communicated internally with DOD employees and externally with non-DOD employees to understand ways in which the value gained by the policy could be improved. In its comments, DOD stated that it had received evidence that the value gained by the policy is significant and cited specific examples to illustrate its point. For example, DOD noted that contact had occurred between the Chairman of the House Armed Services Committee and the Secretary of Defense regarding the costs associated with a particular DOD report, which led to agreement that the report's quality required improvement and could be produced at lower costs. In addition, DOD cited actions taken by DOD decision makers to expand the scope of the policy to include additional types of reports or to establish mechanisms to ensure that reports to Congress are not transmitted without a cost estimate. DOD also noted that DOD decision makers have contacted CAPE staff to commend the Secretary's effort to raise cost awareness and improve transparency through the use of the cost estimating guidance, methods, and tools. Finally, DOD stated that it will continue to monitor the effectiveness of the current policy by engaging with DOD employees, senior-level DOD staff, and individuals outside of DOD. We are encouraged that the department agrees that there could be value gained in evaluating efforts to raise cost awareness and improve transparency of costs by estimating and publishing costs of the DOD studies and reports. However, we disagree with DOD's characterization of the underlying intent of the evaluation that we recommended. As discussed previously, the Secretary included the effort to track and publish costs for reports and studies among efforts intended to address the volume of taskings for reports and studies. Therefore, we believe that in any evaluation of the effort, it would be essential to specifically obtain the views of decision makers on the benefits of estimating report costs, including how such efforts affected their decision making on whether to call for future reports or studies. As stated in our report and as reflected in the examples that DOD described in its response, we recognize that DOD has obtained anecdotal feedback from internal and external sources regarding the cost estimating effort. Given the time and effort that DOD components are expending to use the tool to develop cost estimates, we continue to believe that DOD needs to develop an evaluation approach that will give it a more complete picture of how decision makers have been influenced by the cost estimating policy. Without such information, it will continue to be difficult for DOD to assess the value of the policy, including whether the time and effort it is expending to estimate costs is a worthwhile investment. We further note that our recommendation that DOD evaluate its cost estimating effort did not suggest that such an evaluation would assess whether DOD should apply cost estimating best practices. Rather, we recommended that DOD first evaluate whether its effort is achieving its desired effect. Based on the results of this evaluation, we recommended that if DOD decided to continue the effort, it should modify existing guidance or otherwise take steps to improve its cost estimating approach so that it better aligns with key characteristics of cost estimating best practices. We note that DOD specifically states in its written comments that the department intends to continue the current policy. Therefore, as discussed below in our response to DOD's specific comments regarding that part of our recommendations, we continue to believe that opportunities exist for DOD to improve its cost estimating approach to make it more consistent with relevant best practices. With regard to our recommendations that DOD modify the current guidance or otherwise take steps to improve DOD's cost estimating approach, DOD partially concurred with three of our proposed recommendations and did not concur with the fourth one. * DOD partially concurred with our recommendation to clarify those activities that should be included in the estimate as they are incurred, including time spent at every level to review and approve the report or study. DOD stated that providing clarity to the overall guidance will continue to be a priority, adding that the direction to exclude coordination efforts in the cost estimate is explicitly documented in guidance. As we noted in the draft report, we recognize that DOD's guidance states that manpower costs should not be included when participation only involves normal job activities, such as attending meetings and coordinating studies, reports, or proposals. However, we found that some of the individuals who generated cost estimates found the guidance confusing with regard to whether to include or exclude costs associated with coordination activities that are beyond an individual's normal job activity. As a result, we found inconsistencies in how these individuals decided to account for this activity when generating their cost estimates. * DOD partially concurred with our recommendation requiring that source documentation used to develop the cost estimate is retained and easily accessible for review purposes. DOD stated that its Cost Guidance Portal, which includes the tool, has several features to assist users in obtaining source documentation of their cost estimates, including an automatic e-mail with a detailed summary report of the cost estimate and a hyperlink to each cost estimate. We recognize that these features enable DOD to retrieve the information that was input into the cost estimating tool. However, it does not include a reference to the actual source documents. As we discuss in the report, some individuals responsible for generating the cost estimates that we reviewed had difficulty retrieving the source documentation that they used to develop the estimates. We continue to believe that without specifically requiring individuals to retain source documentation, DOD may not have the information needed so that others can understand how a cost estimate was derived and readily replicate it. * DOD did not concur with our recommendation to establish and implement a verification process to provide reasonable assurance of consistency and completeness of cost inputs used to develop the cost estimate. The department stated that it currently has the ability to pull detailed cost estimates for specific studies and reports, when necessary, and to contact each DOD employee who prepares a cost estimate, which DOD believes provides a reasonable level of accountability. DOD also stated that implementing a verification process similar to the one proposed by GAO would be a costly endeavor. For example, verifying the amount of time spent by each DOD employee preparing a study or report would require the department to reduce time spent on other, higher-priority missions. Rather than implementing a costly verification process for review of cost estimates after they have been submitted, the department will continue to train users prior to their preparation and submission of cost estimates. We agree that the department can pull information for a specific report or study and contact DOD employees regarding estimates. However, we disagree that DOD's ability to take these steps constitutes the same type of accountability that would be present if DOD had a verification process in place. We are not suggesting that such a process would require DOD to verify the time spent by each employee to prepare a report or study. Rather, we would expect that any verification effort would include steps to review whether the estimate was developed in a manner consistent with DOD's guidance. Such a process would provide a greater assurance of consistency and completeness of the cost inputs used to develop the cost estimate. In addition, while we agree that training is important, we note that training is not currently required before an individual is able to use the tool, and individuals may therefore continue to develop cost estimates without the benefit of any training. * DOD partially concurred with our recommendation requiring that the language on the front cover of reports and studies make clear that the estimate is based on certain marginal manpower and nonlabor costs. DOD stated that while the department is open to modifying the phrase that is published on the front cover of reports or studies, the objective should not be to detail the specific costs included or excluded on the front cover and that the cost on the front cover should always represent the marginal costs borne by DOD in preparing the study or report. We agree that DOD does not need to detail all the specific costs included or excluded in the cost estimate on the front cover. Rather, DOD should modify the language on the front cover of reports and studies to make it clear that the estimate is based on certain marginal manpower and nonlabor costs. We are sending copies of this report to appropriate congressional committees and the Secretary of Defense. In addition, this report is available at no charge on the GAO website at [hyperlink, http://www.gao.gov]. If you or your staff have any questions about this report, please contact me at (202) 512-9619 or pickups@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in enclosure VI. Signed by: Sharon L. Pickup: Director: Defense Capabilities and Management: Enclosures - 6: [End of section] Enclosure I: Description of the Four Tracks of the Secretary of Defense's Efficiency Initiative: In May 2010, the Secretary of Defense directed the Department of Defense (DOD) to undertake a departmentwide initiative to assess how the department is staffed, organized, and operated with the goal of reducing excess overhead costs and reinvesting these savings into sustaining DOD's current force structure and modernizing its weapons portfolio. The Secretary's initiative targeted both shorter-and longer- term improvements and set specific goals and targets for achieving cost savings and efficiencies. The Secretary's initiative was organized along four tracks--each of which had a different focus, as shown in table 1.[Footnote 15] Table 1: Description of the Four Tracks of the Secretary of Defense's Efficiency Initiative: Track: Track 1; Purpose: Military services and other components: Tasked to find savings of about $100 billion in overhead costs that are to be reinvested over 5 years, starting with DOD's fiscal year 2012 budget. Track: Track 2; Purpose: External and internal suggestions: Sought ideas, suggestions, and proposals for efficiencies from outside normal, official DOD channels (e.g., "think tanks," industry, external boards, and DOD employees). Track: Track 3; Purpose: Departmentwide review of certain areas: Assessed personnel and health care policies, logistics, and acquisition to identify efficiencies. Track: Track 4; Purpose: Specific issues identified by the Secretary: Focused on specific areas where DOD could take immediate action to reduce inefficiencies and overhead, in particular, to reduce headquarters and support bureaucracies and to instill a culture of cost consciousness and restraint in the department. Source: DOD. Note: Other components include various offices within the Office of the Secretary of Defense, the combatant commands, DOD agencies, and DOD field activities. [End of table] [End of section] Enclosure II: Scope and Methodology: Our objectives were to evaluate DOD's efforts to implement an approach to estimating the costs of selected reports and studies, including (1) whether DOD entities are capturing and presenting costs in a manner consistent with relevant cost estimating best practices; (2) the status of DOD's efforts to track whether organizations are developing cost estimates for all required types of reports and studies; and (3) whether DOD has evaluated the usefulness of its efforts to estimate the cost of reports and studies as one of the means for achieving the Secretary's intended purpose of increasing the transparency of costs, reducing or eliminating reporting requirements, and instilling a culture of cost consciousness across DOD. To determine whether DOD entities are capturing and presenting costs in a manner consistent with relevant cost estimating best practices, we reviewed the cost estimating tool used by DOD entities to generate cost estimates and compared it to several best practices recognized in GAO's Cost Estimating and Assessment Guide and relevant managerial cost accounting guidance--Statement of Federal Financial Accounting Standards No. 4 and relevant sections of DOD's Financial Management Regulation. GAO's Cost Estimating and Assessment Guide reflects best practices for developing and managing cost estimates for capital programs. Because DOD's cost estimating tool covers noncapital costs, we assessed the tool against the best practices in the guide that we identified as most applicable to these types of cost. We compared the tool to three characteristics--comprehensive, well documented, and accurate--in the guide to determine if the estimate generated by the tool is of high quality and reliable. We scored each best practice as not met (the tool provided no evidence that satisfies any of the criterion), minimally met (the tool provided evidence that satisfies a small portion of the criterion), partially met (the tool provided evidence that satisfies about half of the criterion), substantially met (the tool provided evidence that satisfies a large portion of the criterion), and met (the tool provided complete evidence that satisfies the entire criterion). We determined the overall assessment by assigning an individual rating to the best practice: not met = 1, minimally met = 2, partially met = 3, substantially met = 4, and met = 5. To perform this analysis, we reviewed documents provided by the Cost Assessment and Program Evaluation (CAPE) office, including guidance on using the tool and the tool itself, and conducted interviews with DOD officials to corroborate our understanding of the tool and its corresponding guidance. In addition, we compared the cost estimating tool with Statement of Federal Financial Accounting Standards No. 4 and relevant sections of DOD's Financial Management Regulation to determine the extent to which the tool reflects relevant cost accounting standards and cost management guidance. All data elements we assessed are sufficiently reliable for the purpose of assessing the reliability of DOD's cost estimating tool. Further, we had originally intended to review a representative sample of DOD reports and studies with cost estimates published on the front cover. However, DOD could not readily provide us with a comprehensive list of all reports and studies that included cost estimates from which we could select reports and studies to review because DOD could not distinguish between real cost estimates and estimates that were developed while individuals were using the tool for training purposes. DOD was able to provide us information on nine reports prepared for Congress on which Congress had requested more information, and identify the individuals responsible for generating the cost estimates for these reports. Therefore, we reviewed these nine reports. Using semistructured interviews, we held discussions with the key officials responsible for developing the cost estimates for these reports to gain an understanding of their experience with using the tool and corresponding guidance and their level of confidence in the cost estimates generated, among other things. In addition, we reviewed supporting documentation for some of the estimates as well as the snapshot of the data that was entered into the tool. In some cases, officials were not able to readily retrieve documentation; therefore, we were unable to fully evaluate the extent to which they consistently applied DOD's guidance for estimating costs. We were unable to assess the reliability of the data used to generate the cost estimates that were included on these nine reports, or to generalize these findings to all reports, but we feel that the review of these nine reports is sufficient for providing context about the process, and we are making a recommendation about future data collection efforts. To describe DOD's efforts to determine whether organizations are developing cost estimates for all required types of reports and studies, we reviewed guidance and interviewed DOD officials to assess the nature of their tracking efforts. Specifically, we conducted interviews, gathered information and documents regarding internal and external report tracking mechanisms from officials in the CAPE office; the Office of the Assistant Secretary of Defense for Legislative Affairs; the Office of the Director of Administration and Management; Washington Headquarters Services; and the legislative affairs offices of the Departments of the Army, Navy, and Air Force. To determine whether DOD has evaluated the usefulness of its efforts to estimate the cost of reports and studies as one of the means for achieving the Secretary's intended purpose of increasing the transparency of costs, reducing or eliminating reporting requirements, and instilling a culture of cost consciousness across DOD, we reviewed relevant DOD guidance and GAO's Standards for Internal Control in the Federal Government. We also interviewed DOD officials responsible for administering and using the tool, including the individuals who generated the nine reports for Congress, to determine if they had taken steps to obtain feedback from internal and external decision makers regarding the impact of estimating costs and publishing the cost estimate on the front covers of the reports and studies. We conducted this performance audit between October 2011 and May 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Enclosure III: Snapshot of Fields in DOD's Cost Estimating Tool: The tool designed by DOD consists of fields with drop-down menus to allow users to choose the category of costs they wish to input, among other types of information. Additionally, the tool contains information such as the definition of what constitutes a report or study and a real-time calculator. (See figure 2.) Figure 2: Fields in DOD's Cost Estimating Tool: [Refer to PDF for image: illustration] Report/Study Cost estimate (7-202104 PIN 29209): Thursday, 22 September 2011: Purpose: The cost estimate must be prepared as reports and studies are readied for publication. The purpose of the estimate is to improve the transparency of costs associated with reports and studies sponsored or prepared by DoD. All reports and studies sponsored or performed by DoD, including those for special access programs and military intelligence programs, must provide a cost estimate. ODNI will provide separate guidance for NIP programs. All figures should reflect actual costs to the greatest degree possible. Depicted fields are as follows: 1. Report/Study Title: 2. Expected Publication Date: 3. Report/Study Number: 4. This Report/Study was funded and performed during Fiscal Years: 5. Name of Organization(s) Sponsoring the Report/Study: 6. Name of Organization Preparing the Cost Estimate: 7. The report/study is required to comply with DOD regulation: 7b. Enter the DoD regulation(s) required for compliance: 8. The report/study is required to comply with Congressional direction: 8b. Enter the Congressional Report Number: 9. This report/study is required by law: 9b. Enter the Statutory law(s): 10. List costs of all non-labor activities used to prepare and complete the report/study (i.e., contracts, printing, travel, etc.): 11. Total Government Manpower Required for Oversight of Report/Study Execution: 12. Will the cost estimate data be required on a recurring basis? 12b. Enter the frequency of this recurring Report/Study estimate (i.e., biennial, annual, quarterly, etc.): Source: DOD. [End of figure] Enclosure IV: Description of Nine Reports Prepared for Congress That GAO Reviewed: We reviewed information on nine reports DOD prepared for Congress for which cost estimates had been generated in 2011 and interviewed the key officials responsible for developing the cost estimates. The reports we reviewed were prepared by organizations within the Office of the Secretary of Defense, the Navy, and the Air Force. Four of the cost estimates consisted solely of manpower costs, and the other five consisted of manpower and nonlabor costs. (See table 2.) Table 2: Description of Nine DOD Reports Prepared for Congress That GAO Reviewed: Organization that prepared the report: Navy; Report: Integration of Current and Future Unmanned Payloads into Submarines; Date: January 12, 2011; Cost estimate: Manpower: $3,000; Nonlabor: $0; Total: $3,000. Organization that prepared the report: Office of the Under Secretary of Defense for Acquisition, Technology and Logistics; Report: Panel on Contracting Integrity 2010 Report to Congress; Date: January 28, 2011; Cost estimate: Manpower: $18,000; Nonlabor: $0; Total: $18,000. Organization that prepared the report: Navy; Report: Report to Congress on Repair of Naval Vessels in Foreign Shipyards; Date: February 8, 2011; Cost estimate: Manpower: $11,000; Nonlabor: $0; Total: $11,000. Organization that prepared the report: Office of the Under Secretary of Defense for Policy; Report: Report on the Joint Use of Federal Forest Fire Fighting Assets/C-130 Fire Fighting Capabilities; Date: February 9, 2011; Cost estimate: Manpower: $51,000; Nonlabor: $120,000; Total: $171,000. Organization that prepared the report: Office of the Secretary of Defense, Cost Assessment and Program Evaluation; Report: A Review of the Army's Modular Force Structure; Date: February 16, 2011; Cost estimate: Manpower: $149,000; Nonlabor: $460,000; Total: $609,000. Organization that prepared the report: Air Force; Report: Expeditionary Combat Support System (ECSS) Increment 1 Major Automated Information System (MAIS) Critical Change Report; Date: February 18, 2011; Cost estimate: Manpower: $1,232,000 Nonlabor: $1,146,000 Total: $2,378,000. Organization that prepared the report: Navy; Report: Strategic Plan for Homeporting the Littoral Combat Ship; Date: February 24, 2011; Cost estimate: Manpower: $11,000; Nonlabor: $0; Total: $11,000. Organization that prepared the report: Office of the Under Secretary of Defense for Personnel and Readiness; Report: Department of Defense Foreign Language Proficiency Fiscal Year 2010 Report to Congress; Date: March 2011; Cost estimate: Manpower: $78,000; Nonlabor: $23,000; Total: $101,000. Organization that prepared the report: Navy; Report: Littoral Combat Ship (LCS) Monthly Cost Report to Congress; Date: March 24, 2011; Cost estimate: Manpower: $2,000; Nonlabor: $1,000; Total: $3,000. Source: DOD. [End of table] [End of section] Enclosure V: Comments from the Department of Defense: Office of The Secretary Of Defense: Cost Assessment and Program Evaluation: 1800 Defense Pentagon: Washington, D.C. 20301-1800: April 19, 2012: Ms. Sharon Pickup: Director, Defense Capabilities and Management: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Ms. Pickup: This is the Department of Defense (DoD) response to the GAO Draft Report, GAO-12480R, 'Defense Management: Actions Needed to Evaluate the Impact of Efforts to Estimate Costs of Reports and Studies,' dated March 21, 2012 (GAO Code 351661)." The Department appreciates the opportunity to review and comment on this draft report. The Department largely agrees with GAO's recommendation to evaluate the effectiveness of estimating and publishing the costs of DoD studies and reports, and the Department remains committed to improving the existing cost estimating guidance, methods, and tools associated with this policy. As noted in the report, the Department issued a plan in August 2010 to instill a culture of cost-consciousness, in part by estimating the costs of DoD studies and reports and publishing the costs on the cover page of the study or report. The objectives of this policy were to raise cost awareness and transparency of costs throughout the Department. To implement this plan, the Department established the Cost Guidance Group, which consists of representatives from the military departments and other DoD Components. Within weeks of assignment, this group successfully developed cost estimating guidance, methods, and web- based tools to assist DoD employees in estimating costs of studies and reports. The Department directed all DoD employees to use the guidance, methods, and tools beginning in February 2011, and the Cost Guidance Group has continually made iterative improvements to maximize the effectiveness of this policy. In September 2011, the Department learned that the GAO announced a new review titled "Department of Defense's (DoD) Reporting Cost Data Collection." The Department welcomed this outside review of its new policy to estimate costs of studies and reports, but was concerned that the objectives of the policy were misinterpreted since the review's title suggested that the Department was collecting cost data on reports, which is inaccurate. Today, the Department remains concerned because the Draft GAO Report inaccurately states that an objective of the current policy is to "reduce or eliminate reporting requirements." The Department's effort to reduce or eliminate reporting requirements is a separate endeavor entailing the formal submission of legislation to the Congress, and is inappropriately linked in the Draft GAO Report to the initiative to estimate and publish costs of DoD studies and reports. The Department's response to GAO's recommendation is attached. Should you have any questions, please contact my primary action officer, Dr. Bryan Shone, at 703-692-8045 or Bryan.Shone@osd.mil. Sincerely, Signed by: Christine H. Fox: Director: Enclosure: [End of letter] GAO Draft Report Dated March 21, 2012: GAO-12-480R (GAO Code 351661): "Defense Management: Actions Needed To Evaluate The Impact Of Efforts To Estimate Costs Of Reports And Studies" Department Of Defense Comments To The GAO Recommendations: Recommendation 1: To determine whether its effort to estimate costs is having the desired effect of achieving greater transparency, reducing or eliminating reporting requirements and raising cost awareness, GAO recommends that the Secretary of Defense take steps to evaluate the effort, including reaching out to internal and external decision makers to obtain their views on how it has affected their decisions to establishing internal and external reporting requirements and whether they have a common understanding of what types of costs are being reported. Based on the results of this evaluation, if DoD plans to continue the effort to estimate costs for selected reports and studies, we recommend that the Secretary modify the current guidance or otherwise take steps to improve DoD's cost estimating approach, including to: * clarify those activities that should be included in the estimate as they are incurred, including time spent at every level to review and approve the report or study; * require that source documentation used to develop the cost estimate is retained and easily accessible for review purposes; * establish and implement a verification process to provide reasonable assurance of consistency and completeness of cost inputs used to develop the cost estimate; and, * require that the language on the front cover of reports and studies make clear that the estimate is based on certain marginal manpower and non-labor costs. DoD Response: The Department of Defense (DoD) partially concurs with the recommendation set forth by the GAO. The Department agrees that there could be value gained in evaluating efforts to raise cost awareness and improve transparency of costs by estimating and publishing costs of DoD studies and reports. However, any such evaluation must avoid linking the efforts of this policy to objectives that are unrelated to its original intent. Specifically, this policy was not designed to: * Reduce or eliminate reporting requirements. Reducing or eliminating reporting requirements is a separate DoD effort that is not associated with the policy to include a cost estimate on the front cover of DoD studies and reports. * Apply cost estimating best practices from the "GAO's Cost Estimating and Assessment Guide" as cited in the Draft Report. The Department's guidance and methodologies for estimating costs for this application differ from criteria used to estimate costs for other DoD decision- making. The Department would not support a change to the rigor associated with estimating costs of DoD studies and reports unless it is guaranteed that the marginal value gained by increasing the level of accuracy of outweighs the additional efforts required to produce more accurate cost estimates. The Department partially concurs with GAO's recommendation to engage with decision makers (internal and external to the DoD) to better understand the effectiveness of the current policy. This process has already begun. Since the Secretary of Defense implemented this policy in December 2010, members of the Department have communicated internally with DoD employees (both employees who have used the tools to generate cost estimates and senior-level decision makers who have benefited from developments in cost awareness and transparency) as well as externally with non-DoD employees to understand ways in which the value gained by the policy could be improved. The Department has received evidence that the value gained by this policy is significant: * The Chairman of the Armed Services Committee contacted the Secretary of Defense to raise concern over costs associated with a report submitted by the DoD to Congress that was considered unjustified when compared to the value of the report. This initiated a productive dialogue between senior-level DoD decision makers and Congress, with agreement that the report's quality required improvement, and could be produced at lower cost to the Department. * Senior-level DoD decision makers have expanded the scope of the policy beyond congressional and internal DoD reports and studies, mandating that cost estimates be submitted with all studies and reports prepared within their Component (in addition to those mandated by the Secretary of Defense). * Senior-level DoD decision makers have established enforcement mechanisms to uphold the Secretary's direction. For example, the Assistant Secretary of Defense for Legislative Affairs directed her staff to verify that each draft report published on or after February 1, 2011 include a cost estimate before clearing the report for transmittal to the Congress. Similarly, staff members in the immediate Office of the Secretary of Defense (OSD) do not accept studies or reports unless there is a cost estimate included on the front cover. * Senior-level DoD decision makers have personally contacted staff from the OSD' s Cost Assessment and Program Evaluation (CAPE) office commending the Secretary's efforts to raise cost awareness and improve transparency through use of the cost estimating guidance, methods, and tools provided on the Cost Guidance Portal. The Department will continue to monitor the effectiveness of the current policy by engaging with DoD employees, senior-level DoD staff, and individuals outside of the DoD. With respect to the specific recommendations offered by the GAO to modify the current guidance, the Department: * Partially concurs with the recommendation to "clarify those activities that should be included in the estimate as they are incurred, including time spent at every level to review and approve the report or study." Providing clarity to the overall guidance will continue be a priority for the Department. The direction to exclude coordination efforts in the cost estimate is explicitly documented in the guidance.[Footnote 1] * Partially concurs with the recommendation to "require that source documentation used to develop the cost estimate is retained and easily accessible for review purposes." The Cost Guidance Portal includes several features to assist users in obtaining source documentation of their cost estimates. Immediately after completing a cost estimate: - An automatic email with a detailed summary report of the cost estimate is sent to the DoD employee who prepared the cost estimate. - A detailed summary report of the cost estimate is available for DoD employees to save electronically or print for their records. - A unique, permanent hyperlink to each cost estimate is provided for DoD employees to save as a bookmark in their web browser so that they can access and recover their estimate in the future. * Nonconcurs with the recommendation to "establish and implement a verification process to provide reasonable assurance of consistency and completeness of cost inputs used to develop the cost estimate." The Department currently has the ability to pull detailed cost estimates for specific studies and reports when necessary. As mentioned in the GAO Draft Report, the Department has extracted a sample of detailed cost estimates for selected studies and reports to show to the Congress. The Department also has the ability to contact each DoD employee who prepares a cost estimate, which provides a reasonable level of accountability. Implementing a verification process similar to the one proposed by the GAO would be a costly endeavor. For example, verifying the amount of time spent by each DoD employee preparing a study or report would require the Department to reduce time spent on other, higher priority missions. Rather than implementing a costly verification process for review of cost estimates after they have been submitted, the Department will continue to train users prior to their preparation and submission of cost estimates. * Partially concurs with the recommendation to "require that the language on the front cover of reports and studies make clear that the estimate is based on certain marginal manpower and non-labor costs." While the Department is open to modifying the phrase that is published on the front cover of the study or report, the objective should not be to detail the specific costs included or excluded on the front cover— this level of detail is currently provided on the Cost Guidance Portal and in the automatic email sent to the employee who submits the cost estimate. Although each study and report requires a unique level of manpower efforts and non-labor costs, the cost estimate presented on the front cover should always represent the marginal costs borne by the DoD in preparing the study or report. That is, the cost estimate should include any costs associated with activities the Department would not have encountered had the study or reporting requirement not been assigned.[Footnote 2] The Department will continue to implement the current policy with consideration of these specific recommendations to improve the cost estimating guidance, methods, and tools associated with this effort. Enclosure V Footnotes: [1] GAO's draft report states "DoD's guidance was unclear as to whether coordination was limited to the review and approval of a report," (page 11). However, DoD's Studies and Reports Guidance Document states "Manpower costs should not be included when participation only involves normal job activities such as attending meetings, coordinating studies/reports/proposals, completing security reviews of documents, etc." (page 7). [2] There are some exceptions. For example, coordination of draft studies and reports is an internal DoD activity that is brought upon by the DoD's own policies. Therefore, the costs associated with coordination should not be included in the cost of the study or report. [End of section] Enclosure VI: GAO Contact and Staff Acknowledgments: GAO Contact: Sharon L. Pickup, (202) 512-9619 or pickups@gao.gov: Staff Acknowledgments: In addition to the contact named above, key contributors to this report were James A. Reynolds, Assistant Director; Bruce Brown; Frances Abe Dymond; Mae Jones; Terry Richardson; Karen Richey; Amber Lopez Roberts; Robert Sharpe; Amie Steele; Stacey Steele; Sonja S. Ware; and Karen Nicole Willems. [End of section] Footnotes: [1] Remarks as delivered by former Secretary of Defense Robert M. Gates, Abilene, Kansas, May 8, 2010. [2] Remarks as delivered by former Secretary of Defense Robert M. Gates, The Pentagon, August 9, 2010, and DOD News Release No. 706-10, "Sec. Gates Announces Efficiency Initiatives" (Aug. 9, 2010). [3] Guidance for using the tool has been updated and the most recent version can be found in DOD Cost Guidance Group, Studies and Reports Guidance Document: DOD Cost Guidance Portal (Sept. 16, 2011), and DOD Cost Guidance Group, Cost Guidance Frequently Asked Questions: DOD Cost Guidance Portal (Sept. 16, 2011). [4] Secretary of Defense Memorandum, Consideration of Costs in DOD Decision-Making (Dec. 27, 2010). [5] H.R. Rep. No.112-78 (2011), which accompanied a bill for the National Defense Authorization Act for Fiscal Year 2012, Pub. L. No. 112-81 (2012). [6] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, [hyperlink, http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C: March 2009). GAO developed this guide in order to establish a consistent methodology that can be used across the federal government for developing, managing, and evaluating cost estimates. [7] Federal Accounting Standards Advisory Board, Statement of Federal Financial Accounting Standards (SFFAS) No. 4, Managerial Cost Accounting Concepts and Standards for the Federal Government (Washington, D.C.: July 1995), and DOD Financial Management Regulation 7000.14R, Accounting Policy and Procedures, vol. 4 (2012), see ch. 19, "Managerial Cost Accounting," and ch. 22, "Cost Finding." [8] GAO, Standards for Internal Control in the Federal Government, [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] (Washington, D.C.: November 1999). [9] The tool automatically calculates salary costs for military personnel based on composite rates and civilian personnel based on the General Schedule, neither of which includes all costs associated with manpower. [10] Similarly, SFFAS No. 4 permits flexibility when measuring and reporting costs for special purpose cost studies. [11] Grounds rules represent a common set of agreed-on estimating standards that provide guidance and minimize conflicts in definitions. Assumptions represent a set of judgments about past, present, or future conditions postulated as true in the absence of positive proof. [12] Secretary of Defense Memorandum, Track Four Efficiency Initiatives Decisions (Mar. 14, 2011). [13] The Congressional Hearings and Reporting Requirements Tracking System is DOD's official system used to track congressional processes. [14] Remarks as delivered by former Secretary of Defense Robert M. Gates, The Pentagon, August 9, 2010, and DOD News Release No. 706-10, "Sec. Gates Announces Efficiency Initiatives" (Aug. 9, 2010). [15] GAO, Streamlining Government: Key Practices from Select Efficiency Initiatives Should Be Shared Governmentwide, [hyperlink, http://www.gao.gov/products/GAO-11-908] (Washington, D.C.: Sept. 30, 2011). [End of section] GAO’s Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select “E- mail Updates.” Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-4400 U.S. Government Accountability Office, 441 G Street NW, Room 7125 Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548.