Information Technology:

OMB Has Made Improvements to Its Dashboard, but Further Work Is Needed by Agencies and OMB to Ensure Data Accuracy

GAO-11-262: Published: Mar 15, 2011. Publicly Released: Mar 15, 2011.

Additional Materials:

Contact:

David A. Powner
(202) 512-9286
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

Each year the federal government spends billions of dollars on information technology (IT) investments. Given the importance of oversight, the Office of Management and Budget (OMB) established a public Web site, referred to as the IT Dashboard, that provides detailed information on about 800 federal IT investments, including assessments of actual performance against cost and schedule targets (referred to as ratings). In the second of a series of Dashboard reviews, GAO was asked to (1) determine OMB's efforts to improve the Dashboard and how it is using data from the Dashboard, and (2) examine the accuracy of the Dashboard's cost and schedule performance ratings. To do so, GAO analyzed documentation on OMB oversight efforts and Dashboard improvement plans, compared the performance of 10 major investments from five agencies with large IT budgets against the ratings on the Dashboard, and interviewed OMB and agency officials.

Since GAO's first review, in July 2010, OMB has initiated several efforts to increase the Dashboard's value as an oversight tool, and has used the Dashboard's data to improve federal IT management. These efforts include streamlining key OMB investment reporting tools, eliminating manual monthly submissions, coordinating with agencies to improve data, and improving the Dashboard's user interface. Recent changes provide new views of historical data and rating changes. OMB anticipates that these efforts will increase the reliability of the data on the Dashboard. To improve IT management, OMB analysts use Dashboard data to track investment changes and identify issues with performance. OMB officials stated that they use these data to identify poorly performing IT investments for review sessions by OMB and agency leadership. OMB reported that these sessions and other management reviews have resulted in a $3 billion reduction in life-cycle costs, as of December 2010. While the efforts above as well as initial actions taken to address issues GAO identified in its prior review--such as OMB's updated ratings calculations to factor in ongoing milestones to better reflect current status--have contributed to data quality improvements, performance data inaccuracies remain. The ratings of selected IT investments on the Dashboard did not always accurately reflect current performance, which is counter to the Web site's purpose of reporting near real-time performance. Specifically, GAO found that cost ratings were inaccurate for six of the investments that GAO reviewed and schedule ratings were inaccurate for nine. For example, the Dashboard rating for a Department of Homeland Security investment reported significant cost variances for 3 months in 2010; however, GAO's analysis showed lesser variances from cost targets for the same months. Conversely, a Department of Transportation investment was reported as on schedule on the Dashboard, which does not reflect the significant delays GAO has identified in recent work. These inaccuracies can be attributed to weaknesses in how agencies report data to the Dashboard, such as providing erroneous data submissions, as well as limitations in how OMB calculates the ratings. Until the selected agencies and OMB resolve these issues, ratings will continue to often be inaccurate and may not reflect current program performance. GAO is recommending that selected agencies take steps to improve the accuracy and reliability of Dashboard information and OMB improve how it rates investments relative to current performance and schedule variance. Agencies generally concurred with the recommendations; OMB did not concur with the first recommendation but concurred with the second. GAO maintains that until OMB implements both, performance may continue to be inaccurately represented on the Dashboard.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: The Department of Homeland Security (DHS) agreed with our recommendation and, subsequently, in October 2012, updated its Capital Planning and Investment Control (CPIC) Guide with additional policies and procedures to better ensure that IT Dashboard data submissions include complete and accurate investment information. Specifically, DHS's CPIC Guide documents the CPIC Administrator's responsibility for ensuring that the DHS component agencies complete their monthly updates on the IT Dashboard for all investments, that project execution dates are entered for each investment, and that there are no gaps in reporting. Further, the CPIC administrator is to review any data quality issues with component agencies in order to correct any data errors prior to the next monthly update. As a result, the IT Dashboard should reflect more accurate DHS investment information, which may help the Office of Management and Budget, other oversight bodies, and the general public hold the department accountable for progress and results.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Secretary of the Department of Homeland Security should direct the CIO to ensure that investment data submissions include complete and accurate investment information for all required fields.

    Agency Affected: Department of Homeland Security

  2. Status: Closed - Implemented

    Comments: The Department of Homeland Security (DHS) agreed with our recommendation and, subsequently, in October 2012, issued an instruction manual on the management of information technology (IT) investment baselines. These instructions establish responsibilities, processes, and procedures for documenting the baseline standards against which the actual cost and schedule performance of an investment is measured, and are applicable to all major IT investments at DHS. The instructions also identify the responsibilities and timeframes for updates to the IT Dashboard's data including performance metrics, chief information officer ratings, risk data, and rebaseline information. For example, DHS component program managers are responsible for updating investment data on a monthly basis, as well updating contract and risk data when significant changes occur. When a baseline change occurs, program managers are responsible for ensuring that the reason for the change and effective date are reflected on the IT Dashboard. Further, between January 2013 and September 2013, DHS updated the IT Dashboard with rebaseline information, including the reasons for the baseline changes, for 10 investments. As a result of implementing this guidance, DHS's submissions to the IT Dashboard should reflect more accurate performance information and CIO ratings, which may help the Office of Management and Budget, other oversight bodies, and the general public hold the department accountable for progress and results.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Secretary of the Department of Homeland Security should direct the CIO to comply with OMB's guidance on updating the CIO rating as soon as new information becomes available that affects the assessment of a given investment, including when an investment is in the process of a rebaseline.

    Agency Affected: Department of Homeland Security

  3. Status: Closed - Implemented

    Comments: The Department of Homeland Security (DHS) agreed with our recommendation and has since taken action to comply with the Office of Management and Budget's (OMB) guidance on updating investment performance data on the IT Dashboard. Specifically, over a one-year period from September 2012 through August 2013, the department updated the performance data for the Command, Control, Communications, Computers, Intelligence, Surveillance & Reconnaissance (C4ISR) program 12 times, or approximately every month. As a result, the IT Dashboard should reflect more accurate DHS investment performance data, which may help OMB, other oversight bodies, and the general public, hold the department accountable for progress and results.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Secretary of the Department of Homeland Security should direct the CIO to work with C4ISR officials to comply with OMB's guidance on updating investment cost and schedule data on the Dashboard at least monthly.

    Agency Affected: Department of Homeland Security

  4. Status: Closed - Implemented

    Comments: The Department of Transportation (Transportation) agreed to consider our recommendation and has since taken action to comply with the Office of Management and Budget's (OMB) guidance on updating investment performance data on the IT Dashboard. Specifically, over a 12-month period from September 2012 through August 2013, the department updated the performance data for the Automatic Dependent Surveillance-Broadcast program 10 times, or almost every month. As a result, the IT Dashboard should reflect more accurate Transportation performance data, which may help OMB, other oversight bodies, and the general public, hold the department accountable for progress and results.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Secretary of the Department of Transportation should direct the CIO to work with Automatic Dependent Surveillance-Broadcast officials to comply with OMB's guidance on updating investment cost and schedule data on the Dashboard at least monthly.

    Agency Affected: Department of Transportation

  5. Status: Closed - Implemented

    Comments: The Department of the Treasury (Treasury) generally agreed with our recommendation and has since taken action to comply with the Office of Management and Budget's (OMB) guidance on updating the chief information officer (CIO) ratings for its investments by significantly increasing the frequency in which it performed such updates. Specifically, between January 2013 and July 2013, the department updated its CIO ratings for a total of 53 investments on an average of once every 34 days, as compared to calendar year 2010 when the department updated its CIO ratings for a total of 74 investments on an average of only once every 62 days. As a result, the IT Dashboard should reflect more accurate Treasury CIO ratings, which may help OMB, other oversight bodies, and the general public hold the department accountable for progress and results.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Secretary of the Department of the Treasury should direct the CIO to comply with OMB's guidance on updating the CIO rating as soon as new information becomes available that affects the assessment of a given investment, including when an investment is in the process of a rebaseline.

    Agency Affected: Department of the Treasury

  6. Status: Closed - Not Implemented

    Comments: According to the Office of Management and Budget's July 2013 guidance on Exhibit 53s (used by federal agencies to report IT investment information to the IT Dashboard), actual percent complete is no longer required investment data.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Secretary of the Department of the Treasury should direct the CIO to work with Modernized e-File officials to report accurate actual percent complete data for each of the investment's activities.

    Agency Affected: Department of the Treasury

  7. Status: Closed - Implemented

    Comments: The Department of the Treasury (Treasury) generally agreed with our recommendation and, in March 2011, addressed this recommendation by including the following statement in its Chief Information Officer (CIO) comments on the Office of Management and Budget's (OMB) IT Dashboard: "The data quality issues mentioned in the GAO Audit of the IT Dashboard were resolved with the approval of the BCR on January 27, 2011 and subsequent upload to the IT Dashboard." As a result, the IT Dashboard should reflect more accurate Treasury CIO ratings, which may help OMB, other oversight bodies, and the general public hold the department accountable for progress and results.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Secretary of the Department of the Treasury should direct the CIO to work with Payment Application Modernization officials to disclose the extent of this investment's data reliability issues in the CIO rating assessment on the Dashboard.

    Agency Affected: Department of the Treasury

  8. Status: Closed - Implemented

    Comments: The Department of Veterans Affairs (VA) generally concurred with this recommendation and has complied with the Office of Management and Budget's (OMB) guidance on updating CIO ratings. Specifically, since 2012 the average number of days between VA rating updates has decreased from 172 in 2012 to 19 in 2015. As a result, the IT Dashboard should reflect more accurate VA investment performance data, which can help OMB, other oversight bodies, and the general public, hold the department accountable for progress and results.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Secretary of the Department of Veterans Affairs should direct the CIO to comply with OMB's guidance on updating the CIO rating as soon as new information becomes available that affects the assessment of a given investment, including when an investment is in the process of a rebaseline.

    Agency Affected: Department of Veterans Affairs

  9. Status: Closed - Implemented

    Comments: The Department of Veterans Affairs (VA) generally concurred with this recommendation and has complied with the Office of Management and Budget's (OMB) guidance on updating investment performance data on the IT Dashboard. Specifically, the Dashboard shows that the Medical Legacy cost and schedule data have been updated in 11 of the past 12 months, from September 2014 to May 2015. As a result, the IT Dashboard should reflect more accurate VA IT investment performance data, which may help OMB, other oversight bodies, and the general public, hold the department accountable for progress and results.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Secretary of the Department of Veterans Affairs direct the CIO to work with Medical Legacy officials to comply with OMB's guidance on updating investment cost and schedule data on the Dashboard at least monthly.

    Agency Affected: Department of Veterans Affairs

  10. Status: Open

    Comments: The Department of Veterans Affairs generally concurred with this recommendation and has since taken initial steps to submit updated performance data to the Dashboard. Specifically, over a 12-month period from September 2012 to August 2013, the department updated the performance data for the Medical Legacy investment five times. However, more frequent updates are needed to determine whether the IT Dashboard information is consistent with the investment's internal performance information. In September 2015, the department described the investment's performance in a written response, but has yet to provide supporting documentation. Upon receipt of such documentation, we will plan to validate agency actions.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Secretary of the Department of Veterans Affairs direct the CIO to ensure Medical Legacy investment data submitted to the Dashboard are consistent with the investment's internal performance information.

    Agency Affected: Department of Veterans Affairs

  11. Status: Closed - Implemented

    Comments: The Social Security Administration (SSA) agreed with our recommendation and, in October 2014, implemented its updated Performance Measurement Baseline Policy to better ensure that IT Dashboard data submissions include complete and accurate investment information. Specifically, the policy requires that SSA program managers document and approve changes to scope, cost, schedule, or operational performance metrics. Further, it requires the agency to update the IT Dashboard within 30 days with any new agency-approved baseline information. As a result, the IT Dashboard should reflect more accurate SSA investment information, which may help the Office of Management and Budget, other oversight bodies, and the general public hold the department accountable for progress and results.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Commissioner of the Social Security Administration should direct the CIO to ensure that data submissions to the Dashboard include accurate investment information for all required fields.

    Agency Affected: Social Security Administration

  12. Status: Closed - Implemented

    Comments: The Office of Management and Budget (OMB) disagreed with this recommendation, but has since addressed this issue. In particular, OMB has added functionality to the Dashboard which allows Dashboard users to view performance over an investment?s entire lifecycle, or in 3, 6, 9, 12, or 18 month periods. While the current method for performing cost and schedule calculations does not directly address the findings identified in our report, the improved functionality does address our concerns that the performance of recent activities could be masked by prior performance. As a result of this improved functionality, users can now view performance ratings that the investments are presently facing, and can to identify solutions to relevant problems.

    Recommendation: To better ensure that the Dashboard provides meaningful ratings and reliable investment data, the Director of OMB should direct the Federal CIO to develop cost and schedule rating calculations that better reflect current investment performance.

    Agency Affected: Executive Office of the President: Office of Management and Budget

  13. Status: Closed - Implemented

    Comments: The Office of Management and Budget (OMB) agreed with this recommendation and, since 2011, has changed how it performs schedule ratings calculations for the IT Dashboard. As of August 2015, the Frequently Asked Questions for the Dashboard show how the calculations are performed for schedule ratings and recognizes the variance of ongoing, overdue activities. As a result, OMB is more accurately reflecting schedule variances of ongoing activities on the Dashboard, allowing for enhanced oversight of federal IT investments.

    Recommendation: To better ensure that the Dashboard provides meaningful ratings and reliable investment data, the Director of OMB should direct the Federal CIO to update the Dashboard's schedule calculation for in-progress activities to more accurately represent the variance of ongoing, overdue activities.

    Agency Affected: Executive Office of the President: Office of Management and Budget

 

Explore the full database of GAO's Open Recommendations »

Dec 6, 2016

Nov 30, 2016

Nov 16, 2016

Sep 29, 2016

Sep 26, 2016

Sep 15, 2016

Sep 14, 2016

Sep 8, 2016

Jun 29, 2016

Jun 22, 2016

Looking for more? Browse all our products here