Information Technology:

OMB Has Made Improvements to Its Dashboard, but Further Work Is Needed by Agencies and OMB to Ensure Data Accuracy

GAO-11-262: Published: Mar 15, 2011. Publicly Released: Mar 15, 2011.

Additional Materials:

Contact:

David A. Powner
(202) 512-9286
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

Each year the federal government spends billions of dollars on information technology (IT) investments. Given the importance of oversight, the Office of Management and Budget (OMB) established a public Web site, referred to as the IT Dashboard, that provides detailed information on about 800 federal IT investments, including assessments of actual performance against cost and schedule targets (referred to as ratings). In the second of a series of Dashboard reviews, GAO was asked to (1) determine OMB's efforts to improve the Dashboard and how it is using data from the Dashboard, and (2) examine the accuracy of the Dashboard's cost and schedule performance ratings. To do so, GAO analyzed documentation on OMB oversight efforts and Dashboard improvement plans, compared the performance of 10 major investments from five agencies with large IT budgets against the ratings on the Dashboard, and interviewed OMB and agency officials.

Since GAO's first review, in July 2010, OMB has initiated several efforts to increase the Dashboard's value as an oversight tool, and has used the Dashboard's data to improve federal IT management. These efforts include streamlining key OMB investment reporting tools, eliminating manual monthly submissions, coordinating with agencies to improve data, and improving the Dashboard's user interface. Recent changes provide new views of historical data and rating changes. OMB anticipates that these efforts will increase the reliability of the data on the Dashboard. To improve IT management, OMB analysts use Dashboard data to track investment changes and identify issues with performance. OMB officials stated that they use these data to identify poorly performing IT investments for review sessions by OMB and agency leadership. OMB reported that these sessions and other management reviews have resulted in a $3 billion reduction in life-cycle costs, as of December 2010. While the efforts above as well as initial actions taken to address issues GAO identified in its prior review--such as OMB's updated ratings calculations to factor in ongoing milestones to better reflect current status--have contributed to data quality improvements, performance data inaccuracies remain. The ratings of selected IT investments on the Dashboard did not always accurately reflect current performance, which is counter to the Web site's purpose of reporting near real-time performance. Specifically, GAO found that cost ratings were inaccurate for six of the investments that GAO reviewed and schedule ratings were inaccurate for nine. For example, the Dashboard rating for a Department of Homeland Security investment reported significant cost variances for 3 months in 2010; however, GAO's analysis showed lesser variances from cost targets for the same months. Conversely, a Department of Transportation investment was reported as on schedule on the Dashboard, which does not reflect the significant delays GAO has identified in recent work. These inaccuracies can be attributed to weaknesses in how agencies report data to the Dashboard, such as providing erroneous data submissions, as well as limitations in how OMB calculates the ratings. Until the selected agencies and OMB resolve these issues, ratings will continue to often be inaccurate and may not reflect current program performance. GAO is recommending that selected agencies take steps to improve the accuracy and reliability of Dashboard information and OMB improve how it rates investments relative to current performance and schedule variance. Agencies generally concurred with the recommendations; OMB did not concur with the first recommendation but concurred with the second. GAO maintains that until OMB implements both, performance may continue to be inaccurately represented on the Dashboard.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: The Department of Homeland Security (DHS) agreed with our recommendation and, subsequently, in October 2012, updated its Capital Planning and Investment Control (CPIC) Guide with additional policies and procedures to better ensure that IT Dashboard data submissions include complete and accurate investment information. Specifically, DHS's CPIC Guide documents the CPIC Administrator's responsibility for ensuring that the DHS component agencies complete their monthly updates on the IT Dashboard for all investments, that project execution dates are entered for each investment, and that there are no gaps in reporting. Further, the CPIC administrator is to review any data quality issues with component agencies in order to correct any data errors prior to the next monthly update. As a result, the IT Dashboard should reflect more accurate DHS investment information, which may help the Office of Management and Budget, other oversight bodies, and the general public hold the department accountable for progress and results.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Secretary of the Department of Homeland Security should direct the CIO to ensure that investment data submissions include complete and accurate investment information for all required fields.

    Agency Affected: Department of Homeland Security

  2. Status: Closed - Implemented

    Comments: The Department of Homeland Security (DHS) agreed with our recommendation and, subsequently, in October 2012, issued an instruction manual on the management of information technology (IT) investment baselines. These instructions establish responsibilities, processes, and procedures for documenting the baseline standards against which the actual cost and schedule performance of an investment is measured, and are applicable to all major IT investments at DHS. The instructions also identify the responsibilities and timeframes for updates to the IT Dashboard's data including performance metrics, chief information officer ratings, risk data, and rebaseline information. For example, DHS component program managers are responsible for updating investment data on a monthly basis, as well updating contract and risk data when significant changes occur. When a baseline change occurs, program managers are responsible for ensuring that the reason for the change and effective date are reflected on the IT Dashboard. Further, between January 2013 and September 2013, DHS updated the IT Dashboard with rebaseline information, including the reasons for the baseline changes, for 10 investments. As a result of implementing this guidance, DHS's submissions to the IT Dashboard should reflect more accurate performance information and CIO ratings, which may help the Office of Management and Budget, other oversight bodies, and the general public hold the department accountable for progress and results.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Secretary of the Department of Homeland Security should direct the CIO to comply with OMB's guidance on updating the CIO rating as soon as new information becomes available that affects the assessment of a given investment, including when an investment is in the process of a rebaseline.

    Agency Affected: Department of Homeland Security

  3. Status: Closed - Implemented

    Comments: The Department of Homeland Security (DHS) agreed with our recommendation and has since taken action to comply with the Office of Management and Budget's (OMB) guidance on updating investment performance data on the IT Dashboard. Specifically, over a one-year period from September 2012 through August 2013, the department updated the performance data for the Command, Control, Communications, Computers, Intelligence, Surveillance & Reconnaissance (C4ISR) program 12 times, or approximately every month. As a result, the IT Dashboard should reflect more accurate DHS investment performance data, which may help OMB, other oversight bodies, and the general public, hold the department accountable for progress and results.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Secretary of the Department of Homeland Security should direct the CIO to work with C4ISR officials to comply with OMB's guidance on updating investment cost and schedule data on the Dashboard at least monthly.

    Agency Affected: Department of Homeland Security

  4. Status: Closed - Implemented

    Comments: The Department of Transportation (Transportation) agreed to consider our recommendation and has since taken action to comply with the Office of Management and Budget's (OMB) guidance on updating investment performance data on the IT Dashboard. Specifically, over a 12-month period from September 2012 through August 2013, the department updated the performance data for the Automatic Dependent Surveillance-Broadcast program 10 times, or almost every month. As a result, the IT Dashboard should reflect more accurate Transportation performance data, which may help OMB, other oversight bodies, and the general public, hold the department accountable for progress and results.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Secretary of the Department of Transportation should direct the CIO to work with Automatic Dependent Surveillance-Broadcast officials to comply with OMB's guidance on updating investment cost and schedule data on the Dashboard at least monthly.

    Agency Affected: Department of Transportation

  5. Status: Closed - Implemented

    Comments: The Department of the Treasury (Treasury) generally agreed with our recommendation and has since taken action to comply with the Office of Management and Budget's (OMB) guidance on updating the chief information officer (CIO) ratings for its investments by significantly increasing the frequency in which it performed such updates. Specifically, between January 2013 and July 2013, the department updated its CIO ratings for a total of 53 investments on an average of once every 34 days, as compared to calendar year 2010 when the department updated its CIO ratings for a total of 74 investments on an average of only once every 62 days. As a result, the IT Dashboard should reflect more accurate Treasury CIO ratings, which may help OMB, other oversight bodies, and the general public hold the department accountable for progress and results.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Secretary of the Department of the Treasury should direct the CIO to comply with OMB's guidance on updating the CIO rating as soon as new information becomes available that affects the assessment of a given investment, including when an investment is in the process of a rebaseline.

    Agency Affected: Department of the Treasury

  6. Status: Closed - Not Implemented

    Comments: According to the Office of Management and Budget's July 2013 guidance on Exhibit 53s (used by federal agencies to report IT investment information to the IT Dashboard), actual percent complete is no longer required investment data.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Secretary of the Department of the Treasury should direct the CIO to work with Modernized e-File officials to report accurate actual percent complete data for each of the investment's activities.

    Agency Affected: Department of the Treasury

  7. Status: Closed - Implemented

    Comments: The Department of the Treasury (Treasury) generally agreed with our recommendation and, in March 2011, addressed this recommendation by including the following statement in its Chief Information Officer (CIO) comments on the Office of Management and Budget's (OMB) IT Dashboard: "The data quality issues mentioned in the GAO Audit of the IT Dashboard were resolved with the approval of the BCR on January 27, 2011 and subsequent upload to the IT Dashboard." As a result, the IT Dashboard should reflect more accurate Treasury CIO ratings, which may help OMB, other oversight bodies, and the general public hold the department accountable for progress and results.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Secretary of the Department of the Treasury should direct the CIO to work with Payment Application Modernization officials to disclose the extent of this investment's data reliability issues in the CIO rating assessment on the Dashboard.

    Agency Affected: Department of the Treasury

  8. Status: Open

    Comments: The Department of Veterans Affairs (VA) generally concurred with this recommendation and has since taken initial steps to update its chief information officer (CIO) ratings. While VA did not update its CIO ratings data from September 2011 to March 2013, the department has updated its CIO ratings an average of six times for its 24 investments since March 2013. In addition, VA has updated the information for an investment that was in the process of a rebaseline. For example, in June 2013, VA noted that its Medical Legacy investment was baselined in February 2013 when a sub-project was removed due to change in funding. However, in the past year only 4 of the 24 were updated on a monthly basis. We will continue to monitor the department's Dashboard updates.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Secretary of the Department of Veterans Affairs should direct the CIO to comply with OMB's guidance on updating the CIO rating as soon as new information becomes available that affects the assessment of a given investment, including when an investment is in the process of a rebaseline.

    Agency Affected: Department of Veterans Affairs

  9. Status: Open

    Comments: The Department of Veterans Affairs generally concurred with this recommendation and has since taken initial steps to submit updated performance data to the Dashboard. Specifically, from March 2013, when VA restored its ability to update the Dashboard, to June 2014, the department updated the performance data for the Medical Legacy investment five times. However, more frequent updates are needed to determine whether the IT Dashboard information is consistent with the investment's internal performance information. We will continue to monitor the department's actions to provide updated information to the Dashboard regarding this investment.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Secretary of the Department of Veterans Affairs direct the CIO to work with Medical Legacy officials to comply with OMB's guidance on updating investment cost and schedule data on the Dashboard at least monthly.

    Agency Affected: Department of Veterans Affairs

  10. Status: Open

    Comments: The Department of Veterans Affairs generally concurred with this recommendation and has since taken initial steps to submit updated performance data to the Dashboard. Specifically, over a 12-month period from September 2012 to August 2013, the department updated the performance data for the Medical Legacy investment five times. However, more frequent updates are needed to determine whether the IT Dashboard information is consistent with the investment's internal performance information. We will continue to monitor the department's actions to provide updated information to the Dashboard regarding this investment.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Secretary of the Department of Veterans Affairs direct the CIO to ensure Medical Legacy investment data submitted to the Dashboard are consistent with the investment's internal performance information.

    Agency Affected: Department of Veterans Affairs

  11. Status: Open

    Comments: The Social Security Administration (SSA) agreed with our recommendation and, in May 2012, issued a Capital Planning and Investment Control (CPIC) Guide that includes policies and procedures related to updating the IT Dashboard. For example, the SSA's CPIC guide describes the Deputy Commissioner for Systems/Chief Information Officer's responsibility for evaluating and updating IT Dashboard data on a regular basis. However, the guide does not discuss any additional controls to ensure that SSA's data submissions to the IT Dashboard include accurate investment information. Further, during a recent review (GAO-14-64), we identified an issue with SSA?s management of investment baselines, which impacted the reported schedule information. We will continue to monitor the department's progress in implementing this recommendation.

    Recommendation: To better ensure that the Dashboard provides accurate cost and schedule performance ratings, the Commissioner of the Social Security Administration should direct the CIO to ensure that data submissions to the Dashboard include accurate investment information for all required fields.

    Agency Affected: Social Security Administration

  12. Status: Open

    Comments: The Office of Management and Budget (OMB) disagreed with this recommendation, stating that it believed current investment performance was reflected in the ratings since the Dashboard was updated on a monthly basis. Our report noted that the finding was based on the use of historical data in the calculations, not the frequency of Dashboard updates. Since 2011, OMB has changed how it performs cost and schedule ratings calculations for the IT Dashboard. As of August 2014, the Frequently Asked Questions for the Dashboard show how the calculations are performed for both cost and schedule ratings; however, the current method for performing calculations does not address the findings identified in our report. Those findings noted that the performance of ongoing and completed activities was combined, which could mask recent performance and cause Dashboard cost and schedule performance ratings to not always reflect current performance. The cost calculation continues to be a rollup of the costs for all activities, both those ongoing and those completed, which can mask recent project performance. In addition, the schedule calculation continues to focus on completion dates to determine schedule variances. We will monitor OMB's efforts in this area and provide updated information if additional action is taken.

    Recommendation: To better ensure that the Dashboard provides meaningful ratings and reliable investment data, the Director of OMB should direct the Federal CIO to develop cost and schedule rating calculations that better reflect current investment performance.

    Agency Affected: Executive Office of the President: Office of Management and Budget

  13. Status: Open

    Comments: The Office of Management and Budget (OMB) agreed with this recommendation and, since 2011, has changed how it performs schedule ratings calculations for the IT Dashboard. As of August 2014, the Frequently Asked Questions for the Dashboard show how the calculations are performed for schedule ratings; however, the current method for performing schedule calculations does not address the findings identified in our report, namely that OMB's schedule calculation did not recognize the full variance of an overdue activity until it has actually completed. The schedule calculations continue to focus primarily on completed project and activity dates to determine schedule variances, rather than the variance of on-going activities. We will monitor OMB's efforts in this area and provide updated information if additional action is taken.

    Recommendation: To better ensure that the Dashboard provides meaningful ratings and reliable investment data, the Director of OMB should direct the Federal CIO to update the Dashboard's schedule calculation for in-progress activities to more accurately represent the variance of ongoing, overdue activities.

    Agency Affected: Executive Office of the President: Office of Management and Budget

 

Explore the full database of GAO's Open Recommendations »

Sep 25, 2014

Sep 23, 2014

Jun 10, 2014

May 22, 2014

May 12, 2014

May 8, 2014

May 7, 2014

Apr 2, 2014

Looking for more? Browse all our products here