April 2011 — GAO's Comments on Sixth Round of Recipient Reporting
What GAO Found
Analysis of Sixth Round Recipient Reporting Data Shows Data Quality Remains Relatively Stable
According to Recovery.gov as of January 30, 2011, recipients reported on over 209,400 awards across multiple programs indicating that the Recovery Act funded approximately 585,654 jobs during the quarter beginning October 1, 2010, and ending December 31, 2010. This included 2,051 prime reports associated with EECBG recipients. As reported by the Recovery Accountability and Transparency Board, job calculations are based on the number of hours worked in a quarter and funded under the Recovery Act—expressed in FTEs (Full Time Equivalents.)
In examining recipient reports, GAO continued to look for progress in addressing limitations it noted in its prior reports. In those prior rounds, GAO reviewed data logic and consistency and reviewed unusual or atypical data. Data logic and consistency provide information on whether the data are believable, given program guidelines and objectives; unusual or atypical data values indicate potential inaccuracies. As with previous quarterly report rounds, these reviews included (1) the ability to link reports for the same project across quarters and (2) concerns in the data logic and consistency, such as reports marked final that show a significant portion of the award amount not spent. GAO continued to see minor variations in the number or percent of reports appearing atypical or showing some form of data discrepancy. For example, GAO continued to find a small number of prime recipient reports for which there were potential linkage issues across quarters. For this latest round, there was a slight increase from 1.5 percent to 2.2 percent in the number of prime reports appearing across all quarters showing a skip in reporting for one or more quarters. This may impact the ability to track project funding and FTEs over quarters. The number of reports marked "final" for which there appeared to be some discrepancy, such as reports marked "final" but for which project status was marked as less than 50 percent completed, continued to be quite small and unchanged from the previous round.
GAO continued to examine the recipient reports' agency review flag field as part of its examination of data logic and consistency, since GAO has noted inconsistencies between agencies' accounts of their review process and the data shown in that field. Prime recipient report records include a review flag indicating whether or not a federal agency reviewed the record during the data quality review time frames. Prior analyses suggested that, for some agencies, the data in this field might not correctly reflect the extent of their review process. However, this did not seem to be the case for the EECBG program. EECBG program data in this field in this sixth round showed that 93 percent of the prime recipient reports were marked as reviewed by DOE, which was generally consistent with accounts of agency officials about their review process. However, GAO continues to observe some inconsistency when another data field on recipient reports, which shows whether or not a correction was initiated, is considered in conjunction with agency and recipient review flags. A correction could be initiated by either the prime recipient or the reviewing agency. Logically, one might expect that if a correction was made, it would have been initiated by a reviewer, and therefore the review flag should also be set to "yes." In this sixth round, as in the prior round, 10 percent of all prime recipient reports for all programs had this correction flag set to ‘yes' even though the review flags indicated that neither the agency nor prime recipient had reviewed those reports.
For this report, GAO has chosen not to show the count of EECBG FTEs reported, out of concern about the comparability and reliability of the figures across quarterly reporting periods. As GAO noted in its September 2010 report, its field work had shown that the FTE calculations continued to be difficult for some recipients. This concern, while still present, based on information gathered from DOE officials in headquarters and DOE project officers in the field, continues to be addressed by DOE officials at all levels. As GAO noted in September, some confusion may have existed about the acceptability and use of some methods for calculating FTEs over the course of the reporting periods. This decision is also based on some irregularities and inconsistencies GAO observed in its analyses of the FTE data across quarters and the relationship of the hours worked, as reported to DOE by recipients, with the FTE values the recipients directly reported to FederalReporting.gov. DOE officials indicated that they continue to assess compliance with and encourage recipients to follow the DOE and Office of Management and Budget (OMB) guidance on how to correctly report FTEs. Moving forward, as these issues in reporting methods are addressed, it is likely that the comparability and reliability of the figures may improve.
Quality Reviews Performed on EECBG Data by DOE and Prime Recipients Included a Focus on Updated OMB Guidance Requirements
Each quarter, DOE performs quality assurance steps on the data that recipients provide to FederalReporting.gov, including checks that are performed centrally across all their Recovery Act programs and reviews done by EECBG project officers at the program level. Based on these reviews, DOE officials reported that most recipients of Recovery Act funds have reported to FederalReporting.gov in previous rounds and now understand the reporting process, resulting in the reporting proceeding more smoothly.
As in previous rounds, DOE performed several checks of the data centrally as information became available. For example, officials compared the amount recipients reported as funds awarded with agency internal records. They also compared jobs data from DOE's PAGE reporting system with FTEs reported to FederalReporting.gov. When discrepancies were found, project officers were instructed to contact recipients to make the necessary corrections. DOE followed up with grant recipients who did not report to FederalReporting.gov. For the sixth round, DOE reported 36 recipients to OMB as not in compliance. Of these, 34 are EECBG grant recipients. Several are tribal recipients that are in remote locations where reporting online is particularly challenging.
EECBG project officers' efforts also helped ensure the quality of information recipients reported to FederalReporting.gov. For example, one group of project officers GAO interviewed reported spending a large portion of time helping recipients complete reporting requirements and ensuring the quality of reports. Project officers cited helping recipients understand terminology, such as distinguishing between vendors and recipients of subawards. They reported taking steps, including following up when large increases in job numbers were reported, reports were missing, a recipient in a remote location had difficulty submitting reports, or recipients had questions about definitions.
DOE officials notified both recipients and reviewers, such as project officers, of the need to ensure that narrative descriptions met requirements laid out in OMB's September 2010 guidance. On September 29, 2010—a few days after OMB's guidance was released but before recipients started reporting for the quarter—DOE e-mailed both recipients and project officers instructions related to the guidance. The e-mail to recipients informed recipients of the need to provide sufficiently clear descriptions to facilitate the public's understanding, and stated that overly general or unclear award descriptions could be considered material omissions. Similarly, the e-mail to reviewers restated the guidance. It instructed reviewers to make sure they read the descriptions in the narrative data fields, and provide a comment to the recipient if they believed clarification was required. DOE also included this information in its webinars on recipient reporting designed for grant recipients and contractors. Further, it included a step in the reviewers' checklist to determine if the narrative descriptions provided clear and complete information on the award's purpose, scope, and activities.
DOE officials also reported that during the last three quarters' reviews they have focused on ensuring that reports marked "final" correctly reflect that status. They have reached out to educate recipients on what that designation means and to ensure that those marked "final" are correctly identified. This includes looking at the amount reported as spent. DOE's quality assurance process flags reports in which it appears the designation may not be correct based on financial analyses, and encourages recipients to make needed corrections during the continuous corrections process.
What GAO DidBack to top
The recipient reporting section of this report responds to the Recovery Act's mandate that GAO comment on the estimates of jobs created or retained by direct recipients of Recovery Act funds. For its review of the sixth submission of recipient reports, covering the period from October 1, 2010, through December 31, 2010, GAO built on findings from its five prior reviews of the reports, covering the period from February 2009 through September 30, 2010.
To understand how the quality of jobs data reported by EECBG grant recipients has changed over time, GAO compared the six quarters of recipient reporting data that were publicly available at Recovery.gov on February 2, 2011. GAO performed edit checks and other analyses on EECBG grant recipient reports, which included matching DOE-provided data from iPortal and PAGE information systems on EECBG recipients. As part of that matching process, GAO also examined the reliability of recipient data contained in these DOE information systems. GAO's assessment activities included reviewing documentation of system processes, Inspector General reviews of the systems and conducting logic tests for key variables. GAO's matches showed a high degree of agreement between DOE recipient information and the information reported by recipients directly to FederalReporting.gov. However, the magnitude of the differences or lack of agreement with regard to the full-time equivalents (FTE) are not insignificant. In general, GAO considers the data used to be sufficiently reliable, with attribution to official sources for the purposes of providing background information and a general sense of the status of EECBG recipient reporting.
-  Under the continuous corrections period, recipients were allowed to modify submissions from February 2, 2011, to March 21, 2011. The final update of this round of recipient reported data should occur on March 23, 2011.