Moving to Work Demonstration: Opportunities Exist to Improve Information and Monitoring
Highlights
What GAO Found
Public housing agencies (PHA) that participate in the Moving to Work (MTW) program report annually on the performance of their activities, which include efforts to reduce administrative costs and encourage residents to work. But this performance information varies, and the Department of Housing and Urban Development’s (HUD) guidance does not specify that it be quantifiable and outcome oriented. Further, HUD has not identified the performance data that would be needed to assess the results of similar MTW activities or the program as a whole and has not established performance indicators for the program. The shortage of such analyses and indicators has hindered comprehensive evaluation efforts, although such evaluations are key to determining the success of any demonstration program. Further, while HUD has identified some lessons learned from the program, it has no systematic process to identify them and thus has relied primarily on ad hoc information. The absence of a systematic process for identifying lessons learned limits HUD’s ability to promote useful practices that could be more broadly implemented to address the purposes of the program.
HUD generally follows its MTW monitoring policies and procedures, but they could be strengthened. HUD staff review and approve each MTW agency’s annual plan to ensure that planned activities are linked to program purposes and visit each MTW agency annually to provide technical assistance. But HUD has not taken key monitoring steps set out in internal control standards, such as issuing guidance that defines program terms or assessing compliance with all of the requirements. Without clarifying key terms and establishing a process for assessing compliance with statutory requirements, HUD lacks assurance that agencies are actually complying with the statute. Additionally, HUD has not done an annual assessment of program risks despite its own requirement to do so and has not developed risk-based monitoring procedures. Without taking these steps, HUD lacks assurance that it has identified all risks to the program. Finally, HUD does not have policies or procedures in place to verify the accuracy of key information that agencies self-report. For example, HUD staff do not verify self-reported performance information during their reviews of annual reports or annual site visits. Without verifying at least some information, HUD cannot be sure that self-reported information is accurate.
Expanding the MTW program may offer benefits but also raises questions. According to HUD, affordable housing advocates, and MTW agencies, expanding MTW to additional PHAs would allow agencies to develop more activities tailored to local conditions and result in more lessons learned. However, data limitations and monitoring weaknesses raise questions about expansion. HUD recently reported that expansion should occur only if newly admitted PHAs structured their programs to permit high-quality evaluations and ensure that lessons learned could be generalized. Until more complete information on the program’s effectiveness and the extent to which agencies are adhering to program requirements is available, it will be difficult for Congress to know whether an expanded MTW would benefit additional agencies and the residents they serve. Some researchers and MTW agencies suggested alternatives to expansion, including implementing a program that was more limited in scope.
Why GAO Did This Study
HUD’s MTW demonstration program gives participating PHAs the flexibility to create innovative housing strategies through their fiscal year 2018. MTW agencies must create activities linked to three statutory purposes—reducing costs, providing incentives for self-sufficiency, and increasing housing choices—and meet five statutory requirements. Congress is considering expanding MTW and has asked GAO to examine what is known about (1) the program’s success in addressing the three purposes, (2) HUD’s monitoring efforts, and (3) the potential benefits of and concerns about expansion. GAO analyzed the most current annual reports for 30 MTW agencies; compared HUD’s monitoring efforts with internal control standards; and interviewed agency officials, researchers, and industry officials.
Recommendations
GAO makes eight recommendations to HUD: that HUD improve its guidance on reporting performance information, develop a plan for identifying and analyzing standard performance data, establish performance indicators, systematically identify lessons learned, clarify key terms, implement a process for assessing compliance with statutory requirements, do annual assessments of program risks, and verify the accuracy of self-reported data. HUD generally or in part agreed with seven of them. HUD disagreed with our recommendation that it create overall performance indicators. GAO believes, however, that they are critical to demonstrating program results and thus maintains its recommendation.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Housing and Urban Development | To improve what is known about the effectiveness of the MTW program, the Secretary of HUD should improve HUD's guidance to MTW agencies on providing performance information in their annual reports by requiring that such information be quantifiable and outcome-oriented to the extent possible. |
HUD revised the Standard MTW Agreement and the current Reporting Requirements for MTW Agencies (HUD-50900) in May 2013, requiring MTW agencies to report quantifiable and outcome-oriented information on MTW activities. Additionally, HUD issued webcast training on the revised HUD-50900. The revised HUD-50900 includes standard tables for MTW agencies to provide operating information, financial information, and metrics for all MTW activities, and directs MTW agencies to provide quantifiable and outcome-oriented data. Taking these steps should help HUD collect data that reflect outcomes of MTW activities.
|
| Department of Housing and Urban Development | To improve what is known about the effectiveness of the MTW program, the Secretary of HUD should develop and implement a plan for quantitatively assessing the effectiveness of similar activities and the program as a whole including the identification of standard performance data needed. |
HUD developed standard metrics, or measures, that correlate with each of the three statutory purposes, or objectives, of the MTW program. For example, for the statutory objective related to cost-effectiveness, HUD established five measures, including whether there was a decrease in the total cost of a task (in dollars), a decrease in the total time to complete a task (in hours), and an increase in the amount of funds leveraged (in dollars). For the statutory objective related to self-sufficiency, HUD created eight measures; and for the statutory objective related to housing choice, HUD created seven measures. For each measure, HUD will be able to calculate quantitative results and assess whether the result for the MTW program as a whole has been positive, neutral, or negative. Further, HUD will be able to total the results for each individual statutory objective, so that it can assess whether the MTW program as a whole is having a positive, neutral, or negative outcome for each objective.
|
| Department of Housing and Urban Development | To improve what is known about the effectiveness of the MTW program, the Secretary of HUD should establish performance indicators for the MTW program as a whole. |
HUD developed a five-step plan for quantitatively assessing the effectiveness of similar activities and the program as a whole. According to this plan, HUD will assign standard metrics for all of the implemented and proposed activities of each MTW agency, and HUD will ensure that agencies conducting similar activities report annually on the same set of metrics. HUD's plan states that the agency will take the data and enter it into a centralized location, which will allow for aggregation across the MTW program. HUD has taken steps to implement this plan. First, HUD developed standard metrics that correlate with the statutory objectives of the MTW program. Second, HUD issued guidance to MTW agencies on how to assign the published standard metrics to proposed and implemented activities. Third, HUD has begun to compile the results reported by MTW agencies into a centralized location, enabling the aggregation of each metric across the program.
|
| Department of Housing and Urban Development | To enhance the ability to identify MTW practices that could be applied more broadly, the Secretary of HUD should create a process to systematically identify lessons learned. |
HUD created and has begun implementing a process to systematically use the annual performance data that MTW agencies report to identify "promising practices" that could be adopted by other MTW agencies or activities that could potentially be replicated more broadly. In June 2015, HUD provided documentation of the steps it would take to analyze quantitative data from MTW agencies. These steps include analyzing 21 standard metrics to determine whether MTW agencies were achieving negative, neutral, or positive results for each metric. In July 2015, HUD provided updated documentation on the process it had established to identify lessons learned. First, HUD will use the data to identify those standard metrics that suggest positive results. Then, HUD will identify standard metrics with positive results to highlight based on factors such as the degree of positive movement and whether external factors would make a potential practice timely or compelling. According to these procedures, once HUD officials agree on the standard metrics to highlight, HUD will conduct additional analysis to identify specific promising practices and the MTW agency(ies) involved. For example, HUD will consider the length of time that an activity has been implemented, diversity in agency size and geography, and the potential for replicability more broadly (to non-MTW agencies). HUD has taken steps to implement these procedures for those MTW agencies that had submitted performance data to HUD by April 2015. By establishing and implementing these procedures, HUD will have a more systematic and data-driven process for identifying lessons learned from the MTW program that can potentially be applied more broadly.
|
| Department of Housing and Urban Development | To improve HUD's oversight of the MTW program, the Secretary of HUD should issue guidance that clarifies key program terms, such as the three statutory purposes of the program and the five statutory requirements that MTW agencies must meet. |
HUD adopted a revised MTW Standard Agreement (HUD-50900) in May 2013, which provides detailed clarifications on the meaning of the three statutory purposes of the program and relevant metrics. Additionally, the revised HUD-50900 includes standardized tables for agencies to report data related to the requirement to serve a comparable mix of families and provides definitions related to other requirements for categories of households served. By taking these steps, HUD can more effectively determine whether MTW agencies are addressing the program's purposes and meeting requirements and improve its oversight of the MTW program
|
| Department of Housing and Urban Development | To improve HUD's oversight of the MTW program, the Secretary of HUD should develop and implement a systematic process for assessing compliance with statutory requirements. |
Beginning with data for 2013, HUD has begun to systematically track MTW agencies compliance with each of the program's five statutory requirements. For the three requirements that agencies could previously self-certify, HUD has assessed compliance using available data. For example, to assess compliance with the requirement that at least 75 percent of households are very low income, HUD has used income-data for each participating MTW agency. Also, to assess compliance with the requirement for MTW agencies to maintain a comparable mix of families, HUD has begun to use available data to compare the mix of families that MTW agency originally assisted (by family size) to the mix assisted in 2013. According to HUD staff, MTW agencies' compliance with each statutory requirement will be updated throughout each year. As a result of continuing to implement these actions, HUD will be able to determine the extent to which MTW agencies are meeting the program's statutory requirements.
|
| Department of Housing and Urban Development | To improve HUD's oversight of the MTW program, the Secretary of HUD should conduct an annual risk assessment for the MTW program and implement risk-based monitoring policies and procedures such as those currently being considered for site visits. |
HUD's Real Estate Assessment Center's (REAC) Quality Assurance Subsystem (QASS) staff performed an initial risk assessment of the MTW program. QASS staff completed this initial assessment in accordance with the HUD CFO Handbook 1840.1, and issued a report in February 2014. According to HUD, for subsequent years, the agency will assess the risks associated with the MTW program using a "risk register." HUD provided a copy of its 2015 risk register, dated December 2014. This register identified and prioritized risks to the MTW program. Additionally, in March 2015, HUD provided GAO with a document noting the factors that HUD considers when determining whether to conduct a site visit every year, or every other year. HUD considers agency size, whether the agency is new to the MTW program, whether the local HUD field office has raised issues, and the complexity of the MTW program being implemented, among other things. While HUD considers these factors when planning site visits, the agency's ability to conduct site visits is dependent on funding and staffing resources. As a result of conducting annual risk assessments of the MTW program as a whole and conducting risk-based site visits, HUD is taking steps to better assure that it is using its limited monitoring resources efficiently.
|
| Department of Housing and Urban Development | To improve HUD's oversight of the MTW program, the Secretary of HUD should implement control activities designed to verify the accuracy of a sample of the performance information that MTW agencies self-report. |
HUD has taken steps to help ensure that the information that individual MTW agencies report about their MTW activities is assessed against program compliance requirements. In March 2014, HUD published a compliance supplement for financial auditors to use as a part of their annual audits of MTW agencies. This supplement, which became effective in May 2014, provides auditors with guidance on allowable MTW activities, costs, and performance reporting, among other things. In 2015, HUD provided procedures explaining how it will use the results of the annual audits. According to these procedures, auditors are to include their assessment of MTW agencies' compliance with the supplement in their completed audits, and HUD field offices are to review the audits of MTW agencies to determine appropriate action relative to any findings and follow up with MTW agencies to ensure findings are closed. Additionally, according to HUD, if there are audit findings related to the MTW program itself, the HUD field office will inform HUD's MTW office, and these offices will work together to agree upon the recommendations that will be made to the MTW agency to resolve the findings. By implementing these activities, HUD can be better assured of the reliability of the performance information that MTW agencies report.
|