This is the accessible text file for GAO report number GAO-12-490 
entitled 'Moving To Work Demonstration: Opportunities Exist to Improve 
Information and Monitoring' which was released on May 21, 2012. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to Congressional Requesters: 

April 2012: 

Moving To Work Demonstration: 

Opportunities Exist to Improve Information and Monitoring: 

GAO-12-490: 

GAO Highlights: 

Highlights of GAO-12-490, a report to congressional requesters. 

Why GAO Did This Study: 

HUD’s MTW demonstration program gives participating PHAs the 
flexibility to create innovative housing strategies through their 
fiscal year 2018. MTW agencies must create activities linked to three 
statutory purposes—reducing costs, providing incentives for self-
sufficiency, and increasing housing choices—and meet five statutory 
requirements. Congress is considering expanding MTW and has asked GAO 
to examine what is known about (1) the program’s success in addressing 
the three purposes, (2) HUD’s monitoring efforts, and (3) the 
potential benefits of and concerns about expansion. GAO analyzed the 
most current annual reports for 30 MTW agencies; compared HUD’s 
monitoring efforts with internal control standards; and interviewed 
agency officials, researchers, and industry officials. 

What GAO Found: 

Public housing agencies (PHA) that participate in the Moving to Work 
(MTW) program report annually on the performance of their activities, 
which include efforts to reduce administrative costs and encourage 
residents to work. But this performance information varies, and the 
Department of Housing and Urban Development’s (HUD) guidance does not 
specify that it be quantifiable and outcome oriented. Further, HUD has 
not identified the performance data that would be needed to assess the 
results of similar MTW activities or the program as a whole and has 
not established performance indicators for the program. The shortage 
of such analyses and indicators has hindered comprehensive evaluation 
efforts, although such evaluations are key to determining the success 
of any demonstration program. Further, while HUD has identified some 
lessons learned from the program, it has no systematic process to 
identify them and thus has relied primarily on ad hoc information. The 
absence of a systematic process for identifying lessons learned limits 
HUD’s ability to promote useful practices that could be more broadly 
implemented to address the purposes of the program. 

HUD generally follows its MTW monitoring policies and procedures, but 
they could be strengthened. HUD staff review and approve each MTW agency
’s annual plan to ensure that planned activities are linked to program 
purposes and visit each MTW agency annually to provide technical 
assistance. But HUD has not taken key monitoring steps set out in 
internal control standards, such as issuing guidance that defines 
program terms or assessing compliance with all of the requirements. 
Without clarifying key terms and establishing a process for assessing 
compliance with statutory requirements, HUD lacks assurance that 
agencies are actually complying with the statute. Additionally, HUD 
has not done an annual assessment of program risks despite its own 
requirement to do so and has not developed risk-based monitoring 
procedures. Without taking these steps, HUD lacks assurance that it 
has identified all risks to the program. Finally, HUD does not have 
policies or procedures in place to verify the accuracy of key 
information that agencies self-report. For example, HUD staff do not 
verify self-reported performance information during their reviews of 
annual reports or annual site visits. Without verifying at least some 
information, HUD cannot be sure that self-reported information is 
accurate. 

Expanding the MTW program may offer benefits but also raises 
questions. According to HUD, affordable housing advocates, and MTW 
agencies, expanding MTW to additional PHAs would allow agencies to 
develop more activities tailored to local conditions and result in 
more lessons learned. However, data limitations and monitoring 
weaknesses raise questions about expansion. HUD recently reported that 
expansion should occur only if newly admitted PHAs structured their 
programs to permit high-quality evaluations and ensure that lessons 
learned could be generalized. Until more complete information on the 
program’s effectiveness and the extent to which agencies are adhering 
to program requirements is available, it will be difficult for 
Congress to know whether an expanded MTW would benefit additional 
agencies and the residents they serve. Some researchers and MTW 
agencies suggested alternatives to expansion, including implementing a 
program that was more limited in scope. 

What GAO Recommends: 

GAO makes eight recommendations to HUD: that HUD improve its guidance 
on reporting performance information, develop a plan for identifying 
and analyzing standard performance data, establish performance 
indicators, systematically identify lessons learned, clarify key 
terms, implement a process for assessing compliance with statutory 
requirements, do annual assessments of program risks, and verify the 
accuracy of self-reported data. HUD generally or in part agreed with 
seven of them. HUD disagreed with our recommendation that it create 
overall performance indicators. GAO believes, however, that they are 
critical to demonstrating program results and thus maintains its 
recommendation. 

View [hyperlink, http://www.gao.gov/products/GAO-12-490]. For more 
information, contact Mathew J. Scirè at (202) 512-8678 or 
sciremj@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

HUD Has Not Identified Standard Performance Data and Indicators Needed 
to Evaluate the Program: 

HUD Generally Follows Its Monitoring Policies and Procedures, but 
Could Strengthen Them: 

Expanding MTW May Offer Benefits but Also Raises Questions: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Scope and Methodology: 

Appendix II: Comments from the Department of Housing and Urban 
Development: 

Appendix III: GAO Contact and Staff Acknowledgments: 

Table: 

Table 1: Examples of MTW Activities, by Statutory Purpose: 

Figures: 

Figure 1: Timeline of Public Housing Agencies Entering and Exiting the 
MTW Demonstration Program: 

Figure 2: Number of Reported MTW Activities Linked to Each of the 
Statutory Purposes: 

Abbreviations: 

AHSSIA: Affordable Housing and Self-Sufficiency Improvement Act of 
2012: 

HUD: Department of Housing and Urban Development: 

MTW: Moving to Work: 

MTW-PIC: Moving to Work section of the Public and Indian Housing 
Information Center: 

PHA: public housing agency: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

April 19, 2012: 

The Honorable Charles Grassley: 
Ranking Member: 
Committee on the Judiciary: 
United States Senate: 

The Honorable Judy Biggert: 
Chairman: 
Subcommittee on Insurance, Housing and Community Opportunity 
Committee on Financial Services: 
House of Representatives: 

The Honorable Shelley Moore Capito: 
Chairman: 
Subcommittee on Financial Institutions and Consumer Credit: 
Committee on Financial Services: 
House of Representatives: 

The Department of Housing and Urban Development (HUD) provides over 
$25 billion each year to public housing agencies (PHA) to make rental 
housing affordable to about 3.2 million low-income households through 
a variety of programs.[Footnote 1] As of January 2012, 35 of the 
approximately 3,300 PHAs operating in the United States were 
participating in the Moving to Work (MTW) demonstration program until 
the end of their fiscal year 2018. The MTW program has three statutory 
purposes: to reduce costs and achieve greater cost-effectiveness in 
federal housing expenditures, to give families with children 
incentives to obtain employment and become self-sufficient, and to 
increase housing choices for low-income families. It gives 
participating PHAs the flexibility to design and test innovative 
strategies for providing and administering housing assistance in their 
communities. The agencies (MTW agencies) must implement activities 
that are linked to one or more of the program's statutory purposes but 
that are also designed specifically for the communities the agencies 
serve. In fiscal year 2011, MTW agencies received combined funding of 
$3.3 billion for housing activities.[Footnote 2] This represented more 
than 13 percent of HUD's affordable housing expenditures. Legislation 
has recently been proposed to expand the number of MTW agencies. 
However, researchers and organizations that advocate on behalf of 
residents, including legal aid groups, have raised concerns about 
expanding it due to the lack of information on the effect of the 
program as well as concerns about residents. 

In response to your request for information on the effectiveness of 
the MTW program, this report discusses (1) what is known about the 
extent to which the MTW demonstration program is addressing the 
program's statutory purposes, (2) HUD's monitoring of MTW agencies' 
efforts to address these purposes and meet statutory requirements, and 
(3) the potential benefits of and concerns about expanding the number 
of PHAs that can participate in the demonstration program. 

To determine what is known about the extent to which the MTW program 
is addressing statutory purposes, we analyzed the most recent annual 
reports as of January 2012 for 30 agencies.[Footnote 3] We assessed 
the reliability of selected information in the reports by reviewing 
supporting documentation for a sample of seven MTW agencies and 
interviewing officials responsible for preparing and reviewing this 
information. (We selected the sample of MTW agencies to provide 
diversity in factors such as geography, agency size, and length of 
time participating in the program.) We determined that the reports 
were sufficiently reliable for the purposes of our review. We analyzed 
these reports and corresponding annual plans to determine the number 
of ongoing activities and the statutory purpose(s) to which each 
activity was linked. We compared HUD's guidance for MTW agencies on 
the type of performance information they should report with the GPRA 
(Government Performance and Results Act) Modernization Act of 2010. We 
also reviewed evaluations of the program and summarized the challenges 
that HUD and others have faced in conducting such evaluations. 
Finally, we assessed HUD's past efforts to identify lessons learned 
from the MTW program. 

To assess HUD's monitoring of MTW agencies' efforts to address the 
program's statutory purposes and meet requirements, we reviewed 
relevant policies and procedures, including HUD's desk guide for the 
program, the Standard Agreement that HUD executed in 2008 to govern 
participation in the program, and documentation of the steps HUD had 
taken to help ensure compliance with its policies and procedures. We 
also examined the relevant internal control standards that apply to 
federal agencies and HUD's guidance on internal controls and compared 
HUD's monitoring policies and procedures with these standards. To 
discuss the potential benefits and concerns associated with expanding 
the number of PHAs that can participate in the program, we reviewed 
studies, reports, and testimonies by researchers, affordable housing 
advocates, and resident advocates. For all three objectives, we 
interviewed HUD officials, officials from our sample of seven MTW 
agencies, research organizations, affordable housing advocates, and 
organizations that advocate on behalf of residents of federally 
subsidized housing. 

We conducted this performance audit from July 2011 to April 2012 in 
accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. Appendix I 
contains additional information on our scope and methodology. 

Background: 

The MTW demonstration program was authorized by the Omnibus 
Consolidated Rescissions and Appropriations Act of 1996.[Footnote 4] 
The program is intended to give participating agencies the flexibility 
to design and test innovative strategies for providing and 
administering housing assistance. To implement such strategies, 
participating agencies may request waivers of certain provisions in 
the United States Housing Act of 1937, as amended.[Footnote 5] For 
example, agencies may request to waive certain provisions in order to 
combine the funding they are awarded annually from different programs 
into a single, authoritywide funding source. Requirements outside of 
the 1937 Housing Act, such as fair housing rules, cannot be waived 
under MTW. In addition, certain sections of the act, including those 
that cover labor requirements and the demolition and disposition of 
public housing, cannot be waived. The 1996 act that created the 
program requires participating agencies to address three purposes and 
meet five requirements. Specifically, the three statutory purposes are 
to (1) reduce costs and achieve greater cost-effectiveness in federal 
housing expenditures, (2) give families with children incentives to 
obtain employment and become self-sufficient, and (3) increase housing 
choices for low-income families. For example, to reduce administrative 
costs MTW agencies can reduce the frequency of income verifications 
for households with fixed incomes. In making these changes, MTW 
agencies must: 

1. serve substantially the same total number of eligible low-income 
families that they would have served had funding amounts not been 
combined; 

2. maintain a mix of families (by family size) comparable to those 
they would have served without the demonstration; 

3. ensure that at least 75 percent of households served are very low 
income;[Footnote 6] 

4. establish a reasonable rent policy to encourage employment and self-
sufficiency; and: 

5. assure that the housing they provide meets HUD's housing quality 
standards. 

The program's ultimate goal is to identify successful approaches that 
can be applied to PHAs nationwide. 

The 1996 act authorized MTW for 30 agencies. HUD invited PHAs to apply 
for the program and selected an initial cohort of 24 PHAs from among 
the respondents. Six more were added as a result of the Jobs-Plus 
initiative.[Footnote 7] Since then, some agencies have opted not to 
participate, and additional agencies have been added to replace them. 
Other agencies have been added through specific appropriations 
language (see figure 1). In addition, some agencies have completed 
their participation and exited the program.[Footnote 8] As of January 
2012, a total of 39 PHAs were authorized to participate, and 35 were 
participating. 

Figure 1: Timeline of Public Housing Agencies Entering and Exiting the 
MTW Demonstration Program: 

[Refer to PDF for image: time line] 

1996: 
MTW authorized by the Omnibus Consolidated Rescissions and 
Appropriations Act of 1996; the act authorized 30 MTW agencies (24 
PHAs were selected and 6 slots were filled through HUD's Jobs-Plus 
initiative). 

1999: 
MTW implemented; 5 of the 24 PHAs originally selected opted not to 
participate; 1 Jobs-Plus selectee opted not to participate; 2 agencies 
specifically named and authorized to join in the 1999 Appropriations 
Act. 

2000: 
HUD selected 6 MTW agencies through a competitive selection process; 1 
MTW agency added as authorized by 1996 statute.  

2003: 
6 (Jobs-Plus) selectees completed participation.  

2004: 
3 MTW agencies completed participation.  

2008: 
1 MTW agency added as authorized by 1996 statute (slot available from 
Jobs-Plus); 1 former MTW agency returned; 4 PHAs specifically named 
and authorized to join in the 2008 Appropriations Act. 

2009: 
3 PHAs added by HUD through a competitive selection process (the 2009 
Appropriations Act authorized the addition of 3 more MTW agencies, 
expanding the total number of authorized agencies to 33)[A]. 

2010: 
2 MTW agencies selected through a competitive process (the 2010 
Appropriations Act authorized the addition of 3 more agencies, 
expanding the total number of authorized agencies to 36)[B]. 

2011: 
3 MTW agencies authorized (the 2011 Continuing Appropriations Act 
authorized the addition of 3 more MTW the total number of agencies, 
expanding the total authorized agencies to 39)[B]. 

HUD = Department of Housing and Urban Development; 
MTW = Moving to Work; 
PHA = public housing agency. 

Source: HUD.                      

[A] The act specified that the additional agencies must be considered 
"high performers" (as designated by HUD's Public Housing Assessment 
System) with current HOPE VI grants and no more than 5,000 units. 
HUD's Public Housing Assessment System assigns a performance 
designation to each PHA that is based on the physical condition of the 
properties, the agency's financial condition and management 
operations, and a survey of residents' satisfaction with services 
provided. HOPE VI grants fund the revitalization of severely 
distressed public housing. 

[B] The 2010 and 2011 Appropriations Acts specified that the 
additional agencies must be considered "high performers" with no more 
than 5,000 units. 

[End of figure] 

MTW agencies do not receive special funding allocations. Rather, they 
receive funds from the three traditional primary funding sources 
(public housing capital funds, public housing operating funds, and 
Housing Choice Vouchers).[Footnote 9] Traditionally, PHAs have been 
required to use the funds from each separate source only for specific 
purposes, but MTW agencies may combine the money from the three 
sources and use them for a variety of HUD-approved activities. This 
fungibility is intended to give MTW agencies greater flexibility. For 
example, public housing operating funds are traditionally used to make 
up the difference between the rents charged for units and the actual 
cost of operating them. Capital funds are used for modernization 
efforts and management improvements, while voucher funds provide 
rental assistance in the private market. However, by combining funds 
an MTW agency may use public housing capital funds to issue additional 
vouchers or use voucher funds to develop more public housing to better 
fit the needs of its community. MTW agencies also have the authority 
to use their funds to implement innovative activities that differ from 
traditional housing assistance. For instance, an MTW agency can use 
funds to replace public housing with mixed-income communities or reach 
special-needs populations, such as the homeless, using vouchers paired 
with supportive services. 

A Standard Agreement, executed in 2008 to replace individual contracts 
between HUD and participating PHAs, governs the conditions of 
participation in the program.[Footnote 10] HUD enters into this 
agreement with each MTW agency. HUD created the agreement to 
standardize the language in its contracts and its reporting 
requirements and to help create a more operationally sound program. 
The Standard Agreement includes a termination date (the end of each 
agency's 2018 fiscal year) and an attachment that sets out reporting 
requirements (Attachment B). While much of the Standard Agreement is 
the same for all MTW agencies, two sections are tailored to individual 
agencies: a description of the formulas for determining the amounts of 
funding each agency will receive and an optional section that may 
include some agency-specific authorizations. 

The agreement requires all MTW agencies to submit an annual MTW plan 
to HUD for approval and an annual MTW report for acceptance.[Footnote 
11] While the format may vary, HUD has established a standard table of 
contents that lists the information that agencies must include in 
their annual plans and reports. For example, the plan must include, 
among other things, a description of how each planned activity relates 
to at least one of the three purposes of the program; baselines, 
proposed benchmarks, and proposed metrics for assessing the outcomes 
of each activity; citations of the authorizations that give the agency 
the flexibility to conduct the activity; and descriptions of required 
rent reform activities.[Footnote 12] In addition, the plan must 
include a certification that the agency published a notice of plans to 
hold a public hearing on the plan, made the agency's annual plan 
available for public inspection, and conducted a public hearing to 
discuss the annual plan prior to its approval. Similarly, the Standard 
Agreement outlines the information that MTW agencies are required to 
include in annual MTW reports. These reports must include, for 
example, detailed information on the impact of each activity, 
including comparisons of actual outcomes to the benchmarks proposed in 
the annual plan. If the agencies do not achieve the benchmarks or the 
activities are determined to be ineffective, the MTW agency is 
required to describe the challenges, and, if possible, identify 
alternative activities that may be more effective. MTW agencies also 
are required to self-certify that they are in compliance with three 
statutory requirements: assisting substantially the same total number 
of eligible low-income families that they would have served had 
funding amounts not been combined; maintaining a mix of families (by 
family size) comparable to those they would have served had funding 
amounts not been combined under the demonstration; and ensuring that 
at least 75 percent of households served are very low income. 

As well as meeting the requirements in the Standard Agreement, MTW 
agencies must submit tenant-related data into the Moving to Work 
section of the Public and Indian Housing Information Center (MTW-PIC). 
According to HUD officials, the MTW-PIC module was created because the 
standard PIC system that non-MTW agencies use to report tenant data 
could not accommodate some of the activities allowed under MTW, such 
as less frequent tenant income recertifications and rent calculations 
that vary from HUD's standard calculations. The MTW-PIC module was 
created in 2007, and most MTW agencies had transitioned to it by 2008. 

The Office of Public Housing Investments within the Office of Public 
and Indian Housing at HUD headquarters is the designated program 
office for the MTW demonstration program. Within the Office of Public 
Housing Investments is an MTW Office that includes a program director 
and four coordinators who are each assigned to a specific group of MTW 
agencies. The MTW Office is responsible for, among other things, 
processing, reviewing, and approving all annual plans submitted by MTW 
agencies; establishing guidelines for MTW agencies; monitoring 
approved activities and accomplishments; and accepting annual reports. 
Individual MTW coordinators facilitate the reviews of planned and 
implemented activities and are responsible for coordinating with other 
HUD offices, including local HUD field offices, to obtain additional 
input on MTW agencies' planned activities and accomplishments. In 
January 2011, the Office of Public Housing Investments signed a 
Memorandum of Understanding with HUD's Office of Field Operations to 
increase collaboration and formally describe the roles and 
responsibilities of the MTW Office and HUD field office staff. Per 
this memorandum, field office staff assist the MTW Office by reviewing 
and providing input on annual plans and reports, ensuring that 
agencies are reporting tenant information, and participating in annual 
site visits. 

HUD Has Not Identified Standard Performance Data and Indicators Needed 
to Evaluate the Program: 

MTW agencies provide descriptions of their activities and performance 
information in their annual reports to HUD. They show how the 
activities link to the program's statutory purposes in their annual 
plans, as required, and sometimes also in their annual reports. 
However, the type of performance information they provide varies, and 
HUD has provided limited guidance. While varied information on 
individual activities is available, a comprehensive evaluation of the 
MTW program is lacking, in part because HUD does not have a plan for 
identifying and analyzing standard performance data and has not 
established performance indicators for the MTW program as a whole. 
[Footnote 13] Further, HUD does not have a systematic process for 
identifying lessons learned by individual MTW agencies that can be 
replicated at other PHAs. 

Although Information on Activities Is Generally Available at the 
Agency Level, It Varies Due to HUD's Limited Guidance: 

MTW agencies report information on specific activities, including 
descriptions, in their annual reports. Agencies are required in their 
annual plans to link each of their proposed activities to one of the 
program's three statutory purposes, and some agencies also show links 
between ongoing activities and statutory purposes in their annual 
reports. The three statutory purposes are to reduce costs and achieve 
greater cost-effectiveness in federal housing expenditures, give 
families with children incentives to obtain employment and become self-
sufficient, and increase housing choices for low-income families. 
According to the most recently available annual reports, 30 agencies 
have over 360 ongoing activities, including rent reform initiatives 
and work requirements (see table 1).[Footnote 14] 

Table 1: Examples of MTW Activities, by Statutory Purpose: 

Purpose 1: Reduce costs: 

Activity: Biennial recertifications; 
Description (example): Allow elderly and disabled residents, who have 
minimal income and experience few changes in family composition, to be 
recertified every other year instead of annually. 

Activity: New inspections protocol; 
Description (example): Replace annual inspections of all properties in 
the tenant-and project-based voucher programs with inspections of a 
certain percentage of participating properties. 

Activity: Standard deductions for medical expenses (rent reform); 
Description (example): Replace itemized medical and child care 
deductions with a standard deduction in determining rent payments. 

Purpose 2: Encourage self-sufficiency: 

Activity: Minimum rents (rent reform); 
Description (example): Require residents to pay at least a minimum 
rent (MTW agencies can exceed the statutory limit of $50). 

Activity: Work requirement; 
Description (example): Require residents to work or engage in a 
learning or training activity, meet a minimum income level to receive 
housing assistance, or enter and adhere to a local welfare-to-work 
program. In some cases, residents may not be unemployed for more than 
90 days. 

Activity: Enhanced supportive services; 
Description (example): Offer residents supportive services, such as on-
site employment counseling and skill building, classes in household 
budgeting and parenting, and child-care and transportation services. 

Purpose 3: Increase housing choices: 

Activity: Fund affordable housing development activities; 
Description (example): Leverage funds to preserve affordable housing 
resources and create new affordable housing opportunities. 

Activity: Sponsor-based voucher program; 
Description (example): Provide vouchers to service providers who work 
with hard-to-house households to provide intensive supportive services. 

Activity: Site-based waiting list; 
Description (example): Allow residents to apply for the waiting list 
for specific sites or communities where they want to live. 

Source: Most recent MTW annual reports as of January 2012. 

Note: The links between the activities and statutory purposes were 
determined by MTW agencies. 

[End of table] 

According to the most recent annual reports (and corresponding plans) 
for 30 MTW agencies, agencies associated the largest percentage of 
ongoing activities (41 percent) with the statutory purpose of reducing 
costs and improving cost-effectiveness (see figure 2). For example, 
agencies associated changes in certification schedules, inspection 
protocols, and medical deductions with reduced costs. The agencies 
linked 30 percent of their ongoing activities to the statutory purpose 
of increasing housing choices and 24 percent to encouraging self-
sufficiency. The agencies did not link 4 percent of their ongoing 
activities to any purpose in either their most recent annual plan or 
report.[Footnote 15] 

Figure 2: Number of Reported MTW Activities Linked to Each of the 
Statutory Purposes: 

[Refer to PDF for image: pie-chart] 

Purpose 1: Reduce costs: 41%; 
Purpose 2: Encourage self-sufficiency: 24%; 
Purpose 3: Increase housing choices: 30%; 
Not linked: 4%. 

Source: GAO analysis of the most recent MTW annual reports as of 
January 2012 and corresponding annual plans. 

Note: Numbers do not add to 100 percent because of rounding. 

[End of figure] 

In its Standard Agreement, HUD requires agencies to include in their 
annual reports performance information on the impact of each 
implemented activity, including describing the metrics used to assess 
outcomes and comparing actual performance with proposed 
benchmarks.[Footnote 16] While HUD did not define these terms in the 
agreement, it defined them in 2009 training materials. In these 
materials, HUD defined a metric as the "unit of measure that 
quantifies the changes that might occur as a result of the MTW 
activity" and a benchmark as the "projected outcome of the MTW 
activity." Further, in these 2009 training materials HUD defined an 
outcome as the "actual, measured result of the implemented activity." 
As examples, the training materials stated that a metric could be the 
hours of staff time saved, a benchmark could be the number of 
anticipated staff hours saved, and the outcome could be staff hours 
actually saved. HUD directs agencies to develop their own metrics and 
benchmarks for each activity based on local and community 
standards.[Footnote 17] Our analysis of the most recent annual reports 
for 30 MTW agencies showed that the agencies reported performance 
information for 91 percent of the ongoing activities included in the 
reports and used over 1,000 metrics to assess these activities. 
[Footnote 18] MTW agencies collectively met the benchmarks associated 
with 40 percent of these metrics and fell short of meeting 17 percent 
of them.[Footnote 19] For 30 percent of the metrics, it was too soon 
to determine if the benchmarks had been met because the activities 
were not yet completed. For the remaining 13 percent, information 
(either the benchmark or performance data) was lacking to determine 
whether the benchmark was met. 

While MTW agencies are generally devising their own metrics for 
activities and reporting performance information, the usefulness of 
this information is limited because, in some cases, it is not outcome-
oriented. Our analysis of the most recent annual reports for 30 MTW 
agencies showed that the type of information that agencies reported on 
the impact of their activities varied. For example, for similar 
activities designed to promote family self-sufficiency, one MTW agency 
reported only the number of participants, which is generally 
considered an output, and another did not provide any performance 
information.[Footnote 20] In contrast, a third agency reported on, 
among other things, the average income of program graduates, which we 
consider an outcome. 

Internal control standards state that good guidance (information and 
communication) is a key component of a strong internal control 
framework and that there is a need for clear documentation.[Footnote 
21] To be consistent with the GPRA Modernization Act of 2010, HUD's 
guidance on reporting performance information should indicate the 
importance of outcome-oriented information.[Footnote 22] Specifically, 
the act states that an agency should establish efficiency, output, and 
outcome indicators for each program activity. Furthermore, Office of 
Management and Budget guidance on implementation of the act states 
that quantitative and outcome-focused measures are preferred.[Footnote 
23] 

At the time of our review, HUD's guidance did not specify that 
agencies should report quantifiable and outcome-oriented performance 
information. According to the Director of the MTW Office, Attachment B 
of the Standard Agreement is the most current guidance on the 
information that agencies should report in their annual reports. It 
simply states that agencies are to provide detailed information on the 
impact of the activity and compare it against the proposed benchmarks 
to assess outcomes, including whether an activity is on schedule. The 
attachment does not define terms or set expectations for the type of 
information to be reported. After the Standard Agreement was executed 
in 2008, HUD conducted training for participating agencies. As 
previously discussed, the 2009 training materials defined key terms 
such as a metric and outcome and outlined steps agencies could take to 
evaluate their activities. HUD also encouraged the MTW agencies to use 
metrics and benchmarks that did not focus on the number of individuals 
participating in an activity but rather on the objectives of the 
activity and to report quantifiable information. While HUD has posted 
the 2009 training materials on its website, these materials have not 
been incorporated into Attachment B of the Standard Agreement. 

According to the Director of the MTW Office, HUD has not made its 
guidance more specific because agencies are implementing a wide 
variety of activities and thus require some reporting flexibility. We 
acknowledge the need for flexibility, but it is important that HUD 
require agencies to report at least some outcome-oriented performance 
information. Without more specific guidance on the reporting of 
performance information, HUD cannot be assured of collecting 
information that reflects the outcomes of individual activities. Such 
information would help HUD assess the demonstration program and 
whether the activities are furthering program purposes. 

Design and Data Weaknesses Have Prevented a Comprehensive Evaluation 
of MTW: 

As we have previously reported, obtaining performance information from 
demonstration programs that are intended to test whether an approach 
(or any of several approaches) can obtain positive results is 
critical.[Footnote 24] This information is needed to help determine 
whether the program has led to improvements consistent with its 
purposes. HUD has sponsored three broad reviews of the MTW program, 
but these studies are not comprehensive evaluations because of data 
limitations, among other things.[Footnote 25] A 2004 Urban Institute 
evaluation of the MTW program found that most MTW agencies reported 
modest benefits from activities related to administrative streamlining 
and that these results were often not as dramatic as the agencies had 
anticipated.[Footnote 26] The study also noted the difficulty in 
determining whether MTW activities related to employment and income 
had any independent effect and that MTW activities resulted in both 
greater and more limited housing choice.[Footnote 27] 

A second study, conducted in 2007 by Applied Real Estate Analysis, 
Inc. and the Urban Institute, reviewed eight MTW agencies that had 
placed limits on the length of time that residents could receive 
housing assistance.[Footnote 28] The study found that each of the 
approaches varied and concluded that only limited information was 
available with which to evaluate outcomes or establish cause-and-
effect relationships between agencies' policies and recipients' 
experiences. It noted that there were significant limitations to what 
could be learned from these experiences because no evaluative 
framework had been built into the program. Finally, a 2010 HUD Report 
to Congress found that the effects of many MTW activities, especially 
as they related to residents, could not be conclusively identified 
because of the variety of and differences in the activities and 
metrics that MTW agencies were implementing. However, the report did 
identify some results concerning agencies' ability to more efficiently 
allocate resources and engage in strategic long-term 
planning.[Footnote 29] For instance, the study noted that some 
agencies had seen positive effects from combining their traditional 
sources of funding and streamlining their operations--for example, by 
simplifying their housing quality inspections. 

These three studies of the MTW program and our work have identified 
several challenges that have hindered evaluation efforts. These 
challenges include the way the program was initially designed and the 
resulting lack of standard performance data as well as the lack of 
performance indicators for the program as a whole. 

Lack of Program Evaluation Design and Standard Performance Data: 

HUD has taken steps to address the problems noted with MTW's initial 
design and the lack of standard data; however, it has not analyzed the 
data it currently collects or determined whether these data are 
sufficient to evaluate similar activities and the program as a whole. 
As we have previously reported, comparable data are essential to a 
full analysis of programs that incorporate a variety of activities. 
[Footnote 30] We also noted that obtaining performance information 
from demonstration programs that are intended to test whether an 
approach (or any of several approaches) can obtain positive results is 
critical. Finally, we have reported that agencies need to identify any 
data that will be needed to assess the effectiveness of program 
regulations.[Footnote 31] 

Researchers and others have noted the limitations that the program's 
initial design posed to evaluation. In the early years of MTW, 
rigorous evaluation strategies were not required, and the program 
lacked a research design that would have helped in determining 
baseline information.[Footnote 32] The 2004 Urban Institute review of 
MTW concluded that there were limits to what could be learned from its 
review for a variety of reasons, such as the inability to separate 
individual components of agencies' MTW activities for 
analysis.[Footnote 33] As a result, the report is mainly descriptive 
and qualitative.[Footnote 34] HUD's 2010 Report to Congress noted that 
because rent reform activities varied greatly and were not implemented 
using a controlled experimental methodology, the authors were unable 
to recommend specific reforms as best practices.[Footnote 35] 

To help evaluate aspects of the MTW program moving forward, MTW 
officials have added requirements for new agencies. According to HUD's 
2010 Report to Congress, the three agencies admitted to the MTW 
program in 2009 had strong evaluation components.[Footnote 36] Two of 
these agencies have commitments from local universities to evaluate 
their programs. Additionally, HUD has required the two newest agencies 
to participate in a controlled rent reform study. However, these 
improvements will not help evaluate the program as a whole or the 
activities implemented by the 30 other MTW agencies. 

Likely due to the absence of an evaluation framework for the MTW 
program, researchers have noted the lack of standard performance data 
needed to evaluate similar activities and the program as a whole. The 
2004 Urban Institute study noted that the lack of consistent data on 
resident characteristics, incomes, and rent payments prevented the 
authors from being able to determine whether individual agencies were 
able to achieve the goal of increasing self-sufficiency.[Footnote 37] 
Similarly, the HUD Inspector General reported in 2005 that HUD lacked 
the empirical data needed to assess the program as a whole.[Footnote 
38] 

Since these reports, HUD has started collecting some additional data 
from MTW agencies, but it has not yet analyzed the data. HUD created 
the MTW-PIC module to collect tenant characteristics such as household 
size, income, and educational attainment. However, according to MTW 
officials, HUD has not used these data to analyze the program's 
effects, such as changes in resident income. In addition, HUD's 
Standard Agreement has required agencies since 2009 to provide 
information in their annual reports on the impact of activities, 
including benchmarks and metrics. While these reports are informative, 
they do not lend themselves to quantitative analysis because the 
reporting requirements do not call for standardized data, such as the 
number of residents that found employment. 

In addition, whether these data are sufficient to assess similar 
activities and the program as a whole is not clear, and HUD has not 
identified the data it would need for such an assessment. For example, 
neither MTW-PIC nor annual reports capture standard data on 
implemented activities. Further, according to the Director of the MTW 
Office, MTW-PIC does not include information on individuals who 
receive nontraditional services from an MTW agency, such as homeless 
assistance or case management. Representatives from MTW agencies have 
suggested that HUD should collect some standard data for similar 
activities. For example, they noted that if HUD required all agencies 
that implemented rent reform activities to report standard data, the 
results of these efforts could be analyzed even if the specific 
activities varied. The Director of the MTW Office also noted that MTW-
PIC was a potential tool for collecting and analyzing standard 
demographic data. 

The MTW Office has recently developed a statement of work for an 
evaluation of the program, but HUD has not allocated funding for the 
study, according to the Director of the MTW Office. Among other 
things, the proposed evaluation is intended to assess the current 
state of the MTW demonstration and determine the extent to which the 
three statutory purposes have been addressed. The study is also 
expected to include an analysis of outcomes associated with specific 
activities and the demonstration as a whole to identify which MTW 
activities are appropriate for expansion to all PHAs. However, the 
approach envisioned may be limited because it would primarily rely on 
existing data sources. Until HUD develops and implements a plan (that 
includes the identification of standard data) to quantitatively assess 
similar activities and the MTW program as a whole, HUD cannot 
determine their effectiveness. While such analyses may be challenging, 
they would enhance HUD's ability to rigorously assess the 
demonstration. 

Lack of Performance Indicators: 

HUD has not established performance indicators for the MTW program as 
a whole. The GPRA Modernization Act of 2010 requires that federal 
agencies establish efficiency, output, and outcome indicators for each 
program activity as appropriate.[Footnote 39] Internal control 
standards also require the establishment of performance indicators. 
[Footnote 40] In addition, we have previously reported that successful 
performance indicators demonstrate results and provide useful 
information for decision making in order to track how programs and 
activities can contribute to attaining an organization's goals and 
mission, among other things.[Footnote 41] 

As previously discussed, MTW agencies set their own performance 
metrics for activities, but HUD has not established performance 
indicators for the program as a whole. HUD's Fiscal Year 2011 Annual 
Performance Plan established agencywide performance indicators but did 
not explicitly connect the MTW program to any of them.[Footnote 42] 
Although there are no specific targets for the program, the Director 
of the MTW Office noted that the program's activities support some of 
the agencywide indicators. Specific performance indicators for the MTW 
program could be based on the statutory purposes. For example, 
agencies could report on the savings achieved (reducing costs) and the 
number of additional households served (increasing housing choices). 
Without performance indicators for the MTW program, however, HUD 
cannot demonstrate the results of the program as a whole. 

HUD Lacks a Systematic Process for Identifying Lessons Learned: 

While HUD has identified some lessons learned on an ad hoc basis, it 
does not have a systematic process in place for identifying such 
lessons. We have previously reported that obtaining impact information 
from demonstration programs that are intended to test whether an 
approach (or any of several approaches) can obtain positive results is 
critical.[Footnote 43] This information should be gathered to help 
determine whether programs have led to improvements consistent with 
their purposes. 

Since 2000, HUD has identified some activities that could potentially 
be replicated by other PHAs. A HUD-sponsored contractor developed five 
case studies from 2000 to 2002 that were intended to describe some of 
the issues, successes, and challenges involved in implementing the MTW 
demonstration. Four of the case studies were developed around specific 
MTW sites, while the fifth provided an overview of the block grant 
approach. Further, from June 2008 to April 2009 HUD identified 
practices that, with statutory or regulatory changes, could be 
replicated at other PHAs nationwide. These practices, which are posted 
on HUD's website, included implementing savings accounts for public 
housing and voucher recipients to promote resident savings. Most 
recently, HUD's 2010 Report to Congress described promising policies, 
practices, and concerns.[Footnote 44] In addition, officials from some 
of the MTW agencies we interviewed noted that HUD officials had shared 
information on activities that had shown positive effects during site 
visits, quarterly phone calls, newsletters, and annual conferences. 
Finally, HUD's statement of work for its proposed evaluation of the 
MTW program includes the creation of five case studies that would 
review MTW flexibilities. 

However, these efforts have shortcomings. In most cases, the practices 
chosen were based on the opinions of HUD or contracted staff and 
largely involved anecdotal (or qualitative) data rather than 
quantitative data. The lack of standard performance data has affected 
HUD's ability to systematically identify lessons learned. In its 2005 
report on the MTW program, the HUD Inspector General noted that the 
lack of data on the program made it difficult to identify activities 
that could be considered models for addressing the three statutory 
purposes or that could be used to show the importance of individual 
policy changes.[Footnote 45] Further, HUD has not established 
criteria, such as demonstrated performance, for identifying lessons 
learned. Finally, HUD has not made regular efforts to review and 
identify lessons learned. Because HUD does not currently have a 
systematic process for identifying lessons learned, it is limited in 
its ability to promote useful practices that could be implemented more 
broadly. 

HUD Generally Follows Its Monitoring Policies and Procedures, but 
Could Strengthen Them: 

HUD has policies and procedures in place to monitor MTW agencies. 
First, HUD requires program staff to review and comment on agencies' 
annual plans and reports. Second, staff review tenant data submitted 
by MTW agencies. Third, program staff conduct annual site visits at 
each participating agency to provide technical assistance and program 
updates. HUD generally follows these policies and procedures, which 
focus on technical assistance rather than compliance. Due in part to 
this focus, HUD's policies and procedures have several key weaknesses. 
Specifically, HUD has not clarified program terminology, ensured that 
each MTW agency is meeting statutory requirements, performed an annual 
risk assessment, or developed policies and procedures to verify the 
accuracy of key information that MTW agencies self-report. 

HUD Reviews Agency Information and Conducts Annual Site Visits: 

HUD's monitoring policies and procedures for the MTW program are 
contained in a desk guide, which describes the roles and 
responsibilities of HUD staff in reviewing annual plans and reports 
and data submissions, making site visits, and performing other 
monitoring activities. In January 2011, HUD's Office of Public Housing 
Investments and Office of Field Operations signed a Memorandum of 
Understanding documenting the framework for headquarters and field 
staff to follow in overseeing MTW agencies. According to the 
memorandum, the MTW Office (within the Office of Public Housing 
Investments) is responsible for oversight of the MTW program. In many 
cases, the MTW Office works with field offices to jointly develop 
responses to MTW agency issues. Further, a MTW Working Group--
consisting of representatives from Public and Indian Housing programs, 
the Real Estate Assessment Center, and the Office of Policy 
Development and Research--was established to assist with the annual 
plan and report review process.[Footnote 46] As part of the 
memorandum, the Offices of Public Housing Investments and Field 
Operations agreed to the protocols set forth in the desk guide. 

HUD staff from the MTW Office and field offices and the MTW Working 
Group share responsibility for reviewing and commenting on 
participating agencies' annual plans and reports. The Standard 
Agreement (Attachment B) outlines the requirements for annual plans 
and reports that agencies must submit. MTW coordinators, who are each 
responsible for a specific number of MTW agencies, have the lead role 
in reviewing annual plans and reports to determine if they meet the 
requirements of Attachment B and obtaining input from other HUD staff, 
including field offices and the MTW Working Group. Field offices are 
required to review the annual plans and reports submitted by the MTW 
agencies in their jurisdictions and provide their assessment to the 
MTW coordinator. Similarly, the MTW Working Group reviews and provides 
comments to the MTW coordinator. Coordinators summarize the comments 
from the field offices and MTW Working Group and send them to the 
agencies. The coordinators and field office staff work with MTW 
agencies to resolve any outstanding issues. Once such issues have been 
resolved, the MTW Office approves annual plans and accepts annual 
reports.[Footnote 47] 

In 2009, the MTW Office developed review procedures and checklists for 
the coordinators and field office staff to use when reviewing annual 
plans and reports. The checklists provide a framework for MTW 
coordinators to determine whether the annual plans and reports are 
complete and consistent with the requirements of Attachment B. More 
specifically, coordinators are to review plans to ensure that they 
include how proposed activities relate to at least one of the 
statutory purposes and show the anticipated impact on the related 
purpose; corresponding baselines, benchmarks, and metrics; and 
information on rent reform initiatives, among other items. In 
addition, they are to ensure that the plans include a certification 
that the agency published a notice of plans to hold a public hearing 
to discuss its annual plan, made the agency's annual plan available 
for public inspection, and conducted a public hearing to discuss the 
annual plan prior to its approval, among other things.[Footnote 48] 
Further, coordinators are to review reports to ensure that they 
describe the impact of each implemented activity, explain any 
benchmarks that were not achieved and any revised benchmarks or 
metrics, and include the agency's certification that it has met three 
of the five statutory requirements.[Footnote 49] However, these 
procedures do not require MTW coordinators to verify each agency's 
certification that it has met the three statutory requirements. 

Interviews with MTW coordinators and field staff and documentation for 
our sample of seven MTW agencies indicated that HUD generally followed 
these procedures. Documentation we reviewed for the agencies in our 
sample showed that the coordinators generally completed checklists 
while reviewing annual plans and reports. For example, coordinators 
verified that all ongoing activities were reported, ensured the agency 
included its certification that it had met three of the five statutory 
requirements, and made certain the agency certified that it had held a 
public hearing on its annual plan, among other requirements. 
Coordinators also provided comments to agency staff on annual plans 
and reports.[Footnote 50] Once it had determined that the agency had 
addressed all of the comments, the MTW Office notified the agency in 
writing that its plan had been approved and report accepted. 

HUD field staff also monitor MTW agencies' compliance with data 
submission requirements, including the requirement to report 
information on resident characteristics in MTW-PIC. HUD requires MTW 
agencies to report timely, accurate, and complete data on at least 95 
percent of the families receiving housing assistance.[Footnote 51] To 
monitor compliance, field staff review monthly reports from MTW-PIC 
that summarize each participating agency's tenant data reporting 
rates. The reports we reviewed indicated that agencies were complying 
with the reporting requirements. For example, in January 2012, MTW 
agencies overall achieved a 100-percent tenant data submission rate. 

In addition, HUD conducts annual site visits to provide technical 
assistance to each MTW agency. The MTW Office and the local field 
office conduct these visits jointly.[Footnote 52] The MTW Office (in 
particular the coordinator assigned to the agency) takes the lead role 
in conducting the visit, including preparing the agenda, coordinating 
with the local HUD field office, and working with the MTW agency to 
select properties to visit. According to HUD officials, the primary 
objective of the site visit is to provide technical assistance and 
build a working relationship with the participating agencies, not to 
assess compliance with statutory requirements. However, HUD officials 
stated that if compliance issues with statutory purposes are found, 
HUD staff address these issues during the site visit, and coordinators 
often develop timelines for the agency to come into compliance. 

Our analysis of documentation of site visits to participating agencies 
indicated that MTW Office and field staff generally followed HUD's 
annual site visit procedures. Specifically, analysis of site visit 
reports indicated that HUD officials generally discussed the 
effectiveness of activities and helped resolve any outstanding issues. 
For example, as a result of site visits, HUD staff recommended that an 
agency include cost-saving measures in its annual plan, requested 
clarification of output measures, and encouraged one agency to submit 
articles to the MTW newsletter to share its experiences on how rent 
reform encouraged self-sufficiency. 

Interviews with our sample of MTW agencies and corresponding field 
office officials also indicated that HUD was following its policies 
and procedures for annual site visits. MTW agency officials we spoke 
with indicated that the site visits were generally beneficial because 
they provided an opportunity for in-person discussions that helped 
facilitate communication with HUD. HUD's field office staff noted 
their active involvement over the years, which had become more defined 
with the issuance of the desk guide in 2011. According to the Director 
of the MTW Office, the office is considering conducting future site 
visits using a risk-based approach. Using this approach, HUD would 
conduct site visits less frequently but would focus on larger agencies 
that had implemented a wide range of complex activities and newly 
admitted agencies that were implementing new activities. 

To foster information sharing across agencies and provide technical 
assistance, HUD employs a number of additional strategies. For 
example, HUD hosts annual conferences to share information with MTW 
agencies and facilitate information sharing among agencies. The 
conferences cover a variety of topics, and all participating MTW 
agencies are invited to attend. For example, the 2011 conference 
focused on effectively managing funds in a challenging budgetary 
environment. HUD also has engaged participating agencies in quarterly 
conference calls and other training related to program changes such as 
the conversion from PIC to MTW-PIC and the transition to the Standard 
Agreement. Further, HUD issues notices on various topics, such as MTW 
reporting requirements, and publishes quarterly newsletters that 
highlight activities relating to each statutory purpose, among other 
topics. HUD also publishes each agency's annual report and 
researchers' evaluations of MTW activities on its website. 

HUD's Monitoring Policies and Procedures Have Several Key Weaknesses: 

Although HUD follows the policies and procedures that it has in place, 
it could do more to ensure that MTW agencies are demonstrating 
compliance with statutory requirements and to identify possible risks 
relating to activities implemented by each agency, among other things. 
First, HUD has not issued guidance to participating agencies 
clarifying key program terms, including definitions of the purposes 
and statutory requirements of the MTW program. Internal control 
standards require the establishment of clear, consistent goals and 
objectives.[Footnote 53] As previously noted, MTW authorizing 
legislation established three purposes for the program, and agencies 
must link each of their activities to one of these purposes. However, 
HUD has not clearly defined what the language in some of these 
purposes means, such as "increasing housing choices for low-income 
families." MTW agencies have linked activities to this purpose that 
range from using block grant funding to support homeownership programs 
to requiring applicants to complete a renter education program to 
establishing a prisoner re-entry housing program. HUD noted the lack 
of a clear definition in its 2010 Report to Congress but continued to 
require that MTW agencies link activities to this purpose.[Footnote 
54] According to MTW officials, they have not defined what is meant by 
"increasing housing choices" so that agencies have the ability to 
define this term in a manner that fits their local needs. In addition, 
HUD has not clarified what is meant by "serving a comparable mix of 
families" but also requires agencies to comply with this requirement. 
MTW agencies we spoke with described varying interpretations of this 
requirement. For example, officials from one agency told us that they 
observed how family sizes changed in their community and compared 
those changes to changes in families within the MTW program, using 
community survey data and data from the agency's internal system. 
Officials from another agency we spoke with said that over time it had 
become increasingly difficult to determine compliance with this 
statutory requirement. 

HUD has recently taken steps to clarify some terminology, explaining 
how agencies can certify that at least 75 percent of the families they 
serve have very low incomes and that they are serving substantially 
the same number of households under MTW as they did before the 
program. In addition, HUD is revising its reporting requirements for 
MTW agencies. As part of this process, HUD officials told us that they 
plan to update their guidance to more completely collect information 
related to the program's statutory purposes and requirements. They 
acknowledged that the guidance could be strengthened to require MTW 
agencies to provide their agency-specific definition for the three 
statutory purposes. As a first step, they noted that they planned to 
require agencies to define "self-sufficiency" by either choosing one 
of the definitions provided by HUD or creating their own. Similarly, 
the officials stated that they would consider requiring MTW agencies 
to choose between using HUD's definition of increasing housing choices 
or creating their own definition. Although a step in the right 
direction, allowing MTW agencies to create their own definitions of 
key terms would make it difficult to assess the effectiveness of 
efforts to address statutory purposes. HUD officials also said that 
the revised guidance would provide standardized tables for agencies to 
report data related to the requirement to serve a comparable mix of 
families. Until HUD clearly defines what is meant by all of the 
statutory purposes and requirements of the MTW program, HUD cannot 
effectively determine whether agencies are addressing these purposes 
and meeting requirements. 

Second, HUD has only recently assessed agencies' compliance with two 
self-certified requirements and has not assessed compliance with the 
third. Internal control standards require control activities to be in 
place to address program risks.[Footnote 55] In addressing these 
risks, internal control guidance states that management should 
formulate an approach for assessing compliance with program 
requirements.[Footnote 56] While HUD has recently made efforts to 
assess agencies' compliance with two of the three self-certified 
requirements, it does not have a process in place to systematically 
review compliance with all three requirements. In 2011, HUD for the 
first time assessed participating agencies' compliance with the 
requirement to assist "substantially the same" number of eligible 
families that would have been served in the absence of MTW. HUD 
collected data from MTW-PIC, the Voucher Management System, and each 
participating agency's most recent annual report on the number of 
public housing units occupied, vouchers utilized, and other families 
housed and used a formula to compare these data with similar data 
reported before MTW.[Footnote 57] HUD and MTW agency staff we 
interviewed told us that they worked together to discuss discrepancies 
in the calculations. According to the Director of the MTW Office, 
agencies were in compliance with this requirement if they were serving 
at least 95 percent of the number of families in their baseline 
figure. HUD's recent review of each agency's baseline calculation 
indicated that all but one of the agencies were in compliance. Also in 
2011, HUD reviewed MTW-PIC data for the first time to determine 
agencies' compliance with the requirement that at least 75 percent of 
assisted residents be very low income. HUD's analysis of MTW-PIC data 
showed that, as of September 2011, 91 percent of the residents served 
by MTW agencies fell into this category. 

While HUD has taken steps to assess compliance with these two 
statutory requirements, it has not yet developed a methodology for 
assessing agencies' compliance with the requirement to maintain a 
comparable mix of families. The Director of the MTW Office 
acknowledged that self-certifications were not the best means of 
ensuring compliance and told us that the planned revisions to the 
reporting requirements for MTW agencies would help assess compliance 
with the requirements to maintain a comparable mix of families and 
ensure that at least 75 percent of families assisted are very low 
income. Without a process for systematically assessing compliance with 
statutory requirements, HUD lacks assurance that agencies are 
complying with them. 

Third, HUD has not performed an annual assessment of program risks. 
Internal control standards state that an agency should have a risk 
assessment plan that considers internal and external risk factors and 
establishes a control structure to address those risks.[Footnote 58] 
The standards also state that managers should focus on control 
activities to address risks that may involve verifications, 
performance reviews, and documentation, among other things. HUD's own 
internal control standards also require its program offices to perform 
an annual risk assessment of their programs or administrative 
functions using a HUD risk-assessment worksheet.[Footnote 59] These 
standards also stress the importance of performing a risk assessment 
when there are significant program changes. According to the Director 
of the MTW Office, the office has not performed an annual risk 
assessment for the MTW program because it was not aware of this 
requirement. 

In addition, HUD's procedures for monitoring the MTW program are not 
risk-based, meaning that HUD has not tailored its monitoring efforts 
to reflect the perceived risk of each MTW agency.[Footnote 60] MTW 
agencies, unlike other PHAs, are exempt from receiving performance 
scores that reflect their perceived level of risk. While monitoring 
procedures are not risk-based, the Director of the MTW Office stated 
that his office would become aware of risks from HUD's field office 
staff, which have routine responsibility for reviewing financial 
audits and Office of Management and Budget compliance audits. As 
previously discussed, HUD is considering moving toward conducting risk-
based site visits.[Footnote 61] In addition, according to HUD 
officials, the office is considering other methods to more rigorously 
analyze MTW agency risk factors. By not performing an annual risk 
assessment and implementing a risk-based approach to monitoring MTW 
agencies, HUD lacks assurance that it has properly identified and 
addressed risks that may prevent participating agencies from 
addressing program purposes and meeting statutory requirements. HUD 
also lacks assurance that it is efficiently using its limited 
monitoring resources. 

Finally, HUD does not have policies or procedures in place to verify 
the accuracy of key information that MTW agencies self-report. 
Internal control standards and guidance emphasize the need for federal 
agencies to have control activities in place to help ensure that 
program participants report information accurately.[Footnote 62] HUD 
requires agencies to report on benchmarks, metrics, and performance 
information in their annual reports but does not have policies or 
procedures in place to verify the accuracy of this self-reported data. 
Further, MTW coordinators do not verify this information during their 
annual site visits or during their annual report review process, 
preventing HUD from efficiently using its limited monitoring 
resources. As previously noted, HUD relies on the participating agency 
to submit accurate information. We requested documentation from our 
sample of MTW agencies that supported the outcomes they reported in 
their most recent annual report for a selected activity and received 
various types of support. While this information was sufficient for 
our own review, HUD has not taken similar steps to collect evidence to 
substantiate agencies' self-reported information. According to HUD 
officials, the MTW Office is considering ways to verify the accuracy 
of key information that MTW agencies self-report. Although verifying 
all of the self-reported information may be challenging, GAO guidance 
on data reliability recommends tracing a sample of data records to 
source documents to determine whether the data accurately and 
completely reflect the source documents.[Footnote 63] Because HUD does 
not verify the accuracy of any reported performance information, it 
lacks assurance that this information is accurate. To the extent that 
HUD relies on this information to assess program compliance with 
statutory purposes and requirements, its analyses are limited. 

Expanding MTW May Offer Benefits but Also Raises Questions: 

Legislation has been proposed to expand the number of PHAs that can 
participate in the MTW program, and a recent HUD report recommended 
expanding the program up to twice its size. As of March 2012, a 
maximum of 39 PHAs could participate in the program, but a 2011 Senate 
bill would direct HUD to increase that number up to 250.[Footnote 64] 
In addition, legislation has been drafted that would establish MTW as 
a permanent program and eliminate the current restrictions on the 
number of agencies that can participate.[Footnote 65] HUD's 2010 
Report to Congress recommends increasing the number of participating 
agencies to about 60.[Footnote 66] HUD and some stakeholders believe 
that expansion could provide the needed information on the effect of 
the MTW program and allow more PHAs to test innovative ideas, but 
questions remain about the lack of performance information on current 
MTW activities. In addition, alternatives to expansion exist, 
including implementing a more narrowly focused program. 

Current MTW Participants and Affordable Housing Advocates Indicated 
that Expanding MTW Could Help Demonstrate the Program's Effect and 
Increase Innovation: 

According to HUD, some affordable housing advocates, and MTW agencies 
we interviewed, expanding the MTW program could help demonstrate the 
program's effect and increase the number of lessons that can be 
learned from the program. HUD has reported that doubling the number of 
MTW agencies with the use of strategic criteria and program 
implementation could help demonstrate the effects of MTW on a broader 
scale and enable the housing industry to learn even more from the 
demonstration.[Footnote 67] For example, expansion could provide more 
information on how MTW flexibilities would affect a broader group of 
PHAs. The Director of the MTW Office noted that some MTW activities, 
specifically those related to administrative streamlining, had 
influenced the draft Affordable Housing and Self-Sufficiency 
Improvement Act of 2012 (AHSSIA). Some affordable housing advocates 
that we met with emphasized the value of the changes, such as 
decreases in concentrated poverty, that have occurred in some of the 
communities affected by the MTW program and indicated that expansion 
could enable more PHAs to address local needs and therefore benefit 
additional communities. Similarly, officials from MTW agencies that we 
contacted stated that expansion of the program would provide a broader 
testing ground for new approaches and best practices. 

In addition, information from a private research organization, 
affordable housing advocates, and MTW agencies that we met with 
suggests that allowing additional PHAs to participate in the program 
could result in additional opportunities for PHAs to test innovative 
ideas, in part by using the flexibilities provided to tailor their 
housing programs and activities to meet local conditions. In 2004, the 
Urban Institute reported that the local flexibility and independence 
permitted under MTW appeared to allow strong, creative PHAs to 
experiment with innovative solutions to local challenges.[Footnote 68] 
It noted that these PHAs were able to be more responsive to local 
conditions and priorities because they were not as limited by program 
requirements as they might have been before MTW. For example, one 
agency used MTW to increase the proportion of project-based units 
relative to tenant-based assistance. Through MTW, this adjustment was 
done to an extent that would not have been permissible under standard 
rules. The Urban Institute concluded that further deregulation of PHAs 
could yield benefits in terms of design and implementation 
innovations. Officials from an organization that advocates on behalf 
of large PHAs and supports expansion noted that affordable housing 
needs varied by locality and that the MTW program enabled 
participating agencies to design effective approaches based on local 
needs. Similarly, another affordable housing advocacy organization 
told us that they supported expanding MTW not only because it enabled 
participating agencies to tailor activities to local needs but also 
because it involved local communities in the process. 

Officials from several of the MTW agencies we interviewed also noted 
that the MTW program had empowered them to create and implement 
strategies that addressed local issues and said that expanding the 
program would give more PHAs the same flexibility. For example, in one 
northeastern state where the housing stock was relatively old, the MTW 
agency was able to focus on developing new affordable housing. Another 
MTW agency in a western state with mostly newer housing stock chose to 
reduce the frequency of inspections of its properties and focus its 
efforts on administrative streamlining and the disposition of its 
older units. Further, several MTW agencies that we interviewed 
described how they implemented the requirement to establish a rent 
policy that encouraged employment and self sufficiency. For example, 
officials from one MTW agency told us that they believed the 
traditional requirement that residents pay 30 percent of their 
adjusted income in rent was a disincentive to work, because as 
resident income increases so would the payment toward rent. To 
encourage residents to seek work, this agency implemented work 
requirements and a minimum rent. Additionally, some agencies have used 
their MTW status to establish programs that focus on specific 
populations, including working families with children, the elderly and 
disabled, and the homeless. 

Some proponents of expansion that we interviewed also noted that 
expanding the MTW program could provide more PHAs with the ability to 
use funding from different sources more flexibly than possible without 
MTW status. Agencies without MTW status have to implement their 
activities while adhering to the regulations associated with three 
different funding streams, evidence of the fragmented nature of 
housing assistance.[Footnote 69] As we have seen, MTW agencies may 
request waivers of certain provisions of the 1937 Housing Act in order 
to combine annual funding from separate sources into a single 
authoritywide funding source. HUD field office staff with 
responsibility for monitoring MTW agencies observed that the single-
fund flexibility was beneficial because it enabled participating 
agencies to develop supportive service programs, such as job training 
or educational programs, which help move families toward self-
sufficiency. One HUD field office official stated that this 
flexibility would be a significant benefit for other PHAs. An 
affordable housing advocate we met with also noted that this ability 
to use different kinds of funds interchangeably was beneficial because 
it enabled MTW agencies to shift funds based on local priorities. 
Further, officials from the MTW agencies we interviewed agreed that 
this flexibility was beneficial. For example, officials from one MTW 
agency stated it had been able to use the single fund to organize 
itself as a business organization, develop a strategic plan based on 
the housing needs of low-income families in the community, leverage 
public funds and public and private partnerships, and develop mixed-
income communities. Two of the MTW agencies that we interviewed also 
stated that the single-fund flexibility had enabled them to fund 
programs that encouraged self-sufficiency among residents. For 
example, officials explained that they had used funding for coaching 
and counseling services, job training support, and education programs. 
Finally, officials from three of the MTW agencies we interviewed noted 
a related benefit of participation. They said that their MTW status 
had enabled them to respond more quickly to real estate opportunities 
because they do not have to wait for HUD approvals to purchase 
properties. 

Lack of Performance Information and Other Issues Raise Questions about 
Expansion: 

A lack of performance information, limited HUD oversight, and concerns 
about the program's impact on residents raise questions about 
expanding the MTW program. As we noted previously, conclusive 
information about the effectiveness of the MTW program is limited in 
part because HUD does not have a plan for identifying and analyzing 
standard performance data, has not established performance indicators 
for the program as whole, and does not have a systematic process for 
identifying lessons learned. 

HUD's 2010 Report to Congress noted that the conclusive impacts of 
many MTW activities, particularly as they relate to residents, could 
not yet be known.[Footnote 70] For example, the report noted that the 
rent reforms implemented under MTW varied greatly and were not 
implemented using a controlled experimental methodology. As a result, 
which aspects of rent reforms should be recommended for all PHAs were 
not clear. The report also noted the limitations that exist when 
evaluating the outcomes of MTW--limitations that stem from the weak 
initial reporting requirements and lack of a research design. The 
report concluded that, given these limitations, expansion should occur 
only if newly admitted PHAs structure their programs for high-quality 
evaluations that permit lessons learned to be generalized beyond a 
single PHA experience. 

Similarly, affordable housing advocates and legal aid organizations 
that we interviewed stated that because lessons had not been learned 
from MTW, there was no basis for expanding the program. For example, 
officials from a national affordable housing advocacy organization 
stated that some MTW agencies have used their flexibility to establish 
limits on the length of time someone can live in assisted housing, but 
there is little research on the effect of such efforts. The officials 
stated that there was no evidence that this policy had helped anyone 
become self-sufficient and move out of public housing. The officials 
added that data were not available on the extent to which MTW agencies 
have provided incentives for residents to become self-sufficient or 
have increased housing choices. Similarly, an official from a national 
housing law advocacy organization stated that data were not available 
to determine the effect of the MTW program, particularly at the 
national level. 

In addition, our own work, some research organizations, and affordable 
housing advocates question HUD's ability to effectively manage an 
expanded MTW program. As previously noted, HUD's current monitoring 
procedures have several key weaknesses, including the lack of a 
systematic process for assessing agencies' compliance with statutory 
requirements and an assessment of program risks. Some research 
organizations also have questioned HUD's capacity to oversee 
additional MTW agencies. For example, the Urban Institute reported 
that the approval process that HUD was using at the time of the 
institute's 2004 review would not be feasible for an expanded program 
because of the administrative burden involved.[Footnote 71] At the 
time of the 2004 study as well as our review, HUD reviewed each 
individual request to waive specific provisions of the 1937 Housing 
Act before approving annual plans. Staff from another research 
organization questioned whether HUD has the capacity to oversee 
additional agencies. Similarly, one affordable housing advocate that 
we interviewed stated that HUD's capacity to oversee an expanded 
program is not clear, in part because current monitoring activities 
are not transparent. At the time of our review, HUD had four full-time 
MTW coordinators, who each managed from 6 to 10 MTW agencies.[Footnote 
72] According to the Director of the MTW Office, it takes more 
resources for HUD to oversee MTW agencies than non-MTW agencies. Thus, 
if additional agencies were added under the current program design, 
HUD would likely need additional resources. 

Researchers and several of the affordable housing advocates and legal 
aid agencies that we met with also raised concerns that the current 
program, and therefore also an expanded program, could negatively 
affect residents of MTW agencies. For example, two research 
organizations have stated residents could be negatively affected by 
MTW agencies that implement voucher policies that reduce portability--
that is, residents' ability to use their vouchers in an area outside 
of the area where they received it. One of these research 
organizations stated that the differences in the way voucher programs 
were implemented across MTW agencies could reduce residents' ability 
to use vouchers outside of the area where they received the 
assistance. Officials from the other organization noted that some MTW 
agencies had instituted policies that prohibited vouchers from being 
transported out of the originating jurisdictions, thereby limiting 
housing choices. According to HUD officials, MTW agencies with 
policies that limit portability can make exceptions. For example, 
these agencies have made exceptions for residents seeking employment 
opportunities. 

Legal aid organizations that have worked with residents of MTW 
agencies as well as affordable housing advocates told us some of the 
requirements that MTW agencies have implemented, such as work 
requirements, were potentially harmful to residents.[Footnote 73] For 
example, legal aid representatives from one community told us that the 
work requirement was not consistently enforced across various mixed-
income properties that included public housing as well as market-rate 
units. According to these officials, they have had clients who have 
been evicted for not working, even though the client was in school or 
disabled--both exceptions to the work requirement. These officials 
also stated that property managers in the city's various mixed-income 
developments did not implement MTW policies consistently. For example, 
the officials stated that residents have been told by property 
managers that they would be in compliance with the work requirement if 
they were in school or another training program, only to have the MTW 
agency determine that they were not in compliance. According to HUD 
officials, inconsistent enforcement of policies is not unique to MTW 
agencies, and residents would have recourse. Legal aid representatives 
that worked with residents of another MTW agency also told us that the 
work requirement was a punitive policy that negatively affected the 
poorest residents. The officials stated that there were better methods 
for encouraging work and self-sufficiency, such as job training. 
Officials from a national affordable housing advocacy organization 
agreed that work requirements are punitive and stated that they 
disagreed with a policy of making housing assistance contingent on 
other factors, such as having a job. In their view, housing assistance 
should be a stable form of assistance for low-income households. 

Alternatives Exist to Expanding the MTW Program: 

Alternatives to expansion include implementing a program that is 
targeted more to specific activities and waiving some regulations for 
all PHAs as described in proposed legislation. According to the Urban 
Institute, an alternative to expanding MTW could be to systematically 
test a limited number of programmatic alternatives--such as flat 
rents, time limits, or debt financing of capital improvements. 
[Footnote 74] This approach would not allow individual agencies as 
much discretion to design combinations of reforms around local 
conditions and priorities. However, this approach could yield more 
systematic evidence about the costs and benefits of particular program 
reforms if it included a rigorous evaluation design and mandatory data 
collection on key outcomes, such as the number and characteristics of 
participating households. Collection of such information in a 
standardized format would need to be a minimum requirement for 
participation if the point was to learn from the experiences of those 
testing activities. In addition, an official from an affordable 
housing advocacy organization that we met with stated that testing the 
effectiveness of discrete activities on a smaller scale would be 
useful. HUD also noted that altering the scope of the demonstration 
for new participants could improve what was learned from specific 
activities. For example, its 2010 Report to Congress stated that data 
on MTW could be strengthened if the scope of the demonstration were 
altered for new participants by selecting agencies committed to 
testing a particular activity, such as rent reform, and requiring 
rigorous evaluation.[Footnote 75] In December 2011, HUD issued a 
request for proposals for a demonstration that would test alternatives 
to the current rent structure in the voucher program. According to the 
proposal, the demonstration would most likely be undertaken at select 
MTW agencies. In addition, HUD's 2012 appropriations act authorized a 
Rental Assistance Demonstration that would enable HUD to authorize and 
evaluate new approaches to preserving affordable rental housing, 
including converting public housing to project-based rental 
assistance. AHSSIA includes authorization for a revised version of the 
Rental Assistance Demonstration. 

In addition, according to researchers, PHAs could be allowed some 
level of deregulation, so that they could implement the same sort of 
administrative streamlining activities that MTW agencies implement. 
The Urban Institute has reported that further deregulation of PHAs may 
yield benefits in terms of program design and implementation 
innovations but that such deregulation could entail risks and should 
be evaluated.[Footnote 76] Some of the MTW agencies that we met with 
were supportive of the administrative streamlining authorities allowed 
by MTW and described in proposed legislation such as AHSSIA. As 
proposed, AHSSIA would allow PHAs to reduce the frequency of income 
certifications for the elderly and disabled to save on administrative 
costs, among other things. According to the Director of the MTW 
Office, the experiences of MTW agencies informed this policy proposal. 
In addition, HUD's 2010 Report to Congress supported allowing more 
PHAs to participate in the program. Finally, we recently reported on 
cost savings that could be realized from allowing additional housing 
authorities to implement some of the reforms MTW agencies have tested. 
[Footnote 77] 

Conclusions: 

The MTW demonstration is designed to provide participating agencies 
with the flexibility to develop and test activities that achieve cost-
efficiency, encourage residents with children to obtain employment and 
become self-sufficient, and increase housing choices for low-income 
families. While this flexibility has allowed participating agencies to 
implement hundreds of activities, HUD has not done all that it can to 
evaluate the program's effectiveness, identify successful approaches 
that could be applied to public housing agencies more broadly, or 
ensure that MTW agencies comply with program requirements. Because 
Congress is considering expanding the program to many more PHAs, the 
absence of information needed to conduct a comprehensive program 
evaluation and compliance reviews is significant. HUD has recognized 
the importance of rigorous evaluation by requiring newly admitted 
agencies to have strong evaluation components. However, these 
improvements will not help evaluate the program as a whole. Without 
more complete knowledge of the program's effectiveness and the extent 
to which agencies are adhering to program requirements, it is 
difficult for Congress to know whether an expanded MTW will benefit 
additional agencies and the residents they serve. 

Recognizing that it needed to do more to improve what was known about 
the program's effectiveness, HUD started requiring MTW agencies to 
describe the impact of each implemented activity in their annual 
reports beginning in 2009. However, the information that MTW agencies 
reported did not always reflect outcomes, and HUD's guidance does not 
require that information on activities be quantifiable and outcome-
oriented to the extent possible. Without more specific guidance on 
reporting performance information, HUD cannot be assured of collecting 
data that reflects the outcomes of activities. 

Further, challenges such as the lack of analysis of standard data and 
the absence of performance indicators have prevented comprehensive 
evaluations of similar activities and the overall MTW program. HUD has 
recently started collecting additional information through MTW-PIC and 
annual reports, but has not yet analyzed the data. Further, whether 
the data collected are sufficient to assess similar activities and the 
program as a whole is not clear, and HUD has not identified the 
performance data it needs to undertake such analysis. Until HUD has a 
plan (that includes the identification of standard data) to 
quantitatively assess similar activities and the MTW program as a 
whole, HUD cannot determine their effectiveness. Additionally, HUD has 
not established performance indicators specific to MTW. Indicators 
linked to the statutory purposes of reducing costs, encouraging self-
sufficiency, and increasing housing choices would help HUD demonstrate 
that the program has produced desired results. 

Similarly, HUD does not have a systematic process in place to identify 
lessons learned from the MTW demonstration. Identifying activities 
that could be replicated more broadly is a goal of the MTW program and 
could be aided by the analysis of some standard performance data. The 
absence of a criteria-based, regular process for identifying lessons 
learned complicates efforts to determine which MTW activities are most 
effective and should be replicated more broadly. 

At the same time, HUD's monitoring efforts are not as strong as they 
could be. First, because HUD has not clarified key terms related to 
the three statutory purposes and five requirements, it cannot 
effectively determine whether MTW agencies are actually addressing 
these purposes and meeting requirements. Second, HUD does not have a 
process in place to systematically review compliance with all 
requirements. Such a review is especially important to a program like 
MTW that allows participants to self-certify their compliance with 
some program requirements. HUD has begun assessing compliance with two 
of the MTW requirements that call for self-certification, but not the 
third, and thus lacks assurance that agencies are complying with all 
three. 

Moreover, HUD's procedures for monitoring MTW agencies are not risk-
based. It does not conduct an annual assessment of risks and provides 
the same level of monitoring for all agencies, even though some may 
pose greater risks than others. A risk-based approach to monitoring 
would provide greater assurance that HUD has addressed all risks, 
particularly those that may prevent participating agencies from 
addressing program purposes and meeting statutory requirements. 
Further, unless it implements a risk-based approach (such as that 
currently being considered for annual site visits) to monitoring MTW 
agencies, HUD cannot be assured that it is using its limited 
monitoring resources most efficiently. Finally, just as HUD does not 
assess compliance with all three self-certified requirements, it does 
not verify the accuracy of key information that agencies self-report, 
including information on the impact of MTW activities. Annual site 
visits have been used primarily to provide technical assistance rather 
than to assess self-reported information. By not verifying the 
accuracy of any performance information, HUD lacks assurance that this 
information is accurate. 

Recommendations for Executive Action: 

To improve what is known about the effectiveness of the MTW program, 
we recommend that the Secretary of the Department of Housing and Urban 
Development: 

* improve HUD's guidance to MTW agencies on providing performance 
information in their annual reports by requiring that such information 
be quantifiable and outcome-oriented to the extent possible; 

* develop and implement a plan for quantitatively assessing the 
effectiveness of similar activities and the program as a whole 
including the identification of standard performance data needed; and: 

* establish performance indicators for the MTW program as a whole. 

To enhance the ability to identify MTW practices that could be applied 
more broadly, we recommend that the Secretary of the Department of 
Housing and Urban Development create a process to systematically 
identify lessons learned. 

To improve HUD's oversight of the MTW program, we recommend that the 
Secretary of the Department of Housing and Urban Development: 

* issue guidance that clarifies key program terms, such as the three 
statutory purposes of the program and the five statutory requirements 
that MTW agencies must meet; 

* develop and implement a systematic process for assessing compliance 
with statutory requirements; 

* conduct an annual risk assessment for the MTW program and implement 
risk-based monitoring policies and procedures such as those currently 
being considered for site visits; and: 

* implement control activities designed to verify the accuracy of a 
sample of the performance information that MTW agencies self-report. 

Agency Comments and Our Evaluation: 

We provided a draft of this report to HUD. The Assistant Secretary for 
Public and Indian Housing provided written comments, which are 
reprinted in appendix II. HUD disagreed with our recommendation that 
the agency develop performance indicators for the MTW program as a 
whole, was in partial agreement with four recommendations, and 
generally agreed with three. The agency said that developing 
programwide performance measures could be difficult and might be 
contrary to the nature of the demonstration. In addition, HUD 
emphasized the improvements that it had made to its reporting 
requirements in order to collect more consistent, outcome-oriented 
data. We acknowledged these improvements in the draft report, but as 
our recommendations indicated, we saw opportunities for additional 
improvements. HUD also noted that some of our recommendations might be 
a good fit for the existing program but that others might be more 
appropriate for a future expanded demonstration. 

In disagreeing with our recommendation that it establish performance 
indicators for the MTW program as a whole, HUD emphasized the 
difficulty of measuring all activities against the same standard. The 
agency noted that because each MTW agency had implemented a unique 
combination of activities, developing programwide performance measures 
would make determining the impacts of specific activities unclear and 
prevent the identification of individual policies that could be 
applied more broadly. However, the purpose of programwide indicators 
would not be to isolate the impact of individual activities but to 
demonstrate programwide results--including showing the extent to which 
the program was addressing its statutory purposes of achieving greater 
cost-effectiveness in federal housing expenditures, giving families 
with children incentives to obtain employment and become self-
sufficient, and increasing housing choices for low-income families. 
HUD also stated that applying programwide performance measures would 
be complicated by the fact that activities that advance one statutory 
purpose might conflict with other purposes. We agree that it is 
important to evaluate similar activities and have a separate 
recommendation addressing this need. But, the purpose of programwide 
assessment is to demonstrate whether the provision of flexibility in 
itself results in the intended benefits of the MTW program, such as 
cost savings or increased family self-sufficiency. Demonstrating that 
the increased flexibility the program offers has produced the intended 
results is critical, particularly as Congress considers whether to 
expand the program. We continue to believe in the importance of 
demonstrating program results and therefore continue to recommend that 
HUD develop performance indicators for the MTW program as a whole. 

HUD was in partial agreement with four recommendations. 

* First, HUD said that proposed revisions to the reporting 
requirements for MTW agencies had addressed our recommendation that 
the agency improve its guidance to MTW agencies on providing 
performance information in annual reports. HUD's draft guidance is in 
line with our recommendation that HUD require agencies to report 
quantifiable and outcome-oriented information. However, because these 
proposed revisions have yet to be finalized, we did not revise our 
recommendation. 

* Second, HUD agreed that quantitatively assessing the effectiveness 
of similar activities was an important step but noted the difficulties 
associated with assessing the effectiveness of the program as a whole. 
However, as noted above, we continue to believe in the importance of 
demonstrating program results. Consequently, we did not revise our 
recommendation. 

* Third, HUD stated that providing a menu of standard metrics may be 
the best way to clarify the program's statutory purposes and that it 
had made progress in recent years in addressing our recommendation 
that it issue guidance that clarifies the statutory requirements. HUD 
also noted that the proposed revisions to the reporting requirements 
would provide additional clarification on the statutory requirements. 
These efforts, which were acknowledged in the draft report, are a step 
in the right direction, and we encourage HUD to continue finalizing 
this guidance. As noted above, because these proposed revisions have 
yet to be finalized, we did not revise our recommendation. 

* Fourth, HUD described recent efforts to assess compliance with two 
statutory requirements and analysis that it could conduct once 
proposed revisions to reporting requirements for MTW agencies were 
finalized. Because the process used to assess compliance with one of 
the requirements has not been formalized in policy and the proposed 
revisions have not been finalized, we did not revise our 
recommendation that HUD develop and implement a systematic process for 
assessing compliance with statutory requirements. 

HUD generally agreed with the three remaining recommendations. For 
example, HUD agreed that it should proactively identify lessons 
learned and described some of its recent efforts to do so. We 
acknowledged these efforts in our draft report but noted the absence 
of a criteria-based, regular process for identifying lessons learned. 
HUD also described plans to develop a formal risk-based strategy for 
monitoring and, when we asked for further clarification, stated that 
it agreed with our recommendation to conduct an annual risk assessment 
for the MTW program. Finally, HUD discussed potential strategies for 
verifying the information that MTW agencies report using existing or 
planned HUD systems. 

HUD also requested that we consolidate four separate recommendations 
into two, but we continue to believe that maintaining distinctions 
between the separate recommendations is important. First, HUD 
requested that we combine two recommendations: that it create a plan 
to quantitatively assess the effectiveness of similar activities and 
the program as a whole (including identifying the standard performance 
data needed), and that it establish performance indicators for the 
program as a whole. Although related, the two recommendations are 
distinct because the first focuses on the need for program evaluation 
and the second on performance measurement. Program evaluations 
typically examine a broad range of information on program performance, 
while performance measurement generates outcomes that show whether a 
program has achieved specific objectives.[Footnote 78] As a result, we 
did not combine the recommendations. 

Second, HUD requested that we combine the recommendation that it issue 
guidance clarifying key program terms (such as the three statutory 
purposes and five statutory requirements) with the recommendation that 
it implement a systematic process for assessing compliance with 
statutory requirements. However, defining program requirements and 
assessing them are separate and distinct activities. Therefore, we did 
not combine the recommendations. 

As agreed with your offices, unless you publicly announce the contents 
of this report earlier, we plan no further distribution until 30 days 
from the report date. At that time, we will send copies to the 
Secretary of Housing and Urban Development and other interested 
committees. In addition, the report will be available at no charge on 
the GAO website at [hyperlink, http://www.gao.gov]. 

If you or your staff have any questions concerning this report, please 
contact me at (202) 512-8678 or sciremj@gao.gov. Contact points for 
our Offices of Congressional Relations and Public Affairs may be found 
on the last page of this report. Key contributors to this report are 
listed in appendix III. 

Signed by: 

Mathew J. Scirè: 
Director, Financial Markets and Community Investment: 

[End of section] 

Appendix I: Scope and Methodology: 

Our objectives were to examine (1) what is known about the extent to 
which the Moving to Work (MTW) demonstration program is addressing the 
program's statutory purposes, (2) the Department of Housing and Urban 
Development's (HUD) monitoring of MTW agencies' efforts to address 
these purposes and meet statutory requirements, and (3) potential 
benefits of and concerns about expanding the number of public housing 
agencies (PHA) that can participate in the demonstration program. 

To evaluate what is known about the extent to which the MTW program is 
addressing the program's statutory purposes, we reviewed the most 
recent annual reports as of January 2012 for 30 MTW agencies.[Footnote 
79] We reviewed these annual reports and the corresponding annual 
plans to identify the ongoing activities that the agencies were 
implementing, determine the extent to which these activities were 
linked with one or more of the program's statutory purposes, and 
assess the performance information provided for each 
activity.[Footnote 80] To assess the performance information provided 
by MTW agencies, we examined HUD's reporting guidance and compared it 
with internal control standards for federal agencies.[Footnote 81] We 
assessed the reliability of selected information in the reports by 
reviewing supporting documentation from a sample of seven MTW agencies 
and interviewing the officials responsible for preparing and reviewing 
this information. These seven agencies were: 

* Cambridge Housing Authority (Cambridge, Massachusetts), 

* Chicago Housing Authority (Chicago, Illinois), 

* Housing Authority of the City of Atlanta (Atlanta, Georgia), 

* Housing Authority of the City of Pittsburgh (Pittsburgh, 
Pennsylvania), 

* Housing Authority of the County of Santa Clara/Housing Authority of 
the City of San Jose (Santa Clara County and San Jose, California), 

* Lawrence-Douglas County Housing Authority (Lawrence, Kansas), and: 

* Vancouver Housing Authority (Vancouver, Washington).[Footnote 82] 

We selected these agencies to provide diversity in geography, agency 
size, and length of time participating in the program. We determined 
that the reports were sufficiently reliable for the purposes of our 
review. 

Through interviews and a literature search, we identified three 
studies of the MTW program as a whole. We reviewed these studies to 
identify information on the program's effectiveness and any challenges 
associated with assessing it. We determined that these studies were 
methodologically sound and reliable for our purposes. We examined 
HUD's recent efforts to collect data from MTW agencies, including 
documentation on the reporting requirements for MTW agencies. In 
addition, we reviewed HUD's fiscal year 2010-15 strategic plan and 
Fiscal Year 2011 Annual Performance Plan for any performance 
indicators for the MTW program. We also reviewed the GPRA (Government 
Performance and Results Act) Modernization Act of 2010, Office of 
Management and Budget guidance, internal control standards, and a GAO 
report on attributes of successful performance measures.[Footnote 83] 
Further, we identified five studies of specific MTW agencies or 
activities identified by HUD and representatives of the sample of MTW 
agencies we interviewed. Finally, we reviewed published reports and 
HUD's website for information on HUD's efforts to identify lessons 
learned. 

To assess HUD's monitoring of MTW agencies' efforts to address the 
program's statutory purposes and meet requirements, we obtained and 
reviewed documentation of monitoring policies and procedures, 
including the Standard Agreement that HUD executed with MTW agencies 
in 2008, the MTW Desk Guide, a 2011 Memorandum of Understanding 
between HUD's Office of Public Housing Investments and Office of Field 
Operations, and other HUD guidance. Based on these documents and 
interviews with HUD staff, we identified three key monitoring 
processes: the review of annual plans and reports, reviews of data 
entered into the Moving to Work section of the Public and Indian 
Housing Information Center (MTW-PIC), and annual site visits to each 
MTW agency. 

To assess the extent to which HUD staff were following these 
monitoring policies and procedures, we reviewed documentation of 
monitoring activities for our sample of seven MTW agencies. For 
example, to verify the steps HUD had taken to review annual plans and 
reports, we reviewed the checklists that the MTW coordinators used to 
document their review of these plans and reports. We also reviewed 
HUD's comment letters for fiscal year 2011. To verify the steps HUD 
had taken to review data submitted into the MTW-PIC system, we 
reviewed monthly reports that showed the degree to which MTW agencies 
overall complied with reporting requirements from August 2011 through 
January 2012. Finally, to verify that both headquarters and field 
office staff made site visits and the extent to which they made annual 
visits, we reviewed the most recently available site visit reports 
completed by the MTW Office for all agencies as of October 2011. In 
addition, we interviewed the MTW agencies in our sample and the 
corresponding HUD field office officials to discuss the annual site 
visits. 

We also compared HUD's monitoring policies and procedures to internal 
control standards for the federal government and HUD's own program 
management guidance.[Footnote 84] As a part of this analysis, we 
compared HUD's guidance to MTW agencies with the internal control 
requirement for clear goals and objectives. We also reviewed 
information on HUD's efforts to clarify how agencies could certify 
compliance with the requirement to assist "substantially the same" 
number of eligible families that would have been served in the absence 
of MTW. In addition, we compared HUD's efforts to assess agencies' 
compliance with statutory requirements with the internal control 
standard related to assessing compliance with program requirements. 
Further, we reviewed internal control standards for the federal 
government and HUD's own internal control standards and identified the 
requirement that programs have an annual risk assessment. We 
interviewed HUD regarding any risk assessment performed for the MTW 
program. Finally, we interviewed HUD officials to determine whether 
any of the performance information that MTW agencies reported had been 
verified. We compared HUD's lack of verification with the internal 
control standards and guidance that emphasized the need for control 
activities to ensure that program participants report information 
accurately. 

To discuss the potential benefits and concerns associated with 
expanding the number of PHAs that can participate in the program, we 
reviewed studies, reports, and testimonies by researchers, affordable 
housing advocates, resident advocates, and the HUD Office of Inspector 
General. For all three objectives, we interviewed officials from the 
seven MTW agencies in our sample and representatives from affordable 
housing advocacy organizations such as the Council of Large Public 
Housing Agencies, the National Association of Housing and 
Redevelopment Officials, the National Leased Housing Association, and 
the Public Housing Authorities Directors Association. We spoke with 
resident advocacy organizations such as the National Low-Income 
Housing Coalition, the National Housing Law Project, and legal aid 
agencies that represented residents serviced by five of our sample MTW 
agencies.[Footnote 85] We also interviewed staff from the Center for 
Budget Policy and Priorities, a research organization that has studied 
and written about the MTW program; researchers who had evaluated the 
MTW program; and HUD officials from the MTW office and the field 
offices that corresponded to our sample of agencies. During our 
interviews, we discussed the potential benefits of expansion and the 
concerns of these organizations. Based on our review of available 
studies and reports and interviews with the above mentioned 
stakeholders, we identified key benefits and concerns. We also made 
observations based on our findings related to the availability of 
performance information for the program and HUD's monitoring efforts. 

We conducted this performance audit from July 2011 to April 2012 in 
accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. 

[End of section] 

Appendix II: Comments from the Department of Housing and Urban 
Development: 

U.S. Department of Housing and Urban Development: 
Assistant Secretary for Public and Indian Housing: 
Washington, DC 20410-5000: 

April 4, 2012: 

Memorandum For: Matthew J. Scire, Director, Financial Markets and 
Community Investment, Government Accountability Office: 

From: [Signed by] Sandra B. Henriquez, Assistant Secretary for Public
and Indian Housing: 

Subject: Comments Regarding the GAO report entitled Moving to Work
Demonstration: Opportunities Exist to Improve Information and
Monitoring (GAO-12-490) (Engagement code 250626.) 

Thank you for your recent review of the Moving to Work (MTW) program. 
We appreciate the thorough review GAO provided regarding this 
important program, and the thoughtful recommendations you have 
provided. 

Attached please find our comments regarding the above referenced 
report. If you have any questions, please contact Ivan Pour at (202) 
402-2488. 

Attachment. 

[End of letter] 

HUD Response to the Government Accountability Office's Report to 
Congressional Committees Regarding the Moving to Work Demonstration: 

In its analysis of the Moving to Work (MTW) demonstration, the 
Government Accountability Office (GAO) presents a set of 
recommendations designed to improve oversight and monitoring and 
better position the demonstration for a rigorous evaluation. The 
report also recognizes that MTW's initial program design was not 
conducive to the kinds of controls and protocols recommended. Before 
responding to the individual recommendations, it is important to place 
the responses in the context of some historical information as well as 
ongoing discussions about expanding MTW in a way that incorporates the 
lessons learned from the demonstration to date and strengthens it 
going forward. 

In 2008, HUD made significant improvements to its reporting 
requirements in order to collect more consistent, outcome-oriented 
data. The individually tailored agreements of the existing MTW 
agencies (25 at that time) were transferred to a single Standard MTW 
Agreement with the same covenants and requirements for all MTW PHAs. 
As part of this changeover, MTW agencies were required to identify, 
for each MTW activity, the link to an MTW statutory objective and 
outcome metrics appropriately tied to the objective and the activities 
themselves. This was a major change from the way MTW agencies had been 
reporting. Larger MTW agencies that had been combining their funds had 
submitted their MTW Plans and Reports in a format that was largely 
descriptive in nature and not standardized. Smaller MTW agencies 
continued to submit the standard PHA plan with a short addendum 
describing their MTW activities. Neither group was previously required 
to consider or describe outcomes in a meaningful way. 

The Standard MTW Agreement also required agencies to adhere to 
specific reporting and data submission requirements.[Footnote 1] 
Moreover the reporting requirements are now tied to OMB-approved 
information submission requirements, making them dynamic and giving 
HUD the opportunity to modify them as circumstances require.[Footnote 
2] Indeed HUD is completing a revision of these requirements that, as 
described below, addresses many of the recommendations made by GAO in 
this report. 

When HUD overhauled the reporting requirements in 2008, it required 
agencies to develop metrics, but did not prescribe specific metrics at 
that time for two reasons. First, HUD recognized the shift in 
reporting requirements, such as it was, was going to be a major 
undertaking for the MTW agencies. Larger agencies had complex programs 
that now needed to be broken down into identifiable activities with 
associated outcome measures. The smaller agencies had not completed 
Annual MTW Plans or Reports, and so they now needed to engage in a 
much more rigorous planning, data collection and reporting process 
than they had previously done. A second unknown was the ability of 
agencies to define outcome measures with the data systems they 
currently had. Before HUD was in a position to see how agencies would 
be defining their activities, and what capability the agencies had for 
reporting, HUD did not believe it was in a position to prescribe 
specific outcome measures. For these reasons, as well as a recognition 
of the locally-oriented nature of the demonstration, agencies were 
charged with developing their own outcome metrics. 

Over the past few years, HUD has obtained more detailed information, 
providing outcome information regarding the potential success of 
activities at the agency level, and HUD has identified outcome 
measures that appear to be common elements for certain types of 
activities that do not appear to be problematic for agencies to use 
and report. This gives HUD the opportunity to move to standardized 
measures in the revised reporting requirements. The latest iteration 
of the HUD-50900 (Reporting Requirements for MTW Agencies), which 
incorporates the standardized metrics, is included with this response. 
The next step prior to its adoption is for it to go through another 30 
day comment period before receiving final approval by the Office of
Management and Budget (OMB). 

In its 2010 Report to Congress regarding MTW, the Department 
recommended doubling the size of the MTW demonstration to create a 
group of PHAs that would implement activities similar to those already 
being implemented, but with a more rigorous data collection and 
evaluation methodology from the outset in order to properly assess MTW 
activities. Draft legislation has been proposed that would expand MTW 
on a much larger scale, with an emphasis on improved reporting and 
evaluation. The existing MTW agencies thus provide a laboratory to 
begin testing some of the administrative changes that would be 
required of incoming MTW PHAs should the program expand. In the 
meantime, each of GAO's recommendations should take into consideration 
the administrative burden of making these changes roughly fifteen 
years into a twenty-year demonstration. While some recommendations may 
be readily adopted in the existing program, some may be more 
appropriate to set as parameters in a future expanded demonstration. 

GAO Recommendation: Improve guidance to MTW agencies on providing 
performance information in their annual reports by requiring that such 
information be quantifiable and outcome oriented to the extent 
possible. 

HUD Response: HUD believes that with the adoption of the Standard MTW 
Agreement and revised reporting requirements, the recommendation that 
the information be quantifiable and outcome oriented has been 
implemented. Upon approval by OMB of the new reporting requirements in 
Form 50900, more specific instructions and guidance will be provided 
to the agencies. For these two reasons HUD requests that this 
recommendation be considered closed. 

The current Standard MTW Agreement, through Attachment B/Form 50900, 
states that PHAs must "describe baselines, proposed benchmarks and 
metrics to assess outcomes" when proposing activities in MTW Plans, 
and PHAs must "provide detailed information on the impact of the 
activity and compare against the proposed benchmarks and metrics to 
assess outcomes" in MTW Reports. This contractual obligation on the 
part of the MTW agency is reinforced by guidance from HUD in the form 
of: group training opportunities (through the 2009 evaluation training
GAO mentions in several places throughout its report, webcasts and MTW 
conferences), written guidance (training materials published after the 
2009 training mentioned by GAO on page 6, 10 and 12 of this report) 
and technical assistance provided by the MTW Office. For instance, the
GAO notes in footnote 12 on page 6 that "in training materials for the 
MTW program, HUD defines baseline as the performance level that was 
being achieved prior to the implementation of the MTW activity, 
benchmark as the anticipated outcome of a MTW activity, and metric as 
the specific measure that quantifies the changes from an activity." 
The GAO also notes on page 12 that these training materials have not 
been incorporated into Attachment B, but working with PHAs to develop 
quantifiable and outcome oriented metrics is one of the primary 
functions of MTW coordinators is reviewing MTW Plans and Reports and 
providing ongoing technical assistance to PHAs, including at annual 
site visits. 

HUD has further clarified the requirements related to data collection 
in the revised Attachment B/Form 50900, currently pending approval by 
OMB, which explicitly states that "metrics" must be "a numeric value." 
HUD plans to provide further trainings and guidance to MTW agencies 
after the revised Form 50900 is effective. 

Prior to the implementation of the Standard MTW Agreement, 
quantifiable, outcome-oriented data was lacking, and the GAO notes on 
page 14 "the limitations that the program's initial design posed to 
evaluation." However, all MTW PHAs have now been producing 
quantifiable, outcome-oriented data since 2008 or 2009.[Footnote 3] 
The primary barrier to analyzing this data on an ongoing basis has 
been resource limitations. 

GAO Recommendation: Develop and implement a plan for quantitatively 
assessing the effectiveness of similar activities and the program as a 
whole including the identification of standard performance data needed. 

GAO Recommendation: Establish performance indicators for the MTW 
program as a whole. 

HUD Response: The recommendation to implement a plan for assessing the 
program as a whole and the recommendation to establish program 
indicators for the program as a whole are very similar, as any plan 
that "quantitatively assessed effectiveness" of the program as a whole 
would include the "establishment of performance indicators." HUD 
requests these two recommendations be combined, and a consolidated 
response on these two GAO recommendations is provided below. 

Although not yet standardized, the 2008 revisions to the reporting 
requirements moved MTW agencies from a purely descriptive data 
structure to an outcome oriented data structure for the first time 
since the program's inception. Prior to the Standard MTW Agreement, 
MTW agencies were not required to report into either the IMS-PIC 
system, or provide financial information through the Financial 
Assessment Subsystem (FASS). Currently all MTW agencies report into 
these systems, providing standard demographic and financial 
information for both MTW and non-MTW agencies.[Footnote 4] While not 
standardized, extensive quantitative data on many MTW activities 
already exists in the MTW Plans and Reports. Many MTW PHAs 
implementing similar activities use similar and comparable metrics, 
such as time and costs saved for cost-efficiency activities, 
employment status and earnings for self-sufficiency activities and 
additional units created or vouchers leased for housing choice 
activities. HUD systems such as IMS-PIC provide an additional layer of 
verification as well as additional information on the effectiveness of 
certain MTW activities. A comprehensive analysis of outcomes of 
similar activities is possible with the data available at this point. 

In order to enhance the ability to conduct analysis on similar MTW 
activities across MTW agencies, HUD is currently building on the 
metrics created thus far and is developing standard metrics organized 
around the three statutory objectives of the demonstration as part of 
the revised reporting requirements. HUD agrees that the recommendation 
to quantitatively assess the effectiveness of similar activities is an 
important next step. 

HUD agrees that compliance with the statutory requirements should be 
monitored for all PHAs and, as described below, and has already made 
considerable progress in these areas. However, developing more general 
performance indicators for MTW as a whole may be difficult or 
impossible to monitor and contrary to the nature of the demonstration. 

The issue of appropriate performance indicators is not limited to MTW 
agencies. The current Public Housing Assessment System (PHAS) assesses 
both risk and performance. What has been learned since its inception 
is that PHAS works well at assessing risk, but has not always been as 
accurate in assessing performance with MTW agencies. The Standard MTW 
Agreement recognizes this by allowing MTW agencies to opt out of being 
scored by PHAS.[Footnote 5] 

Risk in MTW agencies can be measured today, since each agency is 
required to get a physical inspection and must submit financial 
information. From this information, a physical assessment subscore 
(PASS) can be determined for each property, and a financial score 
(FASS) and management score (MASS) can be determined from the agency's 
financial submissions. In short, all the necessary information to 
determine a PHAS score is available and can be used to assess risk at 
an MTW agency. 

However, these indicators do not present an accurate picture of 
performance. Because each MTW PHA implements a unique combination of 
activities, developing program-wide performance measures may conflate 
the impacts of many different activities, thus preventing HUD from 
identifying the individual policies that can be applied more broadly. 
While HUD supports analysis to determine the extent to which certain 
types of activities are successful in promoting the statutory purposes 
they were designed to address, determining whether the program as a 
whole is advancing these purposes is nearly impossible and would be 
unlikely to produce information that could inform future policy 
decisions. For instance, combining the self-sufficiency outcomes at a 
PHA that has raised rents for most residents with the outcomes at a 
PHA that has decreased rents for most residents would provide 
unconvincing data regarding whether MTW as a whole is promoting self-
sufficiency. 

The difficulty in applying program-wide performance measures is 
further complicated by the fact that activities that advance one of 
the statutory purposes may be in conflict with other purposes. For 
instance, a PHA may have been able to serve 100 families prior to MTW, 
but through streamlining is able to serve 110 (in support of cost-
efficiency). However, rather than serve additional families, the PHA 
chooses to use the funds that could have served the additional ten 
families to provide supportive services to residents (in support of 
self-sufficiency) or provide incentives for residents to move to 
higher-opportunity neighborhoods (in support of housing choice). These 
goals clearly support the MTW statutory purposes, and the PHA would 
still fulfill the statutory requirement to serve substantially the 
same number of families, but by supporting one of the other two 
objectives it may appear that it is not more cost-efficient, as it is 
not serving more families per dollar. 

HUD believes that evaluating the impact of the streamlining 
activities, the supportive services or the mobility program in 
isolation would provide stronger results, and recommends this approach 
for evaluating performance. 

GAO Recommendation: To enhance the ability to identify MTW practices 
that could be applied more broadly, we recommend that the Secretary of 
the Department of Housing and Urban Development create a process to 
systematically identify lessons learned. 

HUD Response: HUD agrees with the recommendation to proactively 
identify lessons learned and has undertaken numerous efforts to that 
effect. The GAO cites its opinion on page 18 that it is a shortcoming 
when best practices are chosen based on the opinions of HUD or 
contracted, staff. HUD has undertaken several initiatives with the 
goal of identifying lessons learned based on the data available, all 
of which are noted in the GAO Report: 

* From June 2008 to April 2009 HUD identified practices that, with 
statutory or regulatory changes, could be replicated at other PHAs 
nationwide. These practices, which are posted on HUD's website, 
included implementing savings accounts for public housing and voucher 
recipients to promote resident savings. 

* Most recently, HUD's 2010 MTW Report to Congress described promising 
policies, practices, and concerns. 

* In addition, interviewed officials from some of the MTW agencies 
noted that HUD officials had shared information on activities that had 
shown positive effects during site visits and annual conferences. The 
MTW Office also recently implemented quarterly conference calls and a 
quarterly newsletter to increase communication about lessons learned 
between MTW agencies and HUD. 

It should be noted that many of the issues that have been raised by 
GAO are the result of the initial program design, which allowed for 
flexibility without protocols in place to ensure that the 
demonstration would eventually produce demonstrable and replicable 
results. HUD has made major improvements in program administration in 
recent years through the adoption of the Standard MTW Agreement, which 
allowed for clearer program parameters and more consistent information 
collection, and the issuance of guidance and other materials to ensure 
compliance with program requirements and improved data collection. 

GAO Recommendation: Issue guidance that clarifies key program terms, 
such as the three statutory purposes of the program and the five 
statutory requirements that MTW agencies must meet.  

GAO Recommendation: Develop and implement a systematic process for 
assessing compliance with statutory requirements.  

HUD Response: Since assessing compliance requires a clear definition 
of the requirements, HUD requests that these two recommendations be 
combined.  

With regard to the statutory purposes, terms like "self-sufficiency" 
and "housing choice" can mean different things in different 
communities or even in different populations. Providing a menu of 
standard preferred metrics to PHAs may be the best option to ensure 
the data collected is useful in a cross-agency evaluation or 
assessment, but does not force PHAs into uniform goals that may not 
meet the needs of their communities. Key terms thus are defined by the 
metrics used to measure them.  

While HUD agrees that greater clarification of the statutory 
requirements can be helpful, this need may be overstated given the 
considerable progress in recent years — particularly with regard to 
defining and monitoring compliance with the statutory requirements. 
For the first time since the demonstration's inception, HUD is 
tracking compliance with the requirement to serve substantially the 
same number of families with a clear methodology and data verified 
through HUD systems and in consultation with local Field Offices. HUD 
is also taking the initial steps, through the pending revised Form 
50900, to require PHAs to provide further information on the 
requirement that 75 percent of assisted families be very low-income 
and that the agencies are serving a comparable mix of families. These 
efforts will establish the metrics used to measure compliance and by 
extension how HUD defines them.  

The following table summarizes the current tracking and monitoring 
mechanisms currently in place, along with anticipated improvements.  

Statutory Requirement: At least 75 percent of the families assisted by 
participating demonstration PHAs shall be very low-income families, as 
defined in section 3(b)(2) of the United States Housing Act of 1937; 
Current Tracking/Monitoring: The revised Attachment B/Form 50900 
(pending OMB approval) requires that agencies report additional 
verifiable information to confirm compliance with this statutory 
requirement. The Public Housing Information Center (PIC) also captures 
information about the income of assisted families. An initial 
assessment conducted by HUD confirms that at the present time 
approximately 90% of the residents at MTW agencies are very-low income. 

Statutory Requirement: Establish a reasonable rent policy, which shall 
be designed to encourage employment and self-sufficiency; 
Current Tracking/Monitoring: Each rent policy to encourage self-
sufficiency is locally defined. This information can be easily 
verified in consultation with MTW Plans and Reports. The few MTW 
agencies that do not yet have such a policy in place have plans to do 
so. 

Statutory Requirement: Continuing to assist substantially the same 
total number of eligible low-income families as would have been served 
had the amounts [of funding] not been combined; 
Current Tracking/Monitoring: HUD has recently conducted an extensive 
effort to provide defensible and quantifiable evidence that MTW 
agencies are meeting this requirement both individually and as a 
whole. HUD determined a baseline number of families that the MTW 
agency was serving upon entry to the demonstration and subsequently 
adjusted the baseline according to a defined set of variables. Now HUD 
maintains information on the actual number of families that an MTW 
agency is serving each year as compared it to the adjusted baseline. 
This results in a ratio that gives the MTW agency's compliance with 
this requirement on an annual basis.  

Not only does this effort ensure compliance with the "substantially 
the same" requirement, it quantifies the extent to which that 
requirement is being met. Because the methodology is consistent across 
agencies and gathered by HUD, aggregate results across the MTW program 
are also available. This systematic process is currently in a 
provisional year in order to test implementation. HUD plans to 
formalize the process further through the publication of a notice. 

Statutory Requirement: Maintaining a comparable mix of families (by 
family size) as would have been provided had the amounts [of funding] 
not been used under the demonstration; 
Current Tracking/Monitoring: HUD has developed a systematic process 
for assessing compliance with this statutory requirement in the 
revised Attachment B/Form 50900, currently under review by OMB. The 
revised Form 50900, which details the planning/reporting requirements 
of MTW agencies and once in effect is incorporated into the Standard 
MTW Agreement as Attachment B, requires MTW agencies to provide 
numerical information that verifies compliance with this statutory 
objective. 

Statutory Requirement: Assuring that housing assisted under the 
demonstration program meets housing quality standards established 
or approved by the Secretary; 
Current Tracking/Monitoring: This statutory requirement is assessed as 
it would be for any other non-MTW agency, as all agencies must meet 
this standard. A systematic process is managed by the Real Estate 
Assessment Center (REAC), which oversees on-site physical inspections. 
Scores for these physical inspections are recorded in HUD's Public 
Housing Assessment System (PHAS) module under the Physical Assessment 
Subsystem (PASS) section. While an MTW agency may elect not to be 
scored in the overall PHAS module (due to other aspects of the 
calculation that do not match how MTW agencies report information) 
PASS sub-scores for physical inspections are still recorded and 
monitored by REAC. 

[End of table] 

GAO Recommendation: Conduct an annual risk assessment for the MTW 
program and implement risk-based monitoring policies and procedures 
such as those currently being considered for site visits.  

HUD Response: Due to constraints on resources, the MTW Office has 
developed a risk-based strategy to determine which MTW agencies will 
receive an in-person site visit and which will be conducted as 
conference calls. The MTW Office is utilizing this approach to develop 
a formal risk based strategy for implementing monitoring policies and 
procedures. Specifically, the MTW Office is developing an approach to 
analyzing MTW activity risks with a focus on the impact of the 
activity on residents.  

Aside from site visits, HUD monitors risk in several other venues:  

* Attachment B/Form 50900 requires more information of activities that 
are deemed "higher risk." The current Attachment B/Form 50900 requires 
agencies to provide hardship case criteria and an impact analysis for 
all MTW activities that involve rent reform. This is based on the risk 
assessment that rent reforms can have significant impacts on 
residents. The impact analysis helps agencies identify the initial 
impacts and monitor the impacts over time. Section VIII of the HUD 
Form 50900 requires that agencies provide in the Report a description 
of progress on the correction or elimination of observed deficiencies 
cited in monitoring visits, physical inspections or other oversight 
and monitoring mechanisms. The MTW Office uses this information to 
assess whether MTW agencies are addressing the issues that have been 
identified through monitoring reviews as in need of correction.  

* As mentioned above, while not provided an overall score, MTW 
agencies are subject to physical inspections, and a physical 
assessment sub-score {PASS) is developed for each PHA property.  

* The MTW Office communicates regularly with HUD's Office of Field 
Operations (OFO) about risks since OFO has the routine responsibility 
for reviewing financial audits and Office of Management and Budget 
compliance audits. The MTW Office is also working in consultation with 
OFO to use the project management tool which compiles information from 
various HUD systems about housing authorities. The MTW Office and ODD 
plan to use this tool to conduct risk-based monitoring of policies and 
procedures. 

* The MTW Office has begun conducting regular meetings with REAC to 
discuss the financial reporting of MTW agencies into the Financial 
Data Schedule (FDS). During FY12, REAC will be providing training for 
the MTW coordinators to increase the staff s ability to analyze 
financial submissions. Staff will then be able to identify factors in 
FDS reporting that signal potential or actual risks to the MTW agency. 

GAO Recommendation: Implement control activities designed to verify 
the accuracy of a sample of the performers' information that MTW 
agencies self-report. 

HUD Response: Information provided by MTW PHAs in Plans and Reports 
may already be corroborated through a variety of venues, including 
traditional HUD monitoring systems (such as IMS-PIC and FDS), as well 
as other traditional monitoring reviews to which non-MTW PHAs are also 
subjected, including physical inspections conducted by the Real Estate 
Assessment Center, Fair Housing reviews, financial audits pursuant to 
OMB Circular A-133 and others. The MTW Working Group[Footnote 6] and 
Field Offices further review MTW Plans and Reports and corroborate 
applicable information. HUD is developing the business rules and 
requirements for a new data collection system, the Next Generation 
Management System (NGMS), and with the implementation of standard 
metrics for the MTW activities and requirements, this information can 
be integrated into the development of those rules, resulting in better 
monitoring and verification at both the field and headquarters level. 

Verifying activity performance data poses greater difficulty, as 
performance measures and data collection protocols are defined 
locally. HUD staff may not be equipped to verify the validity of 
information collected in a manner unique to a specific PHA. Verifying 
the accuracy of activity performance data may be best suited to a 
third-party assessor with experience analyzing a variety of data 
sources. 

Appendix II Footnotes: 

[1] Section VII of the Standard MTW Agreement. 

[2] See Sections VII.A.I.e. and VII.A.2.d. of the Standard MTW 
Agreement. 

[3] In footnote 11 on page 6, the GAO states that MTW agencies with 
less than 10 percent of their housing stock in the MTW program 
continue to submit the 5-year and annual plans required by Section 5A 
of the 1937 Act. However, there are no MTW agencies to which this 
applies; all MTW PHAs are submitting MTW Plans and Reports. 

[4] See the data reporting requirement provision in Section VII.B of 
the Standard MTW Agreement. 

[5] Section II.G of the Standard MTW Agreement. 

[6] In 2008 a working group was established composed of HUD staff from 
the public housing and voucher program offices, the Real Estate 
Assessment Center, field offices with MTW agencies, field operations 
staff, and researchers from the Office of Policy Development and 
Research. The group meets formally on a quarterly basis to discuss the
MTW program. The group members are also part of the Annual MTW Plan 
and Report review process. 

[End of section] 

Appendix III: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Mathew J. Scirè, (202) 512-8678 or sciremj@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, Paige Smith (Assistant 
Director), Anna Carbino, Emily Chalmers, John McGrail, Marc Molino, 
Lisa Moore, Daniel Newman, Lauren Nunnally, and Andrew Stavisky made 
key contributions to this report. 

[End of section] 

Footnotes: 

[1] This amount includes funding for public housing capital and 
operating expenditures and tenant-based rental assistance. HUD 
provides separate funding for project-based rental assistance, which 
is not included in this dollar amount. A PHA is typically a local 
agency created under state law that manages housing for low-income 
residents at rents they can afford. 

[2] Two of the 35 MTW agencies did not begin participating in the MTW 
program until fiscal year 2012 and are therefore not included in this 
dollar amount. 

[3] According to HUD data, as of January 2012 35 agencies were 
participating in the MTW program. Of these, 30 had submitted an annual 
report to HUD. The other five agencies had not been in the program 
long enough to report on their accomplishments. 

[4] Pub. L. No. 104-134, 110 Stat. 1321, 1321-281. 

[5] 42 U.S.C. 1437 et seq. 

[6] Section 3 of the 1937 Housing Act, as amended, defines very low-
income families as families whose incomes do not exceed 50 percent of 
the median family income for the area. 

[7] The Jobs-Plus initiative was a welfare-to-work demonstration aimed 
at significantly increasing employment and income among the residents 
of selected PHAs. HUD considers Jobs-Plus a subset of the MTW 
demonstration. 

[8] According to HUD, 3 of the original 24 MTW agencies completed 
their participation in 2004 and were terminated from the program based 
on the terms of their MTW agreements. 

[9] For additional information on these funding sources, see 
Congressional Research Service, Introduction to Public Housing, R41654 
(Washington, D.C.: Feb. 24, 2011) and Congressional Research Service, 
Section 8 Housing Choice Voucher Program: Issues and Reform Proposals, 
RL34002 (Washington, D.C.: Jan. 6, 2011). 

[10] Earlier agreements varied across participants and had terms that 
ran from 5 to 7 years. Amendments to extend the terms or add 
additional exceptions and flexibilities were common, and over time the 
changes made the agreements difficult for HUD to monitor. 

[11] MTW agencies with less than 10 percent of their housing stock in 
the MTW program continue to submit the 5-Year and annual plans 
required by Section 5A of the 1937 Housing Act. Only information not 
included in these documents would need to be included in a 
supplemental annual MTW plan. 

[12] In training materials for the MTW program, HUD defines "baseline" 
as the performance level that was being achieved prior to the 
implementation of the MTW activity, "benchmark" as the anticipated 
outcome of a MTW activity, and "metric" as the specific measure that 
quantifies the changes from an activity. 

[13] For purposes of this report, we are defining "performance 
indicators" as programwide measures of impact. 

[14] Of the 35 participating agencies, 30 had issued an annual report 
for 2011 as of January 2012. The other five agencies had not been in 
the program long enough to report on their accomplishments. 

[15] According to HUD officials, once an activity is approved, 
agencies may not continue to report its linkages to statutory purposes 
in subsequent plans or reports. 

[16] For purposes of this report, we are defining "performance 
information" as information that MTW agencies include in their annual 
reports on the impact of a specific MTW activity. 

[17] HUD has made some exceptions to its requirement for the reporting 
of impact information. For example, some MTW agencies identified the 
use of funding flexibility as an activity and noted that they were not 
required by HUD to report information on the impact. According to the 
Director of the MTW Office, this exception applies in some instances. 

[18] Agencies often had more than one metric for a single activity. 

[19] HUD does not penalize agencies for not meeting their self-
determined benchmarks. However, if the benchmarks were not achieved or 
if the activity was determined ineffective, HUD requires agencies to 
describe the challenges and, if possible, identify new strategies that 
might be more effective in their annual report. For our analysis, we 
compared the actual, reported performance information with the 
proposed benchmark to determine if the agency met its goal. 

[20] For example, see GAO, Neighborhood Stabilization Program: HUD and 
Grantees Are Taking Actions to Ensure Program Compliance but Data on 
Program Outputs Could be Improved, [hyperlink, 
http://www.gao.gov/products/GAO-11-48] (Washington, D.C.: Dec.17, 
2010). 

[21] GAO, Standards for Internal Control in the Federal Government, 
[hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] 
(Washington, D.C.: November 1999). 

[22] Pub. L. No. 111-352, 124 Stat. 3866 (2011). The GPRA 
Modernization Act of 2010 amends GPRA, Pub. L. No. 103-62, 107 Stat. 
285 (1993). 

[23] OMB Circular A-11, Part 6 ("Preparation and Submission of 
Strategic Plans, Annual Performance Plans, and Annual Program 
Performance Reports"), Section 220.7.IV. 

[24] See GAO, Program Evaluation: Improving the Flow of Information to 
the Congress, [hyperlink, http://www.gao.gov/products/GAO/PEMD-95-1] 
(Washington, D.C.: Jan. 30, 1995). 

[25] We also identified five studies that looked at a specific MTW 
agency or activity. See Thomas Boston and Linje Boston, Monitoring and 
Evaluating the Atlanta Housing Authority's MTW Program: Comprehensive 
Final Report Covering 2004, 2007 and 2010 (Atlanta, GA: Euquant, 
2010); Quadel Consulting Corporation, Performance Benchmarking of the 
Rent Simplification Initiative in the Cambridge Housing Authority's 
Moving to Work Program - Final Report (Cambridge, MA: Cambridge 
Housing Authority, 2008); Building Changes, The South King County 
Housing First Pilot Innovations and Lessons Learned (Seattle, WA: 
Building Changes, May 2010); Sarah Delaney Rosendahl, Housing 
Readiness Program Year One Evaluation for the Housing Authority of the 
County of San Mateo (Oakland, CA: Mills College, May 2010); and Thomas 
D. Boston, Impact of the Mixed-Income Revitalization of Grady Homes: 
Atlanta Housing Authority, 2011 (Atlanta, GA: Housing Authority of the 
City of Atlanta, July 2011). 

[26] Martin D. Abravanel, Robin E. Smith, Margery A. Turner, Elizabeth 
C. Cove, Laura E. Harris, and Carlos A. Manjarrez, Housing Agency 
Responses to Federal Deregulation: An Assessment of HUD's "Moving to 
Work" Demonstration (Washington, D.C.: The Urban Institute, 2004). 

[27] Because of the program's design and lack of controls, the study's 
authors had difficulty isolating the effects of the MTW program from 
other factors that may affect outcomes. 

[28] Robert Miller, Martin D. Abravanel, Helene Berlin, Elizabeth 
Cove, Maria-Alicia Newsome, Carlos A. Manjarrez, Lipi Saikia, Robin E. 
Smith, and Maxine V. Mitchell, The Experiences of Public Housing 
Agencies That Established Time Limits Policies Under the MTW 
Demonstration (Applied Real Estate Analysis, Inc. and The Urban 
Institute, May 2007). 

[29] HUD, Office of Public and Indian Housing and Office of Policy 
Research and Development, Moving to Work: Interim Policy Applications 
and the Future of the Demonstration, a report to Congress (Washington, 
D.C.: 2010). 

[30] [hyperlink, http://www.gao.gov/products/GAO/PEMD-95-1]. 

[31] GAO, Reexamining Regulations: Opportunities Exist to Improve 
Effectiveness and Transparency of Retrospective Reviews, [hyperlink, 
http://www.gao.gov/products/GAO-07-791] (Washington, D.C.: July 16, 
2007). 

[32] In its 2009 training materials, HUD defined a baseline as the 
performance level that was being achieved prior to the implementation 
of the MTW activity. 

[33] Abravanel and others, An Assessment of HUD's Moving to Work 
Demonstration (2004). 

[34] Researchers from the Urban Institute told us that if very 
specific reporting requirements were established and reliable data 
were available, the program could be rigorously assessed. 

[35] HUD, Moving to Work (2010). A controlled experimental methodology 
is a research design that randomly assigns participants to treatment 
and control groups in order to rigorously analyze the effects of the 
studied activity. 

[36] HUD, Moving to Work (2010). 

[37] Abravanel and others, An Assessment of HUD's Moving to Work 
Demonstration (2004). 

[38] HUD, Office of Inspector General, Design and Implementation of 
the Public Housing/Section 8 Moving to Work Demonstration Program, 
2005-SE-001 (Seattle, WA: Apr. 12, 2005). 

[39] Pub. L. No. 111-352, 124 Stat. 3866 (2011). 

[40] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[41] GAO, Tax Administration: IRS Needs to Further Refine Its Tax 
Filing Season Performance Measures, [hyperlink, 
http://www.gao.gov/products/GAO-03-143] (Washington, D.C.: Nov. 22, 
2002). 

[42] HUD, Fiscal Year 2011 Annual Performance Plan (Washington, D.C.: 
2011). 

[43] [hyperlink, http://www.gao.gov/products/GAO/PEMD-95-1]. 

[44] HUD, Moving to Work (2010). 

[45] HUD, Office of Inspector General, 2005-SE-001. 

[46] HUD's Real Estate Assessment Center administers physical 
inspections of HUD's public and multifamily housing properties. The 
Office of Policy Development and Research is responsible for 
maintaining current information on housing needs, market conditions, 
and existing programs, as well as for conducting research on priority 
housing and community development issues. 

[47] According to HUD officials, HUD staff "accept" rather than 
approve annual reports because, unlike the annual plans, the reports 
do not include a request to conduct certain activities. 

[48] Soliciting public comments on the plan is important because, 
unlike other PHAs, MTW agencies are not subject to the Public Housing 
Assessment System, which includes a customer satisfaction survey that 
promotes resident participation. 

[49] MTW agencies certify that they are assisting substantially the 
same total number of eligible low-income families that they would have 
served in the absence of MTW; maintaining a mix of families (by family 
size) comparable to those they would have served without the 
demonstration; and ensuring that at least 75 percent of households 
served are very low income. MTW agencies are not required to certify 
that they meet the other two requirements--that MTW agencies establish 
a reasonable rent policy and assure that housing provided meets HUD's 
quality standards. 

[50] Field office staff that we interviewed said that they reviewed 
annual plans and reports and sent their comments and concerns to MTW 
coordinators. 

[51] MTW agencies are subject to a variety of other reporting 
requirements. For example, MTW agencies are required to report voucher 
utilization in the Voucher Management System. They also must procure a 
public accountant to perform an Office of Management and Budget 
Circular A-133 compliance audit and submit unaudited financial 
statements. In addition, they are subject to HUD physical and 
management inspections of public housing and on-site monitoring 
reviews related to voucher reporting. 

[52] According to HUD officials, the nature of MTW site visits has 
changed since the implementation of the Standard Agreement in 2008. 
Prior to implementation of the Standard Agreement, HUD hired 
contractors to conduct site visits and provided monitoring guidance. 

[53] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[54] This requirement is contained in Attachment B of the Standard 
Agreement. As previously noted, an agency can associate an activity 
with more than one purpose. 

[55] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[56] GAO, Internal Control Management and Evaluation Tool, [hyperlink, 
http://www.gao.gov/products/GAO-01-1008G] (Washington, D.C.: August 
2001). 

[57] The baseline calculation also adjusts for any incremental 
vouchers that the agency may have been awarded over time. 

[58] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[59] HUD, Office of the Chief Financial Officer, Departmental 
Management Control Program, Handbook 1840.1 Rev-3 (Washington, D.C.: 
1999). 

[60] MTW agencies are exempt from scoring in the Public Housing 
Assessment System and the Section 8 Management Assessment System. 
However, MTW agencies are subject to physical inspections conducted by 
the Real Estate Assessment Center under HUD guidelines and issued a 
score. This score is entered into the Public Housing Assessment System 
and can be viewed by MTW staff at any time. A score of 22 or below is 
flagged by the Real Estate Assessment Center (the maximum score is 30) 
and reported to the appropriate field office. 

[61] Under this approach, the MTW Office would focus first on agencies 
that were implementing new activities or significant changes and visit 
these sites. 

[62] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] and 
[hyperlink, http://www.gao.gov/products/GAO-01-1008G. 

[63] GAO, Assessing the Reliability of Computer-Processed Data, 
[hyperlink, http://www.gao.gov/products/GAO-09-680G] (Washington, 
D.C.: July 2009). 

[64] Moving to Work Charter Program Act of 2011 (S. 117). 

[65] Moving to Work Improvement, Expansion, and Permanency Act of 2011 
(draft legislation); Affordable Housing and Self-Sufficiency 
Improvement Act of 2012 (AHSSIA) (draft legislation). 

[66] HUD, Moving to Work (2010). 

[67] HUD, Moving to Work (2010). 

[68] Abravanel and others, An Assessment of HUD's Moving to Work 
Demonstration (2004). 

[69] We recently reported that there were 160 different programs, tax 
expenditures, and other tools that supported homeownership and rental 
housing in fiscal year 2010, including multiple programs that provided 
funding for PHAs. See GAO, 2012 Annual Report: Opportunities to Reduce 
Duplication, Overlap and Fragmentation, Achieve Savings, and Enhance 
Revenue, [hyperlink, http://www.gao.gov/products/GAO-12-342SP] 
(Washington, D.C.: Feb. 28, 2012). 

[70] HUD, Moving to Work (2010). 

[71] Abravanel and others, An Assessment of HUD's Moving to Work 
Demonstration (2004). 

[72] There are three additional coordinators who are each responsible 
for a single MTW agency as designated in the agency's MTW agreement. 
These coordinators have other responsibilities within HUD and are not 
assigned to the MTW Office. 

[73] Work requirements can be met by attending job training programs 
as well as working. 

[74] Abravanel and others, An Assessment of HUD's Moving to Work 
Demonstration (2004). 

[75] HUD, Moving to Work (2010). 

[76] Abravanel and others, An Assessment of HUD's Moving to Work 
Demonstration (2004). 

[77] GAO, Housing Choice Vouchers: Options Exist to Increase Program 
Efficiencies, [hyperlink, http://www.gao.gov/products/GAO-12-300] 
(Washington, D.C.: Mar. 19, 2012). 

[78] GAO, Performance Measurement and Evaluation: Definitions and 
Relationships, [hyperlink, http://www.gao.gov/products/GAO-11-646SP] 
(Washington, D.C.: May 2011). 

[79] Of the 35 participating agencies, 30 had issued an annual report 
for 2011 as of January 2012. The other five agencies had not been in 
the program long enough to report on their accomplishments. 

[80] We reviewed the annual reports to identify ongoing activities. 
For these activities, we reviewed the reports to determine whether the 
activity was linked to a statutory purpose and the corresponding 
annual plan if that information was not available in the report. The 
results from the two analyses were combined. 

[81] GAO, Standards for Internal Control in the Federal Government, 
[hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] 
(Washington, D.C.: November 1999). 

[82] The Housing Authority of the County of Santa Clara and the 
Housing Authority of the City of San Jose are consolidated. We 
considered them as one MTW agency in our sample, as they report the 
information to HUD jointly. 

[83] See [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] 
and GAO, Tax Administration: IRS Needs to Further Refine Its Tax 
Filing Season Performance Measures, [hyperlink, 
http://www.gao.gov/products/GAO-03-143] (Washington, D.C.: Nov. 22, 
2002). 

[84] See [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]; 
GAO, Internal Control Management and Evaluation Tool, [hyperlink, 
http://www.gao.gov/products/GAO-01-1008G] (Washington, D.C.: August 
2001); and HUD, Office of the Chief Financial Officer, Departmental 
Management Control Program, Handbook 1840.1 Rev-3 (Washington, D.C.: 
1999). 

[85] Two legal aid agencies did not respond to our requests for 
interviews. 

[End of section] 

GAO’s Mission: 

The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the 
performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates 
federal programs and policies; and provides analyses, recommendations, 
and other assistance to help Congress make informed oversight, policy, 
and funding decisions. GAO’s commitment to good government is 
reflected in its core values of accountability, integrity, and 
reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each 
weekday afternoon, GAO posts on its website newly released reports, 
testimony, and correspondence. To have GAO e-mail you a list of newly 
posted products, go to [hyperlink, http://www.gao.gov] and select “E-
mail Updates.” 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of 
production and distribution and depends on the number of pages in the 
publication and whether the publication is printed in color or black 
and white. Pricing and ordering information is posted on GAO’s 
website, [hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 
MasterCard, Visa, check, or money order. Call for additional 
information. 

Connect with GAO: 

Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at [hyperlink, http://www.gao.gov]. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; 
E-mail: fraudnet@gao.gov; 
Automated answering system: (800) 424-5454 or (202) 512-7470. 

Congressional Relations: 

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-4400
U.S. Government Accountability Office, 441 G Street NW, Room 7125
Washington, DC 20548. 

Public Affairs: 
Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149 
Washington, DC 20548.