This is the accessible text file for GAO report number GAO-04-657 
entitled 'Workforce Investment Act: States and Local Areas Have 
Developed Strategies to Assess Performance, but Labor Could Do More to 
Help' which was released on June 01, 2004.

This text file was formatted by the U.S. General Accounting Office 
(GAO) to be accessible to users with visual impairments, as part of a 
longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

Report to Congressional Requesters:

United States General Accounting Office:

GAO:

June 2004:

Workforce Investment Act:

States and Local Areas Have Developed Strategies to Assess Performance, 
but Labor Could Do More to Help:

GAO-04-657:

GAO Highlights:

Highlights of GAO-04-657, a report to congressional requesters 

Why GAO Did This Study:

With rising federal deficits and greater competition for public 
resources, it is increasingly important for federal programs, such as 
the Workforce Investment Act (WIA) programs, to show results. This 
report examines (1) how useful WIA performance data are for gauging 
program performance; (2) what local areas are doing to manage their WIA 
performance and assess one-stops on a timely basis, and how states 
assist these efforts; and (3) the extent to which the Department of 
Labor is trying to improve WIA’s performance measurement system and 
assess one-stop success.

What GAO Found:

WIA performance data provide a long-term national picture of outcomes, 
but these data offer little information about current performance and 
represent a small portion of job seekers who received WIA services. 
Unemployment Insurance wage records—the primary data source for 
tracking WIA performance—provide reliable outcome information over 
time. But they have shortcomings, such as not including some categories 
of workers, and considerable time lags before data are available. Many 
states rely on alternative data sources to fill gaps in the wage 
records. However, the time between when a participant receives services 
and when their outcomes are reported to Labor can range from about 1½ 
to 2½ years or longer. In addition, states’ annual reports reflect only 
a small portion of job seekers who receive WIA services because of 
restrictions in the law and policies of Labor. 

With assistance from states, many local areas collect interim outcome 
information from former participants or employers and use other interim 
indicators to track WIA performance levels long before wage record data 
are available. However, states and local areas would like more help 
from Labor in disseminating best practices on interim performance 
measures. In addition, these efforts tell them little about the 
performance of their overall one-stop systems. Many states and local 
areas rely on other indicators—job seeker measures, employer measures, 
program partnership measures, and family and community indicators to 
assess their one-stops. 

Labor has taken steps to improve WIA’s performance system and assess 
one-stops, but could do more. Although Labor is studying adjustment 
methods that could better take into account local differences when 
negotiating performance levels, it has not committed to using such a 
method nationally. Labor also has efforts to improve the quality of 
WIA’s performance data and is developing a set of common measures for 
one-stop partner programs. Yet as part of the common measures, Labor 
plans to restrict the use of alternative data. Labor has also delayed 
plans to conduct an impact evaluation and will not meet its statutory 
requirement to do so by 2005.

Time Delay in Reporting Employment Outcomes is a Minimum of 17 Months: 

[See PDF for image]

[End of figure]

What GAO Recommends:

GAO recommends that Labor continue to allow supplemental data for 
reporting outcomes; assist states and localities in sharing promising 
practices on interim indicators; develop a systematic method to account 
for different populations and economic conditions when negotiating 
performance levels; and expedite steps to implement an impact 
evaluation of WIA services. GAO also suggests that Congress may wish to 
consider requiring that all WIA participants be tracked for reporting 
purposes. Labor generally agreed with our findings and recommendations, 
but it did not agree with our recommendation to expedite WIA’s impact 
evaluation. GAO believes that expediting this evaluation is essential 
to help policymakers assess WIA’s effectiveness.

www.gao.gov/cgi-bin/getrpt?GAO-04-657.

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Dianne Blank at (202) 
512-5654 or blankd@gao.gov.

[End of section]

Contents:

Letter:

Results in Brief:

Background:

WIA Performance Data Provide a Long-Term National Picture of Outcomes, 
but Are Less Useful for Gauging Current Performance and Represent a 
Small Portion of WIA Participants:

States and Local Areas Manage WIA Performance and Assess One-Stops by 
Collecting Timely Data and Making Use of a Range of Performance 
Information:

Labor Has Taken Actions to Improve WIA's Performance Measurement System 
and Assess One-Stops but Could Do More:

Conclusions:

Recommendations For Executive Action:

Matter for Congressional Consideration:

Agency Comments:

Appendix I: Objectives, Scope, And Methodology:

Appendix II: States' Use of UI Wage Records and Other Data Sources for 
Reporting on Employment Outcomes Under WIA:

Appendix III: Comments from the Department of Labor:

Appendix IV: GAO Contacts and Staff Acknowledgments:

GAO Contacts:

Staff Acknowledgments:

GAO Related Products:

Tables:

Table 1: Performance Measures and Allowable Data Sources for the WIA-
Funded Programs:

Table 2. WIA's Mandatory One-Stop Partner Programs and Related Federal 
Agencies:

Table 3: Program Year 2002 National Entered Employment Rates for WIA 
Programs:

Table 4: WIA Performance Measures That Are Tracked or Partially Tracked 
Using UI Wage Records:

Table 5: Strategies Local Areas Use to Hold Service Providers 
Accountable to WIA Performance Levels:

Table 6: Proposed Common Measures and Data Sources:

Table 7: Employment and Training Administration Programs Subject to the 
Common Measures:

Table 8: Survey Numbers and Response Rates:

Table 9: States and Local Areas in Our Study:

Figures:

Figure 1: Number of States That Needed Supplemental Data to Demonstrate 
That They Met Minimum Performance Levels on One or More Measures:

Figure 2: Estimated Number of Months for States to Get Job Placement 
Information from UI Wage Records:

Figure 3: Hypothetical Example Comparing Time Elapsed Between 
Registration and When Outcomes Are Reported for Two Participants:

Figure 4: The Percentage of Local Areas That Report Interim Indicators 
at Least Monthly:

Figure 5: Activities That Local Areas Say Are Greatly Influenced by WIA 
Performance Information:

Figure 6: Four Types of Indicators That States and Local Areas Use to 
Assess Performance of One-Stops:

Figure 7: Information That Local Areas Collect on Job Seekers Who Visit 
One-Stops:

Figure 8: Information That Local Areas Collect on Employers Using One-
Stops:

Figure 9: States That Reported Being at Risk of Not Meeting 80 Percent 
of Negotiated Performance Levels on One or More Measure for Program 
Year 2002:

Abbreviations:

GPRA: Government Performance and Results Act:

IT: information technology:

JTPA: Job Training Partnership Act:

OMB: Office of Management and Budget:

PART: Program Assessment Rating Tool:

TANF: Temporary Assistance for Needy Families:

TEGL: Training and Employment Guidance Letter:

UI: Unemployment Insurance:

WIA: Workforce Investment Act:

WIASRD: Workforce Investment Act Standardized Record Data:

WRIS: Wage Record Interchange System:

United States General Accounting Office:

Washington, DC 20548:

June 1, 2004:

The Honorable Edward M. Kennedy: 
Ranking Minority Member: 
Committee on Health, Education, Labor and Pensions: 
United States Senate:

The Honorable Patty Murray: 
Ranking Minority Member: 
Subcommittee on Employment, Safety and Training: 
Committee on Health, Education, Labor, and Pensions: 
United States Senate:

As the nation faces rising budget deficits and greater competition for 
federal resources to address 21st century challenges, it is becoming 
increasingly important for federal programs to demonstrate that they 
are meeting their long-range goals. Policy makers at all levels are 
seeking to understand whether our current employment and training 
system is making a difference in helping people get and keep jobs. 
Programs whose services are provided through the Workforce Investment 
Act's (WIA) one-stop centers constitute the largest portion of federal 
employment and training funds that provide direct assistance, with the 
three programs funded under WIA--Adults, Dislocated Workers, and Youth-
-totaling about $3.3 billion in fiscal year 2004. With the passage of 
WIA in 1998, lawmakers envisioned a consolidated new system for 
delivering most federally funded employment and training programs--one 
that was more efficient and effective than prior programs. To assess 
whether it is accomplishing its goals, WIA established a rigorous 
performance accountability structure for the programs directly funded 
by WIA--one that emphasized outcomes in areas of job placement, 
retention, earnings, and skill attainment, as well as customer 
satisfaction. From federal long-range goals to local program policies, 
it is critical to demonstrate results at all levels with accurate 
outcome data, timely program management information, and impact studies 
to assess effectiveness.[Footnote 1] But we and others have raised 
questions about whether anyone has a clear picture of what WIA-funded 
programs are achieving.

Because you were concerned whether WIA's performance reporting system 
allows for a useful and accurate assessment of program achievements and 
to better understand how the performance system allows for local 
assessment of program successes, we examined (1) how useful WIA 
performance data are for gauging program performance, (2) what local 
areas are doing to manage their WIA performance and assess one-stop 
success on a timely basis and how states are assisting these efforts, 
and (3) to what extent the Department of Labor is trying to improve 
WIA's performance measurement system and assess one-stop success.

To learn more about what the data tell us about outcomes achieved under 
WIA, we examined states' annual reports, interviewed Labor officials, 
and reviewed federal performance and reporting requirements. In 
addition, to determine how states and local areas track interim 
progress toward meeting their WIA performance levels that were 
negotiated with Labor and to understand what they use to assess overall 
one-stop success, we surveyed the 50 states, as well as all 568 local 
workforce investment areas. We received responses from all 50 states 
and 463 local workforce investment areas (81.5 percent). We also 
visited three states--Florida, Michigan, and Utah--and at least two 
local areas or one-stops in each of these states. We selected these 
states because they are geographically diverse, have implemented 
additional performance measures or strategies to assess one-stop 
success, and have developed integrated statewide data systems. We 
supplemented our site visits with telephone interviews with state and 
local officials in Pennsylvania and interviews with experts in the area 
of workforce development performance measurement. Our review focused 
primarily on the employment-based measures that are tracked or 
partially tracked with the Unemployment Insurance wage records--entered 
employment rate, earnings change/replacement rate, employment 
retention rate, employment and credential rate, and the younger youth 
placement and retention rate. We conducted our work between April 2003 
and April 2004 in accordance with generally accepted government 
auditing standards.

Results in Brief:

WIA performance data are useful for providing a long-term national 
picture of program outcomes; however, these data provide little 
information about current performance and represent only a small 
portion of job seekers that received WIA services. Unemployment 
Insurance wage records--the primary data source for tracking WIA 
performance--have significant advantages over other data sources by 
providing a reliable, cost-effective method to track employment 
outcomes over time. But these data have some shortcomings--about 6 
percent of workers, such as self-employed persons, are not included in 
the wage records, and there are considerable time lags before the data 
are available. To fill this gap, 39 states reported that they use other 
data sources, such as follow-up with participants and employers, to 
report on participants not covered by wage data. Of these 39 states, 23 
relied on these other data sources to demonstrate that they had met 
their minimum performance levels on one or more measures. The time lags 
and other factors affect the timing of states' annual performance 
reports to Labor and, subsequently, Labor's reports to Congress. Most 
of the WIA outcomes data reported in a given program year actually 
reflect participants who left the program in the prior year, limiting 
usefulness for gauging current performance. The minimum time period 
between when participants receive services and when their outcomes are 
reported to Labor is about 1½ years and may be as much as 2½ years, or 
even longer, depending on the length of services received. Further, the 
WIA performance data provide a limited picture of participants served 
because job seekers that use self-directed services and those requiring 
only limited staff assistance are not required to be included in the 
performance reports, and this group likely represents the largest 
portion of job seekers served under WIA.

To manage their WIA performance and assess one-stop performance on a 
timely basis, local areas collect performance information that is not 
readily available from the Unemployment Insurance wage data, with 
states providing technical assistance such as information technology. 
About three-fourths of local areas we surveyed collect interim WIA 
performance data by contacting former participants or employers. Nearly 
all local areas regularly monitor interim information, such as job 
placement and wages at exit, to help them assess whether or not they 
will meet their performance goals. Although both states and local areas 
devote considerable resources to assessing WIA performance, these 
efforts alone tell them little about the value of their entire one-stop 
systems, which is the required service delivery system for WIA and most 
other federally funded employment and training services. Many states 
and local areas reported that they attempt to gauge one-stop success by 
relying on their own measures, using four basic categories of 
indicators--measures of job seekers, such as the number who use the 
one-stop in a single time period; measures of employers, such as the 
number who hire one-stop customers; measures to assess how well program 
partners are working together, such as the number of referrals made to 
one-stop partners; and indicators of how well the one-stops are meeting 
the needs of the family and community, such as changes in average 
household income. States provide assistance to local areas in a variety 
of ways, ranging from supporting their information technology systems 
to training local staff in using the WIA performance measures. While 
states and localities have made progress in developing interim WIA and 
overall one-stop measures, nearly all would like Labor to disseminate 
promising practices information on other interim indicators that they 
might use to gauge their success.

Although Labor has taken actions to improve WIA's performance 
measurement system and assess outcomes across one-stop partners, some 
of these efforts do not go far enough. Many state and local officials 
we interviewed said they think Labor does not adequately consider 
economic and demographic factors when negotiating performance levels 
with states. Labor has commissioned a study of adjustment methods that 
could better take these differences into account. However, even if the 
study produces an acceptable model, Labor has made no commitment to put 
such a standard adjustment method in place nationally. Labor has taken 
steps to improve the collection of outcome data required to assess the 
WIA programs and is defining common measures across partner programs 
that will help assess outcomes for the one-stop system. Yet as part of 
this common measure effort, Labor plans to restrict the use of 
supplemental data for filling gaps in Unemployment Insurance wage 
records because of concerns about the quality of the supplemental data. 
If this restriction is applied to the WIA measures, it may affect the 
ability of states and local areas to demonstrate that they are meeting 
their negotiated performance levels. While Labor has plans to conduct 
impact studies, the department will not meet WIA's requirement to 
conduct at least one multi-site impact study by 2005, and without such 
a study, little will be known about WIA's effectiveness.

To compensate for the impact of changes in the economy and to give 
states and local areas an equal opportunity to meet their performance 
levels, we recommend that the Secretary of Labor continue to allow the 
use of supplemental data for reporting outcomes, provide assistance to 
states and localities in developing and sharing promising practices on 
interim indicators for assessing WIA's performance, and develop an 
adjustment model or other systematic method to account for different 
populations and local economic conditions when negotiating performance 
levels. To comply with statutory requirements and to help federal, 
state, and local policy makers understand what services are most 
effective for improving employment-related outcomes, we recommend that 
the Secretary of Labor expedite steps to design and implement an impact 
evaluation of WIA services. In addition, we are proposing that Congress 
consider requiring that all WIA participants be tracked for reporting 
purposes. In its written comments, Labor generally agreed with our 
findings and recommendations. However, it did not agree with our 
recommendation to expedite efforts to implement an impact evaluation of 
WIA services because Labor believes program consolidation changes in 
proposed reauthorization bills are significant enough to delay the 
evaluation. We disagree that proposed reauthorization changes would 
significantly affect the basic one-stop service delivery structure 
under WIA. Further, we believe that expediting this evaluation is 
essential because the program has been implemented for over 4 years and 
policymakers need impact information to assess WIA's effectiveness and 
inform future discussion.

Background:

Labor required states to implement major provisions of WIA by July 1, 
2000, although some states began implementing provisions of WIA as 
early as July 1999. Services provided under WIA represent a marked 
change from those provided under the previous program, allowing for a 
greater array of services to the general public. WIA requires that many 
federal programs provide employment and training services through one-
stop centers. WIA is designed to provide for greater accountability 
than under previous law: it established new performance measures, a 
requirement to use Unemployment Insurance (UI) wage data to track and 
report on outcomes, and a requirement to conduct at least one multi-
site control group evaluation.

WIA-Funded Services:

When WIA was enacted in 1998, it replaced the Job Training Partnership 
Act (JTPA) programs for economically disadvantaged adults and youth and 
for dislocated workers with three programs--WIA Adult, Dislocated 
Worker, and Youth--that provide a broader range of services to the 
general public, no longer using income to determine eligibility for all 
program services.[Footnote 2] WIA programs provide for three tiers, or 
levels, of service for adults and dislocated workers: core, intensive, 
and training. Core services include basic services such as job searches 
and labor market information. These activities may be self-service or 
require some staff assistance. Intensive services include such 
activities as comprehensive assessment and case management--activities 
that require greater staff involvement. Training services include such 
activities as occupational skills or on-the-job training. Labor's 
guidance provides for monitoring and tracking for the adult and 
dislocated worker programs to begin when job seekers receive core 
services that require significant staff assistance. WIA excludes job 
seekers who receive core services that are self-service and 
informational in nature from being included in the performance 
measures. WIA's youth program does not have three tiers of services, 
but instead requires that 10 youth services, referred to as program 
elements, be made available to all eligible youth. All youth who are 
determined eligible and receive WIA services are included in the 
performance measures.

WIA Performance Measures and a Required Evaluation Are Designed to 
Increase Accountability for Three WIA-Funded Programs:

WIA is designed to provide for greater accountability than its 
predecessor program by establishing new performance measures, a new 
requirement to use UI wage data to track and report on outcomes, and a 
requirement for Labor to conduct at least one multi-site control group 
evaluation. According to Labor, performance data collected from the 
states in support of the measures are intended to be comparable across 
states in order to maintain objectivity in determining incentives and 
sanctions. The performance measures also provide information to support 
Labor's performance goals under the Government Performance and Results 
Act (GPRA),[Footnote 3] the budget formulation process using the Office 
of Management and Budget's (OMB) Program Assessment Rating Tool 
(PART),[Footnote 4] and for program evaluation required under WIA.

In contrast to JTPA, for which data on outcomes were obtained through 
follow-ups with job seekers, WIA requires states to use UI wage records 
to track employment-related outcomes. Each state maintains UI wage 
records to support the process of providing unemployment compensation 
to unemployed workers. The records are compiled from data submitted to 
the state each quarter by employers and primarily include information 
on the total amount of income earned during that quarter by each of 
their employees. Although UI wage records contain basic wage 
information for about 94 percent of workers, certain employment 
categories are excluded, such as self-employed persons, independent 
contractors, federal employees, and military personnel. According to 
Labor's guidance, if a program participant does not appear in the UI 
wage records, states may use supplemental data sources, such as follow-
up with participants and employers, or other administrative databases, 
such as U.S. Office of Personnel Management or U.S. Department of 
Defense records, to track most of the employment-related measures. 
However, only UI wage records may be used to calculate earnings change 
and earnings replacement. (See table 1 for a complete list of the WIA 
performance measures and data sources used for tracking the measures.):

Table 1: Performance Measures and Allowable Data Sources for the WIA-
Funded Programs:

Program: Adult; 
Performance measures: 1. Entered employment rate; 
UI wage records: Yes; 
Supplemental data allowed: Yes; 
Other, such as educational data or survey: No. 

Program: Adult; 
Performance measures: 2. Employment retention rate at 6 months; 
UI wage records: Program: Yes; 
Supplemental data allowed: Program: Yes; 
Other, such as educational data or survey: Program: No. 

Program: Adult; 
Performance measures: 3. Average earnings change in 6 months; 
UI wage records: Program: Yes; 
Supplemental data allowed: Program: No; 
Other, such as educational data or survey: Program: No. 

Program: Adult; 
Performance measures: 4. Entered employment and credential rate; 
UI wage records: Yes; 
Supplemental data allowed: Yes; 
Other, such as educational data or survey: Yes. 

Program: Dislocated worker; 
Performance measures: 5. Entered employment rate; 
UI wage records: Yes; 
Supplemental data allowed: Yes; 
Other, such as educational data or survey: No. 

Program: Dislocated worker; 
Performance measures: 6. Employment retention rate at 6 months; 
UI wage records: Program: Yes; 
Supplemental data allowed: Program: Yes; 
Other, such as educational data or survey: Program: No. 

Program: Dislocated worker; 
Performance measures: 7. Earnings replacement rate in 6 months; 
UI wage records: Program: Yes; 
Supplemental data allowed: Program: No; 
Other, such as educational data or survey: Program: No. 

Program: Dislocated worker; 
Performance measures: 8. Entered employment and credential rate; 
UI wage records: Yes; 
Supplemental data allowed: Yes; 
Other, such as educational data or survey: Yes. 

Program: Youth (age 19-21); 
Performance measures: 9. Entered employment rate; 
UI wage records: Yes; 
Supplemental data allowed: Yes; 
Other, such as educational data or survey: Yes. 

Program: Youth (age 19-21); 
Performance measures: 10. Employment retention rate at 6 months; 
UI wage records: Program: Yes; 
Supplemental data allowed: Program: Yes; 
Other, such as educational data or survey: Program: Yes. 

Program: Youth (age 19-21); 
Performance measures: 11. Average earnings change in 6 months; 
UI wage records: Program: Yes; 
Supplemental data allowed: Program: No; 
Other, such as educational data or survey: Program: Yes. 

Program: Youth (age 19-21); 
Performance measures: 12. Entered employment/ education/training and 
credential rate; 
UI wage records: Yes; 
Supplemental data allowed: Yes; 
Other, such as educational data or survey: Yes. 

Program: Youth (age 14-18); 
Performance measures: 13. Skill attainment; 
UI wage records: No; 
Supplemental data allowed: No; 
Other, such as educational data or survey: Yes. 

Program: Youth (age 14-18); 
Performance measures: 14. Diploma or equivalent; 
UI wage records: Program: No; 
Supplemental data allowed: Program: No; 
Other, such as educational data or survey: Program: Yes. 

Program: Youth (age 14-18); 
Performance measures: 15. Placement and retention rate; 
UI wage records: Yes; 
Supplemental data allowed: Yes; 
Other, such as educational data or survey: Yes. 

Program: All programs; 
Performance measures: 16. Customer satisfaction for participants; 
UI wage records: No; 
Supplemental data allowed: No; 
Other, such as educational data or survey: Yes. 

Program: All programs; 
Performance measures: 17. Customer satisfaction for employers; 
UI wage records: No; 
Supplemental data allowed: No; 
Other, such as educational data or survey: Yes. 

Source: U.S. Department of Labor.

[End of table]

Unlike JTPA, which established expected performance levels using a 
computer model, WIA requires states to negotiate with Labor to 
establish expected performance levels for each measure. States, in 
turn, must negotiate performance levels with each local area. The law 
requires that these negotiations take into account differences in 
economic conditions, participant characteristics, and services 
provided. To derive equitable performance levels, Labor and the states 
primarily rely on historical data to develop their estimates of 
expected performance levels. These estimates provide the basis for 
negotiations.

WIA holds states accountable for achieving their performance levels by 
tying those levels to financial sanctions and incentive funding. States 
that meet their performance levels under WIA are eligible to receive 
incentive grants that generally range from $750,000 to $3 million. 
States that do not meet at least 80 percent of their WIA performance 
levels are subject to sanctions. If a state fails to meet its 
performance levels for 1 year, Labor provides technical assistance, if 
requested. If a state fails to meet its performance levels for 2 
consecutive years, it may be subject to up to a 5-percent reduction in 
its annual WIA formula grant. At the end of program year 2001, four 
states received financial sanctions.

Labor determines incentive grants or sanctions based on the performance 
data that states must submit each December in their annual 
reports.[Footnote 5] States also submit quarterly performance reports, 
which are due 45 days after the end of each quarter. In addition to the 
performance reports, states submit their updates for the Workforce 
Investment Act Standardized Record Data (WIASRD) every January. All 
three submissions primarily represent participants who have exited the 
WIA programs within the previous program year (July 1 - June 30). 
[Footnote 6]

WIA also requires Labor to conduct at least one multi-site control 
group evaluation by the end of fiscal year 2005. WIA requires that 
evaluations address the general effectiveness of programs and 
activities in relation to costs and the impact of these services on the 
community and participants involved.

WIA Requires That Many Federal Programs Work Together to Provide 
Services through the One-Stop System:

WIA requires that states use the one-stop center system to provide 
services for many employment and training programs. Seventeen programs 
funded through four federal agencies are now required to provide 
services through the one-stop center under WIA. Table 2 shows the 
programs that WIA requires to provide services through the one-stop 
centers (termed mandatory programs) and the related federal agencies.

Table 2: WIA's Mandatory One-Stop Partner Programs and Related Federal 
Agencies:

Federal agency: Department of Labor; 
Mandatory one-stop partner programs: 
* WIA Adult; 
* WIA Dislocated Worker; 
* WIA Youth; 
* Employment Service (Wagner-Peyser); 
* Trade Adjustment Assistance programs; 
* Veterans 'employment and training programs; 
* Unemployment Insurance; 
* Job Corps; 
* Welfare-to-Work grant-funded programs; 
* Senior Community Service Employment Program; 
* Employment and training for migrant and seasonal farm workers; 
* Employment and training for Native Americans.

Federal agency: Department of Education; 
Mandatory one-stop partner programs: 
* Vocational Rehabilitation Program; 
* Adult Education and Literacy; 
* Vocational Education (Perkins Act).

Federal agency: Department of Health and Human Services; 
Mandatory one-stop partner programs: 
* Community Services Block Grant.

Federal agency: Department of Housing and Urban Development (HUD); 
Mandatory one-stop partner programs: 
* HUD-administered employment and training. 

Source: U.S. Department of Labor.

[End of table]

Under WIA, employers are expected to play a key role in establishing 
regional workforce development policies, deciding how services should 
be provided in the one-stop, and overseeing one-stop operations. 
Employers, who are encouraged to use the one-stop system to fill their 
job vacancies, are also seen as key one-stop customers under WIA.

WIA Performance Data Provide a Long-Term National Picture of Outcomes, 
but Are Less Useful for Gauging Current Performance and Represent a 
Small Portion of WIA Participants:

WIA performance data are useful for providing a long-term national 
picture of program outcomes; however, these data are less useful for 
providing information about current performance, and represent only a 
small portion of job seekers that received WIA services. UI wage 
records--the primary data source for tracking WIA performance--provide a 
fairly consistent national view of WIA performance and allow for 
tracking outcomes over time. At the same time, the UI wage records have 
some shortcomings--they cannot be used to track job seekers who get 
jobs in other states unless states share data; they do not cover 
certain categories of workers, such as self-employed persons; and they 
are not available on a timely basis. States are making progress in 
overcoming some of these shortcomings by sharing wage data with other 
states and supplementing information on participants not covered by the 
wage data. Despite this progress, time lags and other factors affect 
the timing of states' reports on their annual performance to Labor and, 
subsequently, Labor's reports to Congress. Most of the outcomes data 
reported in a given program year actually reflect participants who left 
the program during the prior year, limiting usefulness for gauging 
current program performance. In addition, the states' annual reports 
reflect only a small portion of job seekers who receive WIA services 
because, under the law and Labor's guidance, not all job seekers who 
utilize one-stop services are required to be included in the 
performance reports.

WIA Annual Performance Data Are Useful for Measuring Outcomes Over the 
Long Term:

WIA annual performance reports--which provide a summary of states' 
performance on the 17 core measures--are useful for providing a national 
perspective of outcomes achieved over time. The information presented 
in the annual reports compares states' negotiated performance levels 
with their actual performance levels. (See table 3 for an example of 
national performance levels for WIA's job placement rate--called the 
entered employment rate--in program year 2002.) These reports provide 
Congress with an annual picture of how well the WIA program is meeting 
its long-range goals to increase the employment, retention, and 
earnings of participants. The WIA performance data are also useful to 
help Labor assess quantitative, outcomes-oriented goals for its 
strategic plans, annual performance plans, and annual performance 
reports required by the Government Performance and Results Act (GPRA). 
In its annual performance report for program year 2002,[Footnote 7] 
Labor used the WIA outcome measures to assess its progress in meeting 
its strategic goals to increase employment, earnings, and assistance to 
adults and increase the number of youth in education or making a 
successful transition to work.[Footnote 8]

Table 3: Program Year 2002 National Entered Employment Rates for WIA 
Programs:

WIA program: Adults; 
Negotiated performance level[A]: 70.6 %; 
Actual performance level: 75.1 %.

WIA program: Dislocated Workers; 
Negotiated performance level[A]: 76.8%; 
Actual performance level: 83.3%.

WIA program: Older Youth (19-21 years old); 
Negotiated performance level[A]: 65.0%; 
Actual performance level: 69.5%.

Source: U.S. Department of Labor.

[A] The national negotiated performance level is calculated as an 
average of all of the states' negotiated performance levels.

[End of table]

Most of the performance outcomes in the annual reports are measured 
using UI wage records--13 of the 17 WIA performance measures rely on UI 
wage records as the primary data source for tracking employment 
outcomes.[Footnote 9] (See table 4.) States maintain UI wage records to 
determine whether unemployed workers qualify for unemployment 
compensation. The records are compiled from data submitted to the state 
each quarter by employers and primarily include information on the 
total amount of wages paid to employees in the quarter. However, UI 
wage records for most states do not include information on the number 
of hours an employee worked during the quarter and when in the quarter 
the wages were earned. For example, the UI wage records for most states 
would not show that one employee may have worked 40 hours a week for 
the entire quarter and another worker may have worked 35 hours a week 
for the last two weeks of the quarter. The UI wage records would 
provide an overall snapshot of the total amount of wages paid to both 
employees for the quarter.

Table 4: WIA Performance Measures That Are Tracked or Partially Tracked 
Using UI Wage Records:

Program: Adult; 
Performance measures: 1. Entered employment rate.

Program: Adult; 
Performance measures: Program: 2. Employment retention rate at 6 
months.

Program: Adult; 
Performance measures: Program: 3. Average earnings change in 6 months.

Program: Adult; 
Performance measures: 4. Entered employment and credential rate.

Program: Dislocated worker; 
Performance measures: 5. Entered employment rate.

Program: Dislocated worker; 
Performance measures: Program: 6. Employment retention rate at 6 
months.

Program: Dislocated worker; 
Performance measures: Program: 7. Earnings replacement rate in 6 
months.

Program: Dislocated worker; 
Performance measures: 8. Entered employment and credential rate.

Program: Youth (19-21 years old); 
Performance measures: 9. Entered employment rate.

Program: Youth (19-21 years old); 
Performance measures: Program: 10. Employment retention rate at 6 
months.

Program: Youth (19-21 years old); 
Performance measures: Program: 11. Average earnings change in 6 
months.

Program: Youth (19-21 years old); 
Performance measures: 12. Entered employment/education/training and 
credential rate.

Program: Youth (14-18 years old); 
Performance measures: 13. Placement and retention rate. 

Source: U. S. Department of Labor.

[End of table]

The UI wage records provide a common yardstick for long-term 
comparisons across states because they contain wage and employment 
information on about 94 percent of the working population in the United 
States, and all states collect and retain these data.[Footnote 10] In 
addition, UI wage records can be used as a common data source to track 
employment outcomes across multiple programs, such as vocational 
education and the Temporary Assistance for Needy Families (TANF) 
[Footnote 11] programs. Further, researchers have found that wage 
record data are more objective and cost-effective than traditional 
survey information.[Footnote 12] For example, one state estimated that 
the cost of doing participant surveys, as was done under JTPA, was 
approximately $13.25 per participant compared with the cost of 
automated record matching to UI wage records, which costs less than 
$.05 per participant. UI wage records make it easier to track longer-
term measures, such as those that assess earnings change, earnings 
replacement, and employment retention 6 months after participants leave 
the program. Without UI wage records, tracking these outcomes would 
require contacting or surveying former participants, perhaps multiple 
times, after they leave the program.

UI Wage Records Have Some Shortcomings, and Many States Rely on 
Supplemental Data to Report on Their Performance:

UI wage records also have some shortcomings. State wage record 
databases only include wage information on job seekers within their 
state; they do not track job seekers who find jobs in other states. 
States cannot readily gain access to UI wage records from other states, 
making it difficult to track individuals who receive services in one 
state but get a job in another. To help gain access to wage information 
in other states, Labor established the Wage Record Interchange System 
(WRIS) --a clearinghouse that makes UI wage records available to states 
seeking employment and wage information on their WIA participants, and 
states are increasingly making use of this option. Nationwide, 38 
states reported that they currently participate in WRIS,[Footnote 13] 
an increase from the 15 states that told us they were planning to or 
participating in WRIS in 2001.[Footnote 14] States may also elect to 
establish their own agreements to share wage information with other 
states--often those that share a common border. Seven of the 38 states 
reported that they maintain their own interstate agreements with other 
states and they also participate in WRIS. One state official we 
interviewed said the state maintains its own agreements in addition to 
WRIS so that the state can get data more quickly than through WRIS. 
According to a Labor official, states often retrieve wage record data 
from other states within a matter of days using WRIS. However, the 
process can take much longer--up to a couple of weeks--if participating 
states take longer to respond to requests.

In addition, even though UI wage records contain information on about 
94 percent of workers, they do not contain information on certain 
employment categories of workers, such as self-employed persons, most 
independent contractors, military personnel, federal government 
workers, and postal workers. To compensate for the 6 percent of workers 
who are not in the UI wage records, Labor allows states to report 
employment outcomes using other data sources--for example, by contacting 
participants after they leave the program--to track WIA participants who 
are employed in these uncovered occupations. We found that 39 states 
reported relying on this supplemental information to report on 
participants not covered by the wage data. Twenty-three states told us 
that without the supplemental data, they would not have been able to 
show that they met minimum performance levels on at least one measure, 
and 10 of these states said they would not have been able to show that 
they met minimum performance levels on 10 of the measures in program 
year 2001. (See fig. 1.):

Figure 1: Number of States That Needed Supplemental Data to Demonstrate 
That They Met Minimum Performance Levels on One or More Measures:

[See PDF for image]

[End of figure]

Labor also allows states to use other employment and administrative 
data sources to track employees excluded from the UI wage records, such 
as the U.S. Office of Personnel Management, the U.S. Postal Service, 
and the U.S. Department of Defense. Eight states reported that they 
currently fill gaps in coverage using other administrative and 
employment data sources. Labor has recently established an agreement 
with the U.S. Office of Personnel Management and is working on 
agreements with the U.S. Department of Defense and the U.S. Postal 
Service to obtain employment data through a clearinghouse similar to 
WRIS to help more states obtain this outcome data. Labor plans to begin 
testing this new clearinghouse in program year 2004. (See app. II for a 
detailed listing of states' use of UI wage records and other data 
sources for reporting on WIA outcomes.):

Time Lags Affect When Outcomes Are Reported and Limit Data's Usefulness 
for Gauging Current Performance:

UI wage records also suffer significant time delays between the time an 
individual gets a job and the time it appears in the UI wage records. 
State procedures for collecting and compiling wage information from 
employers can be slow and time-consuming. Data are collected from 
employers only once every quarter, and employers in most states have 30 
days after the quarter ends to report the data to the state. For 
example, the wage report for the last calendar quarter of the year 
(ending on December 31) is due to the state on January 31. After the 
state receives the wage report, the data must be processed. Many 
employers report the data electronically, but some employers--especially 
small businesses--are allowed to submit data in paper format, which then 
must be converted to electronic media. After data entry, information 
must be checked for errors and corrected. All these steps take time, 
which can delay the availability of the wage record data for reporting 
on outcomes for several months. According to our survey, 28 states 
estimated they get information on job placement within 4 months after 
participants exit the program, and 44 states have this information 
within 6 months. (See fig. 2.):

Figure 2: Estimated Number of Months for States to Get Job Placement 
Information from UI Wage Records:

[See PDF for image]

Note: One state reported that it could not estimate the number of 
months it takes to get job placement information.

[End of figure]

The time lags in receiving wage data, together with the use of longer-
term outcome measures, affect when outcomes are reported and limit the 
data's usefulness for gauging current performance. All 13 of WIA's 
employment-related outcomes are measured after participants leave--or 
exit--the program, and some measures, such as those that assess wage 
changes and employment retention, require a 6-month wait.[Footnote 15] 
To compensate for time lags, Labor devised a reporting structure that 
reaches back to the prior year to provide a complete year's worth of 
outcome data on WIA participants for the annual reports. For example, 
for the employment-based measures, participants who are reported on in 
the program year 2002 annual report, provided to Labor in December 
2003, left WIA in the four quarters between October 2001 and September 
2002 and may have received services much earlier. The amount of time 
between when participants receive services and when their outcomes are 
reported to Labor varies, but it is about 1½ years at a minimum. A 
hypothetical example will illustrate this point by showing two 
participants that would be included in the program year 2002 report. 
Sue registered in April 2001, participated in the program for at least 
6 months, and left between October and December 2001, taking about 32 
months from the time of registration until her outcomes were reported. 
Joe, on the other hand, did not register until July 2002, participated 
and left the program within 3 months, taking about 17 months from the 
time of registration until his outcomes were reported. (See fig. 3.):

Figure 3: Hypothetical Example Comparing Time Elapsed Between 
Registration and When Outcomes Are Reported for Two Participants:

[See PDF for image]

[End of figure]

WIA Performance Data Represent a Small Proportion of the Population 
Provided Services under WIA, Making It Difficult to Know What the 
Program Is Achieving:

WIA performance data represent a small proportion of the job seeker 
population receiving services at one-stops, making it difficult to know 
what the overall WIA program is achieving. Most one-stop customers who 
participate in self-directed services and only receive limited staff 
assistance, for example to conduct a job search, are not reflected in 
the WIA performance reports. This group is estimated to be the largest 
portion of customers served under WIA. For example, one of the local 
areas in our study that tracks each one-stop customer told us that only 
about 5.5 percent of the individuals who walked into their one-stops in 
fiscal year 2003 were registered for WIA services. The current law 
excludes job seekers who receive services that are self-service and 
informational in nature. Labor's guidance tells states to register 
adults and dislocated workers who receive core services that require 
significant staff assistance designed to help with job seeking or 
acquiring occupational skills, but states have flexibility in deciding 
what constitutes significant staff assistance.[Footnote 16] As a result 
of this flexibility, some local areas register a smaller proportion of 
participants than others, and in an earlier report, we said the local 
areas differed on when they registered WIA customers.[Footnote 17] In 
our recent visits to 4 states, we found that states and localities 
still differ on whom they track--some local officials said they register 
job seekers who received core services that required significant staff 
assistance, and others said they do not register participants until 
they receive intensive services. In addition, 21 of the 50 states we 
surveyed reported that they have instituted their own policies to more 
specifically define when registration should occur, suggesting that 
there is variation in interpreting Labor's guidance. Some experts told 
us that local workforce areas do not get adequate credit for serving 
everyone, making it difficult to show what is being achieved with 
available funding.

States and Local Areas Manage WIA Performance and Assess One-Stops by 
Collecting Timely Data and Making Use of a Range of Performance 
Information:

With assistance from states, local areas manage WIA performance and 
assess one-stop centers by collecting timely performance data and 
making use of a variety of performance information. To understand how 
well they are doing in meeting their performance levels, most local 
areas directly contact former participants or employers to collect 
interim WIA performance data that are not readily available from UI 
wage records. States provide assistance to local areas in a variety of 
ways, ranging from supporting their information technology (IT) systems 
to training local area staff. While states and local areas must meet 
performance goals for WIA, no similar goals exist for the overall one-
stop system--the service delivery system required under WIA for most 
federally funded employment and training services. Nonetheless, some 
states and many local areas have developed a range of measures to help 
them assess how well the one-stop is doing. Despite the progress states 
and local areas have made in developing and using interim outcome 
information, states and local areas told us they would like more help 
from Labor in collecting and disseminating promising practices on 
interim ways to assess WIA performance.

Many States Help Local Areas Monitor Their WIA Performance Levels:

According to our survey, many states play an active role in helping 
local areas monitor how well they are doing in meeting their 
performance levels. The assistance they provide ranges from ensuring 
that local areas have ready access to participants' UI wage records to 
developing IT systems and training local area staff on implementing WIA 
performance measures. To ensure local areas have ready access to wage 
record information on their participants, 23 states reported that they 
give local areas some form of electronic access to UI wage data for 
their WIA participants. Ten of these states give local areas direct 
online access to the UI wage reporting system, making information 
available to local officials as quickly as it is reported to the state; 
the others give local areas access once information has been merged 
into the statewide WIA reporting system. According to officials in 
Florida, having direct access to UI wage data allows them to not only 
monitor performance levels but also develop industry and wage profiles, 
tailor training programs to meet regional needs, and obtain contact 
information for former participants, facilitating follow-up with 
individuals that would not be found otherwise. When states do not give 
local areas ready access to UI wage data, as might occur in states with 
restrictive privacy laws, state officials usually provide local areas 
with standard reports on their WIA progress, either for individual WIA 
participants or, most often, aggregated across all WIA participants in 
the local area.

In addition to helping provide timely information, almost all states 
are supporting local areas' IT efforts. According to our survey, 47 
states have established or are in the process of establishing statewide 
IT systems to help local areas organize, track, and report WIA 
performance data. In about three-fourths of these states, the statewide 
IT systems allow the local areas to produce special reports that are 
tailored to local tracking needs and can report information for the 
local areas to use at the one-stop center, service provider, or case 
manager level. Although most local areas reported that they use a 
statewide system to help meet federal reporting requirements, half also 
use a locally developed IT system in combination with a statewide IT 
system. Local officials we met with often commented that they use a 
separate IT system because they do not find their state systems useful 
for managing the day-to-day operations of a WIA program. As a result of 
needing both a statewide and a local system, almost half of local areas 
reported that at least some, if not all, of their one-stop staff must 
enter the same WIA information into at least two IT systems.

Most states provide a range of other support services to local areas to 
help them manage their WIA performance requirements and to understand 
what implementation approaches work better than others in providing 
one-stop services, according to our survey. States reported they most 
often provide local areas with more specific written guidance or 
notices that explain federal guidance on the WIA performance measures 
and performance reports. In addition, about 90 percent of local areas 
told us that their states conduct training, make presentations, and 
hold regular meetings with local staff about WIA performance measures. 
To help local areas better understand what implementation approaches 
work better than others, several states have conducted special studies 
of the one-stop system. Sixteen states told us they have recently 
conducted studies on program implementation and processes; 11 states 
told us they have done return-on-investment studies; and 4 states have 
done impact evaluations that use control groups. Nationwide, half of 
the local areas believe that having a strong relationship with their 
state greatly helps them achieve their WIA levels.

To Help Them Meet Their WIA Performance Levels and Manage Their 
Programs, Most Local Areas Collect Outcome Information from a Variety 
of Sources and Actively Monitor Their Progress:

Because UI wage data suffer from time delays, about three-fourths of 
local areas collect outcome information from other sources to help them 
assess whether they are meeting their WIA performance levels and to 
help them manage their programs. Over 75 percent of local areas 
reported that they directly follow up with participants after they 
leave the program, collecting job placement or earnings information to 
help fill gaps until the data are available from the UI wage records. 
Sometimes local officials will also follow up with employers to verify 
employment or collect other documentation, such as pay stubs or W-2 
forms, as verification of employment. If outcomes do not appear in UI 
wage records over time, many local areas will report the findings from 
these other data sources in their WIA performance reports to the state. 
Local officials in a rural area of Pennsylvania told us that collecting 
this interim outcome data is important to help them assess their 
progress in meeting their performance levels, so much so that they 
provide small gift certificates to former participants who periodically 
report back to WIA staff. According to these officials, this strategy 
of obtaining follow-up data saves considerable staff time as well as 
increases their performance levels by more completely capturing 
information on participants.

Nearly all of the local areas reported on our survey that they track 
other types of interim indicators to manage their WIA programs--most 
often the number of registered WIA participants, services provided to 
WIA participants, number of participants that have completed training, 
and number of WIA exiters. Over half of these local areas report these 
data to decision makers on at least a monthly basis. About 80 percent 
of local areas track some kind of cost information, such as cost per 
participant or cost per outcome, and 24 percent report this information 
at least monthly. (See fig. 4.) Although these indicators may not be 
directly tracked and reported under WIA, they are useful for helping 
local officials know the number of participants that will be counted in 
their WIA measures. Furthermore, in some cases, these interim 
indicators also help the local areas predict their WIA performance 
outcomes. For example, one local official told us that knowing the 
number of participants who complete training helps him predict the 
number of participants who will find a job. Overall, nearly half of 
local areas reported that this type of interim information greatly 
helped them meet or exceed their performance levels. Despite the 
progress states and local areas have made in developing and using 
interim outcome information, nearly all states and local areas reported 
they would like more help from Labor in collecting and disseminating 
promising practices on interim indicators to assess WIA performance.

Figure 4: The Percentage of Local Areas That Report Interim Indicators 
at Least Monthly:

[See PDF for image]

[End of figure]

Because meeting WIA performance levels may affect future funding, most 
local areas hold service providers accountable and actively monitor 
their WIA performance levels. Through our survey, we found that over 80 
percent of local areas hold their service providers accountable by 
incorporating negotiated performance levels in their contracts. In 
addition, nearly 80 percent of local areas establish goals for the 
number of participants who are registered in or exited from WIA. A 
lesser number of local areas--24 percent--establish pay for performance 
contracts, and 18 percent provide financial incentives to their service 
providers. (See table 5.) Officials from one local area that we visited 
told us they provide monetary bonuses to providers that exceed their 
WIA goals and withhold 20 percent of their payments for those providers 
that do not reach their WIA goals. In addition, over 80 percent of 
local areas nationwide reported that having staff devoted to monitoring 
and managing WIA performance greatly helps them achieve or exceed their 
levels.

Table 5: Strategies Local Areas Use to Hold Service Providers 
Accountable to WIA Performance Levels:

Strategies: Incorporate the negotiated WIA performance levels into 
contracts; 
Percentage of local areas: 83.

Strategies: Establish other goals such as number of participants that 
registered or exited from WIA; 
Percentage of local areas: 78.

Strategies: Establish interim, real-time goals in contracts; 
Percentage of local areas: 65.

Strategies: Set goals for serving target populations; 
Percentage of local areas: 58.

Strategies: Establish pay-for-performance contracts based on meeting 
goals for WIA participants; 
Percentage of local areas: 24.

Strategies: Provide financial incentives for meeting or exceeding 
goals; 
Percentage of local areas: 18. 

Source: GAO survey of local areas.

[End of table]

Once final WIA performance information is available, local areas use 
this information to assess program services over time and to guide 
future program development. Most often local areas reported they use 
WIA performance information to modify their programs. We found that 
about two-thirds of local areas use performance information to a great 
extent to help them identify areas for program improvement and adopt 
new program approaches. Over half of local areas use their WIA 
performance information to analyze trends over time and prepare 
strategic plans. (See fig. 5.):

Figure 5: Activities That Local Areas Say Are Greatly Influenced by WIA 
Performance Information:

[See PDF for image]

[End of figure]

To Gauge One-Stop Performance, Many Local Areas Use Indicators in 
Addition to Those Required by WIA:

While WIA requires that officials monitor outcomes for all job seekers 
who receive staff-assisted core, intensive, and training services 
funded by WIA, there is no requirement to track those who receive self-
directed core services, which may be the majority served under WIA. In 
addition, Labor does not require that states and local areas measure 
the overall performance of the one-stop system. Nonetheless, most 
states and local areas have developed ways to assess the performance of 
their one-stops, using four basic types of indicators--job seeker 
measures, employer measures, program partnership measures, and family 
and community indicators. (See fig. 6.):

Figure 6: Four Types of Indicators That States and Local Areas Use to 
Assess Performance of One-Stops:

[See PDF for image]

[End of figure]

Job Seeker Measures:

Even without a federal requirement to do so, according to our survey, 
almost 90 percent of local areas gather information on one-stop job 
seekers, even if they are not registered and participating in any 
particular federal program. Most often local areas reported that they 
require the one-stop centers to track and report the number of job 
seekers who visit the one-stop in a single time period, usually through 
a paper and pencil or computer log. We also found that 58 percent of 
the local areas are collecting information on job seekers that 
repeatedly visit one-stop centers, sometimes through electronic means. 
About 20 percent of all local areas reported using electronic swipe 
cards to track job seekers in their one-stop centers. These swipe 
cards, similar to membership or grocery store discount cards, are 
issued to each job seeker using the one-stop and contain unique 
identifying information that can be read each time the job seeker 
accesses services. For example, according to local officials in 
Philadelphia, they issue swipe cards to job seekers and scan these 
cards to record both the services received--such as using computers in 
the resource room, attending orientation workshops, or talking with the 
case managers--and the date and time the services were provided. Using 
data from this system, one-stop managers can assess traffic flow and 
schedule staff accordingly, and may eventually be able to link 
participants and services to outcomes achieved. Officials also told us 
they are using demographic information from an analysis of swipe card 
data to target marketing efforts and to develop services more 
strategically.

In addition to counting the number of job seekers who visit the one-
stop center, we found that local areas are tracking information on how 
many program referrals they receive, how satisfied they are with 
services, and what types of outcomes they achieve. Over half of local 
areas reported that they survey job seekers who visit the one-stop to 
gauge their satisfaction with services.[Footnote 18] For example, a 
one-stop center in Utah that we visited not only uses a one-stop 
satisfaction survey, but officials also periodically contact one-stop 
customers to ask how they liked the services. According to our survey, 
some of the local areas said that having job seeker satisfaction 
information was one of the best ways to assess the one-stop system. 
Many local areas collect more in-depth information on all one-stop job 
seekers--over one-third collect demographic characteristics, and over 
one-fourth monitor outcomes, such as whether job seekers got a job and 
at what wages. (See fig. 7.):

Figure 7: Information That Local Areas Collect on Job Seekers Who Visit 
One-Stops:

[See PDF for image]

[End of figure]

Employer Measures:

Many local areas also track information on employers' use of one-stops. 
About 70 percent of local areas nationwide reported that they require 
one-stop centers to track some type of employer measure, such as the 
number of employers that use one-stop services, how many hire one-stop 
customers, and the type of services that employers use. To gauge 
employer involvement, local areas most often require the one-stops to 
count and report the number of employers that use one-stop services. 
Over 40 percent of local areas require one-stops to track the number of 
employers that repeatedly use one-stop services. For example, a one-
stop center in Utah we visited tracks employers that repeatedly use 
one-stop services and those that have not used services in a while. It 
uses this information to reach out to employers who have not returned 
for services, encouraging them to use one-stop services again. To 
understand how employers view the one-stop services they received, 60 
percent of local areas reported they collect information on employer 
satisfaction.[Footnote 19] A smaller number--about 20 percent of local 
areas--track information on market penetration, such as the number of 
employers in the labor market that could potentially use one-stop 
services. For example, Philadelphia officials told us they measure 
market penetration by comparing the number of employers that use the 
one-stop center with the number of employers in the community as a 
whole. [Footnote 20] (See fig. 8.):

Figure 8: Information That Local Areas Collect on Employers Using One-
Stops:

[See PDF for image]

[End of figure]

Program Partnership Measures:

Most of the programs that provide services through the one-stop system 
have their own performance measures, but as we have reported in the 
past, these measures cannot be readily summed to obtain an overall 
measure of one-stop performance.[Footnote 21] However, one-third of the 
local areas told us that they combine in one report some of the key 
federal measures for the various one-stop programs--including wages at 
employment or other earnings indicators--and use this report to assess 
the one-stop system as a whole. For example, Florida officials produce 
a report--called the Red and Green report--that assembles for each local 
area outcomes on 22 measures from different one-stop programs, such as 
WIA, Wagner-Peyser, and TANF. Weaker program outcomes are identified in 
red and stronger outcomes are identified in green. They use this report 
to assess performance, diagnose weak spots, and predict long-term 
outcomes across one-stop partners.

More often local areas have gone a step further and have identified 
outcomes they consider to be key, developing common definitions for 
these measures to be used across programs. Just over half of the local 
areas reported in our survey that they track cross-cutting employment 
measures, such as job placement, and a little less than half said they 
track wages at placement and employment retention across programs. For 
example, Utah developed a set of outcome, process, efficiency, and 
activity measures to gauge the performance of all of their one-stops 
and to ensure alignment with agency goals and objectives. These 
measures include entered employment, earnings increase, and employment 
retention across Wagner Peyser, WIA, TANF, Trade Adjustment Assistance, 
and Food Stamp Employment and Training programs.

In addition to tracking outcomes for the various one-stop partners, 
some local areas assess their one-stop systems by measuring the level 
of coordination among one-stop partners, as well as the range and 
quality of services they provide. Nearly 40 percent of local areas we 
surveyed said that they use indicators, such as increased coordination 
among partners and number of referrals partners made, to assess how 
well the overall one-stop system is operating. For example, one local 
area reported it is developing a one-stop report card that will track 
the flow of customers through the system and monitor each program's 
contribution to the services provided, including the results of program 
referrals. They will use this report card to target areas that need 
attention. To ensure the one-stop system is providing quality services, 
some local areas we visited also conduct mystery shopper reviews 
wherein individuals posing as employers or job seekers evaluate the 
quality of the services they receive. Michigan conducts such mystery 
shopper visits of all their one-stops over the course of a year to 
assess the quality of customer services, including how courteous, 
professional, and knowledgeable one-stop staff are. The state receives 
a comprehensive report of each visit and uses this information to 
target technical assistance.

Family and Community Indicators:

A few local areas look outside their one-stops to assess how well one-
stop services are meeting the needs of the family and the community. In 
their written comments to our survey, several local areas told us that 
they consider some type of community indicator, such as changes in the 
local unemployment rate or increases in the average household income in 
the local area, to be the best way to determine the overall 
effectiveness of their one-stop system. Some local areas focus on 
indicators of family well-being, such as family self-sufficiency--or the 
ability of families to financially support themselves--to assess whether 
their one-stop systems are meeting family needs. One rural one-stop in 
Michigan even uses some indicators that are not related to income. 
These local officials told us that their indicators include a 
collection of family indicators, including whether families are getting 
the child care they need and how well the children are doing at home 
and at school, to understand how well the one-stop is meeting the needs 
of the family.

Labor Has Taken Actions to Improve WIA's Performance Measurement System 
and Assess One-Stops but Could Do More:

Although Labor has taken steps to improve WIA's performance measurement 
system and assess one-stops, some of its efforts do not go far enough. 
Labor has commissioned a study of adjustment methods that would better 
take into account economic and demographic differences when negotiating 
performance levels. However, even if an acceptable model is developed, 
Labor has made no commitment to put a standard adjustment method in 
place nationally. To improve the quality of WIA's performance data, 
Labor has initiated a data validation project. Labor is taking a 
significant step toward measuring one-stop outcomes, but a planned 
change may lead to restricting the use of supplemental data to fill 
gaps in UI wage records. While Labor has plans to conduct impact 
studies, the department will not meet WIA's requirement to conduct an 
impact study by 2005, and without such a study, little will be known 
about WIA's effectiveness.

Labor Has Commissioned a Study of Adjustment Methods That Could Be Used 
in Negotiating Performance Levels, but This Effort Is Limited:

Labor has commissioned a study of adjustment methods that could be used 
to set expected performance levels during the negotiations process, but 
this effort does not go far enough. WIA requires that annual 
negotiations to establish expected performance levels consider 
differences in economic conditions, participant characteristics, and 
services provided--factors that can have a significant effect on the 
performance levels states and local areas are expected to achieve. 
However, many of the state and local officials we interviewed said they 
did not think these factors were adequately addressed in the 
negotiations process, and as a result they think some of their 
performance levels were set too high for the current economy. For 
example, some local officials said that their negotiated performance 
levels on the earnings change and earnings replacement measures were 
based on a stronger economy and did not reflect recent increases in the 
unemployment rate. Nationwide, 22 states reported that they are at risk 
of not meeting at least 80 percent of their negotiated performance 
levels on one or more of the WIA measures for program year 2002. (See 
fig. 9.) Further, 10 states reported that they are at risk of receiving 
financial sanctions on one or more measures for program year 
2002.[Footnote 22]

Figure 9: States That Reported Being at Risk of Not Meeting 80 Percent 
of Negotiated Performance Levels on One or More Measure for Program 
Year 2002:

[See PDF for image]

Note: One state did not provide a response to this question.

[End of figure]

To address states' concerns, Labor has commissioned a study of 
adjustment methods, such as the type of model used under JTPA--one that 
adjusted for factors beyond the control of local programs, such as high 
unemployment or a high concentration of non-English-speaking program 
applicants. The JTPA model assigned adjustment factors and weights for 
each performance measure using a multiple regression analysis, 
predicting how well a local area might do based on the relevant 
factors.[Footnote 23] For example, the model would assign a lower 
expected performance level to a local program serving extremely 
disadvantaged participants in an economically depressed area and a 
higher expected performance level to a local program serving job 
seekers who are nearly ready to get a job in an area with good economic 
conditions. All states and nearly all local areas we surveyed told us 
they would like Labor to use a model that can adjust for varying 
economic and population factors. Although Labor is studying adjustment 
methods, even if an acceptable model is developed, it has made no 
commitment to implement such an adjustment method nationally.

Some states currently use their own adjustment model or other methods 
in the negotiation process to account for factors beyond the control of 
local programs, but Labor has not yet taken steps to increase 
consistency across states as it did under JTPA. According to our 
survey, we found that nine states used a regression model or other 
method to a great extent to establish their performance levels for 
negotiating their program year 2004 performance levels with Labor. 
Under JTPA, Labor allowed states flexibility to develop their own 
adjustment procedures, but it established standard parameters to govern 
the adjustment methods used by states. These parameters addressed the 
procedures for adjusting performance levels, the quality of data, and 
factors that could be used for adjustments. For example, the procedures 
for adjusting performance levels were required to be objective and 
equitable across all local areas. In addition, Labor developed optional 
adjustment models that could be used by states because it recognized 
that not all states and local areas have the expertise and resources 
necessary to develop adjustment procedures. Without standard 
parameters, the process will lack consistency, and some states may be 
at a disadvantage in the process of negotiating their performance 
levels.

Concerns Have Been Raised about the Quality of WIA's Performance Data, 
and Labor Has Begun a Major Initiative to Address These Concerns:

Issues have been raised about the quality of performance data that 
Labor uses to assess program performance. As we mentioned previously, 
Labor allows flexibility in determining which participants to track for 
reporting purposes. This flexibility leads to variations in reporting, 
which raises questions about both the accuracy and the comparability of 
states' performance data.[Footnote 24] In addition, we recently 
reported that performance data submitted by states in quarterly and 
annual reports were not sufficiently reliable to determine outcomes for 
the WIA programs.[Footnote 25] Furthermore, Labor's Office of Inspector 
General has said that there is little assurance that the states' 
performance data for all WIA programs are either accurate or complete 
because of inadequate oversight of data collection and management at 
the federal, state, and local levels.[Footnote 26]

Labor has initiated a new data validation project to improve the 
quality of the performance information collected and reported under 
WIA. Labor's data validation project includes developing procedures and 
accuracy standards to help states validate that WIA performance and 
participant data are correctly reported. For this project, Labor 
developed data validation handbooks and software and required states to 
begin validating program year 2002 data, which were reported to Labor 
on December 1, 2003. States are required to conduct two types of data 
validation: (1) review samples of WIA participant files and (2) assess 
whether reporting software accurately calculated the performance 
measures. Labor provided software to help states generate the aggregate 
information required for performance reports, such as performance 
outcomes. If states elect to use Labor's software, they are not 
required to validate the calculations. At the time of our 
survey--December 2003 through February 2004--we found that 41 states had 
begun using Labor's data validation software. Labor also plans to hold 
states accountable for meeting accuracy standards, beginning in the 
third year of validation. Once these accuracy standards are in place, 
states failing to meet the standards may lose eligibility for incentive 
awards or, in cases with significant deviations from the standards, 
states may be sanctioned.

Labor Is Taking Steps to Measure One-Stop Outcomes, but a Planned 
Change May Lead to Restricting the Use of Supplemental Data to Fill 
Gaps in UI Wage Records:

Labor is taking a significant step toward measuring outcomes across 
one-stop partners by developing definitions for a set of common 
performance measures. The Office of Management and Budget established a 
set of common measures to be applied to all federal employment and 
training programs administered by Labor, Education, Health and Human 
Services, Veterans Affairs, Interior, and Housing and Urban 
Development. (See table 6.) Labor has developed standard definitions 
for calculating these measures across all of its Employment and 
Training Administration programs. (See table 7.) This will allow Labor 
to sum outcomes across all its programs to provide a more uniform 
picture of outcomes achieved. According to a department official, Labor 
worked with other federal agencies to get agreement on common data 
sources and common language, where possible. For example, Labor is 
working on developing a process, using WRIS, that would allow other 
federal programs to use UI wage records to track outcomes. As part of 
the common measures, Labor plans to require one-stops to track all 
participants who walk through the door of a one-stop center and receive 
any one-stop service, regardless of which program provides the 
service.[Footnote 27] According to Labor, tracking all one-stop job 
seekers will enable officials to obtain information about who is 
served, what services are provided, which partner programs provided 
services, and what outcomes are achieved. While these changes can 
provide more information on job seekers, there is no provision for any 
measure of employer involvement in the one-stops, and experts and state 
and local officials we interviewed said that at least one measure is 
needed to address employer usage.

Table 6: Proposed Common Measures and Data Sources:

Measures: Adult measures: 

Measures: Adult measures: Entered employment; 
Data Source: UI wage records.

Measures: Adult measures: Employment retention; 
Data Source: UI wage records.

Measures: Adult measures: Earnings increase; 
Data Source: UI wage records.

Measures: Adult measures: Efficiency (program appropriation level/
number of program participants); 
Data Source: Administrative records.

Measures: Youth measures: Placement in employment or education; 
Data Source: UI wage records and administrative records.

Measures: Youth measures: Attainment of a degree or certificate; 
Data Source: Administrative records.

Measures: Youth measures: Literacy and numeracy gains; 
Data Source: Assessment instrument.

Measures: Youth measures: Efficiency (program appropriation level/
number of program participants); 
Data Source: Administrative records.

Source: U.S. Department of Labor, TEGL 15-03.

[End of table]
 

Table 7: Employment and Training Administration Programs Subject to the 
Common Measures:

Employment and Training Administration program: WIA Adult; 
Program's target population: Adult.

Employment and Training Administration program: WIA Dislocated Workers 
(including National Emergency Grants); 
Program's target population: Adult.

Employment and Training Administration program: WIA Youth; 
Program's target population: Youth.

Employment and Training Administration program: Labor Exchange; 
Program's target population: Adult.

Employment and Training Administration program: Trade Adjustment 
Assistance; 
Program's target population: Adult.

Employment and Training Administration program: Migrant and Seasonal 
Farmworkers; 
Program's target population: Adult.

Employment and Training Administration program: Native American 
Employment and Training; 
Program's target population: Adult and youth.

Employment and Training Administration program: Senior Community 
Service Employment programs; 
Program's target population: Adult.

Employment and Training Administration program: H-1B Technical Skills 
Training; 
Program's target population: Adult.

Employment and Training Administration program: Job Corps; 
Program's target population: Youth.

Employment and Training Administration program: Responsible 
Reintegration of Youthful Offenders; 
Program's target population: Youth. 

Source: U.S. Department of Labor, TEGL 15-03.

[End of table]

While most of Labor's policies for the common measures can advance 
measurement across one-stop partners, Labor plans to rely almost 
entirely on the UI wage records and discontinue the use of supplemental 
data for filling gaps in UI wage records. Labor officials tell us that 
they are making this change to address concerns about the quality of 
supplemental data being collected. Under Labor's current guidance, 
supplemental data must be documented. However, the department has no 
systematic process in place to monitor the accuracy of these 
supplemental data.[Footnote 28] If Labor elects to replace the current 
definitions of the WIA entered employment rate and earnings retention 
measure with the common measure definitions, this restriction on the 
use of supplemental data could have a significant impact on the ability 
of states and local areas to meet their negotiated performance levels. 
In addition, Labor's new data validation project could help ensure the 
accuracy of supplemental data that is collected at the local level.

Labor Will Not Meet WIA's Requirement to Conduct an Impact Study by 
2005, and Without Such a Study, Little Will Be Known about WIA's 
Effectiveness:

While Labor has plans to conduct impact studies, the department will 
not meet WIA's requirement to conduct at least one multi-site control 
group evaluation by fiscal year 2005. This type of impact study is 
important because outcome measures alone cannot show whether an outcome 
is a direct result of program participation or whether it is a result 
of other influences, such as the state of the local economy. Labor 
officials said they did not initiate impact studies of WIA within the 
first few years after WIA passed to allow states and local areas time 
to implement the considerable changes that were required under WIA. 
According to officials, Labor had planned to initiate an impact 
evaluation of the WIA adult and dislocated worker programs in 2004, but 
this plan is currently on hold because Labor is anticipating changes to 
these programs as a result of reauthorization. Once WIA is 
reauthorized, Labor officials told us that they would likely allow 2 or 
3 years for changes to be implemented before initiating an impact 
evaluation. The evaluation itself will take 5 to 6 years, but Labor 
plans to issue interim reports on the findings once the study is under 
way. Even though the House passed a reauthorization bill, the Workforce 
Reinvestment and Adult Education Act of 2003 (HR1261), and the Senate 
passed a bill, the Workforce Investment Act of 2003 (S1627), passage of 
a final bill has stalled. Both bills propose changes to WIA, but most 
of the basic one-stop service delivery and governance structure would 
stay the same in both bills. Given that these changes will not likely 
affect the fundamental service delivery and structure of WIA, it is 
unclear why Labor has not proceeded with its evaluations of WIA as 
planned.

Conclusions:

When WIA was implemented nearly 4 years ago, it fundamentally changed 
the way federally funded employment and training services are provided 
to job seekers, the way the system engages employers, and the way it 
measures performance. Making this shift has taken time and some trial 
and error as state and local policy makers and one-stop service 
providers learned what type of service structure met local needs. Since 
implementation, states and local areas have made great progress in 
retooling their systems and in gathering all the data needed to report 
on their performance to Labor. But, only recently are we getting a 
nationwide glimpse of outcomes achieved under WIA.

The requirement to use UI wage data is a step in the right direction by 
providing a reasonably consistent look at national program results over 
time. Historically, there have been data quality issues with outcome 
data collected directly from participants, as was done prior to WIA. 
The UI wage data provide a level of credibility that other data sources 
do not have. States have made progress in accessing data from other 
states. But in order to meet their performance levels, some states must 
continue to rely on other data sources to fill gaps. Out of concern for 
data comparability, Labor is proposing to limit the use of data from 
other sources. This decision, if applied to the WIA programs, will 
hinder the ability of some states to demonstrate that they have met 
their expected performance levels and may cause one-stops to focus 
their efforts on only those occupations covered by the UI wage records. 
This policy also seems overly restrictive, given that Labor is 
implementing data validation procedures that could be used to ensure 
the accuracy and validity of supplemental data. Even with the 
capability to use supplemental data, some states and local areas have 
failed to demonstrate that they met their negotiated performance levels 
for 2 years in a row and have suffered financial sanctions--often 
citing local economic conditions as the cause. The development of a 
method to systematically adjust for economic and demographic factors 
outside the control of the local area in setting expected performance 
levels could help mitigate these concerns.

While the use of UI wage records has improved the quality of the data 
that are used to track outcomes under WIA, this information alone does 
little for real-time program management. We found that state and local 
officials have made significant strides in collecting their own data to 
assess whether they are likely to meet their federally required 
performance levels, manage their programs on a real-time basis, and 
track a broader one-stop population than just registered WIA 
participants. In some ways, the WIA performance measures based on the 
UI wage records and the interim data collected at the state and local 
level provide a useful system to cross check these data. However, not 
all states and local areas have determined what interim information is 
necessary, nor have they had the benefit of learning from their peers. 
Without some additional information or the sharing of promising 
practices, these states and local areas will be at a disadvantage in 
monitoring their progress and, perhaps, in meeting their minimum 
performance levels. Further, Labor has also failed to meet WIA's 
requirement to conduct a systematic evaluation of WIA. Plans to do an 
evaluation have been delayed until reauthorization is complete, even 
though the proposed bills would retain most of the WIA service delivery 
and governance structure. Delays in committing to an evaluation now may 
be costly because policy makers will not be able to benefit from an 
understanding of WIA's effectiveness.

Without clear guidance from Labor, states and local areas continue to 
struggle with determining who should be tracked in the WIA performance 
measures. At the same time, even if states and localities had a common 
understanding of whom to track and were consistently reporting on the 
same categories of customers, they would only be reporting on a small 
portion of overall one-stop customers. While a requirement to track all 
job seekers who visit the one-stops may appear to be a major change, we 
found that over half the local areas already collect information on job 
seekers that repeatedly use one-stops, suggesting that some local areas 
are already equipped to uniquely identify and track each job seeker. It 
may take time and resources for local areas to fully develop the 
capability to collect data on each job seeker, but this may be the best 
way to start gauging the value of one-stops overall. As long as the law 
excludes individuals who participate in self-service and informational 
services, it will be difficult to understand the full reach of WIA.

Recommendations For Executive Action:

To compensate for the impact of changes in the economy and to give 
states and local areas an equal opportunity to meet their performance 
levels, we recommend that the Secretary of Labor:

* continue to allow the use of supplemental data for reporting 
outcomes, but develop more stringent guidance and monitoring of these 
data;

* provide assistance to states and localities in developing and sharing 
promising practices on interim indicators for assessing WIA's 
performance; and:

* develop an adjustment model or other systematic method to account for 
different populations and local economic conditions when negotiating 
performance levels.

To comply with statutory requirements and to help federal, state, and 
local policy makers understand what services are most effective for 
improving employment-related outcomes, we recommend that the Secretary 
of Labor expedite efforts to design and implement an impact evaluation 
of WIA services.

Matter for Congressional Consideration:

We suggest that Congress may wish to consider requiring that 
information be collected and reported on all WIA participants, 
including those who only receive self-service and informational 
services, so that Congress may have a better understanding of the full 
reach of WIA and the one-stop system.

Agency Comments:

We provided a draft of this report to Labor for review and comment. 
Labor generally agreed with recommendations about continuing the use of 
supplemental data, sharing promising practices on interim performance 
indicators, and developing an adjustment model or other systematic 
method for use in negotiating performance levels. In addition, Labor 
agreed with our matter for Congressional consideration that information 
be collected and reported on all WIA participants. However, Labor 
disagreed with our recommendation to expedite efforts to design and 
implement an impact evaluation of WIA services. We have incorporated 
Labor's comments in our report, as appropriate. A copy of Labor's 
response is in appendix III.

On our recommendation regarding the use of supplemental data for 
reporting outcomes under WIA, Labor responded that it will continue to 
allow supplemental wage data except when calculating results on the 
common measures that are reported to the Office of Management and 
Budget. Labor also told us that its ongoing data validation effort will 
collect additional information that will help assess the quality of 
supplemental wage data that states are reporting. We continue to 
believe that when assessing state and local progress toward meeting 
WIA's expected performance levels, supplemental data will be essential 
to gather a more complete picture of WIA outcomes.

On our recommendation to develop and share promising practices on 
interim indicators for assessing WIA's performance, Labor noted some of 
the efforts currently under way to facilitate information exchange, 
including state and local peer-to-peer alliances, Labor's promising 
practices web site, and a Performance Enhancement Project for states to 
share ideas and promising practices. However, despite Labor's ongoing 
efforts to facilitate information exchange, nearly all states and local 
areas reported on our survey that they would like more help from Labor 
in collecting and disseminating information on promising practices on 
interim indicators to assess WIA performance.

Regarding our recommendation to develop an adjustment or other 
systematic method for use in negotiating performance levels, Labor 
agreed with the importance of taking economic conditions and 
characteristics of the population into account when setting performance 
expectations. Labor noted the study it has commissioned on adjustment 
models that we cited in our report and said the results of this study 
are not yet available. Labor expressed concern that any systematic 
method for taking economic and demographic factors into account must 
not diminish the role of the states and local areas in setting 
strategic goals. Our recommendation for a systematic approach would not 
replace any state and local efforts to establish their own goals, but 
it could help make the national process for setting goals more uniform 
and provide tools for states and local areas that do not have the 
resources to develop their own adjustment procedures.

In response to our recommendation to expedite the design and 
implementation of an impact evaluation of WIA services, Labor told us 
that it believes the program consolidation changes proposed in the 
reauthorization bill passed by the House are significant enough to 
delay the multi-site evaluation required by WIA. However, we disagree 
that proposed reauthorization changes would significantly affect the 
basic one-stop service delivery structure under WIA. It is now 4 years 
past the full implementation of WIA and a well-designed evaluation 
would help inform policymakers in the future. Waiting for the 
implementation of any changes resulting from the current 
reauthorization cycle would likely delay the start of an evaluation at 
least 2 years, thus not having results available until after another 
reauthorization cycle has passed.

We are sending copies of this report to the Secretary of Labor, 
relevant congressional committees, and others who are interested. 
Copies will also be made available to others upon request. The report 
is also available on GAO's home page at http://www.gao.gov.

Please contact me on (202) 512-7215 if you or your staff have any 
questions about this report. Other major contributors to this report 
are listed in appendix III.

Signed by: 

Sigurd R. Nilsen, 
Director, Education, Workforce, and Income Security Issues:

[End of section]

Appendix I: Objectives, Scope, And Methodology:

We examined (1) how useful WIA performance data are in gauging program 
performance, (2) what local areas are doing to manage their WIA 
performance and assess one-stop success on a timely basis and how 
states are assisting these efforts, and (3) to what extent Labor is 
trying to improve WIA's performance measurement system and assess one-
stop success. Our review focused primarily on the employment-based 
measures that rely on UI wage records--entered employment rate, 
earnings change/replacement rate, employment retention rate, 
employment and credential rate, and the younger youth placement and 
retention rate. To address these questions, we conducted two surveys--
one of state WIA officials and one of local area workforce officials; 
reviewed different types of literature about WIA and the WIA 
performance measurement system; interviewed experts and Department of 
Labor officials; interviewed state and local WIA officials; and visited 
three states and two local areas or one-stops within each state. We 
supplemented our site visits with telephone interviews with state and 
local officials in Pennsylvania:

We provided a draft of this report to officials at the Department of 
Labor for their review and incorporated their comments where 
appropriate. We conducted our work from April 2003 through April 2004 
in accordance with generally accepted government auditing standards.

Research and Experts:

To obtain further information on the area of WIA performance 
management, we reviewed and analyzed numerous studies, reports, and 
other literature, and we interviewed experts on WIA and workforce 
development performance measurement. We reviewed a Department of Labor 
study that discussed costs of data collection and found it sufficiently 
reliable for the purpose of comparing costs of surveys and automated 
record matching to UI wage records. We also interviewed Department of 
Labor officials, as well as representatives of the National Governors' 
Association and the National Association of Workforce Boards.

Surveys:

To determine how useful WIA performance data are in gauging program 
performance and what states and local areas are doing to manage and 
assess WIA programs and one-stop systems, we surveyed all 50 states and 
the District of Columbia,[Footnote 29] as well as all existing local 
workforce investment areas, using similar but not identical 
questionnaires. We conducted both surveys via the Internet. We asked 
both groups to provide information on issues related to the WIA 
performance measures, such as state or local policies, the availability 
and use of UI data, WIA performance levels; management practices; 
information technology systems, efforts to monitor and manage their WIA 
programs and one-stop systems, factors that adversely affected their 
ability to assess their one-stops systems, and the types of technical 
assistance that would help with managing their one-stop systems' 
performance.

We pre-tested the questionnaires used for each of the surveys at least 
three times. Table 8 provides survey numbers and response rates for 
both surveys.

Table 8: Survey Numbers and Response Rates:

Survey of: States; 
Number of survey recipients: 51; 
Number of surveys completed: 51; 
Response rate (percent): 100.

Survey of: Local workforce investment areas; 
Number of survey recipients: 568; 
Number of surveys completed: 463; 
Response rate (percent): 81.5. 

Source: GAO analysis.

[End of table]

Because these were not sample surveys, there are no sampling errors. 
However, the practical difficulties of conducting any survey may 
introduce errors, commonly referred to as nonsampling errors. For 
example, difficulties in how a particular question is interpreted, in 
the sources of information that are available to respondents, or how 
the data are entered into a database can introduce unwanted variability 
into the survey results. We took steps in the development of the 
questionnaires, the data collection, and data analysis to minimize 
these nonsampling errors. For example, as already noted, we pretested 
the questionnaires to ensure that questions were clear and 
understandable. In that these were Web-based surveys whereby 
respondents entered their responses directly into our database, there 
was little possibility of data entry error. In addition, we verified 
that the computer programs used to analyze the data were written 
correctly.

Site Visits:

We visited three states--Florida, Michigan, and Utah--and traveled to 
at least two local areas or one-stop centers in each of these 
states.[Footnote 30] We supplemented our site visits with telephone 
interviews with state and local officials in Pennsylvania. (See table 9 
for a list of the states and local areas in our study.) Based on input 
from recognized experts and our literature review, we selected these 
states because they are geographically diverse, have experience in 
implementing additional performance measures to assess one-stop 
success, and have developed integrated statewide data systems.

Table 9: States and Local Areas in Our Study:

State: Site visits: Florida; 
Local Area: Citrus Levy Marion Workforce Development Board; 
City: Ocala.

State: Site visits: Florida; 
Local Area: First Coast Workforce Development, Inc.; 
City: Jacksonville.

State: Site visits: Michigan; 
Local Area: Capital Area Michigan Works!; 
City: Lansing.

State: Site visits: Michigan; 
Local Area: Area Community Services Employment and Training Council; 
City: Grand Rapids.

State: Site visits: Utah; 
Local Area: Metro Region; 
City: Salt Lake City.

State: Site visits: Utah; 
Local Area: American Fork; 
City: American Fork.

State: In-depth telephone interviews: Pennsylvania; 
Local Area: North Central Workforce Investment Board; 
City: Ridgway.

State: In-depth telephone interviews: Pennsylvania; 
Local Area: Philadelphia Workforce Investment Board; 
City: Philadelphia. 

Source: GAO analysis.

[End of table]

In each state, we interviewed state officials responsible for 
monitoring local areas' WIA programs and analyzing and reporting on the 
state's WIA performance data, as well as other state WIA and IT 
officials and staff of the state's Workforce Investment Board. At the 
local areas, we interviewed WIA officials and staff, including service 
providers, staff responsible for performance management issues, IT 
staff, case managers and other frontline staff, as well as staff of the 
local area Workforce Investment Board. The state and local interviews 
were administered using a semi-structured interview guide that we 
developed through a review of relevant literature and discussions with 
recognized experts on WIA performance management.

Information that we gathered on our site visits represents only the 
conditions present in the states and local areas at the time of our 
site visits, from June through October 2003. We cannot comment on any 
changes that may have occurred after our fieldwork was completed. 
Furthermore, our fieldwork focused on in-depth analysis of only a few 
selected states and local areas or sites. Based on our site visit 
information, we cannot generalize our findings beyond the states and 
local areas or sites we visited.

[End of section]

Appendix II: States' Use of UI Wage Records and Other Data Sources for 
Reporting on Employment Outcomes Under WIA:

State: Alabama; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: Alaska; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: Yes; 
Uses supplemental data to close gaps in UI wage data: No; 
Provides local areas online or automated access to UI 
wage records: Yes.

State: Arizona; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: Yes.

State: Arkansas; 
Uses other states' UI 
wage records: No; 
Obtains UI 
wage records through WRIS: No; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: Yes.

State: California; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: Yes.

State: Colorado; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: Connecticut; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: No; 
Provides local areas online or automated access to UI 
wage records: No.

State: Delaware; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: Yes; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: District of Columbia; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: No; 
Uses administrative databases: Yes; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: Florida; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: No; 
Provides local areas online or automated access to UI 
wage records: Yes.

State: Georgia; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: Yes.

State: Hawaii; 
Uses other states' UI 
wage records: No; 
Obtains UI 
wage records through WRIS: No; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: Idaho; 
Uses other states' UI 
wage records: No; 
Obtains UI 
wage records through WRIS: No; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: Illinois; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: No; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: Yes.

State: Indiana; 
Uses other states' UI 
wage records: No; 
Obtains UI 
wage records through WRIS: No; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: Iowa; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: No; 
Provides local areas online or automated access to UI 
wage records: No.

State: Kansas; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: Yes.

State: Kentucky; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: Yes.

State: Louisiana; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: Yes.

State: Maine; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: Yes.

State: Maryland; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: Yes; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: Massachusetts; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: Michigan; 
Uses other states' UI 
wage records: No; 
Obtains UI 
wage records through WRIS: No; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: Minnesota; 
Uses other states' UI 
wage records: No; 
Obtains UI 
wage records through WRIS: No; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: No; 
Provides local areas online or automated access to UI 
wage records: No.

State: Mississippi; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: Yes.

State: Missouri; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: Yes; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: Montana; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: Nebraska; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: Nevada; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: No; 
Provides local areas online or automated access to UI 
wage records: No.

State: New Hampshire; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: No; 
Provides local areas online or automated access to UI 
wage records: No.

State: New Jersey; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: New Mexico; 
Uses other states' UI 
wage records: No; 
Obtains UI 
wage records through WRIS: No; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: New York; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: North Carolina; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: Yes; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: North Dakota; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: Ohio; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: Oklahoma; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: Yes.

State: Oregon; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: Pennsylvania; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: No; 
Provides local areas online or automated access to UI 
wage records: No.

State: Rhode Island; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: Yes.

State: South Carolina; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: South Dakota; 
Uses other states' UI 
wage records: No; 
Obtains UI 
wage records through WRIS: No; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: No; 
Provides local areas online or automated access to UI 
wage records: Yes.

State: Tennessee; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: Texas; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: Yes.

State: Utah; 
Uses other states' UI 
wage records: No; 
Obtains UI 
wage records through WRIS: No; 
Uses administrative databases: Yes; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: Vermont; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: No; 
Provides local areas online or automated access to UI 
wage records: No.

State: Virginia; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: No; 
Uses administrative databases: Yes; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

State: Washington; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: Yes; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: Yes.

State: West Virginia; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: Yes.

State: Wisconsin; 
Uses other states' UI 
wage records: Yes; 
Obtains UI 
wage records through WRIS: Yes; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: No; 
Provides local areas online or automated access to UI 
wage records: No.

State: Wyoming; 
Uses other states' UI 
wage records: No; 
Obtains UI 
wage records through WRIS: No; 
Uses administrative databases: No; 
Uses supplemental data to close gaps in UI wage data: Yes; 
Provides local areas online or automated access to UI 
wage records: No.

Total; 
Uses other states' UI 
wage records: 41; 
Obtains UI 
wage records through WRIS: 38; 
Uses administrative databases: 9; 
Uses supplemental data to close gaps in UI wage data: 40; 
Provides local areas online or automated access to UI 
wage records: 18. 

Source: GAO survey of states.

[End of table]

[End of section]

Appendix III: Comments from the Department of Labor:

U.S. Department of Labor:

MAY 12 2004:

Assistant Secretary for Employment and Training 
Washington. D.C. 20210:

Mr. Sigurd R. Nilsen 
Director:
Education, Workforce, and Income Security Issues 
U.S. General Accounting Office 
441 G Street, NW:
Washington, D.C. 20548:

Dear Mr. Nilsen:

The Employment and Training Administration is in receipt of the draft 
General Accounting Office (GAO) report, "States and Local Areas Have 
Developed Strategies to Assess Performance, but Labor Could Do More to 
Help," GAO-04-657. The objectives of the study were to determine: (1) 
how useful Workforce Investment Act (WIA) performance data are for 
gauging program performance; (2) what local areas are doing to manage 
their WIA performance and assess One-Stop success on a timely basis, 
and how states are assisting these efforts; and (3) the extent to which 
the Department of Labor is trying to improve WIA's performance 
measurement system and assess One-Stop success.

The report includes four specific recommendations for executive action. 
These include: (1) continuing to allow the use of supplemental data for 
reporting outcomes; (2) provide assistance in developing and sharing 
promising practices on interim performance indicators; (3) develop an 
adjustment model or other systematic method to account for variances in 
population and local economic conditions when negotiating performance 
levels; and (4) expedite efforts to design and implement an impact 
evaluation of WIA services.

We believe the enclosed comments can improve the final report. If you 
would like additional information, please don't hesitate to contact me 
at (202) 693-2700.

Sincerely, 

Signed by: 

Emily Stover DeRocco: 

Enclosure: 

Draft General Accounting Office (GAO) Report "States and Local Areas 
Have Developed Strategies to Assess Performance, but Labor Could Do 
More to Help":

Comments on the Overall Report:

* ETA recommends that the title of the report be changed. In addition to 
deleting the phrase "but Labor Could Do More to Help," we suggest that 
the title more accurately reflect the purpose of the report. The 
current proposed title only reflects one purpose of the study. The 
report also examines the usefulness of WIA data in gauging program 
performance and what the Department of Labor is doing to improve WIA's 
performance measurement system.

* We recommend the last sentence of the first full paragraph on page 7 
be changed to read: "While Labor has plans to conduct impact studies, 
the Department has delayed implementation of such studies because of 
anticipated changes to WIA that may significantly change the workforce 
investment system. As a result, the Department will not meet WIA's 
requirement to conduct at least one multi-site impact study by 2005."

The GAO report characterization of the requirement as "an evaluation to 
assess the achievements of the three WIA-funded programs" (pages 8, 9, 
11, and 34) is incorrect. The requirement is that ETA conduct "at least 
1 multi-site control group evaluation" (WIA, Section 172 (c)). 
Similarly, in several places the report refers to "multi-state" instead 
of "multi-site". There is no requirement to assess the achievement of 
all three WIA-funded programs.

* On Page 16, the GAO reports that 38 states are members of the Wage 
Record Interchange System (WRIS). The correct number is 44. An 
additional five states are negotiating agreements to participate in 
WRIS, to bring the total number of states to 49 by the end of 2004.

* On page 39, the report states: "there is no provision for any measure 
of employer involvement in One-Stops." While the current proposed 
statutory language does not measure process or "involvement", ETA is 
proposing a new reporting system that will collect data on types of 
employers served and services provided to them.

ETA disagrees with the following statement made in the last sentence of 
the first paragraph on page 41: "these [consolidation] changes will not 
likely affect the fundamental service delivery and structures of WIA". 
We believe the changes proposed to consolidate all adult programs, 
including Wagner-Peyser, and the changes in the youth program would 
lead to significant changes in the system structure and delivery of 
services. It is for this reason that ETA has proposed a delay to multi-
site evaluation. ETA sees the possible restructuring of WIA as a 
significant change and, as a result, ETA invested its evaluation funds 
in other significant areas including evaluations of Trade Adjustment 
Assistance (TAA), Unemployment Insurance (UI), and other significant 
programs.

* On page 43, GAO has recommended that information be collected and 
reported on all WIA participants. ETA agrees with this recommendation 
and is developing a new reporting system that would enable states to 
report activity and outcomes:

for all WIA participants, including those in non-WIA funded programs 
such as Wagner-Peyser. Also, ETA recognizes a need to issue additional 
guidance on the definition of a participant and registration policies.

On page 43 GAO states: "To comply with statutory requirements and to 
help federal, state and local policy makers understand what services 
are most effective for improving employment-related outcomes, we 
recommend that the Secretary of Labor expedite efforts to design and 
implement an impact evaluation of WIA services." We disagree with the 
recommendation as stated and believe the recommendation should focus on 
the suggestion that ETA initiate a multi-site impact study of WIA-
related activities and services within one year of implementation of 
the reauthorized WIA.

Specific Comments on the GAO Recommendations:

Recommendation 1: DOL should continue to allow the use of supplemental 
data for reporting outcomes, but develop more stringent guidance and 
monitoring of these data.

* The common measures policy requires the use of Unemployment 
Insurance:

(UI) wage records for reporting on employment, earnings and retention 
measures. ETA is not disallowing the use of supplemental wage data, 
except for calculating results on the common measures that will be 
reported to the Office of Management and Budget. In ETA's Training and 
Employment Guidance Letter 15-03, Common Measures Policy, we encourage 
states to collect and report additional information to sufficiently 
describe program outcomes. Further guidance is being provided for using 
supplemental wage data, and states will be allowed to use and report 
supplemental wage data where appropriate. The Data Validation process 
identifies supplemental wage data and the source documentation used to 
support such supplemental wage data.

* Program operators have the latitude to devise their own performance 
management procedures and many have done so successfully. They can also 
use supplemental data to obtain more timely feedback on outcomes 
achieved for their own performance tracking purposes.

Recommendation 2: DOL should provide assistance to states and 
localities in developing and sharing promising practices on interim 
indicators for assessing WIA's performance.

* To lend balance to this discussion, many states and locals have formed 
peer-to-peer alliances to share ideas, solve common problems, and swap 
best practices concerning performance management as well as other WIA 
issues. The Department has facilitated some of this connectivity. With 
leaner public resources available, DOL encourages these collaborations.

ETA has created a system-based Web site, www.promising practices.org, 
where successful practices in this and other program areas can be 
posted by program operators and can be viewed by those seeking to learn 
what others are doing.

ETA is currently in the process of gathering information on two related 
topics:

1) Identifying state agencies with some methodology for tracking all 
participants. This study will be completed the end of May and ETA plans 
to share its findings with the workforce system.

2) Identifying states that have problems accessing wage data and the 
problems they are experiencing. ETA expects to have the results in late 
May. We anticipate providing assistance to those agencies in 
negotiating the sharing of such wage data. We also note that we have 
already identified states that are not experiencing any significant 
problems in accessing wage data and will share their information with 
the state agencies that are currently having problems.

Additionally, ETA funded the Performance Enhancement Project, through 
which a number of states have been able to connect to others states in 
their regions to share ideas and promising practices related to 
improving performance.

Recommendation 3: DOL should develop an adjustment model or other 
systematic method to account for different populations and local 
economic conditions when negotiating performance levels.

ETA agrees that it is important to take into account characteristics of 
the populations served and economic conditions when setting performance 
expectations and interpreting results achieved against those 
expectations. We have made efforts to help states understand the types 
of analyses that are useful when negotiating performance levels. 
Further, states are able to re-negotiate their performance levels to 
reflect changes that occur in economic conditions or client 
characteristics between the time levels were originally negotiated and 
when states report final outcomes against those measures.

* ETA currently has a project in which the State of Michigan is taking 
the lead to develop an adjustment model and coordinate it with other 
states to ensure buy-in and understanding of the model. We do not yet 
have the results of the ongoing project. Certainly, any adjustment 
model that is considered would have to incorporate and account for the 
statistically significant variables that impact performance results. In 
fact, ETA is aware of studies that suggest that local economic 
variables do not have a statistically significant impact on results 
achieved for many of the performance measures, such as entered 
employment rates, employment retention rates, and credential/skill 
attainment rates.

ETA does not believe it is appropriate to diminish the importance of 
state and local ownership of the goal setting process. Any systematic 
method for taking into account the economy and demographics of an area 
must not replace or supercede the responsibility of the Governor and 
the boards to set strategic goals.

Recommendation 4: DOL should expedite efforts to design and implement 
an impact evaluation of WIA services.

* An impact evaluation only has value if the program being evaluated is 
fully implemented and there is some stability in its operations 
throughout the evaluation's follow-up period. Further, to measure WIA's 
impacts on participants' post-program earnings and employment would 
require a follow-up period of at least three years. Therefore, any re-
authorization legislation during the next three years which altered 
WIA's service provision or target group would undermine the value of an 
impact evaluation begun during that period. We believe the proposal to 
consolidate the various funding streams would be such a change. 
Therefore, we propose to begin an impact evaluation of WIA only after 
any changes embodied in reauthorization have been implemented.

[End of section]

Appendix IV: GAO Contacts and Staff Acknowledgments:

GAO Contacts:

Dianne Blank, Assistant Director (202) 512-5654 Laura Heald, Analyst-
in-Charge (202) 512-8701:

Staff Acknowledgments:

Carolyn S. Blocker and Cheri Harrington made significant contributions 
to all phases of the effort. Stu Kaufman made significant contributions 
in the design and administration of the surveys. In addition, Jessica 
Botsford provided legal support; Avrum Ashery and Barbara Hills 
provided graphic design assistance; and Elizabeth Curda, Patricia 
Dalton, Catherine Hurley, and Shana Wallace also provided key technical 
assistance.

[End of section]

GAO Related Products:

National Emergency Grants: Labor is Instituting Changes to Improve 
Award Process, but Further Actions Are Required to Expedite Grant 
Awards and Improve Data. GAO-04-496. Washington D.C.: April 16, 2004.

Workforce Investment Act: Labor Actions Can Help States Improve Quality 
of Performance Outcome Data and Delivery of Youth Services. GAO-04-308. 
Washington D.C.: February 23, 2004.

Workforce Training: Almost Half of States Fund Employment Placement and 
Training and Employment through Employer Taxes and Most Coordinate with 
Federally Funded Programs. GAO-04-282. Washington D.C.: February 13, 
2004.

Workforce Investment Act: One-Stop Centers Implemented Strategies to 
Strengthen Services and Partnerships, but More Research and Information 
Sharing Is Needed. GAO-03-725. Washington, D.C.: June 18, 2003.

Workforce Investment Act: Issues Related to Allocation Formulas for 
Youth, Adults, and Dislocated Workers. GAO-03-636. Washington, D.C.: 
April 25, 2003.

Workforce Training: Employed Worker Programs Focus on Business Needs, 
but Revised Performance Measures Could Improve Access for Some Workers. 
GAO-03-353. Washington, D.C.: February 14, 2003.

Older Workers: Employment Assistance Focuses on Subsidized Jobs and Job 
Search, but Revised Performance Measures Could Improve Access to Other 
Services. GAO-03-350. Washington, D.C.: January 24, 2003.

Workforce Investment Act: States' Spending Is on Track, but Better 
Guidance Would Improve Financial Reporting. GAO-03-239. Washington, 
D.C.: November 22, 2002.

Workforce Investment Act: States and Localities Increasingly Coordinate 
Services for TANF Clients, but Better Information Needed on Effective 
Approaches. GAO-02-696. Washington, D.C.: July 3, 2002.

Workforce Investment Act: Youth Provisions Promote New Service 
Strategies, but Additional Guidance Would Enhance Program Development. 
GAO-02-413. Washington, D.C.: April 5, 2002.

Workforce Investment Act: Better Guidance and Revised Funding Formula 
Would Enhance Dislocated Worker Program. GAO-02-274. Washington, D.C.: 
February 11, 2002.

Workforce Investment Act: Improvements Needed in Performance Measures 
to Provide a More Accurate Picture of WIA's Effectiveness. GAO-02-275. 
Washington, D.C.: February 1, 2002.

Workforce Investment Act: Better Guidance Needed to Address Concerns 
Over New Requirements. GAO-02-72. Washington, D.C.: Oct. 4, 2001.

Workforce Investment Act: Implementation Status and the Integration of 
TANF Services. GAO/T-HEHS-00-145. Washington, D.C.: June 29, 2000.

FOOTNOTES

[1] Program impact evaluations establish the causal connection between 
outcomes and program activities, separate out the influence of 
extraneous factors, develop explanations for why those outcomes 
occurred, and thus isolate the program's contribution to those changes.

[2] WIA's youth program uses low income as an eligibility requirement.

[3] GPRA is intended to focus government decision making, management, 
and accountability on the results and outcomes achieved by federal 
programs.

[4] OMB developed PART as a diagnostic tool meant to provide a 
consistent approach to evaluating federal programs as part of the 
executive budget formulation process. PART includes questions to 
address whether programs are meeting their long-term and annual goals.

[5] Beginning in program year 2003, annual reports will be due on 
October 1.

[6] Exited from the program means that an individual has completed 
program or other one-stop services or has not received any services for 
90 days and is not scheduled for future services. 

[7] Program year 2002 ran from July 2002 to June 2003, and the annual 
performance report was submitted in December 2003.

[8] OMB also assessed the WIA performance levels for the dislocated 
worker and youth programs in the PART process used to formulate the 
President's fiscal year 2005 budget request.

[9] For the two wage change measures and the wage replacement measure, 
the only data source Labor allows is the UI wage records.

[10] Most states have the last five completed calendar quarters of wage 
record data available online and archive wage records prior to that. 
However, the archived data are not readily accessible and are more 
costly to access than online information. The length of time states 
retain wage data varies by state, depending on their resources, state 
records retention laws, and the frequency with which they dispose of 
old records. 

[11] TANF provides low-income families with income support and 
employment-related assistance.

[12] National Commission for Employment Policy, A Feasibility of the 
Use of Unemployment Wage-Record Data as an Evaluation Tool for JTPA: 
Report on Project's Phase I Activities, Research Report. Number 90-02 
(Jan. 1991), and U.S. Department of Labor, Employment and Training 
Administration, A Field Guide to Automated Follow-Up: Cost-Effective 
Collection of Performance Information (July 1998). 

[13] Since our survey, Labor reports that 44 states are now 
participating in WRIS and an additional 5 states are negotiating 
agreements to participate in WRIS by the end of 2004.

[14] U.S. General Accounting Office, Workforce Investment Act: 
Improvements Needed in Performance Measures to Provide a More Accurate 
Picture of WIA's Effectiveness, GAO-02-275 (Washington, D.C.: February 
2002).

[15] Most of the participants being reported on in a given program year 
actually exited the program during the prior program year.

[16] All youth who receive WIA-funded services are required to be 
registered.

[17] See GAO-02-275. Other reports have cited disincentives in the 
performance measures to serve certain workers using WIA funds. See, for 
example, U.S. General Accounting Office, Older Workers: Employment 
Assistance Focuses on Subsidized Jobs and Job Search, but Revised 
Performance Measures Could Improve Access to Other Services, GAO-03-350 
(Washington, D.C.: January 24, 2003); and U.S. General Accounting 
Office, Workforce Training: Employed Worker Programs Focus on Business 
Needs, but Revised Performance Measures Could Improve Access for Some 
Workers.,GAO-03-353 (Washington, D.C.: February 14, 2003).

[18] While WIA requires that all states track job seeker customer 
satisfaction, Labor does not require a sufficient sample size to be 
useful to each local area.

[19] As with job seeker satisfaction, WIA requires that states track 
employer customer satisfaction, but Labor does not require a sufficient 
sample size to be useful to each local area.

[20] For this measure local officials define employers as a firm with 
10 or more employees.

[21] See GAO-02-275. 

[22] States are sanctioned when they do not meet at least 80 percent of 
their negotiated performance levels on the same measure or measures 2 
years in a row.

[23] Under JTPA, performance standards were only set with local areas, 
not states.

[24] GAO-02-275.

[25] U.S. General Accounting Office, Workforce Investment Act: Labor 
Actions Can Help States Improve Quality of Performance Outcome Data and 
Delivery of Youth Services, GAO-04-308 (Washington, D.C.: February 
2004).

[26] U.S. Department of Labor, Office of Inspector General, Workforce 
Investment Act Performance Outcomes Reporting Oversight. 06-02-006-03-
390 (Washington, D.C., Sept. 30. 2002).

[27] Labor officials said they are proposing a new reporting system 
that would enable states to report on activity and outcomes for all WIA 
participants, including those in non-WIA funded programs such as 
Wagner-Peyser.

[28] Labor will continue to allow the use of employment administrative 
databases such as the U.S. Office of Personnel Management records to 
fill gaps in the wage records. 

[29] We surveyed Washington, D.C., as part of our state survey and we 
report some of its data in appendix II. However, because of its unique 
status of not having both state and local governance structures, its 
data were not included in the state data presented in the report.

[30] Utah is a single workforce investment area.

GAO's Mission:

The General Accounting Office, the investigative arm of Congress, 
exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony:

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics.

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading.

Order by Mail or Phone:

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to:

U.S. General Accounting Office

441 G Street NW,

Room LM Washington,

D.C. 20548:

To order by Phone: 	

	Voice: (202) 512-6000:

	TDD: (202) 512-2537:

	Fax: (202) 512-6061:

To Report Fraud, Waste, and Abuse in Federal Programs:

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470:

Public Affairs:

Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.

General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.

20548: