This is the accessible text file for GAO report number GAO-03-389 
entitled 'Title I: Characteristics of Tests Will Influence Expenses; 
Information Sharing May Help States Realize Efficiencies' which was 
released on May 08, 2003.

This text file was formatted by the U.S. General Accounting Office 
(GAO) to be accessible to users with visual impairments, as part of a 
longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

Report to Congressional Requesters:

United States General Accounting Office:

GAO:

May 2003:

TITLE I:

Characteristics of Tests Will Influence Expenses; Information Sharing 
May Help States Realize Efficiencies:

GAO-03-389:

GAO Highlights:

Highlights of GAO-03-389, a report to Congressional Requesters 

Why GAO Did This Study:

The No Child Left Behind Act of 2001 (NCLBA) reauthorized the 
$10 billion Title I program, which seeks to improve the educational 
achievement of 12.5 million students at risk. In passing the 
legislation, Congress increased the frequency with which states are to 
measure student achievement in mathematics and reading and added 
science as another subject.  Congress also authorized funding to 
support state efforts to develop and implement tests for this purpose.

Congress mandated that GAO study the costs of implementing the required 
tests. This report describes characteristics of states’ Title I tests, 
provides estimates of what states may spend to implement the required 
tests, and identifies factors that explain variation in expenses. 


What GAO Found:

The majority of states administer statewide tests and customize 
questions to measure student learning against their state standards. 
These states differ along other characteristics, however, including the 
types of questions on their tests and how they are scored, the extent 
to which actual test questions are released to the public following 
the tests, and the number of new tests they need to develop to comply 
with the NCLBA. 

GAO provides three estimates of total expenditures between fiscal year
2002 and 2008, based on different assumptions about the types of test 
questions states may choose to implement and how they are scored. The 
method by which tests are scored largely explains the differences in 
GAO’s estimates.

If all states use tests with multiple-choice questions, which are 
machine scored, GAO estimates that the total state expenditures will be 
about $1.9 billion. If all states use tests with a mixture of multiple-
choice questions and a limited number of open-ended questions that 
require students to write their response, such as an essay, which are 
hand scored, GAO estimates spending to be about $5.3 billion. GAO 
estimates that spending will be at about $3.9 billion, if states keep 
the mix of question types states reported to GAO. In general, hand 
scoring is more expensive and time and labor intensive than machine 
scoring. Benchmark funding for assessments as specified in NCLBA will 
cover a larger percentage of estimated expenditures for tests comprised 
of multiple-choice questions and a smaller percentage of estimated 
expenditures for tests comprised of a mixture of multiple-choice and 
open-ended questions. Several states are exploring ways to reduce 
assessment expenses, but information on their experiences is not 
broadly shared among states.

What GAO Recommends:

Given that significant expenses may be associated with testing, GAO is 
recommending that Education facilitate the sharing of information on 
states’ experiences in attempting to reduce expenses. Education agreed 
with GAO’s recommendation but raised concerns about GAO’s methodology 
for estimating expenditures.

www.gao.gov/cgi-bin/getrpt?GAO-03-389.

To view the full report, including the scope
and methodology, click on the link above.
For more information, contact Marnie S. Shaul at (202) 512-7215 or 
shaulm@gao.gov.

[End of section]

Contents:

Letter:

Results in Brief:

Background:

States Generally Report Administering Statewide Assessments Developed 
to Measure Their State Standards, but Differ Along Other 
Characteristics:

Estimates of Spending Driven Largely by Scoring Expenditures:

Conclusions:

Recommendation:

Agency Comments:

Appendix I: Objectives, Scope, and Methodology:

Appendix II: Accountability and Assessment Requirements under
the 1994 and 2001 Reauthorizations of Title I:

Appendix III: Number of Tests States Reported They Need to
Develop or Augment to Comply with NCLBA (as of March 2003):

Appendix IV: Estimates of Assessment Expenditures NCLBA Required, but Not 
in Place at the Time of Our Survey, FY 2002-08:

Appendix V: State Development and Nondevelopment Estimates:

Appendix VI: Fiscal Years 2002-08 Estimated Expenditures for 
Each Question Type:

Appendix VII: Comments from the Department of Education:

Appendix VIII: GAO Contacts and Staff Acknowledgments:

GAO Contacts:

Staff Acknowledgments:

Tables:

Table 1: Number of Assessments and Subject Areas Required by the 1994 
and 2001 ESEA Reauthorizations:

Table 2: Assessment Minimum Amounts under NCLBA:

Table 3: The Number of Tests States Reported Needing to Develop or 
Augment Varies:

Table 4: Estimated Expenditures by States for Title I Assessments, 
Fiscal Years 2002-08:

Table 5: Estimated Total Expenditures for Test Development Are Lower 
Than for Test Administration, Scoring, and Reporting:

Table 6: Total Estimated Expenditures by States for Title I 
Assessments, Fiscal Years 2002-08:

Table 7: States Selected for Study:

Table 8: Examples of Assessment Expenditures:

Table 9: Average Annual Expenditures for the 7 States (adjusted to 2003 
dollars):

Table 10: Estimated Expenditures to Implement Title I Assessments in a 
Given Year:

Table 11: Estimates of Expenditures for the Assessments Required by 
NCLBA That Were Not in Place at the Time of Our Survey, Fiscal Years 
2002-08:

Table 12: Estimates by State, Development, and Nondevelopment 
Expenditures:

Table 13: Estimated Expenditures for Each Question Type, Fiscal Years 
2002-08:

Figures:

Figure 1: The Majority of States Report They Currently Use Statewide 
Tests and Plan to Continue to Do So:

Figure 2: The Majority of States Reported That They Currently Use and 
Plan to Develop New Tests That Are Customized to Measure Their State's 
Standards:

Figure 3: The Majority of States Reported They Use a Combination of 
Multiple-choice and Open-ended Questions on Their Tests, but Many 
States Are Uncertain about Question Type on Future Tests:

Figure 4: States Split in Decision to Release Test Questions to the 
Public Following Tests:

Figure 5: Estimated Scoring Expenditures Per Assessment Taken for 
Selected States, Fiscal Year 2002:

Figure 6: Various Factors Are Likely to Affect What States Spend on 
Title I Assessments:

Figure 7: Total Expenditures Likely to Be Lower in First Few Years and 
Benchmark Funding in NCLBA Estimated to Cover Most of Expenditures in 
First Few Years:

Abbreviations:

ESEA: Elementary and Secondary Education Act
LEA: local educational agency
NASBE: National Association of State Boards of Education
NCLBA: No Child Left Behind Act:

United States General Accounting Office:

Washington, DC 20548:

May 8, 2003:

The Honorable Judd Gregg
Chairman, Committee on Health, Education,
 Labor, and Pensions
United States Senate:

The Honorable Edward M. Kennedy
Ranking Minority Member, Committee on
 Health, Education, Labor, and Pensions
United States Senate:

The Honorable John A. Boehner
Chairman, Committee on Education
 and the Workforce
House of Represenatives:

The Honorable George Miller
Ranking Minority Member, Committee on
 Education and the Workforce
House of Representatives:

Title I, the largest source of federal funding for primary and 
secondary education, provided states $10.3 billion in fiscal year 2002 
to improve the educational achievement of 12.5 million students at 
risk. In passing the No Child Left Behind Act of 2001 (NCLBA), Congress 
increased funding for Title I and placed additional requirements on 
states and schools for improving student performance. To provide an 
additional basis for making judgments about student progress, NCLBA 
increased the frequency with which states are to assess students in 
mathematics and reading and added science as another subject. Under 
NCLBA, states can choose to administer statewide, local, or a 
combination of state and local assessments, but these assessments must 
measure states' content standards for learning. If a state fails to 
fulfill NCLBA requirements, the Department of Education (Education) can 
withhold federal funds designated for state administration until the 
requirements have been fulfilled. To support states in developing and 
implementing their assessments, Congress authorized specific funding to 
be allocated to the states between fiscal year 2002 and 2007.

NCLBA requires that states test all students annually in grades 3 
through 
8 in mathematics and reading or language arts and at least once in one 
of the high school grades by the 2005-06 school year. It also requires 
that states test students in science at least once in elementary, 
middle, and high school by 2007-08. Some states have already developed 
assessments in many of the required subjects and grades.

In the conference report accompanying passage of the NCLBA, Congress 
mandated that we do a study of the anticipated aggregate cost to 
states, between fiscal year 2002 and 2008, for developing and 
administering the mathematics, reading or language arts, and science 
assessments required under section 1111(b) of the act. As agreed with 
your offices, this report (1) describes characteristics of states' 
Title I assessments and (2) provides estimates of what states may spend 
to implement the required assessments between fiscal year 2002 and 2008 
and identifies factors that explain variation in expenses.[Footnote 1]

To determine the characteristics of states' Title I assessments, we 
collected information through a survey sent to the 50 states, the 
District of Columbia, and Puerto Rico; all 52 responded to our survey. 
We also reviewed published studies detailing the characteristics of 
states' assessments. To estimate projected expenditures all states are 
expected to incur, we reviewed 7 states' expenditures--all of which had 
implemented the 6 assessments required by the 1994 Elementary and 
Secondary Education Act (ESEA) reauthorization and were testing 
students in many of the additional subjects and grades required by 
NCLBA. The 7 states were Colorado, Delaware, Maine, Massachusetts, 
North Carolina, Texas, and Virginia. To estimate projected expenditure 
ranges for all states, we used expenditures from these 7 states coupled 
with key information gathered through a survey completed by each 
state's assessment director. We estimated projected state expenditures 
for test development, administration, scoring, and reporting results 
for both assessments that states need and assessments that states 
currently have in place. Our methodology for estimating expenditures 
was reviewed by several internal and external experts and their 
suggestions have been incorporated as appropriate. Education officials 
were also briefed on our methodology and raised no substantial 
concerns. As agreed with your offices, we did not determine 
expenditures for alternate assessments for students with disabilities 
nor expenditures for English language proficiency testing. In addition, 
we did not determine the expenditures local school districts may incur 
with respect to these assessments. To determine what factors account 
for variation in projected expenditures, we reviewed the 7 states' 
expenditures, noting the test characteristics that were associated with 
specific types and levels of expenditure. We supplemented our 
examination of state expenditures with interviews of test publishers 
and contractors and state assessment officials in these states 
regarding the factors that account for price and expenditure variation. 
The expenditure data that we received were not audited. Actual 
expenditures may vary from projected amounts, particularly when events 
or circumstances are different from those assumed. All estimates are 
reported in nominal dollars unless otherwise noted.

We conducted our work in accordance with generally accepted government 
auditing standards between April 2002 and March 2003. 
(See app. I for more details about our scope and methodology.):

Results in Brief:

The majority of states share two characteristics--they administer 
statewide assessments rather than individual local assessments and use 
customized questions to measure the content taught in the state schools 
rather than questions from commercially available tests. However, 
states differ in many other respects. For example, some states use 
assessments that include multiple-choice questions and other states 
include a mixture of multiple-choice questions and a limited number of 
questions that require students to write their response, such as an 
essay. Many states that use questions that require students to write 
their response believe that such questions enable them to more 
effectively measure certain skills, such as writing. However, others 
believe that multiple-choice questions also allow them to assess such 
skills. In addition, some states make actual test questions available 
to the public after testing but differ with respect to the percentage 
of test questions they publicly release and consequently, the number of 
questions they will need to replace. States also vary in the number of 
new tests they reported needing to develop to comply with the NCLBA, 
which ranged from 0 to 17.

We provide three estimates--$1.9, $3.9, and $5.3 billion--of total 
spending by states between fiscal year 2002 and 2008, with the method 
by which assessments are scored largely explaining the differences in 
our estimates. These estimates are based on expenditures associated 
with new assessments as well as existing assessments. The $1.9 billion 
estimate is based on the assumption that all states will use multiple-
choice questions, which are machine scored. The $3.9 billion estimate 
is based on the assumption that all states keep the mix of question 
types--whether multiple-choice or a combination of multiple-choice and 
open-ended--states reported to us. The $5.3 billion estimate is based 
on the assumption that all states will use a combination of multiple-
choice questions and questions that require students to write their 
response, such as an essay, which are hand scored. Several states are 
exploring ways to reduce assessment expenses. This information could be 
beneficial to others, however, it is currently not being broadly 
shared. Given that significant expenses may be associated with testing, 
we are recommending that Education facilitate the sharing of 
information on states' experiences as they attempt to reduce expenses. 
Education agreed with our recommendation, but raised concerns about our 
methodology for estimating expenditures.

Background:

Enacted as part of President Johnson's War on Poverty, the original 
Title I program was created in 1965, but the 1994 and most recently, 
the 2001 reauthorization of ESEA, mandated fundamental changes to 
Title I. The 1994 ESEA reauthorization required states to develop state 
standards and assessments to ensure that students served by Title I 
were held to the same standards of achievement as other students. Some 
states had already implemented assessments prior to 1994, but they 
tended to be norm referenced--a student's performance was compared to 
the performance of all students nationally. The 1994 ESEA 
reauthorization required assessments that were criterion referenced--
students' performance was to be judged against the state standards for 
what children should know and be able to do.[Footnote 2] In passing the 
NCLBA, Congress built on the 1994 requirements by, among other things, 
increasing the number of grades and subject areas in which states were 
required to assess students, as shown in table 1. NCLBA requires annual 
testing of students in third through eighth grades, in mathematics and 
reading or language arts. It also requires mathematics and reading or 
language arts testing in one of the high school grades (10-12). States 
must also assess students in science at least once in elementary (3-5), 
middle (6-9), and high school (10-12). NCLBA gives the states until the 
2005-06 school year to administer the additional mathematics and 
reading or language arts assessments and until the 2007-08 school year 
to administer the science assessments (see app. II for a summary of 
Title I assessment requirements).

Table 1: Number of Assessments and Subject Areas Required by the 1994 
and 2001 ESEA Reauthorizations:

Subject: Reading or language arts; Number of required assessments: 1994 
ESEA reauthorization: 3; Number of required assessments: 2001 ESEA 
reauthorization: 7.

Subject: Mathematics; Number of required assessments: 1994 ESEA 
reauthorization: 3; Number of required assessments: 2001 ESEA 
reauthorization: 7.

Subject: Science; Number of required assessments: 1994 ESEA 
reauthorization: 0; Number of required assessments: 2001 ESEA 
reauthorization: 3.

Subject: Total; Number of required assessments: 1994 ESEA 
reauthorization: 6; Number of required assessments: 2001 ESEA 
reauthorization: 17.

Source: P.L. No. 103-382 (1994) and P.L. No. 107-110 (2001).

[End of table]

Unlike the 1994 ESEA reauthorization, NCLBA does not generally permit 
Education to allow states additional time to implement these 
assessments beyond the stated time frames.[Footnote 3] Under the 1994 
ESEA reauthorization, Congress allowed states to phase in the 1994 ESEA 
assessment requirements over time, giving states until the beginning of 
the 2000-01 school year to fully implement them with the possibility of 
limited time extensions. In April 2002, we reported that the majority 
of states were not in compliance with the Title I accountability and 
assessment provisions required by the 1994 law.[Footnote 4]

Every state applying for Title I funds must agree to implement the 
changes described in the 2001 act, including those related to the 
additional assessments. In addition to the regular Title I state grant, 
NCLBA authorizes additional funding to states for these assessments 
between fiscal year 2002 and 2007.[Footnote 5] These funds are to be 
allocated each year to states, with each state receiving $3 million, 
regardless of its size, plus an amount authorized based on its share of 
the nation's school age population. States must use the funds to pay 
the cost of developing the additional state standards and assessments. 
If a state has already developed the required standards and 
assessments, it may use these funds to, among other things, develop 
challenging state academic content and student academic achievement 
standards in subject areas other than those required under Title I and 
to ensure the validity and reliability of state assessments. NCLBA 
authorized $490 million for fiscal year 2002 for state assessments and 
such funds as may be necessary through fiscal year 
2007. However, if in any year Congress appropriates less than the 
amounts shown in table 2, states may defer or suspend testing; however, 
states are still required to develop the assessments. In fiscal year 
2002, states received $387 million for assessments.

Table 2: Assessment Minimum Amounts under NCLBA:

Fiscal year: 2002; Appropriation benchmark: $370,000,000.

Fiscal year: 2003; Appropriation benchmark: 380,000,000.

Fiscal year: 2004; Appropriation benchmark: 390,000,000.

Fiscal year: 2005; Appropriation benchmark: 400,000,000.

Fiscal year: 2006; Appropriation benchmark: 400,000,000.

Fiscal year: 2007; Appropriation benchmark: 400,000,000.

Fiscal year: Total; Appropriation benchmark: $2.34 billion.

Source: P.L. No. 107-110 (2001).

[End of table]:

Other organizations have provided cost estimates of implementing the 
required assessments. The National Association of State Boards of 
Education (NASBE) estimated that states would spend between 
$2.7 to $7 billion to implement the required assessments. 
AccountabilityWorks estimated that states would spend about $2.1 
billion.[Footnote 6]

States can choose to use statewide assessments, local assessments, or 
both to comply with NCLBA. States can also choose to develop their own 
test questions or augment commercially available tests with questions 
so that they measure what students are actually taught in school. 
However, NCLBA does not permit states to use commercially available 
tests that have not been augmented.

NCLBA provides Education a varied role with respect to these 
assessments. Education is responsible for determining whether or not 
states' assessments comply with Title I requirements. States submit 
evidence to Education showing that their systems for assessing students 
and holding schools accountable meet Title I requirements, and 
Education contracts with individuals who have expertise in assessments 
and Title I to review this evidence. The experts provide Education with 
a report on the status of each state regarding the degree to which a 
state's system for assessing students meets the requirements and, 
therefore, warrants approval. Under NCLBA, Education can withhold 
federal funds provided for state administration until Education 
determines that the state has fulfilled those requirements.[Footnote 7] 
Education's role also includes reporting to Congress on states' 
progress in developing and implementing academic assessments, and 
providing states, at the state's request, with technical assistance in 
meeting the academic assessment requirements. It also includes 
disseminating information to states on best practices.

States Generally Report Administering Statewide Assessments Developed 
to Measure Their State Standards, but Differ Along Other 
Characteristics:

The majority of states report using statewide assessments developed to 
measure student learning against the content they are taught in the 
states' schools, but their assessments differ in many other ways. For 
example, some states use assessments that include multiple-choice 
questions, while others include a mixture of multiple-choice questions 
and questions that require students to write their answer by composing 
an essay or showing how they calculated a math answer. In addition, 
some states make actual test questions available to the public but 
differ with respect to the percentage of test questions they publicly 
release. Nearly all states provide accommodations for students with 
disabilities and some states report offering their assessments in 
languages other than English. States also vary in the number of new 
tests they will need to develop to comply with the NCLBA.

The Majority of States Use Statewide Tests That They Report Are Written 
to Their State Standards:

Forty-six states currently administer statewide tests to students and
44 plan to continue using statewide tests for future tests NCLBA 
requires them to add.[Footnote 8] (See fig. 1.) Only 4 states--Idaho, 
Kansas, Pennsylvania, and Nebraska--currently use a combination of 
state and local assessments and only Iowa currently uses all local 
assessments.

Figure 1: The Majority of States Report They Currently Use Statewide 
Tests and Plan to Continue to Do So:

[See PDF for image]

Note: Percentages do not add to 100 because of rounding.

[End of figure]
:

The majority of states (31) report that all of the tests they currently 
use consist of questions customized, that is, developed specifically to 
assess student progress against their state's standards for learning 
for every grade and subject tested. (See fig. 2.) Many of the remaining 
states are using different types of tests for different grades and 
subjects. For example, some states are using customized tests for some 
grades and subjects and commercially available tests for other grades 
and subjects. Seven states reported using only commercially available 
tests in all the grades and subjects they tested.

In the future, the majority of states (33) report that all of their 
tests will consist of customized questions for every subject and grade. 
Moreover, those states that currently use commercially available tests 
report plans to replace these tests with customized tests or augment 
commercially available tests with additional questions to measure what 
students are taught in schools, as required by NCLBA.

Figure 2: The Majority of States Reported That They Currently Use and 
Plan to Develop New Tests That Are Customized to Measure Their State's 
Standards:

[See PDF for image]

Note: Percentages do not add to 100 due to rounding. In the current 
period, "other" includes states that reported using commercially 
available tests for all grades and subjects tested that had not been 
augmented with additional questions to measure state standards. These 
states reported plans to augment these tests with additional questions 
or replace them with customized tests.

[End of figure]

States Vary in Approach to Specific Accommodations:

In developing their assessments, nearly all states (50) reported 
providing specific accommodations for students with 
disabilities.[Footnote 9] These often include Braille, large print, and 
audiotape versions of their assessments for visually impaired students, 
as well as additional time and oral administration.

About a quarter of the states (12) report offering these assessments in 
languages other than English, typically Spanish. Both small and larger 
states scattered across the United States offer assessments in 
languages besides English. For example, states such as Wyoming and 
Delaware and large states such as Texas and New York offer Spanish 
language versions of their assessments. New York and Minnesota offer 
their assessments in as many as four other languages besides 
English.[Footnote 10] While a quarter of the states currently translate 
or offer assessments in languages other than English, additional states 
may provide other accommodations for students with limited English 
proficiency, such as additional time to take the test, use of bilingual 
dictionaries, or versions of the test that limit use of idiomatic 
expressions.

States Are Using Different Types of Questions to Assess Students:

Thirty-six states report they currently use a combination of multiple-
choice and a limited number of open-ended questions for at least some 
of the assessments they give their students. (See fig. 3.) For example, 
in Florida, third grade students' math skills are assessed using 
multiple-choice questions, while fifth grade students' math skills are 
assessed using a combination of multiple-choice and open-ended 
questions. Twelve states reported having tests that consist entirely of 
multiple-choice questions. For example, all of Georgia's and Virginia's 
tests are multiple-choice. Almost half of the states reported that they 
had not made a decision about the ratio of multiple-choice to open-
ended questions on future tests. Of the states that had made a 
decision, most reported plans to develop assessments using the same 
types of questions they currently use.

Figure 3: The Majority of States Reported They Use a Combination of 
Multiple-choice and Open-ended Questions on Their Tests, but Many 
States Are Uncertain about Question Type on Future Tests:

[See PDF for image]

[End of figure]


States choose to use a mixture of question types on their tests for 
varying reasons. For example, some officials believe that open-ended 
questions, requiring both short and long student responses, more 
effectively measure certain skills such as writing or math computation 
than multiple-choice questions. Further, they believe that different 
question types will render a more complete measure of student knowledge 
and skills. In addition, state laws sometimes require test designers to 
use more than one type of question. In Maine, for example, state law 
requires that all state and local assessments employ multiple measures 
of student performance.

States Split as to Whether They Make Actual Test Questions Available to 
the Public Following Tests:

Slightly over half of the states currently release actual test 
questions to the public, but differ in the percent of questions they 
release. (See fig. 4.) Texas, Massachusetts, Maine, and Ohio release 
their entire tests to the public following the tests, allowing parents 
and other interested parties to see every question their children were 
asked. Other states, such as New Jersey and Michigan release only a 
portion of their tests. Moreover, even those states that do not release 
questions to the general public may release a portion of the questions 
to teachers, as does North Carolina, so that they can better understand 
areas where students are having the most difficulty, and improve 
instructions. States that release questions must typically replace them 
with new questions.

Figure 4: States Split in Decision to Release Test Questions to the 
Public Following Tests:

[See PDF for image]

[End of figure]


Often, states periodically replenish their tests with new questions to 
improve test security. For example, states like Florida, Kentucky, 
Maryland, and South Carolina that do not release test questions, 
replenish or replace questions periodically.

In addition to replenishing test items, many states use more than one 
version for each of their tests and do so for various reasons. For 
example, Virginia gives a different version of its test to students who 
may have been absent. Some states use multiple test versions of their 
high school tests to allow those students who do not pass it to take it 
multiple times. Still other states, such as Massachusetts and Maine, 
use multiple versions to enable the field testing of future test 
questions.

States Vary in the Number of Additional Tests They Reported They Need 
to Develop or Augment:

States differ in the number of additional tests they reported they need 
to meet NCLBA requirements, with some having all of the tests needed 
while others will need to develop new tests or augment commercially 
available tests with additional questions to fulfill the new 
requirements for a total of 17 tests. (See table 3.) Appendix III has 
information on the number of tests each state needs to develop or 
augment to comply with NCLBA.

The majority of states (32) report they will need to develop or augment 
9 or fewer tests and the rest (20) will need to develop or augment 10 
or more tests. Eight states--Alabama, New Mexico, Montana, South 
Dakota, Idaho, West Virginia, Wisconsin, and the District of Columbia 
report that they need to develop or augment all 17 tests. Maryland is 
also replacing a large number of its tests (15); although its 
assessments were certified as compliant with the 1994 law, the tests 
did not provide scores for individual students. Although Education 
waived the requirement that Maryland's tests provide student level 
data, Maryland is in the process of replacing them so that it can 
provide such data, enabling parents to know how well their children are 
performing on state tests.

Table 3: The Number of Tests States Reported Needing to Develop or 
Augment Varies:

Range in number of test states need to comply with NCLBA: None; Number 
of states: 5.

Range in number of test states need to comply with NCLBA: 1-3; Number 
of states: 4.

Range in number of test states need to comply with NCLBA: 4-6; Number 
of states: 6.

Range in number of test states need to comply with NCLBA: 7-9; Number 
of states: 17.

Range in number of test states need to comply with NCLBA: 10-12; Number 
of states: 10.

Range in number of test states need to comply with NCLBA: 13 or more; 
Number of states: 10.

Source: GAO survey.

[End of table]:

Most states reported plans to immediately begin developing the tests, 
which according to many of the assessment directors we spoke with, 
typically take 2 to 3 years to develop. For example, most states 
reported that by 2003 they will have developed or will begin developing 
the reading and mathematics tests that must be administered by the 
2005-06 school year. Similarly, most states reported that by 2005 they 
will have developed or will begin developing the science tests that 
must be administered by the 2007-08 school year.

To help them develop these tests, most states report using one or more 
outside contractors to help manage testing programs. Nearly all states 
report that developing, administering, scoring, and reporting will be a 
collaborative effort involving contractors and state and local 
education agencies. However, while states report that contractors and 
state education agencies will share the primary role in developing, 
scoring, and reporting new assessments, local education agencies will 
have the primary role in administering the assessments.

Estimates of Spending Driven Largely by Scoring Expenditures:

We provide three estimates--$1.9, $3.9, and $5.3 billion--of total 
state spending between fiscal years 2002 and 2008 for test development, 
administration, scoring, and test reporting. These figures include 
estimated expenses for assessments states will need to add as well as 
continuing expenditures associated with assessments they currently have 
in place. The method of scoring largely explains the differences in the 
estimates. However, various other factors, such as the extent to which 
states release assessment questions to the public after testing and 
therefore need to replace them, also affect expenditures. Between 
states, however, the number of students assessed will largely explain 
variation in expenditures. Moreover, because expenditures for test 
development are small in relation to test administration, scoring, and 
reporting (nondevelopment expenditures), we estimate that state 
expenditures may be lower in the first few years when states are 
developing their assessments and higher in subsequent years as states 
begin to administer and score them and report the results.

Different Estimates Primarily Reflect Differences in How Assessments 
Are Scored:

We estimate that states may spend $1.9, $3.9, or $5.3 billion on 
Title I assessments between fiscal years 2002 through 2008, with 
scoring expenditures largely accounting for differences in our 
estimates. 
Table 4 shows total state expenditures for the 17 tests required by 
Title I. In appendix IV, we also provide separate estimates for 
expenses associated with the subset of the 17 assessments that states 
reported they did not have in place at the time of our survey but are 
newly required by NCLBA.

Table 4: Estimated Expenditures by States for Title I Assessments, 
Fiscal Years 2002-08:

Question type: Multiple-choice; Estimate: $1.9 billion; 
Questions and scoring methods used: Estimate assumes that all states 
use machine-scored multiple-choice questions.

Question type: Current question type; Estimate: $3.9 billion; 
Questions and scoring methods used: Estimate assumes that states use 
the mix of question types reported in our survey.

Question type: Multiple-choice and open-ended; Estimate: $5.3 billion; 
Questions and scoring methods used: Estimate assumes that all 
states use both machine-scored multiple-choice questions and some hand 
scored open-ended questions.

Source: GAO projections based on state assessment plans and 
characteristics and expenditure data gathered from 7 states.

[End of table]

The $1.9 billion estimate assumes that all states will use multiple-
choice questions on their assessments. Multiple-choice questions can be 
scored by scanning machines, making them relatively inexpensive to 
score. For instance, North Carolina, which uses multiple-choice 
questions on all of its assessments and machine scores them, spends 
approximately 
$0.60 to score each assessment.

The $3.9 billion estimate assumes that states will implement 
assessments with questions like the ones they currently use or plan to 
use based on state education agency officials' responses to our survey. 
However, 
25 states reported that they had not made final decisions about 
question type for future assessments. Thus, the types of questions 
states ultimately use may be different from the assessments they 
currently use or plan to use.

Finally, the $5.3 billion estimate assumes that all states will 
implement assessments with both multiple-choice and open-ended 
questions. Answers to open-ended questions, where students write out 
their responses, are typically read and scored by people rather than by 
machines, making them much more expensive to score than answers to 
multiple-choice questions. We found that states using open-ended 
questions had much higher scoring expenditures per student than states 
using multiple-choice questions, as evidenced in the states we visited, 
as shown in figure 5.[Footnote 11] For example, Massachusetts, which 
uses many open-ended questions on its Title I assessments, spends about 
$7.00 to score each assessment. Scoring students' answers to open-ended 
questions in Massachusetts involves selecting and training people to 
read and score the answers, assigning other people to supervise the 
readers, and providing a facility where the scoring can take place. In 
cases where graduation decisions depend in part on a student's score on 
the assessment, the state requires that two or three individuals read 
and score the student's answer. By using more than one reader to score 
answers, officials ensure consistency between scorers and are able to 
resolve disagreements about how well the student performed.

Figure 5: Estimated Scoring Expenditures Per Assessment Taken for 
Selected States, Fiscal Year 2002:

[See PDF for image]

[End of figure]


We estimate that, for most states, much of the expense associated with 
assessments will be related to test scoring, administration, and 
reporting, not test development, which includes such expenses as 
question development and field testing.[Footnote 12] (See table 5.) In 
Colorado, for example, test administration, scoring, and reporting 
expenditures comprise 
89 percent of the total expenditures, while test development 
expenditures comprised only 11 percent. (See app. V for our estimates 
of development and nondevelopment expenditures by state.):

Table 5: Estimated Total Expenditures for Test Development Are Lower 
Than for Test Administration, Scoring, and Reporting:

In millions.

Development; Multiple-choice: $668; Current: question type: $706; 
Multiple-choice and open-ended: $724.

Administration, scoring, and reporting; Multiple-choice: 1,233; 
Current: question type: 3,237; Multiple-choice and open-ended: 4,590.

Total; Multiple-choice: $1,901; Current: question type: $3,944; 
Multiple-choice and open-ended: $5,313.

Source: GAO projections based on state assessment plans and 
characteristics and expenditure data gathered from 7 states.

[End of table]

Various Factors are Likely to Affect Expenditures for Title I 
Assessments:

While the scoring method explains a great deal of the variation in 
expenditures among states, other factors are likely to affect 
expenditures. These factors include the number of different test 
versions used, the extent to which the state releases assessment 
questions to the public after testing, fees for using copyrighted 
material, and factors unique to the state. (See fig. 6.) For example, 
states that use multiple test versions will have higher expenditures 
than those that have one. Massachusetts used 
24 different test versions for many of its assessments and spent 
approximately $200,000 to develop each assessment. Texas used only 
1 version for its assessments and spent approximately $60,000 per 
assessment. In addition, states that release test items to the public 
or require rapid reporting of student test scores are likely to have 
higher expenditures than states that do not because they need to 
replace these items with new ones to protect the integrity of the tests 
and assign additional staff to more rapidly score the assessments by 
the specified time frame. States that customize their assessments may 
have higher expenditures than states that augment commercially 
available tests. Moreover, factors unique to the state may affect 
expenditures. Maine, which had one of the lowest assessment development 
expenses of all of the states we visited (about $22,000 per 
assessment), has a contract with a nonprofit testing company.

Between states, the number of students tested generally explains much 
of the variation in expenditures, particularly when question types are 
similar. States with large numbers of students tested will generally 
have higher expenditures than states with fewer students.

Figure 6: Various Factors Are Likely to Affect What States Spend on 
Title I Assessments:

[See PDF for image]

[End of figure]

Benchmark Amounts in NCLBA Will Cover Varying Portions of States' 
Estimated Expenditures and Amount Covered Will Vary Primarily by Type 
of Test Questions States Use:

Using the benchmark funding levels specified in NCLBA, we estimate that 
these amounts would cover varying portions of estimated expenditures. 
(See table 6.) In general, these benchmark amounts would cover a larger 
percentage of the estimated expenditures for states that choose to use 
multiple-choice tests. To illustrate, we estimated that Alabama would 
spend $30 million if it continued to use primarily multiple-choice 
questions, but $73 million if the state used assessments with both 
multiple-choice and open-ended questions. The specified amount would 
cover 
151 percent of Alabama's estimated expenditures if it chose to use all 
multiple-choice questions, but 62 percent if the state chose to use 
both multiple-choice and open-ended questions.

Table 6: Total Estimated Expenditures by States for Title I 
Assessments, Fiscal Years 2002-08:

Alabama; Estimates (in millions): Multiple-choice: $30; Estimates (in 
millions): Current question type: $30; Estimates (in millions): 
Multiple-choice and open-ended: $73; Appropriation benchmark 
(in millions)[A]: $46; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 151%; Appropriation benchmark as 
percent of estimated expenses: Current question type: 151%; 
Appropriation benchmark as percent of estimated expenses: Multiple-
choice and open-ended: 62%.

Alaska; Estimates (in millions): Multiple-choice: 17; Estimates (in 
millions): Current question type: 25; Estimates (in millions): 
Multiple-choice and open-ended: 28; Appropriation benchmark 
(in millions)[A]: 26; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 154; Appropriation benchmark as 
percent of estimated expenses: Current question type: 106; 
Appropriation benchmark as percent of estimated expenses: Multiple-
choice and open-ended: 93.

Arizona; Estimates (in millions): Multiple-choice: 39; Estimates (in 
millions): Current question type: 108; Estimates (in millions): 
Multiple-choice and open-ended: 108; Appropriation benchmark 
(in millions)[A]: 51; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 132; Appropriation benchmark as 
percent of estimated expenses: Current question type: 47; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 47.

Arkansas; Estimates (in millions): Multiple-choice: 23; Estimates (in 
millions): Current question type: 42; Estimates (in millions): 
Multiple-choice and open-ended: 53; Appropriation benchmark 
(in millions)[A]: 37; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 158; Appropriation benchmark as 
percent of estimated expenses: Current question type: 88; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 70.

California; Estimates (in millions): Multiple-choice: 178; Estimates 
(in millions): Current question type: 235; Estimates (in millions): 
Multiple-choice and open-ended: 632; Appropriation benchmark 
(in millions)[A]: 219; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 123; Appropriation benchmark as 
percent of estimated expenses: Current question type: 93; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 35.

Colorado; Estimates (in millions): Multiple-choice: 32; Estimates (in 
millions): Current question type: 87; Estimates (in millions): 
Multiple-choice and open-ended: 87; Appropriation benchmark 
(in millions)[A]: 46; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 145; Appropriation benchmark as 
percent of estimated expenses: Current question type: 53; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 53.

Connecticut; Estimates (in millions): Multiple-choice: 28; Estimates 
(in millions): Current question type: 68; Estimates (in millions): 
Multiple-choice and open-ended: 68; Appropriation benchmark 
(in millions)[A]: 41; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 147; Appropriation benchmark as 
percent of estimated expenses: Current question type: 59; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 59.

Delaware; Estimates (in millions): Multiple-choice: 14; Estimates (in 
millions): Current question type: 24; Estimates (in millions): 
Multiple-choice and open-ended: 24; Appropriation benchmark 
(in millions)[A]: 26; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 183; Appropriation benchmark as 
percent of estimated expenses: Current question type: 106; 
Appropriation benchmark as percent of estimated expenses: Multiple-
choice and open-ended: 106.

District of Columbia; Estimates (in millions): Multiple-choice: 13; 
Estimates (in millions): Current question type: 13; Estimates (in 
millions): Multiple-choice and open-ended: 17; Appropriation 
benchmark 
(in millions)[A]: 24; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 184; Appropriation benchmark as 
percent of estimated expenses: Current question type: 184; 
Appropriation benchmark as percent of estimated expenses: Multiple-
choice and open-ended: 144.

Florida; Estimates (in millions): Multiple-choice: 83; Estimates (in 
millions): Current question type: 211; Estimates (in millions): 
Multiple-choice and open-ended: 281; Appropriation benchmark 
(in millions)[A]: 102; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 123; Appropriation benchmark as 
percent of estimated expenses: Current question type: 48; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 36.

Georgia; Estimates (in millions): Multiple-choice: 54; Estimates (in 
millions): Current question type: 54; Estimates (in millions): 
Multiple-choice and open-ended: 174; Appropriation benchmark 
(in millions)[A]: 67; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 124; Appropriation benchmark as 
percent of estimated expenses: Current question type: 124; 
Appropriation benchmark as percent of estimated expenses: Multiple-
choice and open-ended: 39.

Hawaii; Estimates (in millions): Multiple-choice: 17; Estimates (in 
millions): Current question type: 31; Estimates (in millions): 
Multiple-choice and open-ended: 31; Appropriation benchmark 
(in millions)[A]: 28; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 162; Appropriation benchmark as 
percent of estimated expenses: Current question type: 91; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 91.

Idaho; Estimates (in millions): Multiple-choice: 18; Estimates (in 
millions): Current question type: 23; Estimates (in millions): 
Multiple-choice and open-ended: 30; Appropriation benchmark 
(in millions)[A]: 30; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 167; Appropriation benchmark as 
percent of estimated expenses: Current question type: 131; 
Appropriation benchmark as percent of estimated expenses: Multiple-
choice and open-ended: 98.

Illinois; Estimates (in millions): Multiple-choice: 65; Estimates (in 
millions): Current question type: 164; Estimates (in millions): 
Multiple-choice and open-ended: 211; Appropriation benchmark 
(in millions)[A]: 92; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 141; Appropriation benchmark as 
percent of estimated expenses: Current question type: 56; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 44.

Indiana; Estimates (in millions): Multiple-choice: 40; Estimates (in 
millions): Current question type: 113; Estimates (in millions): 
Multiple-choice and open-ended: 113; Appropriation benchmark 
(in millions)[A]: 56; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 140; Appropriation benchmark as 
percent of estimated expenses: Current question type: 49; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 49.

Iowa; Estimates (in millions): Multiple-choice: 24; Estimates (in 
millions): Current question type: 62; Estimates (in millions): 
Multiple-choice and open-ended: 62; Appropriation benchmark 
(in millions)[A]: 38; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 158; Appropriation benchmark as 
percent of estimated expenses: Current question type: 62; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 62.

Kansas; Estimates (in millions): Multiple-choice: 23; Estimates (in 
millions): Current question type: 36; Estimates (in millions): 
Multiple-choice and open-ended: 51; Appropriation benchmark 
(in millions)[A]: 38; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 164; Appropriation benchmark as 
percent of estimated expenses: Current question type: 106; 
Appropriation benchmark as percent of estimated expenses: Multiple-
choice and open-ended: 73.

Kentucky; Estimates (in millions): Multiple-choice: 28; Estimates (in 
millions): Current question type: 62; Estimates (in millions): 
Multiple-choice and open-ended: 71; Appropriation benchmark 
(in millions)[A]: 43; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 155; Appropriation benchmark as 
percent of estimated expenses: Current question type: 70; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 61.

Louisiana; Estimates (in millions): Multiple-choice: 31; Estimates (in 
millions): Current question type: 81; Estimates (in millions): 
Multiple-choice and open-ended: 81; Appropriation benchmark 
(in millions)[A]: 49; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 158; Appropriation benchmark as 
percent of estimated expenses: Current question type: 60; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 60.

Maine; Estimates (in millions): Multiple-choice: 18; Estimates (in 
millions): Current question type: 33; Estimates (in millions): 
Multiple-choice and open-ended: 33; Appropriation benchmark 
(in millions)[A]: 29; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 159; Appropriation benchmark as 
percent of estimated expenses: Current question type: 86; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 86.

Maryland; Estimates (in millions): Multiple-choice: 35; Estimates (in 
millions): Current question type: 91; Estimates (in millions): 
Multiple-choice and open-ended: 91; Appropriation benchmark 
(in millions)[A]: 51; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 146; Appropriation benchmark as 
percent of estimated expenses: Current question type: 56; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 56.

Massachusetts; Estimates (in millions): Multiple-choice: 38; Estimates 
(in millions): Current question type: 109; Estimates (in millions): 
Multiple-choice and open-ended: 109; Appropriation benchmark 
(in millions)[A]: 55; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 144; Appropriation benchmark as 
percent of estimated expenses: Current question type: 50; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 50.

Michigan; Estimates (in millions): Multiple-choice: 57; Estimates (in 
millions): Current question type: 177; Estimates (in millions): 
Multiple-choice and open-ended: 177; Appropriation benchmark 
(in millions)[A]: 80; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 140; Appropriation benchmark as 
percent of estimated expenses: Current question type: 45; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 45.

Minnesota; Estimates (in millions): Multiple-choice: 34; Estimates (in 
millions): Current question type: 91; Estimates (in millions): 
Multiple-choice and open-ended: 91; Appropriation benchmark 
(in millions)[A]: 51; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 149; Appropriation benchmark as 
percent of estimated expenses: Current question type: 56; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 56.

Mississippi; Estimates (in millions): Multiple-choice: 25; Estimates 
(in millions): Current question type: 63; Estimates (in millions): 
Multiple-choice and open-ended: 63; Appropriation benchmark 
(in millions)[A]: 39; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 154; Appropriation benchmark as 
percent of estimated expenses: Current question type: 61; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 61.

Missouri; Estimates (in millions): Multiple-choice: 36; Estimates (in 
millions): Current question type: 99; Estimates (in millions): 
Multiple-choice and open-ended: 99; Appropriation benchmark 
(in millions)[A]: 54; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 150; Appropriation benchmark as 
percent of estimated expenses: Current question type: 54; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 54.

Montana; Estimates (in millions): Multiple-choice: 18; Estimates (in 
millions): Current question type: 28; Estimates (in millions): 
Multiple-choice and open-ended: 29; Appropriation benchmark 
(in millions)[A]: 27; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 149; Appropriation benchmark as 
percent of estimated expenses: Current question type: 97; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 94.

Nebraska; Estimates (in millions): Multiple-choice: 18; Estimates (in 
millions): Current question type: 34; Estimates (in millions): 
Multiple-choice and open-ended: 34; Appropriation benchmark 
(in millions)[A]: 32; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 177; Appropriation benchmark as 
percent of estimated expenses: Current question type: 93; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 93.

Nevada; Estimates (in millions): Multiple-choice: 21; Estimates (in 
millions): Current question type: 26; Estimates (in millions): 
Multiple-choice and open-ended: 45; Appropriation benchmark 
(in millions)[A]: 33; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 152; Appropriation benchmark as 
percent of estimated expenses: Current question type: 125; 
Appropriation benchmark as percent of estimated expenses: Multiple-
choice and open-ended: 72.

New Hampshire; Estimates (in millions): Multiple-choice: 17; Estimates 
(in millions): Current question type: 32; Estimates (in millions): 
Multiple-choice and open-ended: 32; Appropriation benchmark 
(in millions)[A]: 29; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 168; Appropriation benchmark as 
percent of estimated expenses: Current question type: 92; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 92.

New Jersey; Estimates (in millions): Multiple-choice: 43; Estimates (in 
millions): Current question type: 127; Estimates (in millions): 
Multiple-choice and open-ended: 127; Appropriation benchmark 
(in millions)[A]: 67; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 153; Appropriation benchmark as 
percent of estimated expenses: Current question type: 53; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 53.

New Mexico; Estimates (in millions): Multiple-choice: 21; Estimates (in 
millions): Current question type: 39; Estimates (in millions): 
Multiple-choice and open-ended: 41; Appropriation benchmark 
(in millions)[A]: 33; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 155; Appropriation benchmark as 
percent of estimated expenses: Current question type: 84; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 81.

New York; Estimates (in millions): Multiple-choice: 83; Estimates (in 
millions): Current question type: 276; Estimates (in millions): 
Multiple-choice and open-ended: 276; Appropriation benchmark 
(in millions)[A]: 121; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 146; Appropriation benchmark as 
percent of estimated expenses: Current question type: 44; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 44.

North Carolina; Estimates (in millions): Multiple-choice: 49; Estimates 
(in millions): Current question type: 49; Estimates (in millions): 
Multiple-choice and open-ended: 152; Appropriation benchmark 
(in millions)[A]: 65; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 132; Appropriation benchmark as 
percent of estimated expenses: Current question type: 132; 
Appropriation benchmark as percent of estimated expenses: Multiple-
choice and open-ended: 43.

North Dakota; Estimates (in millions): Multiple-choice: 16; Estimates 
(in millions): Current question type: 23; Estimates (in millions): 
Multiple-choice and open-ended: 23; Appropriation benchmark 
(in millions)[A]: 26; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 162; Appropriation benchmark as 
percent of estimated expenses: Current question type: 109; 
Appropriation benchmark as percent of estimated expenses: Multiple-
choice and open-ended: 109.

Ohio; Estimates (in millions): Multiple-choice: 55; Estimates (in 
millions): Current question type: 171; Estimates (in millions): 
Multiple-choice and open-ended: 171; Appropriation benchmark 
(in millions)[A]: 86; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 158; Appropriation benchmark as 
percent of estimated expenses: Current question type: 50; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 50.

Oklahoma; Estimates (in millions): Multiple-choice: 27; Estimates (in 
millions): Current question type: 37; Estimates (in millions): 
Multiple-choice and open-ended: 66; Appropriation benchmark 
(in millions)[A]: 42; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 156; Appropriation benchmark as 
percent of estimated expenses: Current question type: 114; 
Appropriation benchmark as percent of estimated expenses: Multiple-
choice and open-ended: 63.

Oregon; Estimates (in millions): Multiple-choice: 28; Estimates (in 
millions): Current question type: 28; Estimates (in millions): 
Multiple-choice and open-ended: 70; Appropriation benchmark 
(in millions)[A]: 40; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 145; Appropriation benchmark as 
percent of estimated expenses: Current question type: 145; 
Appropriation benchmark as percent of estimated expenses: Multiple-
choice and open-ended: 57.

Pennsylvania; Estimates (in millions): Multiple-choice: 58; Estimates 
(in millions): Current question type: 162; Estimates (in millions): 
Multiple-choice and open-ended: 181; Appropriation benchmark 
(in millions)[A]: 87; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 150; Appropriation benchmark as 
percent of estimated expenses: Current question type: 54; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 48.

Puerto Rico; Estimates (in millions): Multiple-choice: 28; Estimates 
(in millions): Current question type: 28; Estimates (in millions): 
Multiple-choice and open-ended: 70; Appropriation benchmark 
(in millions)[A]: 47; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 167; Appropriation benchmark as 
percent of estimated expenses: Current question type: 167; 
Appropriation benchmark as percent of estimated expenses: Multiple-
choice and open-ended: 67.

Rhode Island; Estimates (in millions): Multiple-choice: 17; Estimates 
(in millions): Current question type: 28; Estimates (in millions): 
Multiple-choice and open-ended: 28; Appropriation benchmark 
(in millions)[A]: 27; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 161; Appropriation benchmark as 
percent of estimated expenses: Current question type: 98; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 98.

South Carolina; Estimates (in millions): Multiple-choice: 31; Estimates 
(in millions): Current question type: 82; Estimates (in millions): 
Multiple-choice and open-ended: 85; Appropriation benchmark 
(in millions)[A]: 43; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 139; Appropriation benchmark as 
percent of estimated expenses: Current question type: 53; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 51.

South Dakota; Estimates (in millions): Multiple-choice: 18; Estimates 
(in millions): Current question type: 18; Estimates (in millions): 
Multiple-choice and open-ended: 27; Appropriation benchmark 
(in millions)[A]: 26; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 145; Appropriation benchmark as 
percent of estimated expenses: Current question type: 145; 
Appropriation benchmark as percent of estimated expenses: Multiple-
choice and open-ended: 97.

Tennessee; Estimates (in millions): Multiple-choice: 33; Estimates (in 
millions): Current question type: 33; Estimates (in millions): 
Multiple-choice and open-ended: 85; Appropriation benchmark 
(in millions)[A]: 52; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 158; Appropriation benchmark as 
percent of estimated expenses: Current question type: 158; 
Appropriation benchmark as percent of estimated expenses: Multiple-
choice and open-ended: 61.

Texas; Estimates (in millions): Multiple-choice: 126; Estimates (in 
millions): Current question type: 232; Estimates (in millions): 
Multiple-choice and open-ended: 441; Appropriation benchmark 
(in millions)[A]: 147; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 116; Appropriation benchmark as 
percent of estimated expenses: Current question type: 63; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 33.

Utah; Estimates (in millions): Multiple-choice: 24; Estimates (in 
millions): Current question type: 44; Estimates (in millions): 
Multiple-choice and open-ended: 61; Appropriation benchmark 
(in millions)[A]: 37; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 154; Appropriation benchmark as 
percent of estimated expenses: Current question type: 84; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 60.

Vermont; Estimates (in millions): Multiple-choice: 16; Estimates (in 
millions): Current question type: 25; Estimates (in millions): 
Multiple-choice and open-ended: 25; Appropriation benchmark 
(in millions)[A]: 25; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 155; Appropriation benchmark as 
percent of estimated expenses: Current question type: 102; 
Appropriation benchmark as percent of estimated expenses: Multiple-
choice and open-ended: 102.

Virginia; Estimates (in millions): Multiple-choice: 43; Estimates (in 
millions): Current question type: 60; Estimates (in millions): 
Multiple-choice and open-ended: 129; Appropriation benchmark 
(in millions)[A]: 59; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 136; Appropriation benchmark as 
percent of estimated expenses: Current question type: 99; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 46.

Washington; Estimates (in millions): Multiple-choice: 41; Estimates (in 
millions): Current question type: 118; Estimates (in millions): 
Multiple-choice and open-ended: 118; Appropriation benchmark 
(in millions)[A]: 55; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 135; Appropriation benchmark as 
percent of estimated expenses: Current question type: 47; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 47.

West Virginia; Estimates (in millions): Multiple-choice: 23; Estimates 
(in millions): Current question type: 23; Estimates (in millions): 
Multiple-choice and open-ended: 43; Appropriation benchmark 
(in millions)[A]: 31; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 135; Appropriation benchmark as 
percent of estimated expenses: Current question type: 135; 
Appropriation benchmark as percent of estimated expenses: Multiple-
choice and open-ended: 72.

Wisconsin; Estimates (in millions): Multiple-choice: 29; Estimates (in 
millions): Current question type: 66; Estimates (in millions): 
Multiple-choice and open-ended: 72; Appropriation benchmark 
(in millions)[A]: 53; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 180; Appropriation benchmark as 
percent of estimated expenses: Current question type: 80; Appropriation 
benchmark as percent of estimated expenses: Multiple-choice and open-
ended: 73.

Wyoming; Estimates (in millions): Multiple-choice: 15; Estimates (in 
millions): Current question type: 21; Estimates (in millions): 
Multiple-choice and open-ended: 21; Appropriation benchmark 
(in millions)[A]: 25; Appropriation benchmark as percent of 
estimated expenses: Multiple-choice: 171; Appropriation benchmark as 
percent of estimated expenses: Current question type: 119; 
Appropriation benchmark as percent of estimated expenses: Multiple-
choice and open-ended: 119.

Total; Estimates (in millions): Multiple-choice: $1,901; Estimates (in 
millions): Current question type: $3,944; Estimates (in millions): 
Multiple-choice and open-ended: $5,313; Appropriation 
benchmark (in millions)[A]: $2,733; Appropriation benchmark as percent 
of estimated expenses: Multiple-choice: 144%; Appropriation benchmark 
as percent of estimated expenses: Current question type: 69%; 
Appropriation benchmark as percent of estimated expenses: Multiple-
choice and open-ended: 51%.

Source: GAO analysis.

[A] Figures in these columns are based largely on benchmark funding 
levels in NCLBA. If Congress appropriates less than the benchmark 
amounts, states may defer test administration. For fiscal years 2002 
and 2003, however, we used the actual appropriation. In addition, 
because we were mandated to estimate spending for fiscal year 2008, for 
purposes of this analysis, we assumed a fiscal year 2008 benchmark of 
$400 million, the same amount as for fiscal years 2005, 2006, and 2007. 
It should be noted, however, that Congress has not authorized funding 
past fiscal year 2007, when Title I would be reauthorized. Benchmarks 
by state were calculated based on the formula in NCLBA for allocating 
assessment funds to the states.

[End of table]

Total Expenditures Likely to Be Lower in the First Few Years, 
Increasing Over Time as States Begin to Administer, Score, and Report 
Additional Assessments:

Estimated expenditures are likely to be lower in the first few years 
when tests are being developed and increase in later years when greater 
numbers of tests are administered, scored, and reported. As a result, 
the benchmark funding amounts in NCLBA would cover a larger percentage 
of estimated expenditures in the first few years. Under some 
circumstances, the funding benchmarks in NCLBA exceed estimated state 
expenditures. For example, as shown in figure 7, the fiscal year 2002 
allocation would more than cover all of the estimated expenses if all 
states were to use multiple-choice questions or continue with the types 
of questions they currently use. If all states were to choose to use a 
mixture of multiple-choice and open-ended questions, the most expensive 
option, fiscal year 2002 funding would cover 84 percent of states' 
total expenditures. We estimate a similar pattern for fiscal year 2003. 
(See app. VI for fiscal year 2002 through 2008 estimated expenditures 
for each question type.):

In fiscal year 2007 and 2008, benchmark funding would continue to cover 
all of the estimated expenditures if all states were to use all 
multiple-choice questions, about two-thirds of estimated expenditures 
if all states continued using their current mix of questions, and a 
little over 50 percent of estimated expenditures if all states were to 
use a mixture of question types, the most expensive option.

Figure 7: Total Expenditures Likely to Be Lower in First Few Years and 
Benchmark Funding in NCLBA Estimated to Cover Most of Expenditures in 
First Few Years:

[See PDF for image]

[End of figure]

Opportunities May Exist to Share Information on Efforts to Reduce 
Testing Expenditures:

Some states are exploring ways to control expenses related to 
assessments and their experiences may provide useful information to 
other states about the value of various methods for controlling 
expenditures. Recently, several states, in conjunction with testing 
industry representatives, met to discuss ways of reducing test 
expenditures. For example, the group discussed a range of possible 
options for reducing expenditures, including computer-administered 
tests; commercially available tests that can be customized to states 
standards by adding additional questions; computerized scoring of 
written responses, and computer scanning of students' written 
responses. Information about individual states experiences as they 
attempt to reduce expenses could benefit other states. However, such 
information is currently not systematically shared.

Conclusions:

The 1994 and 2001 ESEA reauthorizations raised student assessments to a 
new level of importance. These assessments are intended to help ensure 
that all students are meeting state standards. Congress has authorized 
funding to assist states in developing and implementing these 
assessments. We estimate that federal funding benchmarks in NCLBA will 
cover a larger percentage of expenses in the first few years when 
states are developing their assessments, with the covered percentage 
decreasing as states begin to administer, score, and report the full 
complement of assessments. Moreover, the choices states make about how 
they will assess students will influence expenditures. Some states are 
investigating ways to reduce the expenses, but currently information on 
states' experiences in attempting to reduce expenses is not broadly 
shared. We believe states could benefit from information sharing.

Recommendation:

Given the large federal investment in testing and the potential for 
reducing test expenditures, we recommend that Education use its 
existing mechanisms to facilitate the sharing of information on states' 
experiences as they attempt to reduce expenses.

Agency Comments:

The Department of Education provided written comments on a draft of 
this report, which we have summarized below and incorporated in the 
report as appropriate. (See app. VII for agency comments.) Education 
agreed with our recommendation, stating that it looks forward to 
continuing and enhancing its efforts to facilitate information sharing 
that might help states contain expenses. However, Education raised 
concerns about our methodology, noted the availability of additional 
federal resources under ESEA that might support states' assessment 
efforts, and pointed out that not all state assessment costs are 
generated by NCLBA.

With regard to our estimates, we have confidence that our methodology 
is reasonable and provides results that fairly represent potential 
expenditures based on the best available information. Education's 
comments focus on the uncertainties that are inherent in estimation of 
any kind--the necessity of assumptions, the possibility of events or 
trends not readily predicted, and other potential sources of error that 
are acknowledged in the report--without proposing an alternative 
methodology. Because of the uncertainty, we produced three estimates 
instead of one. In developing our approach, we solicited comments from 
experts in the area and incorporated their suggestions as appropriate. 
We also discussed our estimation procedures with Education staff, who 
raised no significant concerns. Second, Education cites various other 
sources of funds that states might use to finance assessments. While 
other sources may be available, we focused primarily on the amounts 
specifically authorized for assessments in order to facilitate their 
comparison to estimated expenses and because they are the minimum 
amounts that Congress must appropriate to ensure that states continue 
to develop as well as implement the required assessments.

We are sending copies of this report to the Secretary of Education, 
relevant congressional committees, and other interested parties. Please 
contact me on (202) 512-7215 or Betty Ward-Zukerman on (202) 512-2732 
if you or your staff have any questions about this report. In addition, 
the report will be available at no charge on GAO's Web site at http://
www.gao.gov. Other GAO contacts and staff acknowledgments are listed in 
appendix VIII.

Marnie S. Shaul, Director
Education, Workforce and Income Security Issues:

Signed by Marnie S. Shaul:

[End of section]

Appendix I: Objectives, Scope, and Methodology:

The objectives of this study were to provide information on the basic 
characteristics of Title I assessments, and to estimate what states 
would likely spend on Title I assessments between fiscal year 2002 and 
2008, and identify factors that explain variation in estimated 
expenditures. To address the first objective, we collected information 
from a survey sent to the 50 states, the District of Columbia, and 
Puerto Rico, and reviewed documentation from state education agencies 
and from published studies detailing the characteristics of states' 
assessments. To address the second objective, we collected detailed 
assessment expenditure information from 7 states, interviewed officials 
at state education agencies, discussed cost factors with assessment 
contractors, and estimated assessment expenditures under three 
different scenarios. The methods we used to address the objectives were 
reviewed by several external reviewers, and we incorporated their 
comments as appropriate. This appendix discusses the scope of the 
study, the survey, and the methods we used to estimate assessment 
expenditures.

Providing Information on the Basic Characteristics of Title I 
Assessments:

We surveyed all 50 states, the District of Columbia, and Puerto Rico, 
all of which responded to our survey. We asked them to provide 
information about their Title I assessments, including the 
characteristics of current and planned assessments, the number and 
types of new tests they needed to develop to satisfy No Child Left 
Behind Act (NCLBA) requirements, when they planned to begin developing 
the new assessments, the types of questions on their assessments, and 
their use of contractors. We also reviewed documentation from several 
states about their assessment programs and published studies detailing 
the characteristics of states' assessments.

Estimating Assessment Expenditures and Explaining Variation in the 
Estimates:

This study estimates likely expenditures on Title I assessments by 
states between fiscal year 2002 and 2008, and identifies factors that 
may explain variation in the estimates. It does not estimate 
expenditures for alternate assessments for students with disabilitiess 
for English language proficiency testing, or expenditures incurred by 
school districts.[Footnote 13] Instead, we estimated expenses states 
are expected to incur based on expenditure data obtained for this 
purpose from 7 states combined with data on these and other states' 
assessment plans and characteristics obtained through a 
survey.[Footnote 14] In the 7 states, we requested information and 
documentation on expenditures in a standard set of areas, met with 
state officials to discuss the information and asked that they review 
our subsequent analysis of information regarding their state. The 
expenditure data that we received from the 7 states were not audited. 
Moreover, actual expenditures may vary from projected amounts, 
particularly when events or circumstances are different from those 
assumed, such as changes in the competitiveness of the market for 
student assessment or changes in assessment technology.

Selection of 7 States:

We selected 7 states that had assessments in place in many of the 
grades and subjects required by the NCLBA from the 17 states with 
assessment systems that had been certified by Education as in 
compliance with requirements of the Improving America's Schools Act of 
1994 when we began our work. We included states with varying student 
enrollments, including 2 states with relatively small numbers of 
students. The states we selected were Colorado, Delaware, Maine, 
Massachusetts, North Carolina, Texas and Virginia. (See table 7 for 
information about the selected states.):

Table 7: States Selected for Study:

State: Colorado; Date approved by Education: July 2001; Number of 
students: 724,508; Number of assessments: Reading 
(out of 7): 7; Number of assessments: Math 
(out of 7): 5; Number of assessments: Science 
(out of 3): 1; Total: 13.

State: Delaware; Date approved by Education: December 2000; Number of 
students: 114,676; Number of assessments: Reading 
(out of 7): 7; Number of assessments: Math 
(out of 7): 7; Number of assessments: Science 
(out of 3): 3; Total: 17.

State: Maine; Date approved by Education: February 2002; Number of 
students: 207,037; Number of assessments: Reading 
(out of 7): 3; Number of assessments: Math 
(out of 7): 3; Number of assessments: Science 
(out of 3): 3; Total: 9.

State: Massachusetts; Date approved by Education: January 2001; Number 
of students: 975,150; Number of assessments: Reading 
(out of 7): 5; Number of assessments: Math 
(out of 7): 4; Number of assessments: Science 
(out of 3): 3; Total: 12.

State: North Carolina; Date approved by Education: June 2001; Number of 
students: 1,293,638; Number of assessments: Reading 
(out of 7): 7; Number of assessments: Math 
(out of 7): 7; Number of assessments: Science 
(out of 3): 0; Total: 14.

State: Texas; Date approved by Education: March 2001; Number of 
students: 4,059,619; Number of assessments: Reading 
(out of 7): 7; Number of assessments: Math 
(out of 7): 7; Number of assessments: Science 
(out of 3): 2; Total: 16.

State: Virginia; Date approved by Education: January 2001; Number of 
students: 1,144,915; Number of assessments: Reading 
(out of 7): 4; Number of assessments: Math 
(out of 7): 4; Number of assessments: Science 
(out of 3): 3; Total: 11.

Source: U.S. Department of Education, National Center for Education 
Statistics, and state education agencies.

[End of table]:

Collection of Expenditure Information from 7 States:

We collected detailed assessment expenditure information from officials 
in the 7 states. We obtained actual expenditures on contracts and state 
assessment office budget expenditures for fiscal year 2002 for all 7 
states and for previous years in 4 states.[Footnote 15] In site visits 
to the 7 states, we interviewed state education agency officials who 
explained various elements of their contracts with assessment 
publishing firms and the budget for the state's assessment office. To 
the extent possible, we collected expenditure data, distinguishing 
expenditures for assessment development from expenditures for 
assessment administration, scoring, and reporting, because 
expenditures vary differently between these two expenditure categories. 
Assessment development expenditures vary with the number of assessments 
while administration, scoring, and reporting expenditures vary with the 
number of students taking the assessments. (See table 8 for examples of 
expenditures.):

Table 8: Examples of Assessment Expenditures:

(Continued From Previous Page)

Type of expenditure: Development; Example of expenditure: Question 
writing; Question review (e.g., for bias).

Type of expenditure: Administration; Example of expenditure: Printing 
and delivering assessment booklets.

Type of expenditure: Scoring; Example of expenditure: Scanning 
completed booklets into scoring machines.

Type of expenditure: Reporting; Example of expenditure: Producing 
individual score reports.

Source: State education agencies.

[End of table]

Calculation of Averages for Development and for Administration, 
Scoring, and Reporting:

Using annual assessment expenditures for all 7 states, the number of 
assessments developed and implemented, and the number of students who 
took the assessments, we calculated average expenditures for ongoing 
development (assessments past their second year of development) and 
average expenditures for administration, scoring, and reporting for 
each state. (See table 9.):

Table 9: Average Annual Expenditures for the 7 States (adjusted to 2003 
dollars):

State: Colorado; Average development expenditures (per ongoing 
assessment): $72,889; Average expenditures for administration, 
scoring, and reporting (per assessment taken): $10.35; Both multiple-
choice and open-ended questions: Yes; Multiple-choice questions: [Empty].

State: Delaware; Average development expenditures (per ongoing 
assessment): $66,592; Average expenditures for administration, 
scoring, and reporting (per assessment taken): $8.78; Both multiple-
choice and open-ended questions: Yes; Multiple-choice questions: [Empty].

State: Maine; Average development expenditures (per ongoing 
assessment): $22,295; Average expenditures for administration, 
scoring, and reporting (per assessment taken): $9.96; Both multiple-
choice and open-ended questions: Yes; Multiple-choice questions: [Empty].

State: Massachusetts; Average development expenditures (per ongoing 
assessment): $190,870; Average expenditures for administration, 
scoring, and reporting (per assessment taken): $12.45; Both multiple-
choice and open-ended questions: Yes; Multiple-choice questions: [Empty].

State: North Carolina; Average development expenditures (per ongoing 
assessment): $104,181; Average expenditures for administration, 
scoring, and reporting (per assessment taken): $1.85; Both multiple-
choice and open-ended questions: Multiple-choice questions: Yes.

State: Texas; Average development expenditures (per ongoing 
assessment): $61,453; Average expenditures for administration, 
scoring, and reporting (per assessment taken): $4.72; Both multiple-
choice and open-ended questions: Multiple-choice questions: Yes.

State: Virginia; Average development expenditures (per ongoing 
assessment): $78,489; Average expenditures for administration, 
scoring, and reporting (per assessment taken): $1.80; Both multiple-
choice and open-ended questions: Multiple-choice questions: Yes.

Source: GAO analysis of state education agency information.

Note: We were able to obtain data for more than 1 year for Colorado, 
Delaware, Maine, Massachusetts, and Texas. For these states, we 
adjusted their average expenditures to 2003 dollars and then averaged 
these adjusted expenditures across the years that data were collected. 
North Carolina did not distinguish Title I assessments from other 
assessments it offers.

[End of table]

Estimating States' Likely Expenditures for 17 Title I Assessments:

We provide three estimates of what all states are likely to spend on 
all of the required 17 assessments using the average development 
expenditure and average expenditures for administration, scoring, and 
reporting by question type (multiple-choice or multiple-choice with 
some open-ended questions). One estimate assumes that all states use 
only multiple-choice questions, the second assumes that states will use 
the types of questions state officials reported they use or planned to 
use, and the third assumes that all states will use both multiple-
choice and a limited number of long and short open-ended questions. All 
estimates reflect states' timing of their assessments (for example, 
that science assessments are generally planned to be developed and 
administered later than assessments for reading and mathematics).

To estimate what states would spend under the assumption that they use 
only multiple-choice questions, we took the mean of the average annual 
expenditures per assessment for North Carolina, Texas, and Virginia, 
states that use multiple-choice assessments. To compute an estimate 
that reflected the types of questions states used or planned to use, we 
used the appropriate averages. To illustrate, California reported 15 
multiple-choice tests and 2 tests that include a combination of 
multiple-choice and open-ended questions. For the 15 multiple-choice 
tests, we used the mean from the multiple-choice states (North 
Carolina, Texas, and Virginia). For the 
2 multiple-choice and open-ended tests, we used the mean from the 
states that had both question types (Colorado, Delaware, Maine, and 
Massachusetts). To estimate what states would spend, assuming that all 
states use both multiple-choice and open-ended questions, we used the 
mean of the average annual expenditures for Colorado, Delaware, Maine, 
and Massachusetts, states that use both types of questions.

Estimating Development Expenditures:

To estimate development expenditures, we obtained information from each 
state regarding the number of assessments it needed to develop, the 
year in which it planned to begin development of each new assessment, 
and the number of assessments it already had. For each assessment the 
state indicated it needed to develop, we estimated initial development 
expenditures beginning in the year the state said it would begin 
development and also for the following year because interviews with 
officials revealed that developing an entirely new assessment takes 
approximately 2 to 3 years. For the 7 states that provided data, we 
were typically not able to separate expenditures for new test 
development from expenditures for ongoing test development. Where such 
data were available, we determined that development expenses for new 
assessments were approximately three times the expense of development 
expenses for ongoing assessments, and we used that approximation in our 
estimates.

For each state each year, we multiplied the number of tests in initial 
development by three times the average ongoing development expenditure 
to reflect that initial development of assessments is more expensive 
than ongoing development.[Footnote 16] We multiplied the number of 
ongoing tests by the average ongoing development expenditure. The sum 
of these two products provides a development expenditure for each state 
in each year and provides a total development estimate. We calculated 
three estimates as follows:

* using the expenditure information from states that use multiple-
choice questions, we produced a lower estimate;

* using the information from the state survey on the types of tests 
they planned to develop (some indicated both open-ended/multiple-choice 
tests and some multiple-choice), we produced a middle 
estimate;[Footnote 17] and:

* using the expenditure information from the states that use open-ended 
and multiple-choice questions, we produced the higher estimate.

Estimating Administration, Scoring, and Reporting Expenditures:

To produce an estimate for administration, scoring, and reporting, we 
used three variables: the average number of students in a grade; the 
number of administered assessments; and the average administration, 
scoring, and reporting expenditure per assessment taken. We calculated 
the average number of students in a grade in each year using data from 
the National Center for Education Statistics' Common Core of Data for 
2000-01 and their Projection of Education Statistics to 2011. We 
obtained data on the number of administered assessments from our state 
education agency survey. Data on average expenditures come from the 
states in which we collected detailed expenditure information.

For each state in each year, we multiplied the average number of 
students in a grade by the number of administered assessments and by 
the appropriate average assessment expenditure. Summing over states and 
years provided a total estimate for administration, scoring, and 
reporting. As above, we performed these calculations, using the 
expenditure information from multiple-choice states to produce the 
lower estimate, using the information from the state survey and 
expenditure information from both combination and multiple-choice 
states to produce a middle estimate, and using the expenditure 
information from the combination states to produce the higher estimate. 
We also estimated what states are likely to spend on the assessments 
that states did not have in place at the time of our survey, but are 
required by NCLBA, using the same basic methodology. Table 10 provides 
an overview of our approach to estimating states' likely expenditures 
on Title I assessments.

Table 10: Estimated Expenditures to Implement Title I Assessments in a 
Given Year:

[See PDF for image]

Source: GAO analysis.

[End of table]

We conducted our work in accordance with generally accepted government 
auditing standards between April 2002 and March 2003.

[End of section]

Appendix II: Accountability and Assessment Requirements under the 1994 
and 2001 Reauthorizations of Title I:

Developing standards for content and performance:

Requirements for 1994: Develop challenging standards for what students 
should know in mathematics and reading or language arts. In addition, 
for each of these standards, states should develop performance 
standards representing three levels: partially proficient, proficient, 
and advanced. The standards must be the same for all children. If the 
state does not have standards for all children, it must develop 
standards for Title I children that incorporate the same skills, 
knowledge, and performance expected of other children; Requirements 
for 2001: In addition, develop standards for science content by 2005-
06. The same standards must be used for all children.

Implementing and administering assessments: 

Requirements for 1994: Develop and implement assessments aligned with 
the content and performance standards in at least mathematics and 
reading or language arts; Requirements for 2001: Add assessments 
aligned with the content and performance standards in science by the 
2007-08 school year. These science assessments must be administered at 
some time in each of the following grade ranges: grades 3 through 5, 6 
through 9, and 10 through 12.

Requirements for 1994: Use the same assessment system to measure Title 
I students as the state uses to measure the performance of all other 
students. In the absence of a state system, a system that meets Title I 
requirements must be developed for use in all Title I schools; 
Requirements for 2001: Use the same assessment system to measure Title 
I students as the state uses to measure the performance of all other 
students. If the state provides evidence to the Secretary that it lacks 
authority to adopt a statewide system, it may meet the Title I 
requirement by adopting an assessment system on a statewide basis and 
limiting its applicability to Title I students or by ensuring that the 
Title I local educational agency (LEA) adopts standards and aligned 
assessments.

Requirements for 1994: Include in the assessment system multiple 
measures of student performance, including measures that assess higher 
order thinking skills and understanding; Requirements for 2001: 
Unchanged.

Requirements for 1994: Administer assessments for mathematics and 
reading in each of the following grade spans: grades 3 through 5, 6 
through 9, and 10 through 12; Requirements for 2001: Administer 
reading and mathematics tests annually in grades 
3 through 8, starting in the 2005-06 school year (in addition to the 
assessments previously required sometime within grades 10 through 12); 
States do not have to administer mathematics and reading or language 
arts tests annually in grades 3 through 8 if Congress does not provide 
specified amounts of funds to do so, but states have to continue to 
work on the development of the standards and assessments for those 
grades; Have students in grades 4 and 8 take the National Assessment 
of Educational Progress examinations in reading and mathematics every 
other year beginning in 2002-03, as long as the federal government pays 
for it.

Requirements for 1994: Assess students with either or both criterion 
referenced assessments and assessments that yield national norms. 
However, if the state uses only assessments referenced against national 
norms at a particular grade, those assessments must be augmented with 
additional items as necessary to accurately measure the depth and 
breath of the state's academic contents standards; Requirements for 
2001: Unchanged.

Requirements for 1994: Assess students with statewide, local, or a 
combination of state and local assessments. However, states that use 
all local or a combination of state and local assessments, must ensure, 
among other things, such assessments are aligned with the state's 
academic content standards, are equivalent to one another, and enable 
aggregation to determine whether the state has made adequate yearly 
progress; Requirements for 2001: Unchanged.

Requirements for 1994: Implement controls to ensure the quality of the 
data collected from the assessments; Requirements for 2001: Unchanged.

Including students with limited English proficiency and with 
disabilities in assessments:

Requirements for 1994: Assess students with disabilities and limited 
English proficiency according to standards for all other students; 
Provide reasonable adaptations and accommodations for students with 
disabilities or limited English proficiency to include testing in the 
language and form most likely to yield accurate and reliable 
information on what they know and can do; Requirements for 2001: By 
2002-03 annually assess the language proficiency of students with 
limited English proficiency. Students who have attended a U.S. school 
for 3 consecutive years must be tested in English unless an individual 
assessment by the district shows testing in a native language will be 
more reliable.

Reporting data:

Requirements for 1994: Report assessment results according to the 
following: by state, local educational agency (LEA), school, gender, 
major racial and ethnic groups, English proficiency, migrant status, 
disability, and economic disadvantage; Requirements for 2001: 
Unchanged.

Requirements for 1994: LEAs must produce for each Title I school a 
performance profile with disaggregated results and must publicize and 
disseminate these to teachers, parents, students, and the community. 
LEAs must also provide individual student reports, including test 
scores and other information on the attainment of student performance 
standards; Requirements for 2001: Provide annual information on the 
test performance of individual students and other indicators included 
in the state accountability system by 2002-03. Make this annual 
information available to parents and the public and include data on 
teacher qualifications. Compare high-and low-poverty schools with 
respect to the percentage of classes taught by teachers who are "highly 
qualified," as defined in the law, and conduct similar analyses for 
subgroups listed in previous law.

Measuring improvement:

Requirements for 1994: Use performance standards to establish a 
benchmark for improvement referred to as "adequate yearly progress." 
All LEAs and schools must meet the state's adequate yearly progress 
standard, for example, having 90 percent of their students performing 
at the proficient level in mathematics. LEAs and schools must show 
continuous progress toward meeting the adequate yearly progress 
standard. The state defines the level of progress a school or LEA must 
show. Schools that do not make the required advancement toward the 
adequate yearly progress standard can face consequences, such as the 
replacement of the existing staff; Requirements for 2001: In addition 
to showing gains in the academic achievement of the overall school 
population, schools and districts must show that the following 
subcategories of students have made gains in their academic 
achievement: pupils who are economically disadvantaged, have limited 
English proficiency, are disabled, or belong to a major racial or 
ethnic group. To demonstrate gains among these subcategories of 
students, school districts measure their progress against the state's 
definition of adequate yearly progress; States have 12 years for all 
students to perform at the proficient level.

Consequences for not meeting the adequate yearly 
progress standard:

Requirements for 1994: LEAs are required to identify for improvement 
any schools that fail to make adequate yearly progress for 2 
consecutive years and provide technical assistance to help failing 
schools develop and implement required improvement plans. After a 
school has failed to meet the adequate yearly progress standard for 3 
consecutive years, LEAs must take corrective action to improve the 
school; Requirements for 2001: New requirements are more specific as 
to what actions an LEA must take to improve failing schools. Actions 
are defined for each year the school continues to fail leading up to 
the 5th year of failure when a school may be restructured by changing 
to a charter school, replacing school staff, or state takeover of the 
school administration. The new law also provides that LEAs offer 
options to children in failing schools. Depending on the number of 
years a school has been designated for improvement, these options may 
include going to another public school with transportation paid by the 
LEA or using Title I funds to pay for supplemental help.

Source: P. L. No. 103-382 (1994) and Pub.L No. 107-110 (2001).


[End of table]

[End of section]

Appendix III: Number of Tests States Reported They Need to Develop or 
Augment to Comply with NCLBA (as of March 2003):

State: Alabama; Number of tests needed: 17.

State: Alaska; Number of tests needed: 9.

State: Arizona; Number of tests needed: 9.

State: Arkansas; Number of tests needed: 9.

State: California; Number of tests needed: 5.

State: Colorado; Number of tests needed: 4.

State: Connecticut; Number of tests needed: 8.

State: Delaware; Number of tests needed: 0.

State: District of Columbia; Number of tests needed: 17.

State: Florida; Number of tests needed: 0.

State: Georgia; Number of tests needed: 0.

State: Hawaii; Number of tests needed: 9.

State: Idaho; Number of tests needed: 17.

State: Illinois; Number of tests needed: 6.

State: Indiana; Number of tests needed: 9.

State: Iowa; Number of tests needed: 0.

State: Kansas; Number of tests needed: 11.

State: Kentucky; Number of tests needed: 8.

State: Louisiana; Number of tests needed: 8.

State: Maine; Number of tests needed: 8.

State: Maryland; Number of tests needed: 15.

State: Massachusetts; Number of tests needed: 6.

State: Michigan; Number of tests needed: 8.

State: Minnesota; Number of tests needed: 11.

State: Mississippi; Number of tests needed: 3.

State: Missouri; Number of tests needed: 8.

State: Montana; Number of tests needed: 17.

State: Nebraska; Number of tests needed: 11.

State: Nevada; Number of tests needed: 11.

State: New Hampshire; Number of tests needed: 9.

State: New Jersey; Number of tests needed: 10.

State: New Mexico; Number of tests needed: 17.

State: New York; Number of tests needed: 8.

State: North Carolina; Number of tests needed: 3.

State: North Dakota; Number of tests needed: 11.

State: Ohio; Number of tests needed: 8.

State: Oklahoma; Number of tests needed: 8.

State: Oregon; Number of tests needed: 6.

State: Pennsylvania; Number of tests needed: 11.

State: Puerto Rico; Number of tests needed: 10.

State: Rhode Island; Number of tests needed: 11.

State: South Carolina; Number of tests needed: 3.

State: South Dakota; Number of tests needed: 17.

State: Tennessee; Number of tests needed: 15.

State: Texas; Number of tests needed: 1.

State: Utah; Number of tests needed: 0.

State: Vermont; Number of tests needed: 9.

State: Virginia; Number of tests needed: 6.

State: Washington; Number of tests needed: 8.

State: West Virginia; Number of tests needed: 17.

State: Wisconsin; Number of tests needed: 17.

State: Wyoming; Number of tests needed: 11.

[End of table]

Source: GAO survey.

[End of section]

Appendix IV: Estimates of Assessment Expenditures NCLBA Required, but 
Not in Place at the Time of Our Survey, FY 2002-08:

Table 11 provides estimates of assessment expenditures states may incur 
for grades and subjects they reported they would need to add to meet 
the additional assessment requirements under NCLBA. These estimates do 
not include any expenditures for continuing development or 
administration of assessments in grades and subjects already included 
in states' reported assessment program, unless states indicated plans 
to replace its existing assessments. Estimates reflect total 
expenditures between fiscal year 
2002 and 2008, and are based on the assumptions we made regarding 
question types.

Table 11: Estimates of Expenditures for the Assessments Required by 
NCLBA That Were Not in Place at the Time of Our Survey, Fiscal Years 
2002-08:

Dollars in billions: Question type: Multiple-choice; Dollars in 
billions: Estimate: $0.8; Questions and scoring methods used: 
Estimate assumes that all states use machine-scored multiple-choice 
questions.

Dollars in billions: Question type: Current question type; Dollars in 
billions: Estimate: $1.6; Questions and scoring methods used: 
Estimate assumes that states use the mix of question types they 
reported in our survey.

Dollars in billions: Question type: Multiple-choice and open-ended; 
Dollars in billions: Estimate: $2.0; Questions and scoring 
methods used: Estimate assumes that all states use both machine scored 
multiple-choice questions and some hand scored open-ended questions.

Source: GAO.

Note: Projections based on state assessment plans and characteristics 
and expenditure data gathered from 7 states.

[End of table]

[End of section]

Appendix V: State Development and Nondevelopment Estimates:

Table 12 provides test development and nondevelopment expenditures by 
state between fiscal year 2002-08. Test development estimates reflect 
expenditures associated with both new and existing tests. 
Nondevelopment expenditures reflect expenditures associated with 
administration, scoring, and reporting of results for both new and 
existing assessments.

Table 12: Estimates by State, Development, and Nondevelopment 
Expenditures:

Dollars in millions:

Alabama; Multiple-choice and open-ended: Development: $16; Multiple-
choice and open-ended: Non-development: $57; Current question 
type: Development: $15; Current question type: Non-development: $15; 
Multiple-choice: Development: $15; Multiple-choice: Non-
development: $15.

Alaska; Multiple-choice and open-ended: Development: 15; Multiple-
choice and open-ended: Non-development: 14; Current question 
type: Development: 15; Current question type: Non-development: 10; 
Multiple-choice: Development: 13; Multiple-choice: Non-
development: 4.

Arizona; Multiple-choice and open-ended: Development: 15; Multiple-
choice and open-ended: Non-development: 93; Current question 
type: Development: 15; Current question type: Non-development: 93; 
Multiple-choice: Development: 14; Multiple-choice: Non-
development: 25.

Arkansas; Multiple-choice and open-ended: Development: 14; Multiple-
choice and open-ended: Non-development: 39; Current question 
type: Development: 14; Current question type: Non-development: 28; 
Multiple-choice: Development: 13; Multiple-choice: Non-
development: 10.

California; Multiple-choice and open-ended: Development: 13; Multiple-
choice and open-ended: Non-development: 619; Current question 
type: Development: 12; Current question type: Non-development: 223; 
Multiple-choice: Development: 12; Multiple-choice: Non-
development: 166.

Colorado; Multiple-choice and open-ended: Development: 13; Multiple-
choice and open-ended: Non-development: 74; Current question 
type: Development: 13; Current question type: Non-development: 74; 
Multiple-choice: Development: 12; Multiple-choice: Non-
development: 20.

Connecticut; Multiple-choice and open-ended: Development: 14; 
Multiple-choice and open-ended: Non-development: 54; Current 
question type: Development: 14; Current question type: Non-development: 
54; Multiple-choice: Development: 13; Multiple-choice: Non-
development: 15.

Delaware; Multiple-choice and open-ended: Development: 12; Multiple-
choice and open-ended: Non-development: 13; Current question 
type: Development: 12; Current question type: Non-development: 13; 
Multiple-choice: Development: 11; Multiple-choice: Non-
development: 3.

District of Columbia; Multiple-choice and open-ended: Development: 13; 
Multiple-choice and open-ended: Non-development: 4; Current 
question type: Development: 12; Current question type: Non-development: 
1; Multiple-choice: Development: 12; Multiple-choice: Non-
development: 1.

Florida; Multiple-choice and open-ended: Development: 12; Multiple-
choice and open-ended: Non-development: 269; Current question 
type: Development: 12; Current question type: Non-development: 200; 
Multiple-choice: Development: 11; Multiple-choice: Non-
development: 72.

Georgia; Multiple-choice and open-ended: Development: 12; Multiple-
choice and open-ended: Non-development: 162; Current question 
type: Development: 11; Current question type: Non-development: 44; 
Multiple-choice: Development: 11; Multiple-choice: Non-
development: 44.

Hawaii; Multiple-choice and open-ended: Development: 14; Multiple-
choice and open-ended: Non-development: 17; Current question 
type: Development: 14; Current question type: Non-development: 17; 
Multiple-choice: Development: 13; Multiple-choice: Non-
development: 5.

Idaho; Multiple-choice and open-ended: Development: 15; Multiple-
choice and open-ended: Non-development: 16; Current question 
type: Development: 14; Current question type: Non-development: 9; 
Multiple-choice: Development: 14; Multiple-choice: Non-
development: 4.

Illinois; Multiple-choice and open-ended: Development: 13; Multiple-
choice and open-ended: Non-development: 198; Current question 
type: Development: 13; Current question type: Non-development: 151; 
Multiple-choice: Development: 12; Multiple-choice: Non-
development: 53.

Indiana; Multiple-choice and open-ended: Development: 14; Multiple-
choice and open-ended: Non-development: 99; Current question 
type: Development: 14; Current question type: Non-development: 99; 
Multiple-choice: Development: 13; Multiple-choice: Non-
development: 27.

Iowa; Multiple-choice and open-ended: Development: 12; Multiple-choice 
and open-ended: Non-development: 50; Current question type: 
Development: 12; Current question type: Non-development: 50; 
Multiple-choice: Development: 11; Multiple-choice: Non-development: 
14.

Kansas; Multiple-choice and open-ended: Development: 14; Multiple-
choice and open-ended: Non-development: 37; Current question 
type: Development: 13; Current question type: Non-development: 23; 
Multiple-choice: Development: 13; Multiple-choice: Non-
development: 10.

Kentucky; Multiple-choice and open-ended: Development: 14; Multiple-
choice and open-ended: Non-development: 58; Current question 
type: Development: 14; Current question type: Non-development: 48; 
Multiple-choice: Development: 13; Multiple-choice: Non-
development: 16.

Louisiana; Multiple-choice and open-ended: Development: 14; Multiple-
choice and open-ended: Non-development: 67; Current question 
type: Development: 14; Current question type: Non-development: 67; 
Multiple-choice: Development: 13; Multiple-choice: Non-
development: 18.

Maine; Multiple-choice and open-ended: Development: 14; Multiple-
choice and open-ended: Non-development: 19; Current question 
type: Development: 14; Current question type: Non-development: 19; 
Multiple-choice: Development: 13; Multiple-choice: Non-
development: 5.

Maryland; Multiple-choice and open-ended: Development: 16; Multiple-
choice and open-ended: Non-development: 75; Current question 
type: Development: 16; Current question type: Non-development: 75; 
Multiple-choice: Development: 15; Multiple-choice: Non-
development: 20.

Massachusetts; Multiple-choice and open-ended: Development: 13; 
Multiple-choice and open-ended: Non-development: 96; Current 
question type: Development: 13; Current question type: Non-development: 
96; Multiple-choice: Development: 12; Multiple-choice: Non-
development: 26.

Michigan; Multiple-choice and open-ended: Development: 14; Multiple-
choice and open-ended: Non-development: 163; Current question 
type: Development: 14; Current question type: Non-development: 163; 
Multiple-choice: Development: 13; Multiple-choice: Non-
development: 44.

Minnesota; Multiple-choice and open-ended: Development: 15; Multiple-
choice and open-ended: Non-development: 76; Current question 
type: Development: 15; Current question type: Non-development: 76; 
Multiple-choice: Development: 14; Multiple-choice: Non-
development: 20.

Mississippi; Multiple-choice and open-ended: Development: 13; 
Multiple-choice and open-ended: Non-development: 51; Current 
question type: Development: 13; Current question type: Non-development: 
51; Multiple-choice: Development: 12; Multiple-choice: Non-
development: 14.

Missouri; Multiple-choice and open-ended: Development: 14; Multiple-
choice and open-ended: Non-development: 85; Current question 
type: Development: 14; Current question type: Non-development: 85; 
Multiple-choice: Development: 13; Multiple-choice: Non-
development: 23.

Montana; Multiple-choice and open-ended: Development: 16; Multiple-
choice and open-ended: Non-development: 13; Current question 
type: Development: 16; Current question type: Non-development: 12; 
Multiple-choice: Development: 15; Multiple-choice: Non-
development: 3.

Nebraska; Multiple-choice and open-ended: Development: 13; Multiple-
choice and open-ended: Non-development: 21; Current question 
type: Development: 13; Current question type: Non-development: 21; 
Multiple-choice: Development: 12; Multiple-choice: Non-
development: 6.

Nevada; Multiple-choice and open-ended: Development: 14; Multiple-
choice and open-ended: Non-development: 31; Current question 
type: Development: 13; Current question type: Non-development: 13; 
Multiple-choice: Development: 13; Multiple-choice: Non-
development: 8.

New Hampshire; Multiple-choice and open-ended: Development: 13; 
Multiple-choice and open-ended: Non-development: 18; Current 
question type: Development: 13; Current question type: Non-development: 
18; Multiple-choice: Development: 12; Multiple-choice: Non-
development: 5.

New Jersey; Multiple-choice and open-ended: Development: 14; Multiple-
choice and open-ended: Non-development: 113; Current question 
type: Development: 14; Current question type: Non-development: 113; 
Multiple-choice: Development: 13; Multiple-choice: Non-
development: 30.

New Mexico; Multiple-choice and open-ended: Development: 16; Multiple-
choice and open-ended: Non-development: 25; Current question 
type: Development: 16; Current question type: Non-development: 24; 
Multiple-choice: Development: 15; Multiple-choice: Non-
development: 7.

New York; Multiple-choice and open-ended: Development: 14; Multiple-
choice and open-ended: Non-development: 262; Current question 
type: Development: 14; Current question type: Non-development: 262; 
Multiple-choice: Development: 13; Multiple-choice: Non-
development: 70.

North Carolina; Multiple-choice and open-ended: Development: 13; 
Multiple-choice and open-ended: Non-development: 139; Current 
question type: Development: 12; Current question type: Non-development: 
37; Multiple-choice: Development: 12; Multiple-choice: Non-
development: 37.

North Dakota; Multiple-choice and open-ended: Development: 14; 
Multiple-choice and open-ended: Non-development: 9; Current 
question type: Development: 14; Current question type: Non-development: 
9; Multiple-choice: Development: 13; Multiple-choice: Non-
development: 2.

Ohio; Multiple-choice and open-ended: Development: 13; Multiple-choice 
and open-ended: Non-development: 158; Current question type: 
Development: 13; Current question type: Non-development: 158; 
Multiple-choice: Development: 12; Multiple-choice: Non-development: 
42.

Oklahoma; Multiple-choice and open-ended: Development: 14; Multiple-
choice and open-ended: Non-development: 53; Current question 
type: Development: 13; Current question type: Non-development: 24; 
Multiple-choice: Development: 13; Multiple-choice: Non-
development: 14.

Oregon; Multiple-choice and open-ended: Development: 14; Multiple-
choice and open-ended: Non-development: 57; Current question 
type: Development: 13; Current question type: Non-development: 15; 
Multiple-choice: Development: 13; Multiple-choice: Non-
development: 15.

Pennsylvania; Multiple-choice and open-ended: Development: 15; 
Multiple-choice and open-ended: Non-development: 166; Current 
question type: Development: 15; Current question type: Non-development: 
147; Multiple-choice: Development: 14; Multiple-choice: Non-
development: 45.

Puerto Rico; Multiple-choice and open-ended: Development: 14; Multiple-
choice and open-ended: Non-development: 56; Current question 
type: Development: 13; Current question type: Non-development: 15; 
Multiple-choice: Development: 13; Multiple-choice: Non-
development: 15.

Rhode Island; Multiple-choice and open-ended: Development: 14; 
Multiple-choice and open-ended: Non-development: 13; Current 
question type: Development: 14; Current question type: Non-development: 
13; Multiple-choice: Development: 13; Multiple-choice: Non-
development: 4.

South Carolina; Multiple-choice and open-ended: Development: 13; 
Multiple-choice and open-ended: Non-development: 73; Current 
question type: Development: 13; Current question type: Non-development: 
70; Multiple-choice: Development: 12; Multiple-choice: Non-
development: 19.

South Dakota; Multiple-choice and open-ended: Development: 17; 
Multiple-choice and open-ended: Non-development: 10; Current 
question type: Development: 15; Current question type: Non-development: 
3; Multiple-choice: Development: 15; Multiple-choice: Non-
development: 3.

Tennessee; Multiple-choice and open-ended: Development: 15; Multiple-
choice and open-ended: Non-development: 70; Current question 
type: Development: 14; Current question type: Non-development: 19; 
Multiple-choice: Development: 14; Multiple-choice: Non-
development: 19.

Texas; Multiple-choice and open-ended: Development: 12; Multiple-
choice and open-ended: Non-development: 429; Current question 
type: Development: 11; Current question type: Non-development: 221; 
Multiple-choice: Development: 11; Multiple-choice: Non-
development: 115.

Utah; Multiple-choice and open-ended: Development: 12; Multiple-choice 
and open-ended: Non-development: 50; Current question type: 
Development: 12; Current question type: Non-development: 33; 
Multiple-choice: Development: 11; Multiple-choice: Non-development: 
13.

Vermont; Multiple-choice and open-ended: Development: 15; Multiple-
choice and open-ended: Non-development: 10; Current question 
type: Development: 15; Current question type: Non-development: 10; 
Multiple-choice: Development: 14; Multiple-choice: Non-
development: 3.

Virginia; Multiple-choice and open-ended: Development: 13; Multiple-
choice and open-ended: Non-development: 116; Current question 
type: Development: 12; Current question type: Non-development: 48; 
Multiple-choice: Development: 12; Multiple-choice: Non-
development: 31.

Washington; Multiple-choice and open-ended: Development: 14; Multiple-
choice and open-ended: Non-development: 104; Current question 
type: Development: 14; Current question type: Non-development: 104; 
Multiple-choice: Development: 13; Multiple-choice: Non-
development: 28.

West Virginia; Multiple-choice and open-ended: Development: 17; 
Multiple-choice and open-ended: Non-development: 26; Current 
question type: Development: 16; Current question type: Non-development: 
7; Multiple-choice: Development: 16; Multiple-choice: Non-
development: 7.

Wisconsin; Multiple-choice and open-ended: Development: 15; Multiple-
choice and open-ended: Non-development: 57; Current question 
type: Development: 15; Current question type: Non-development: 51; 
Multiple-choice: Development: 14; Multiple-choice: Non-
development: 15.

Wyoming; Multiple-choice and open-ended: Development: 14; Multiple-
choice and open-ended: Non-development: 7; Current question 
type: Development: 14; Current question type: Non-development: 7; 
Multiple-choice: Development: 13; Multiple-choice: Non-
development: 2.

Total; Multiple-choice and open-ended: Development: $724; Multiple-
choice and open-ended: Non-development: $4,590; Current 
question type: Development: $706; Current question type: Non-
development: $3,237; Multiple-choice: Development: $668; 
Multiple-choice: Non-development: $1,233.

Source: GAO estimates based on state assessment plans and 
characteristics and expenditure data gathered from 7 states.

[End of table]

[End of section]

Appendix VI: Fiscal Years 2002-08 Estimated Expenditures for Each 
Question Type:

Table 13 provides estimates for each question type and the benchmark 
appropriations by fiscal years from 2002 through 2008. Each estimate 
reflects assumptions about the type of questions on the assessments. 
For example, the multiple-choice estimate assumes that all states will 
use assessments with only multiple-choice questions. These estimates 
also assume that states implement the assessment plans reported to us. 
The benchmark appropriation is based on actual appropriations in 2002 
and 2003 and on the benchmark funding level in NCLBA for 2004-07. We 
assumed a benchmark of $400 million in 2008, the same as in 2005, 2006, 
and 2007.

Table 13: Estimated Expenditures for Each Question Type, Fiscal Years 
2002-08:

Question type.

Multiple-choice; Fiscal year (in millions): 2002: $165; Fiscal year (in 
millions): 2003: 237; Fiscal year (in millions): 2004: 288; Fiscal year 
(in millions): 2005: 291; Fiscal year (in millions): 2006: 293; Fiscal 
year (in millions): 2007: 308; Fiscal year (in millions): 2008: 318; 
Total: $1,901.

Current question type; Fiscal year (in millions): 2002: $324; Fiscal 
year (in millions): 2003: 442; Fiscal year (in millions): 2004: 572; 
Fiscal year (in millions): 2005: 615; Fiscal year (in millions): 2006: 
633; Fiscal year (in millions): 2007: 665; Fiscal year (in millions): 
2008: 692; Total: $3,944.

Multiple-choice and open-ended; Fiscal year (in millions): 2002: $445; 
Fiscal year (in millions): 2003: 586; Fiscal year (in millions): 2004: 
761; Fiscal year (in millions): 2005: 824; Fiscal year (in millions): 
2006: 855; Fiscal year (in millions): 2007: 903; Fiscal year (in 
millions): 2008: 941; Total: $5,313.

Benchmark appropriation; Fiscal year (in millions): 2002: $366; Fiscal 
year (in millions): 2003: 376; Fiscal year (in millions): 2004: 390; 
Fiscal year (in millions): 2005: 400; Fiscal year (in millions): 2006: 
400; Fiscal year (in millions): 2007: 400; Fiscal year (in millions): 
2008: 400; Total: $2,733.

Source: GAO estimates based on state assessment plans and 
characteristics and expenditure data gathered from 7 states.

Note: Fiscal years 2002 through 2008 sums may not equal the total 
because of rounding.

[End of table]

[End of section]

Appendix VII: Comments from the Department of Education:

UNITED STATES DEPARTMENT OF EDUCATION:

THE UNDER SECRETARY:

April 29, 2003:

Ms. Marnie S. Shaul Director:

Education, Workforce, and Income Security Issues United States General 
Accounting Office Washington, DC 20548:

Dear Ms. Shaul:

I am writing in response to the General Accounting Office's (GAO) draft 
report, "Title I: Characteristics of Tests Will Influence Expenses; 
Guidance May Help States Realize Efficiencies." We appreciate the 
opportunity to review and respond.

Problems with Estimating the Costs of Testing:

While the draft report contains some useful information on the 
estimated costs of testing in the seven States studied, the report goes 
on to project these estimates on to all other States, which makes the 
report much less valuable and possibly misleading. We are very 
concerned about the inclusion and the weight given to the estimates of 
costs for each State based on estimates of the costs in the particular 
circumstances of only seven States studied in depth by GAO. In effect, 
this section of the draft report uses multiple levels of assumptions, 
which results in estimates that have the potential to be substantially 
in error. The GAO report ends up with three specific cost estimates for 
each State that have a ring of authority that we believe is 
significantly out of proportion to the confidence one can place in 
them.

While the other forty-five "States" (including the District of Columbia 
and Puerto Rico) apparently responded to survey questions, it does not 
appear that they provided the level of detailed cost information used 
in the draft report on the costs for each of the States. As the study 
acknowledges, the factors in computing and estimating costs are very 
specific to the circumstances of each State, and cannot be generalized.

Many factors can affect the costs in different States to make the 
estimates wrong and misleading. For example, the report cites the types 
of questions included in State assessments as one of the main reasons 
for different costs; yet 48 percent of the States reported that they 
are uncertain about the type of questions they will include on future 
tests, thus making projected costs in those States suspect.	The report 
also cites other factors such as the number of different forms of 
assessments used, and the extent of public release of questions. We 
believe that there are many other factors that may also be crucial, 
such as the scoring of assessments through outside contracts versus the 
scoring of assessments by in-house staff, the expertise and experience 
of State staff, and many other individual characteristics of a State, 
including specific characteristics of its student population.

The number of new tests that each State reported it would need to 
develop is found in Table 3 and used subsequently to estimate 
development costs by multiplying the number of tests in initial 
development by 3 times the average ongoing development expenditure. The 
assumption behind this calculation is questionable because: a) the GAO 
reports that they were typically not able to separate the costs of new 
test development from ongoing test development, and b) the costs of 
initial test development will vary tremendously by the nature of the 
test. For instance, to develop an 8th-grade reading test when grades 3-
7 already are being tested should be a trivial expense compared to 
developing a science assessment when none currently exists. A more 
reasonable estimate for test development could be derived from the 
total test development expenditure in the seven States surveyed, since 
that includes initial and ongoing test development.

We also question the draft report's analysis of question type --i.e., 
multiple-choice versus open-ended questions --as a key determinant of 
costs. The draft report fails to differentiate open-ended questions 
that involve short factual answers from open-ended questions that 
involve lengthy writing samples. The costs of the latter will be quite 
high compared to the costs of the former. The draft report also does 
not consider the proportion of open-ended questions employed in an 
assessment. The functions of open-ended questions can be provided by 
relatively small proportions of such questions compared to multiple-
choice questions. By not taking into account the nature of open-ended 
questions, and by not adjusting for the ability of States to retain 
open-ended questions while lowering costs through reducing the 
proportion of such questions, the draft report likely over-estimates 
substantially the assessment costs in the upper two of its three 
estimates.

The draft report assumes that costs will be lower in the first few 
years of test administration and will increase in later years when more 
tests are being developed and administered. One could reasonably argue 
the opposite, that costs are always greater at the outset and that 
States are likely to combine their test development process in a single 
content area such as reading across grades 3-8 in the initial years. As 
a result "out year" costs would be lower. Moreover, GAO projections do 
not take into account the results of increasing competition as more 
companies enter the burgeoning State assessment market. Likewise, no 
provision is made for advances in technology. There are already 
companies in the market that are capable of administering State 
assessments with handheld computers. Software currently available can 
score open-ended questions. The forces of competition and technology 
almost surely will drive down costs in the development and 
administration of State assessments.

Even though the draft report, in a footnote on page 26, acknowledges 
that the estimates "may be biased," based on the many problems noted in 
this response, we strongly recommend the deletion of the information on 
estimates of the costs for the States not studied directly.

We believe the report is also misleading in suggesting that all of the 
estimated costs are generated by the testing requirements in the No 
Child Left Behind Act. Educational assessments are an inherent 
responsibility of the States, and many States (and, in many cases, 
school districts) have already developed and administered tests that 
would meet No Child Left Behind requirements. Many of these costs have 
been borne or would be borne by the States irrespective of the No Child 
Left Behind Act. We think the report needs to make the point that not 
all of these costs are incremental costs generated by the Act.

Problems in Indicating the Sources of Federal Funding:

In addition, the draft report contains information on sources of 
Federal funding, but does not, by any means, provide a complete 
picture. For example, it appears to focus on the funding under one 
Federal program under which testing costs are allowable. But it does 
not include other funding sources in the No Child Left Behind Act under 
which testing costs would also be allowable, such as Title I, Part A 
administrative costs, consolidated administrative costs under Section 
9201 of the Elementary and Secondary Education Act (ESEA), Title V of 
ESEA, and the additional funds that may be transferred under Title VI 
of ESEA to various funding sources. Thus, it seriously understates 
available resources provided under Federal programs.

The Recommendation for Sharing Information:

We have no problems with the one recommendation contained in the report 
--that the Department of Education use its existing mechanisms to 
facilitate the sharing of information among States regarding assessment 
development and administration as States attempt to reduce expenses. As 
one example of information sharing already undertaken and facilitated 
by the Department, we suggest that you include in your report 
information on the Enhanced Assessment Grants, as authorized under 
Title VI of the No Child Left Behind Act. In February 2003, the 
Department awarded $17 million to fund projects aimed at improving the 
quality of State assessment instruments, especially for students with 
disabilities and students of limited English proficiency. In selecting 
grant recipients, the Department awarded priority points to 
applications submitted by consortia of States. All nine awards went to 
State consortia, ranging from three to fifteen States per consortium. 
The Department takes very seriously its commitment to provide technical 
assistance to State and local grantees and looks forwards to continuing 
and enhancing its efforts to share information on assessment 
development and administration to help States reduce costs and make the 
accountability provisions of the No Child Left Behind Act as effective 
and efficient as possible.

We additionally suggest a change in the title of the report to more 
accurately reflect the report's content. We suggest substituting 
"Information Sharing" for "Guidance" so that the title reads 
"Characteristics of Tests Will Influence Expenses: Information Sharing 
May Help States Realize Efficiencies.":

We appreciate the opportunity to provide comments on this draft report, 
and would be glad to work with your office to make the report more 
reliable and useful.

Sincerely,

Eugene W. Hickok:

Signed by Eugene W. Hickok:

[End of section]

Appendix VIII: GAO Contacts and Staff Acknowledgments:

GAO Contacts:

Sherri Doughty (202) 512-7273
Jason Palmer (202) 512-3825:

Staff Acknowledgments:

In addition to those named above, Lindsay Bach, Cindy Decker, and 
Patrick DiBattista made important contributions to this report. 
Theresa Mechem provided assistance with graphics.

FOOTNOTES

[1] NCLBA authorizes funding through fiscal year 2007 for assessments. 
However, consistent with the mandate for this study, we examined 
expenditures between fiscal years 
2002 through 2008, enabling us to more fully capture expenditures 
associated with the science assessments, which are required to be 
administered in school year 2007-08.

[2] A norm referenced test evaluates an individual's performance in 
relation to the performance of a large sample of others, usually 
selected to represent all students nationally in the same grade or age 
range. Criterion referenced tests are assessments that measure the 
mastery of specific skills or subject content and focus on the 
performance of an individual as measured against a standard or 
criterion rather than the performance of others taking the test.

[3] The Secretary of Education may provide states 1 additional year if 
the state demonstrates that exceptional or uncontrollable 
circumstances, such as a natural disaster or precipitous and unforeseen 
decline in the financial resources of the state prevented full 
implementation of the academic assessments by the deadlines.

[4] U.S. General Accounting Office, Title I: Education Needs to Monitor 
States' Scoring of Assessments, GAO-02-393 (Washington, D. C.: Apr. 1, 
2002).

[5] According to Education, there are also other sources of funding in 
NCLBA that states may draw upon for assessment related expenses.

[6] NASBE and AccountabilityWorks made different assumptions regarding 
what costs would vary with the number of students tested and which 
would be invariant costs. For example, NASBE assumed that development 
costs would vary by the number of students taking the test and 
AccountabilityWorks did not. Additionally, AccountabilityWorks reports 
having verified its assumptions with officials from two states, while 
the authors of the NASBE study do not report having verified their 
assumption with state officials.

[7] This amount is generally 1 percent of the amount that states 
receive under Title I or $400,000, whichever is greater.

[8] The District of Columbia and Puerto Rico are included in our state 
totals.

[9] Two states reported that they did not provide accommodations for 
students with disabilities at the state level, however, accommodations 
may have been provided at the local school level.

[10] New York offers its assessments in Spanish, Korean, Haitian 
Creole, and Russian and Minnesota offers its mathematics assessments in 
Spanish, Hmong, Somali, and Vietnamese.

[11] In Texas and Colorado, we were unable to separate scoring 
expenditures from other types of expenditures.

[12] This may not be true for smaller states because they may have 
fewer assessments to administer, score, and report.

[13] The study also does not estimate the opportunity costs of 
assessments.

[14] Because our expenditure data were limited to 7 states, our 
estimates may be biased. For example, if the 7 states we selected had 
higher average development expenditures per ongoing assessment than the 
average state, then our estimate of development expenditures would be 
biased upwards.

[15] We were unable to obtain information on personnel expenditures 
from 5 of the 7 states, and so we did not include personnel 
expenditures in our analysis. In the 2 states in which we obtained 
personnel expenditures, such expenditures were a relatively small part 
of the assessment budget. 

[16] We found estimates were not sensitive to changes in assumptions 
regarding development costs, partly because they proved to be a 
generally small portion of overall expenses.

[17] For states that reported that they did not know the kinds of 
question they would use on future tests, we assumed that future test 
would be the same as they currently use. Where data were missing, we 
assumed that states would use assessments with both multiple-choice and 
open-ended questions, potentially biasing our estimates upward.

GAO's Mission:

The General Accounting Office, the investigative arm of Congress, 
exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony:

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics.

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to daily E-mail alert for newly 
released products" under the GAO Reports heading.

Order by Mail or Phone:

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to:

U.S. General Accounting Office

441 G Street NW,

Room LM Washington,

D.C. 20548:

To order by Phone: 	

	Voice: (202) 512-6000:

	TDD: (202) 512-2537:

	Fax: (202) 512-6061:

To Report Fraud, Waste, and Abuse in Federal Programs:

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470:

Public Affairs:

Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.

General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.

20548: