This is the accessible text file for GAO report number GAO-08-1037R entitled 'Military Personnel: Evaluation Methods Linked to Anticipated Outcomes Needed to Inform Decisions on Army Recruitment Incentives' which was released on September 18, 2008. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. GAO-08-1037R: United States Government Accountability Office: Washington, DC 20548: September 18, 2008: Congressional Committees: Subject: Military Personnel: Evaluation Methods Linked to Anticipated Outcomes Needed to Inform Decisions on Army Recruitment Incentives: Since the September 11, 2001 attacks on the World Trade Center and the Pentagon, the United States has launched several military operations that have dramatically increased the operations tempo of the military services and required the large-scale mobilization of reservists. These factors have particularly affected the active Army, Army Reserve, and Army National Guard, which have shouldered the bulk of the personnel burden associated with ongoing operations in Iraq. A 2007 Congressional Research Service report notes that many observers have expressed concern that these factors might lead to lower recruiting and retention rates, thereby jeopardizing the vitality of today's all-volunteer military.[Footnote 1] Additionally, in 2004 the Army began its modular force transformation to restructure itself from a division-based force to a more agile and responsive modular brigade-based force--an undertaking it considers to be the most extensive reorganization of its force since World War II. Both ongoing military operations and transformation have prompted the Army to increase its recruitment efforts. To encourage military service, Congress, through Section 681 of the National Defense Authorization Act for Fiscal Year 2006,[Footnote 2] temporarily authorized the Army to provide not more than four new recruitment incentives and directed the Secretary of the Army to submit to Congress a plan for each recruitment incentive it develops under the authority provided. Section 681 states that each plan should include (1) a description of the incentive, including its purpose and the potential recruits to be addressed by the incentive; (2) a description of the provisions of the U.S. Code relevant to the military[Footnote 3] that would need to be waived in order for the Army to provide the incentive and an explanation of why these provisions would need to be waived; (3) a statement of the anticipated outcomes as a result of providing the incentive; and (4) the method to be used to evaluate the effectiveness of the incentive. The Army is also required to submit an annual report to Congress on each of the recruitment incentives developed under this authority; this report is to include a description of the incentives and an assessment of their impact on recruitment during the previous fiscal year. The Army began providing recruitment incentives under this authority in June 2006 and is currently using it to pilot three recruitment incentives. Under Section 681, the Army's authority to provide these new recruitment incentives expires on December 31, 2009. The Army may modify, expand, or take steps to make permanent some or all of these recruitment incentives, based on the data it collects during this pilot phase. Section 681 also directs the Secretary of the Army to submit each recruitment incentive plan to the Comptroller General and requires GAO to report to Congress on the expected outcomes of each recruitment incentive in terms of cost effectiveness and mission achievement as soon as practicable after receipt of each plan. For this report, we (1) identified and described the recruitment incentives the Army has developed under Section 681 of the National Defense Authorization Act for Fiscal Year 2006 and (2) assessed the extent to which the plans for each incentive included anticipated outcomes and a methodology for evaluating these outcomes. To address our first objective, we obtained information about the recruitment incentives that the Army has developed under Section 681 through the two guiding documents for each incentive--the notification letter and the accompanying plan provided to Congress.[Footnote 4] For our second objective, we analyzed the extent to which the plans that the Army developed for each incentive included anticipated outcomes and a methodology for evaluating these outcomes. For the one incentive developed pursuant to Section 681 that the Army has offered for more than 1 year, we reviewed the letter to Congress which comprises the Army's required annual report; further, we reviewed the two assessments that provided input to this letter. We also obtained information from interviews and correspondence with officials from the Army's Office of the Deputy Chief of Staff for Personnel; the Army National Guard; Army General Counsel; U.S. Army Accessions Command; U.S. Army Recruiting Command; the Office of the Assistant Secretary of the Army for Manpower and Reserve Affairs; and the Office of the Undersecretary of Defense for Personnel and Readiness. We conducted this performance audit from April 2008 through September 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Results in Brief: The Secretary of the Army has notified Congress--as the law requires-- of the three recruitment incentives that the Army has developed under Section 681 of the National Defense Authorization Act for Fiscal Year 2006. As further required by Section 681, the Army has developed a plan for each of these incentives. * The Recruiter Incentive Pay Pilot Program, which the Army began offering in June 2006, provides monthly, quarterly, and annual bonus pay to Army, Army Reserve, and Army National Guard recruiters--of both officers and enlisted soldiers--who exceed their recruitment goals. * The Army Advantage Fund Pilot Program, which the Army began offering in February 2008, provides payment to enlisted active Army and Army Reserve recruits with no prior service, following their completion of service, to be used for home ownership or small business development. * The Officer Accession Pilot Program, which the Army began offering in August 2008, offers three options--two that target Reserve Officer Training Corps (ROTC) cadets and one that targets certain qualified medical professionals and chaplains between the ages of 43 and 60. One of the ROTC options provides a monetary bonus to cadets who attend the Leader Training Course, a 4-week program that ROTC students take during the summer between their sophomore and junior years of college. The second ROTC option provides a monthly stipend to cadets who participate satisfactorily in approved language immersion programs, programs of study abroad, or academic courses involving instruction in a foreign language of strategic importance to DOD. Both of these options require cadets to sign a contract to serve as commissioned officers after they graduate. The third option reduces the mandatory length of the military service obligation from 8 years to 2 years for certain qualified medical professionals and chaplains. Army officials with whom we spoke stated that the Army does not currently expect to develop a fourth recruitment incentive. However, the Army noted in technical comments on our draft report that it reserves the option to develop one more recruitment incentive pursuant to Section 681. Two of the Army's three recruitment incentive plans do not include anticipated outcomes and none describes a methodology to be used to evaluate its effectiveness. Section 681 requires that each plan the Secretary of the Army submits to Congress include, among other things, (1) a statement of the outcomes the Army anticipates will result from the incentive offered, and (2) a description of the method to be used to evaluate the incentive's effectiveness. Although the Army did not include this information in the plans it sent to Congress for the first two incentives, the Army did include information about anticipated outcomes in accompanying notification letters. In addition, the Army assigned responsibility for developing evaluation methodologies and assessing the Recruiter Incentive Pay Pilot Program to the U.S. Army's Accession Command's Center for Accession Research and for the Army Advantage Fund Pilot Program to RAND. For the Officer Accession Pilot Program, the Army did include anticipated outcomes in the plan, and it assigned the evaluation responsibilities to the U.S. Army Accessions Command, the Army Surgeon General, and the Army Chief of Chaplains. However, the information on anticipated outcomes is limited for all three incentives in that it does not include the assumptions behind the numbers, and, therefore, the link between the desired outcomes and the methodology to be used to assess progress is not clear. For example, while the U.S. Army Accessions Command's Center for Accessions Research recommended that the Army stop offering the Recruiter Incentive Pay Pilot Program based on its assessment of the incentive on the active Army and Army Reserve, the National Guard Bureau recommended that the Army continue to offer it based on the Bureau's assessment of the incentive's effect on the Army National Guard. Because information on this incentive's anticipated outcomes is limited, Army officials continue to lack the information they need to make informed, objective decisions regarding the extent to which--and the reasons why--the program did or did not meet its intended goals. In this case, for the Recruiter Incentive Pay Pilot Program, the Secretary of the Army determined that further study was required and modified the incentive. The results of an initial status report on the Army Advantage Fund Pilot Program were similarly inconclusive because the anticipated outcomes that the Army included in the notification letter to Congress were not applicable to the incentive as implemented. After this initial status report, the Army determined it would need more time before it could provide information about the effectiveness of this incentive. Because Army officials responsible for managing the recruitment incentives did not incorporate all of the necessary elements into their recruitment incentive plans, the plans do not present a clear and convincing business case for the incentives, and it is difficult for those exercising oversight to make an independent assessment of the extent to which the incentives have been a success. To improve management of the incentives and ensure that the required annual assessments will be of use to Army decision makers, we are recommending that--for each recruitment incentive developed pursuant to Section 681--the Secretary of the Army issue guidance to clearly specify anticipated outcomes, describe the assumptions behind these anticipated outcomes, identify the evaluation method to be used to assess progress toward these outcomes, and link anticipated outcomes to the stated evaluation method. We are also recommending that the Secretary of the Army ensure that this information is specifically addressed in the required annual assessments for each recruitment incentive. Additionally, should the Army decide--based on the results of the annual assessments of each incentive--to seek to make any of these incentives permanent, we further recommend that the Secretary of the Army, for any proposals the Army may wish to develop, include in its business case clearly specified anticipated outcomes for each incentive, a description of the assumptions behind these anticipated outcomes, an identification of the evaluation method to be used to assess progress toward these outcomes, and an explanation that links anticipated outcomes to the stated evaluation method. In commenting on a draft of our report, DOD concurred with each of our recommendations. The department's comments are reprinted in enclosure I. Background: To meet its human capital needs, the Army must annually recruit and retain more than twice the number of uniformed personnel needed by any other military service. In fiscal year 2008, the Army's recruiting mission was over 167,000. Like the other services, the Army employs three primary types of tools: (1) military recruiters, (2) advertising, and (3) incentives such as enlistment bonuses, educational benefits, and military pay. The Army has about 14,050 recruiters, most located in the United States, who recruit from the local population. To address its recruiting needs, the Army may raise or lower the size of the recruiter force and may use other methods to enhance recruiter effectiveness and efficiency. In terms of advertising, the Army manages its own program and works closely with its own contracted advertising agency. In fiscal year 2007, the Army spent $476 million for advertising to support recruiting efforts. Finally, the Army offers enlistment bonuses to attract high-quality recruits; it also uses educational benefits such as the Montgomery GI Bill, the Army College Fund, tuition assistance, and college loan repayment. In fiscal year 2007, the Army spent approximately $473 million on enlistment bonuses for active Army recruits and for Army Reserve and Army National Guard recruits who were mobilized to support ongoing operations. Generally, before the Army can offer a new recruitment incentive, DOD policy requires that it develop and submit a legislative proposal through DOD's Unified Legislation and Budgeting Process. DOD's Office of the Undersecretary of Defense for Personnel and Readiness established this process in 1993 to ensure the quality of personnel- related proposals developed by the services and to align these proposals with DOD's budgeting process. According to DOD policy, [Footnote 5] this process provides a joint forum to develop, review, and coordinate legislative initiatives that affect personnel and readiness.[Footnote 6] As part of this process, the services are expected to develop a description of the personnel initiative, including a business case and an estimate of the approximate cost per year. Only after legislation is passed by Congress and signed into law by the president can the military services offer the incentive or other personnel initiative. According to DOD officials, because Section 681 directly authorized the Secretary of the Army to develop new recruitment incentives, the Army's new incentives were not reviewed through DOD's Unified Legislation and Budgeting process. On prior occasions, the Office of the Under Secretary of Defense for Personnel and Readiness has proposed to Congress that it be granted similar pilot program authority, but Congress has not, to date, included these proposals in legislation. According to Personnel and Readiness officials, this type of authority would enhance the Office of the Secretary of Defense's ability to assess the effectiveness of personnel programs and hence allow for better informed investment decisions throughout the services. The Army Developed Three New Recruitment Incentive Programs under Section 681: The Secretary of the Army has notified Congress of the three new recruitment incentives it has developed pursuant to Section 681. These are (1) the Recruiter Incentive Pay Pilot Program, (2) the Army Advantage Fund Pilot Program, and (3) the Officer Accession Pilot Program. As required by Section 681, the Army has developed a plan for each of these incentives. Each plan cites the authority for the incentive and describes the incentive offered, the purpose of the incentive, and the types of individuals eligible for the incentive. Table 1 highlights key elements of the three incentives. The Army has the authority to develop no more than four recruitment incentives under Section 681 and has already developed three.[Footnote 7] While Army officials with whom we spoke stated that the Army does not currently expect to develop any additional recruitment incentives under Section 681, the Army noted in technical comments on our draft report that it reserves the option to develop one more recruitment incentive pursuant to Section 681. Table 1: Description of Recruitment Incentives. Program: Recruiter Incentive; Pay Pilot Program; Description, availability, and applicability: * Provides bonus pay to active Army, Army Reserve, and Army National Guard recruiters nationwide; * Assumes additional pay will motivate recruiters to exceed recruitment goals; * Provides bonuses that range from $100 to $8,600 per year, depending on the extent to which recruiters exceed set benchmarks. Program: Army Advantage Fund Pilot Program; Description, availability, and applicability: * Provides recruits payment, following their completion of service, intended for home ownership or small business development; * Provides up to $40,000 for active Army recruits and up to $20,000 for Army Reserve recruits who enlist at selected recruiting battalions. Program: Officer Accession Pilot Program; Description, availability, and applicability: * Option A provides a $5,000 bonus to students who complete the ROTC Leader Training Course and sign a contract to serve in the Army as commissioned officers after they graduate; * Option B provides between $100 and $250 per month, up to a limit of $3,000 per academic year, to senior ROTC students who are enrolled in an approved language immersion program, program of study abroad, or academic course involving instruction in a foreign language of strategic importance to DOD and who sign a contract to serve in the Army as commissioned officers after they graduate; * Option C waives the required 8-year military service obligation for qualified medical and religious professionals between the ages of 43 and 60 and authorizes a 2-year military service obligation. Source: GAO analysis of Army data. [End of table] The Army began offering bonuses under the Recruiter Incentive Pay Pilot Program in June 2006. This incentive provides bonus pay to active Army, Army Reserve, and Army National Guard recruiters--of both officers and enlisted soldiers--nationwide who exceed their recruitment goals. The incentive's purpose is to increase the number and quality of new Army recruits. In general, DOD and the Army measure enlisted recruit quality based on two criteria: graduation from high school and Armed Forces Qualification Test score. Of the new recruits enlisting in the active Army, Navy, Air Force, and Marine Corps each year, DOD's goal is that at least 90 percent meet the graduation criteria and at least 60 percent score above average on the Armed Forces Qualification Test. To help ensure quality, the plan for the Recruiter Incentive Pay Pilot Program requires that at least 50 percent of the new recruits signed by a recruiter meet the Army's quality benchmarks. Originally, the Army designed the incentive to provide bonuses to recruiters based solely on individual performance. Subsequently, in May 2008, the Army modified the incentive in response to a Secretary of the Army decision related to the Army's assessment of the program's effect during its first year of implementation. The modified incentive continues to provide bonuses to most recruiters based solely on individual performance, although for recruiters in one of the Army's six recruiting brigades, the recruiting station as a whole must meet its goal in order for individual recruiters to receive bonus pay for exceeding their recruitment goals. The Army began offering enlistment bonuses under the Army Advantage Fund Pilot Program in February 2008. Like the Recruiter Incentive Pay Pilot Program, the Army Advantage Fund Pilot Program provides a monetary incentive; however, rather than providing a monetary incentive to recruiters, the Army Advantage Fund Pilot Program provides incentive pay to high-quality recruits who enlist for specified time frames. Specifically, it provides payment to enlisted active Army and Army Reserve recruits with no prior service, after they have completed their period of service, to be used for home ownership or small business development.[Footnote 8] The incentive also has an advertising component. The Army is currently offering the incentive through 5 of 45 Army recruiting battalions nationwide, so that it can assess the incentive's effect before considering further expansion. The Army is initially offering the Army Advantage Fund at the following locations: Albany, New York; Cleveland, Ohio; Montgomery, Alabama; San Antonio, Texas; and Seattle, Washington. The Army selected Albany, Cleveland, and Seattle because the demographics of these recruiting battalions reflect nationwide demographics, better allowing it to project the results of the incentive nationwide. According to Army officials, the Army selected Montgomery and San Antonio in order to obtain information about the incentive's effect in areas with larger populations of African Americans and Hispanics. The incentive's purpose is to give the Army a competitive advantage over the other services in attracting individuals to military service. The Army began offering incentives under the Officer Accession Pilot Program in August 2008, targeting individuals qualified to accept officer commissions. As we have previously reported, the Army faces unique officer accession challenges in the future.[Footnote 9] These challenges are the result of the Army's extensive role in ongoing military operations, as well as the expansion of the officer corps as part of the congressionally authorized 30,000-soldier increase to the Army's end strength and the Army's need for more officers as part of its ongoing transformation effort.[Footnote 10] The Officer Accession Pilot Program provides both monetary and nonmonetary incentives through three options. Option A pays a bonus to ROTC cadets who complete the Leader Training Course and sign a contract to serve in the Army as commissioned officers after they graduate. The Leader Training Course is a 4-week, field-oriented program that ROTC students take during the summer between their sophomore and junior years of college. Option B provides a stipend to senior ROTC students who participate satisfactorily in approved language immersion programs, programs of study abroad, or an academic course involving instruction in a foreign language of strategic importance to DOD and who sign a contract to serve in the Army as commissioned officers after they graduate. The list of approved languages will be reviewed annually. Currently, approved languages are Arabic, Persian-Farsi, Persian-Dari, Chinese- Mandarin, Pashto, Urdu, Indonesian, Swahili, Hausa, and Korean. Option C reduces the military service obligation for certain qualified medical and religious professionals between the ages of 43 and 60 from 8 years to 2 years. The 8-year military service obligation is regarded as a barrier to recruiting older, experienced professionals for initial appointment and service in a critical career field for which they are otherwise well qualified. An individual may qualify under only one of these options. Section 681 specifies that the Army may not continue to provide any of these incentives after December 31, 2009. According to officials from the Office of the Under Secretary of Defense for Personnel and Readiness, to make any of the incentives piloted under section 681 permanent, the Army would need to submit a legislative proposal to Congress, as it does for other recruitment incentives. The Army would need to use DOD's Unified Legislation and Budgeting Process to do so. It generally takes about 2 years for a proposal for a personnel initiative to go through this process and only after legislation is passed by Congress and signed into law by the president can the military services offer the incentive or other personnel initiative. As a result, the potential may exist for a gap between the December 31, 2009 expiration of the authority provided in Section 681 and the date on which the Army can again offer any incentives initially developed under Section 681 that it seeks to make permanent. Two Plans for Recruitment Incentives Lack Outcomes and None Includes Evaluation Methodologies: Two of the Army's three recruitment incentive plans do not include anticipated outcomes and none describes a methodology to be used to evaluate its effectiveness. Section 681 requires the Army to include in the plan it submits to Congress before implementing each incentive (1) a statement of the outcomes the Army anticipates will result from the incentive offered and (2) a description of the method to be used to evaluate the incentive's effectiveness. In addition, Section 681 requires the Army to develop an annual report to Congress on each incentive, to include both a description of the recruitment incentives provided under Section 681 during the previous fiscal year and an assessment of each incentive's impact on the recruitment of individuals as officers or enlisted members of the Army. Although the Army did not include this information in the plans it developed for the first two incentives, the Army did include information about anticipated outcomes in its notification letters to Congress. In addition, the Army assigned responsibility for developing evaluation methodologies and assessing the Recruiter Incentive Pay Pilot Program to the U.S. Army's Accessions Command's Center for Accession Research and for the Army Advantage Fund Pilot Program to RAND. For the Officer Accession Pilot Program, the Army did include anticipated outcomes in the plan, and it assigned the assessment responsibilities to the U.S. Army Accessions Command, the Army Surgeon General, and the Army Chief of Chaplains. All of these organizations have conducted, are currently conducting, or plan to conduct assessments of each of the recruitment incentives. The Secretary of the Army and Army officials responsible for the incentives intend to use these assessments, along with other data collected as part of the assessment process, to inform their decisions about each incentive's future--for example, whether to expand the incentive nationwide and whether to develop a legislative proposal to make the incentive permanent. Prior GAO work shows that one of the benefits of piloting programs is that they can enable organizations to make choices based on rigorous evaluation,[Footnote 11] and Army Regulation 5-1 states that data- driven management decision making is one of the core values of the Army's performance improvement criteria.[Footnote 12] But if officials do not clearly establish at the outset the metrics on which they intend to base their decisions about the pilot programs, they may not be in a position to ensure that the data collected and the analysis produced will enable them to make informed, objective decisions regarding the extent to which such programs are meeting their goals or to judge their relative effectiveness and cost, both compared to each other and to existing programs. Moreover, since the methodologies to evaluate the effectiveness of recruitment incentives developed pursuant to Section 681 are--like all evaluation designs--constrained by a number of factors, it is important to recognize these factors candidly and describe their effect. For example, these methodologies should disclose the extent to which factors other than the Section 681 incentives themselves are controlled for, and the extent to which these factors could independently influence any changes in recruitment numbers. Because the Army did not incorporate all of the necessary elements into its recruitment incentive plans, the plans do not present a clear and convincing business case for the incentives, making it difficult for those exercising oversight to independently assess the extent to which the incentives have been a success. Recruiter Incentive Pay Pilot Program: The March 2006 plan for the Recruiter Incentive Pay Pilot Program does not include a statement of anticipated outcomes or a methodology to be used to evaluate its effectiveness. However, in an April 2006 notification letter to Congress, the Secretary of the Army estimated that the incentive would result in an additional 6,500 active Army accessions, an additional 2,000 Army Reserve accessions, and an additional 5,100 Army National Guard accessions per year.[Footnote 13] But the letter does not include information on the process--or assumptions--that the Army used to develop these anticipated outcomes. Because the Army's plan does not include the outcomes, it is also unclear whether the Army intended to assess the incentive's effectiveness against the estimated accessions stated in the letter. Because it is the only one of the three incentives that has been offered for more than 1 year, the Recruiter Incentive Pay Pilot Program is the only one for which the Army has completed its required annual assessment, the results of which it has provided to Congress. Based on the results of this assessment, the Secretary of the Army concluded that further study of the incentive's effect was required. However, neither the plan for the incentive nor the April 2006 notification letter to Congress identified the assumptions that were used to develop the program's anticipated outcomes. Similarly, these documents did not identify key factors--such as the ways in which the Army National Guard's recruiting environment differs from that of the active Army and the Army Reserve. As a result, Army officials lack the information they need to make informed, objective decisions regarding the extent to which--and the reasons why--the program did or did not meet its intended goals. Specifically, U.S. Army Accessions Command's Center for Accessions Research assessed the incentive's effect on the active Army and Army Reserve and concluded that the Recruiter Incentive Pay Pilot Program (1) did not increase either the quality or the number of new recruits, (2) encouraged recruiters to focus on nonquality recruits, and (3) did not appear to be cost-effective, at a cost of approximately $3.7 million for the first year. Accordingly, the U.S. Army Accessions Command's Center for Accessions Research recommended that the Army stop offering the incentive as of December 2007. In contrast, based on the results of its assessment of the incentive's effect on the Army National Guard, the National Guard Bureau recommended that the Army continue to offer the incentive. A March 2008 letter, which comprises the Army's annual report to Congress on the incentive, combines the results of the two. In the letter, the Army compared the number of individuals signing contracts in the year after it began to offer bonuses under the Recruiter Incentive Pay Pilot Program to the number of individuals who had joined during the year before. This comparison revealed that during the first year of the incentive, the active Army had 2,899 fewer accessions than the previous year and the Army Reserve 1,362 fewer, while the Army National Guard had 6,711 more. To assist in further assessing the incentive, the Army modified the incentive--as discussed earlier--to include a team concept in selected areas. However, because this modification does not apply to the Army National Guard, it will not provide Army officials additional insight into what factors may have contributed to positive assessment of the program for the Army National Guard and the negative assessment for the active Army and the Army Reserve. Army Advantage Fund Pilot Program: The plan developed for the Army Advantage Fund Pilot Program does not include a statement of anticipated outcomes or a methodology to be used to evaluate its effectiveness. In its notification letter to Congress, the Army estimated the incentive would result in an additional 1,500 active Army accessions, an additional 900 Army Reserve accessions, and an additional 350 Army National Guard accessions during fiscal year 2008.[Footnote 14] Because the letter does not discuss how the Army developed these anticipated outcomes, the extent to which they are appropriate to measuring the incentive's effectiveness is unclear. For example, an official from the organization conducting the assessment for the Army Advantage Fund Pilot Program said that these numbers likely assume that the incentive will be implemented nationwide, but to date the incentive has only been implemented in five locations for purposes of the pilot, as discussed earlier. Moreover, because the assumptions underlying this estimate are not made clear in either of the two guiding documents for the incentive--the notification letter and the plan--the extent to which the assumptions that were used to develop these estimated outcomes are also being used in ongoing assessments of the incentive is unclear. In the plan for the Army Advantage Fund Pilot Program, the Army assigned RAND the responsibility for assessing the incentive's effectiveness, and the Army obtained information on RAND's 3-month assessment in June 2008. RAND is continuing to collect information about the program. The plan states that Army leaders will determine if the Army Advantage Fund Pilot Program should be modified, depending on the success of the incentive, and officials responsible for managing the incentive indicated that they intend to use assessment data to make decisions about the incentive's future--including whether and when they will offer the incentive nationwide. Officer Accession Pilot Program: The plan developed for the Officer Accession Pilot Program does not include a methodology to be used to evaluate its effectiveness. It does state anticipated outcomes for each of the incentive's three options in terms of numbers of officer accessions and assigns responsibility to other organizations for assessing each option. The Army anticipates that Option A, which provides a bonus to ROTC cadets who attend the Leader Training Course--a requirement for the ROTC Advanced Course-- will result in the contracting of between 250 and 290 additional cadets and other ROTC Advanced Course participants per year. The Army expects Option B, which provides a monthly stipend to students enrolled in the study of languages of strategic importance to DOD--and determined by the Secretary of the Army to also be of importance to the Army--to result in the contracting and commitment to military service of between 200 and 300 additional students per year. Option B is expected to motivate students with an interest in or pre-existing knowledge of strategic foreign languages to participate in Senior ROTC and commit to a period of military service. Finally, the Army anticipates that Option C, which reduces the military service obligation for qualified midlife medical and religious professionals, will increase the number of appointments by an additional 20 to 30 officers per year. For none of the three options does the plan include an explanation about how the anticipated outcomes were developed, although conversations with Army officials indicate that the estimates are based on relevant data, such as the number of additional students that the Army's Leader Training Course, as currently structured, could accommodate. Nonetheless, based on the plan alone, it is unclear what assumptions the Army used to develop the outcomes, and what data the Army intends to collect and analyze in order to assess each option's effectiveness. The incentive's plan assigns responsibility to U.S. Army Accessions Command for assessing Options A and B, and the Army Chief of Chaplains and the Office of the Army Surgeon General responsibility for assessing the effectiveness of Option C. The incentive's plan states that the Army's Deputy Chief of Staff for Personnel, in determining the overall effect of the Officer Accession Pilot Program, will analyze reports on the effect of each option of the incentive and that the Secretary of the Army or his designee will consider these data and other information in deciding whether or not to continue the incentive. The Army began offering the program in August 2008 and so has not yet conducted any assessments. Conclusions: The Army's role in--among other things--simultaneously supporting multiple military operations means that it must recruit substantially greater numbers of individuals than the other military services. To address this challenge, Section 681 of the National Defense Authorization Act for Fiscal Year 2006 authorizes the Army to develop not more than four new recruitment incentives. The Army has established three such incentives. However, because the Army began offering these incentives before it presented a sound business case that clearly defined desired outcomes with transparent assumptions and established a methodology for evaluating performance linked to those desired outcomes, its basis for proposing the three incentives remains somewhat unclear. Additionally, because the plans do not provide the necessary information, by relying on the plans alone it is not possible to assess either each recruitment incentive's cost effectiveness or the extent to which it has achieved its intended mission. Unless changes are made, such challenges will continue to hamper the Army's ability to assess the effectiveness of any recruitment incentive developed under Section 681 that it decides to seek to make permanent. Furthermore, because Army officials have yet to develop an approach to systematically evaluate the data that the Army collects on the incentives, there is currently no assurance that the Army's decisions about any incentives they decide to seek to make permanent will be informed by a sound business case. Recommendations for Executive Action: To improve management of the incentives and ensure that the required annual assessments will be of use to Army decision makers, we recommend that--for each recruitment incentive developed pursuant to Section 681- -the Secretary of the Army take the following two actions: Issue guidance to clearly specify anticipated outcomes for each incentive, describe the assumptions behind these anticipated outcomes, identify the evaluation method to be used to assess progress toward these outcomes, and link anticipated outcomes to the stated evaluation method. Ensure that this guidance is specifically addressed in the required annual assessments for each recruitment incentive. Additionally, should the Army decide--based on the results of the annual assessments of each incentive--to seek to make any of these incentives permanent, we further recommend that the Secretary of the Army, for any proposals the Army may wish to develop, include in its business case clearly specified anticipated outcomes for each incentive, a description of the assumptions behind these anticipated outcomes, an identification of the evaluation method to be used to assess progress toward these outcomes, and an explanation that links anticipated outcomes to the stated evaluation method. Agency Comments and Our Evaluation: In commenting on a draft of our report, DOD concurred with our three recommendations and specifically stated that it would take actions intended to improve management of the incentives and help ensure that the required annual assessments will be of use to Army decision makers. We have reprinted DOD's official comments as enclosure I. DOD also provided technical comments, which we have incorporated in the report as appropriate. We are sending copies of this report to the Secretary of Defense and the Secretaries of the Army, Navy, and Air Force. We will also make copies available to others on request. In addition, the report will be available at no charge on the GAO Web site at [hyperlink, http://www.gao.gov]. Should you or your staff have any questions on the matters discussed in this report, please contact me at (202) 512-3604 or farrellb@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this letter. GAO staff who made key contributions to this report are listed in enclosure II. Signed by: Brenda S. Farrell: Director, Defense Capabilities and Management: Enclosures - 2: List of Congressional Committees: The Honorable Carl Levin: Chairman: The Honorable John McCain: Ranking Member: Committee on Armed Services: United States Senate: The Honorable Ike Skelton: Chairman: The Honorable Duncan L. Hunter: Ranking Member: Committee on Armed Services: House of Representatives: Enclosure I: Comments from the Department of Defense: Department Of The Army: Office Of The Deputy Chief Of Staff, G-1: Directorate Military Personnel Management: 300 Army Pentagon: Washington, DC 20310-0300: September 9, 2008: Brenda S. Farrell: Director, Defense Capabilities and Management: U.S. Government Accountability Office: 441 G Street, N.W., Washington, DC 20548: Dear Ms. Farrell: This is the Department of Defense (DoD) response to the GAO draft report, GAO-08-1037, "Military Personnel: Evaluation Methods Linked to Anticipated Outcomes Needed to Inform Decisions on Army Recruitment Incentives," dated August 25, 2008 (GAO Code 351200). DoD appreciates the opportunity to review and comment on the draft report. Detailed comments on the GAO recommendations and technical comments are enclosed. For further questions concerning this report please contact my action officer, Lieutenant Colonel Denise Corley, (703) 695-6673. Signed by: Gina S. Farrisee: Brigadier General, U.S. Army: Director of Military Personnel Management: Enclosures: GAO Draft Report - Dated August 25, 2008: GAO Code 351200/GAO-08-1037: "Military Personnel: Evaluation Methods Linked to Anticipated Outcomes Needed to Inform Decisions on Army Recruitment Incentives" Department Of Defense Comments To The Recommendations: Recommendation 1: The GAO recommends that for each recruitment incentive developed pursuant to Section 681 of the National Defense Authorization Act for FY 2006, the Secretary of the Army issue guidance to: * clearly specify anticipated outcomes for each incentive; * describe the assumptions behind these anticipated outcomes; * identify the evaluation method to be used to assess progress toward these outcomes; and; * link anticipated outcomes to the stated evaluation method. DOD Response: Concur. The Secretary of the Army will ensure that for each of the recruitment incentives developed pursuant to Section 681 of the National Defense Authorization Act for FY 2006, that the anticipated outcomes, underlying assumptions, evaluation methods and linkage between the anticipated outcome and evaluation method are clearly articulated in the annual report to Congress on each recruiting incentive. Recommendation 2: The GAO recommends that the Secretary of the Army ensure that the guidance referred to in Recommendation 1 is specifically addressed in the required annual assessments for each recruitment incentive. DOD Response: Concur. The Secretary of the Army will ensure that the anticipated outcomes, underlying assumptions, evaluation methods and linkage between the anticipated outcome and evaluation method are specifically addressed in the annual report to Congress on each recruiting incentive and serve as the basis for continuing, modifying, or discontinuing any or all of the incentive programs. Recommendation 3: Should the Army decide - based on the results of the annual assessments of each incentive - to seek to make any of these incentives permanent, the GAO recommends that the Secretary of the Army, for any proposals the Army may wish to develop, include in its business case: * clearly specified anticipated outcomes for each incentive; * a description of the assumptions behind the anticipated outcomes; * an identification of the evaluation method to be used to assess progress toward these outcomes; and; * an explanation that links anticipated outcomes to the stated evaluation method. DOD Response: Concur. Should the Secretary of the Army decide to seek to make any of the recruitment incentives permanent, the Army will develop a business case that includes clearly specified anticipated outcomes, a description of the assumptions behind the anticipated outcomes, an identification of the evaluation method to be used to assess progress toward these outcomes, an explanation that links anticipated outcomes to the stated evaluation method. [End of enclosure] Enclosure II: GAO Contacts and Staff Acknowledgements: GAO Contact: Brenda S. Farrell, (202) 512-3604 or farrellb@gao.gov: Acknowledgements: In addition to the individual named above, David Moser (Assistant Director), Catherine H. Brown, Charles Perdue, Joanne Landesman, Julia Matta, Terry Richardson, and Barbara J. Williams made key contributions to this report. [End of enclosure] Footnotes: [1] Congressional Research Service, The Library of Congress, Recruiting and Retention: An Overview of FY2005 and FY2006 Results for Active and Reserve Component Enlisted Personnel (Washington, D.C.: Updated Jan. 26, 2007). [2] Pub. L. No. 109-163 (2006). [3] See titles 10 and 37, U.S. Code. [4] Section 681 requires the Army to notify Congress by submitting a plan for each recruitment incentive. The Army must then wait 45 days before providing the new recruitment incentive. [5] DOD Directive 5500.20, Unified Legislation and Budgeting Process (Mar. 28, 2005). [6] The Army and other services may also submit proposals directly to DOD's Office of General Counsel. [7] The Officer Accession Pilot Program provides both monetary and nonmonetary incentives through three options. According to the General Counsel of the Department of the Army, although this incentive is comprised of three options, it counts as a single incentive for several reasons; among the five reasons cited by the General Counsel of the Department of the Army is that the options within the Officer Accession Pilot Program are mutually exclusive, meaning a recruit cannot select more than one. [8] The incentive amount varies by length of service and by component. An individual enlisting in the active Army for 3 years receives $25,000; an individual enlisting for 4 years receives $35,000; and an individual enlisting for 5 years receives $40,000. An individual enlisting in the Army Reserve for 3, 4, or 5 years receives $10,000, $15,000, or $20,000, respectively. Currently, recruits who accept the Army Advantage Fund are not required to provide evidence that they intend to use the bonus for home ownership or small business development. If the incentive is made permanent, the Army expects to both make such evidence a requirement and develop an associated funding mechanism. [9] GAO, Military Personnel: Strategic Plan Needed to Address Army's Emerging Officer Accession and Retention Challenges, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-224] (Washington, D.C.: Jan. 19, 2007). [10] As part of its modular force transformation effort, the Army redesigned its 10 active duty division force into a force of 42 or 43 brigade combat teams. The redesign has resulted in entirely new organizational structures and increased officer requirements by 4,131 spaces from 2001 to July 2006. [11] GAO, Comptroller General's Forum, High-Performing Organizations: Metrics, Means, and Mechanisms for Achieving High Performance in the 21st Century Management Environment, [hyperlink, http://www.gao.gov/cgi- bin/getrpt?GAO-04-343SP] (Washington, D.C.: Feb. 13, 2004). [12] Army Regulation 5-1, Total Army Quality Management (Mar. 15, 2002). [13] For the active Army, "accessions" are individuals who have begun their military service, as distinguished from those who have signed a contract to serve but who have not yet begun their service. For the Army Reserve and Army National Guard, the term has a broader meaning; accession can occur shortly after an individual signs a contract, when he or she is "shipped" to basic training, or when a service member transfers from the active Army to the Army Reserves or Army National Guard. [14] Recruits who enlist in the Army National Guard are not currently eligible for the Army Advantage Fund Pilot Program. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office: 441 G Street NW, Room LM: Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: