This is the accessible text file for GAO report number GAO-13-714R entitled 'Department of Veterans Affairs: Available Data Not Sufficiently Reliable to Describe Use of Consulting Services' which was released on July 30, 2013. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. GAO-13-714R: GAO: United States Government Accountability Office: 441 G St. N.W. Washington, DC 20548: July 30, 2013: The Honorable Mike Coffman: Chairman: Subcommittee on Oversight and Investigations: Committee on Veterans' Affairs: House of Representatives: The Honorable Bill Johnson: House of Representatives: Department of Veterans Affairs: Available Data Not Sufficiently Reliable to Describe Use of Consulting Services: You requested that GAO review the Department of Veterans Affairs' (VA) use of consultants. This review focuses on how VA has used consulting services and what is known about the costs of these services. However, available data were not sufficiently reliable to identify the costs of consulting services used by VA. Therefore, this report outlines the data limitations related to consulting services for VA as well as some of the uses of consulting contracts. To identify how VA has used consulting services and their costs, we identified contracts in the federal government's procurement information system--referred to as the Federal Procurement Data System- Next Generation (FPDS-NG)--that VA awarded in fiscal years 2011 and 2012 and had been assigned North American Industry Classification System (NAICS) codes that contained the word consulting. According to the United States Census Bureau, NAICS is the standard used by federal statistical agencies in classifying business establishments for the purpose of collecting, analyzing, and publishing statistical data related to the U.S. business economy. From these contracts, we identified seven contracting offices within VA that were among the largest users of consulting services as measured by the dollar value of contracts awarded. These seven contracting offices included three within VA's Office of Acquisition, Logistics, and Construction: (1) the Acquisition Business Service, Washington, D.C.; (2) the Strategic Acquisition Service, Frederick, Maryland; and (3) the Technology Acquisition Center, Eatontown, New Jersey. The remaining four contracting offices were within the Veterans Health Administration (VHA)--two within the Program Contracting Activity Central and two Veteran Integrated Service Network contracting offices.[Footnote 1] We interviewed contracting officials at each of the seven offices to obtain an overview of how the offices contracted for consulting services and for what purposes. We also discussed the officials' perspectives on what they considered to be a consulting service. We also selected 24 contracts--4 contracts each for six of the seven contracting offices--to obtain a better understanding of how VA uses consulting services and to determine if officials considered these contracts to be for consulting. We selected the contracts from fiscal year 2011 and 2012 data using criteria that allowed us to select contracts with a variety of characteristics.[Footnote 2] First, we chose the contract with the highest value, as measured by the dollar value of the contracts, and then we randomly selected three other contracts with the following characteristics: (1) competitively awarded, (2) awarded noncompetitively, and (3) coded as being closely associated with an inherently governmental function.[Footnote 3] We later excluded 2 of the 24 contracts in our sample because no work was performed, leaving us with 22 contracts in our sample. For the seventh office, the Strategic Acquisition Service, we used different selection criteria for a pretest and so we excluded those 4 contracts from our contract analyses. To determine if our use of NAICS codes would enable us to identify a group of consulting contracts that was reliable for our intended purposes, we asked agency officials to identify if the NAICS code associated with each of our 22 sample contracts correctly classified the service provided and if the officials considered the contract to be for consulting services. We conducted this performance audit from November 2012 through July 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Results in Brief: Two primary factors limited our ability to identify a list of consulting contracts that was sufficiently accurate and complete for identifying the types and costs of consulting services used by VA. First, there is no consistently used definition for consultant or consulting services across VA that we could apply for our intended purposes.[Footnote 4] Second, there are data limitations associated with identifying consulting services at VA. NAICS codes can be used to classify both consulting and nonconsulting services and according to VA officials, in some cases, contracting officers had applied the incorrect NAICS code to contracts. Nevertheless, VA offices were able to provide information on how they used consulting services and identified contracts among those in our sample that they considered to be consulting services. Background: VA is one of the U.S. government's largest agencies, and administers directly, or with the assistance of other agencies, programs that provide health care, benefits, and other services to service members, veterans, their dependents, and survivors. VA consists of three primary organizations: the Veterans Benefits Administration, VHA, and the National Cemetery Administration. In fiscal year 2012, VA contracted for about $8.95 billion in services. VA's Office of Acquisition, Logistics, and Construction provides acquisition, logistics, construction, and leasing support to VA administrations and VA staff offices, and awards about 80 percent of all contracts assigned a NAICS code that includes the word consulting. Key Factors Limit the Ability to Identify Consulting Services and Costs: Officials across major VA offices did not use a consistent definition of consulting services and had different perspectives on what types of services they considered consulting. For example, officials from one office included in our study said they considered all advisory and assistance service contracts--contracts used by federal agencies to acquire three broad areas of services: management and professional support; studies, analyses, and evaluations; and engineering and technical services--to be consulting contracts, but did not believe that all consulting contracts could be classified as advisory and assistance service contracts.[Footnote 5] In contrast, officials from several other contracting offices indicated that while consulting services could be classified as advisory and assistance service contracts, not all advisory and assistance service contracts were for consulting services. In addition, a few contracting officials noted that they considered "advice" to be consulting, but not the work conducted to inform the advice or any resulting written product. Based on our interviews and analyses, we determined that using NAICS code descriptors would not produce a set of contracts that we could reliably use to describe or analyze the costs of VA consulting services. Further, there is no means for contracting officers to track obligations for such contracts in FPDS-NG, as there is no specific definition or field for coding consulting contracts in FPDS-NG. Of the 22 contracts we selected based on our criteria--including NAICS code descriptors containing the word "consulting"--agency officials considered 14 contracts not to be for consulting services. We asked agency officials responsible for each contract to identify if the NAICS code was assigned correctly and if they considered the contract to be for consulting. We asked both of these questions because of the possibility that a contract's services could be correctly classified with a NAICS code for consulting but the agency officials might not consider the contract to be for consulting. See table 1 for a summary of their responses. Table 1: Number of Contracts VA Officials Considered to be for Consulting: NAICS code assigned correctly: Considered to be for consulting: 6; Not considered to be for consulting: 10. NAICS code assigned incorrectly: Considered to be for consulting: 2; Not considered to be for consulting: 4. Total: Considered to be for consulting: 8; Not considered to be for consulting: 14. Source: GAO summary of VA officials' responses. [End of table] As indicated in table 1 above, in 10 cases, we found that although contracting officers reported that they had assigned a correct NAICS code to a contract, one that could be used to classify consulting services, they did not necessarily consider the contract to be for consulting. For several of these contracts, we found that a single NAICS code was used to classify both consulting and nonconsulting services. For example, one computer-related NAICS code can be used to classify both consulting and nonconsulting services for computer system planning and design and was applied to a contract to acquire computer help desk support staffed by technical professionals. Contracting officials explained that they did not consider such support as a consulting service but that this NAICS code best described the service acquired. Further, contracting officers reported that office staff had assigned an incorrect NAICS code to six other contracts we selected, and of these, four were considered not to be for consulting services. In two of these cases, the contracting officers reported that the NAICS codes were incorrect because their task order contracts were issued under a governmentwide contract awarded by the General Services Administration. For example, one task order was for the installation of solar panels at a VA facility and the officials did not consider the work performed to be a consulting service. They explained that NAICS codes assigned by the General Services Administration to its contracts are pre-populated to the task orders subsequently issued by individual government agencies, and when not suitable, can be difficult to change. VA Uses Consulting Services Contracts for a Variety of Purposes: For those contracts that VA officials considered to be for consulting, VA procured a variety of services, including program evaluations, engineering support services, and gathering data to support decision making. For example, an Office of Acquisition, Logistics, and Construction official identified a contract used to acquire expertise to facilitate a major human resource initiative as a consulting service. The contracting officer considered this to be consulting because the contractor produced recommendations. In another example of a consulting service, a VHA office arranged for a contractor to test the operational readiness of an emergency department information system and provide training to VA staff. In this instance, the contractor was responsible for developing instructional guides for staff and conducting follow-up training sessions. Agency Comments and Our Evaluation: In written comments to a draft of this product (reproduced in the enclosure), the Department of Veterans Affairs generally concurred with our finding that the data reviewed could not be used to describe the use of consulting services--the desired outcome of the analysis. The agency also noted that it disagreed with our characterization that the data are unreliable. However, based on generally accepted government auditing standards, data that would not achieve the desired outcome of the analysis are unreliable. VA officials reported that they searched the FPDS-NG for contracts with NAICS codes that refer to consulting and that such a search does identify applicable contracts and obligations. As we reported, using essentially this methodology, VA contracting officials considered 14 of the 22 contracts we selected with NAICS codes that contained the word "consulting" not to be for consulting even though they were coded as such. This data limitation coupled with the fact that there was no consistently used definition for consultant or consulting services across VA limited our ability to identify a set of consulting contracts that was sufficiently accurate and complete to describe the types and costs of consulting services used by VA. Therefore, we maintain that the available data were not sufficiently reliable for our purposes of identifying consulting contracts for VA. VA also provided technical comments, which we have incorporated as appropriate. We are sending copies of this report to interested congressional committees and the Secretary of Veterans Affairs. In addition, the report will be available at no charge on the GAO website at [hyperlink, http://www.gao.gov]. If you or your staff have questions about this report, please contact me at (202) 512-4841 or mackinm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report were LaTonya Miller (Assistant Director), Marie P. Ahearn, Sonja Bensen, Andrea Bivens, Virginia A. Chanley, Danielle Greene, and Julia M. Kennon. Sincerely yours, Signed by: Michele Mackin: Director, Acquisition and Sourcing Management: [End of section] Enclosure: Enclosure I: Comments from the Department of Veterans Affairs: Department Of Veterans Affairs: Washington, DC 20420: July 19, 2013: Ms. Michele Mackin: Director, Acquisition and Sourcing Management: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Ms. Mackin: The Department of Veterans Affairs (VA) has reviewed the Government Accountability Office's (GAO) draft report, "Department of Veterans Affairs: Available Data Not Sufficiently Reliable to Describe Use of Consulting Services" (GAO-13-714R). VA generally concurs with GAO that the data reviewed would not achieve the desired outcome of the analysis. Thus, while VA disagrees with GAO's characterization that the data is "unreliable," we believe meaningful insights could be achieved with a refined scope and methodology. To demonstrate the broad nature of the data, VA's Office of Acquisition, Logistics, and Construction conducted a search of the North American Industry Classification System (NAICS) codes containing the words "consulting," "consulting services," or "consultant," which matched 188 NAICS codes. Federal contracting officers are required to select the most appropriate of these 188 NAICS codes based on the type of industry doing the work to be performed. These codes would include consulting efforts ranging from routine technical advice on specific issues to broad-ranging advice on policy or operational issues. The enclosure contains technical comments related to the draft report. VA appreciates the opportunity to comment on your draft report. Sincerely, Signed by: Jose D. Riojas: Interim Chief of Staff: Enclosure: Department of Veterans Affairs (VA) Response to Government Accountability Office (GAO) Draft Report: "Department of Veterans Affairs: Available Data Not Sufficiently Reliable to Describe Use of Consulting Services" (GAO-13-714R): Technical Comments: Page 3, "Background" section, last sentence: VA recommends replacing "is responsible for" with "awards" to differentiate that the VA office (Office of Acquisition, Logistics, and Construction) that awards the majority of VA consulting contracts frequently does so on behalf of other VA Staff Offices and Administrations, which are the responsible parties. Page 4, second paragraph, second sentence: VA disagrees with the statement regarding the Federal Procurement Data System-Next Generation (FPDS-NG) providing "no means to track obligations" for consulting contracts. Using relevant North American Industry Classification System codes, an FPDS-NG search does identify applicable contracts and obligations. Further, additional contracting and financial details are tracked in other internal VA systems. Based on the broad scope of the request, the resulting data captured from the procurement system provides a wide range of results. Further refinement of the request could allow VA to more effectively provide the appropriate and targeted data. [End of section] Footnotes: [1] For the purpose of selecting contracting offices to include in our review, we counted VHA's Program Contracting Activity Central as two contracting offices because in the FPDS-NG separate office contracting codes were assigned to offices responsible for energy and health technology contracting. The network contracting offices included in our review performed contracting for Veteran Integrated Service Network 5 (VA Capitol Health Care Network) and Veteran Integrated Service Network 22 (Desert Pacific Healthcare Network). [2] In the case of two offices, where we confirmed that the contracts were not for consulting prior to the interview, we selected other contracts relevant to our purposes as substitutes. [3] For new civilian service contracts awarded on or after March 1, 2012, the Office of Federal Policy (OFPP) instructed agencies to indicate in FPDS-NG "if the services provided under the contract are predominantly for functions closely associated to inherently governmental ones." OFPP Memorandum, "Service Contract Inventories," (Dec. 19, 2011). Also, see GAO, Managing Service Contracts: Recent Efforts to Address Associated Risks Can Be Further Enhanced, [hyperlink, http://www.gao.gov/products/GAO-12-87] (Washington, D.C.: Dec. 7, 2011) for additional information on the risks of using such contracts. [4] Federal Acquisition Regulation (FAR) § 31.205-33 defines "professional and consultant services" but for the purpose of identifying the allowability of certain contractor costs. We did not find this definition useful for the purpose of identifying consulting contracts at VA because of its narrow focus. [5] FAR Subpart 37.2--Advisory and Assistance Services. The FAR definition of "advisory and assistance services" states that all advisory and assistance services are classified in one of the three defined subdivisions, which are the three areas of services listed above. Also, see GAO, Federal Contracting: Congressional Action Needed to Address Long-standing Problems with Reporting of Advisory and Assistance Services, [hyperlink, http://www.gao.gov/products/GAO-08-319] (Washington, D.C.: Mar. 31, 2008) where we reported that agencies reporting of advisory and assistance service contract obligations were inaccurate to the point of being meaningless and are not used for management purposes. [End of section] GAO’s Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select “E-mail Updates.” Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, DC 20548. [End of document]