This is the accessible text file for GAO report number GAO-13-130 entitled 'VA Health Care: Reliability of Reported Outpatient Medical Appointment Wait Times and Scheduling Oversight Need Improvement' which was released on January 18, 2012. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to Congressional Requesters: December 2012: VA Health Care: Reliability of Reported Outpatient Medical Appointment Wait Times and Scheduling Oversight Need Improvement: GAO-13-130: GAO Highlights: Highlights of GAO-13-130, a report to congressional requesters. Why GAO Did This Study: VHA provided nearly 80 million outpatient medical appointments to veterans in fiscal year 2011. While VHA has reported continued improvements in achieving access to timely medical appointments, patient complaints and media reports about long wait times persist. GAO was asked to evaluate VHA’s scheduling of timely medical appointments. GAO examined (1) the extent to which VHA’s approach for measuring and monitoring medical appointment wait times reflects how long veterans are waiting for appointments; (2) the extent to which VAMCs are implementing VHA’s policies and processes for appointment scheduling, and any problems encountered in ensuring veterans’ access to timely medical appointments; and (3) VHA’s initiatives to improve veterans’ access to medical appointments. To conduct this work, GAO made site visits to 23 clinics at four VAMCs, the latter selected for variation in size, complexity, and location. GAO also reviewed VHA’s policies and data, and interviewed VHA officials. What GAO Found: Outpatient medical appointment wait times reported by the Veterans Health Administration (VHA), within the Department of Veterans Affairs (VA), are unreliable. Wait times for outpatient medical appointments—- referred to as medical appointments-—are calculated as the number of days elapsed from the desired date, which is defined as the date on which the patient or health care provider wants the patient to be seen. The reliability of reported wait time performance measures is dependent on the consistency with which schedulers record the desired date in the scheduling system in accordance with VHA’s scheduling policy. However, VHA’s scheduling policy and training documents for recording desired date are unclear and do not ensure consistent use of the desired date. Some schedulers at Veterans Affairs medical centers (VAMC) that GAO visited did not record the desired date correctly. For example, three schedulers changed the desired date based on appointment availability; this would have resulted in a reported wait time that was shorter than the patient actually experienced. VHA officials acknowledged limitations of measuring wait times based on desired date, and described additional information used to monitor veterans’ access to medical appointments, including patient satisfaction survey results. Without reliable measurement of how long patients are waiting for medical appointments, however, VHA is less equipped to identify areas that need improvement and mitigate problems that contribute to wait times. While visiting VAMCs, GAO also found inconsistent implementation of VHA’s scheduling policy that impedes VAMCs from scheduling timely medical appointments. For example, four clinics across three VAMCs did not use the electronic wait list to track new patients that needed medical appointments as required by VHA scheduling policy, putting these clinics at risk for losing track of these patients. Furthermore, VAMCs’ oversight of compliance with VHA’s scheduling policy, such as ensuring the completion of required scheduler training, was inconsistent across facilities. VAMCs also described other problems with scheduling timely medical appointments, including VHA’s outdated and inefficient scheduling system, gaps in scheduler and provider staffing, and issues with telephone access. For example, officials at all VAMCs GAO visited reported that high call volumes and a lack of staff dedicated to answering the telephones impede scheduling of timely medical appointments. In January 2012, VHA distributed telephone access best practices that, if implemented, could help improve telephone access to clinical care. VHA is implementing a number of initiatives to improve veterans’ access to medical appointments such as expanded use of technology to interact with patients and provide care, which includes the use of secure messaging between patients and their health care providers. VHA also is piloting a new initiative to provide health care services through contracts with community providers that aims to reduce travel and wait times for veterans who are unable to receive certain types of care within VHA in a timely way. What GAO Recommends: GAO recommends that VHA take actions to (1) improve the reliability of its medical appointment wait time measures, (2) ensure VAMCs consistently implement VHA’s scheduling policy, (3) require VAMCs to allocate staffing resources based on scheduling needs, and (4) ensure that VAMCs provide oversight of telephone access and implement best practices to improve telephone access for clinical care. VA concurred with GAO’s recommendations. View [hyperlink, http://www.gao.gov/products/GAO-13-130]. For more information, contact Debra Draper at (202) 512-7114 or draperd@gao.gov. [End of section] Contents: Letter: Background: Reported Medical Appointment Wait Times Are Unreliable; VHA Uses Additional Methods to Monitor Patients' Access to Medical Appointments: Inconsistent Implementation of VHA's Scheduling Policy and Other Problems Impede VAMCs' Ability to Schedule Timely Medical Appointments: VHA Is Implementing a Number of Initiatives to Improve Access to Medical Appointments: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Comments from the Department of Veterans Affairs: Appendix II: GAO Contact and Staff Acknowledgments: Related GAO Products: Tables: Table 1: Selected Fiscal Year 2012 VHA Medical Appointment Wait Time Performance Measures: Table 2: Number of Schedulers at Each VAMC Visited Who Incorrectly Recorded the Medical Appointment Desired Date, by Error Type: Table 3: Number of Clinics That Did Not Implement Selected Elements of the VHA's Scheduling Policy, by VAMC Visited: Abbreviations: CBOC: community-based outpatient clinic: FDPP: Facility Director Performance Plan: NDPP: Network Director Performance Plan: OIG: Office of Inspector General: PACT: Patient-Aligned Care Team: PAR: Performance and Accountability Report: Project ARCH: Access Reached Closer to Home: VA: Department of Veterans Affairs: VAMC: VA Medical Center: VHA: Veterans Health Administration: VISN: Veterans Integrated Service Network: VistA: Veterans Health Information Systems and Technology Architecture: [End of section] United States Government Accountability Office: Washington, DC 20548: December 21, 2012: Congressional Requesters: The Veterans Health Administration (VHA), within the Department of Veterans Affairs (VA), operates one of the nation's largest health care systems. Its medical facilities include 152 VA medical centers (VAMC), as well as more than 800 community-based outpatient clinics (CBOC) and ambulatory care centers that operate under supervision of the VAMCs.[Footnote 1] At these facilities, VHA provides outpatient care through primary and specialty care clinics.[Footnote 2] In fiscal year 2011, there were more than 8 million veterans enrolled in VHA's health system, and VHA provided nearly 80 million outpatient medical appointments to veterans for primary and specialty care.[Footnote 3] Although access to timely medical appointments is critical to ensuring that veterans obtain needed medical care, long wait times and inadequate scheduling processes at VHA medical facilities have been long-standing problems. For example, in 2001, GAO reviewed the timeliness of medical appointments and found that two-thirds of the specialty care clinics visited had wait times longer than 30 days, although some clinics had made progress in reducing wait times, primarily by improving their scheduling processes and making better use of their staff.[Footnote 4] Later, in 2007, the VA Office of Inspector General (OIG) reported that VHA facilities did not always follow VHA's scheduling policies and processes and that the accuracy of VHA's reported wait times for medical appointments was unreliable. [Footnote 5] Most recently, in 2012, the VA OIG reported that VHA was not providing all new veterans with timely access to full mental health evaluations, and had overstated its success in providing veterans with timely new and follow-up appointments for mental health treatment.[Footnote 6] VHA has reported continued improvements in measuring and achieving timely access to medical appointments. For example, in fiscal year 2011, VHA had a goal of scheduling medical appointments within 14 days of the patient's or provider's desired medical appointment date, [Footnote 7] and in that year, VA reported that it completed 95 percent of specialty care medical appointments and 94 percent of primary care medical appointments within this time frame. However, patient complaints and media reports about long wait times have persisted, prompting renewed concerns about excessive medical appointment wait times. You asked us to evaluate VHA's scheduling of timely medical appointments. We examined (1) the extent to which VHA's approach for measuring and monitoring medical appointment wait times reflects how long veterans are waiting; (2) the extent to which VAMCs are implementing VHA's policies and processes for medical appointment scheduling, and any problems encountered in ensuring veterans' access to timely medical appointments as identified by VAMCs; and (3) VHA's initiatives to improve veterans' access to medical appointments. To address all three objectives, we interviewed VHA central office officials responsible for medical appointment scheduling policy, medical appointment wait time measurement, and initiatives to improve access to timely medical appointments. We also conducted site visits to four VAMCs selected for variation in size and complexity, geographic location, or role as a pilot site for a VHA initiative to improve access to timely medical appointments.[Footnote 8] These four VAMCs were located in Dayton, Ohio; Fort Harrison, Montana; Los Angeles, California; and Washington, D.C. At each site, we visited the VAMC as well as the affiliated CBOCs that had among the highest volume of medical appointments.[Footnote 9] We also visited the highest volume ambulatory care center at one VAMC that had such facilities. [Footnote 10] At each VAMC, CBOC, and ambulatory care center, we visited an outpatient primary care clinic, and where available, one or more outpatient specialty care clinics for a total of 23 clinics--9 primary care and 14 specialty care clinics--the clinics were among those with the highest medical appointment volume.[Footnote 11] Results from our site visits cannot be generalized to other VAMCs. We also interviewed the directors and relevant staff of the four Veterans Integrated Service Networks (VISN), or regional networks of care, for the sites we visited.[Footnote 12] To examine the extent to which VHA's approach for measuring and monitoring medical appointment wait times reflects how long veterans are waiting, we reviewed VHA's outpatient medical appointment scheduling policy and processes and training documents based on the policy, as well as documents related to performance accountability that include wait time measures.[Footnote 13] At each of the four VAMCs we visited, we interviewed the leadership team, scheduling managers, and managers from all of our selected clinics about oversight activities to ensure the accuracy of scheduling data, and about what data and measures they use to manage clinics and improve medical appointment timeliness. In addition, we interviewed schedulers from the 9 primary care clinics and 10 of the 14 specialty care clinics we visited using a structured protocol to determine the accuracy with which schedulers determined and recorded medical appointment data for hypothetical patient medical appointments into the scheduling system in accordance with VHA's scheduling policy. [Footnote 14] We also compared reported medical appointment wait times for the clinics we visited to information gathered during our site visits.[Footnote 15] To determine the extent to which VAMCs are implementing VHA's scheduling policy and processes, and to gather information on problems encountered in scheduling timely medical appointments, we reviewed VHA's scheduling policy, interviewed VHA central office officials responsible for the scheduling policy, and obtained information about scheduling practices from officials at each of the four VAMCs we visited. Specifically, for each of the VAMCs, we interviewed leadership, scheduling managers, clinic managers, patient advocates, and case managers. We obtained information about the VAMCs' oversight to ensure compliance with policy, and about problems staff at these facilities say they experience in scheduling timely medical appointments. We reviewed the implementation of selected elements of VHA's scheduling policy at both the VAMC and individual clinic level. These elements included the use of VHA's scheduling software to schedule medical appointments, the use of the electronic wait list[Footnote 16] for tracking patients new to a clinic that are waiting for medical appointments, and the use of the recall/reminder software to facilitate reminders for patients that need to return to the clinic for follow-up medical appointments. We also obtained from VHA and reviewed VAMCs' fiscal year 2011 certifications of compliance with VHA's scheduling policy--a required annual self-certification--as well as documentation of scheduler training completion obtained from the four VAMCs visited. Finally, we reviewed patient complaints about telephone responsiveness collected by each VAMC's Office of the Patient Advocate. To examine VHA's initiatives to improve veterans' access to timely medical appointments, we interviewed VHA central office officials to obtain information about selected initiatives and reviewed relevant VHA documents outlining these initiatives. We also interviewed officials at the VAMCs we visited about the implementation of these initiatives and officials at the Billings Clinic, a non-VA health care facility involved in the implementation of one of the initiatives. [Footnote 17] We conducted this performance audit from February 2012 to December 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: VHA's health care system is geographically divided into 21 VISNs, each of which is headed by a VISN director. Each VISN is comprised of a network of VAMCs, and the VISN office serves as the basic budgetary and decision-making unit for providing health care services to veterans within that geographical area. Each VAMC and its affiliated CBOCs and ambulatory care centers are headed by a VAMC director who manages administrative functions, and a chief of staff who manages clinical functions for these facilities. VHA's Central Office establishes system-wide scheduling policy. VHA Medical Appointment Scheduling Policy: VHA's scheduling policy establishes processes and procedures for scheduling medical appointments, and for ensuring the competency of staff directly or indirectly involved in the scheduling process. This policy is designed to help VAMCs meet VHA's commitment to scheduling medical appointments with no undue waits or delays. Specifically, VHA's scheduling policy includes, but is not limited to, the following requirements: * Requires VAMCs to use VHA's Veterans Health Information Systems and Technology Architecture (VistA) medical appointment scheduling system to schedule medical appointments.[Footnote 18] * Requires VAMCs to keep appointment schedules open and available for patients to make medical appointments at least 3 to 4 months into the future. * Requires schedulers to record in the VistA scheduling system the date on which the patient or provider wants the patient to be seen as the desired date. To determine the desired date, schedulers should be in communication with the patient when scheduling the medical appointment. * Requires schedulers to record the desired date correctly and describes how to determine and record the desired date for new patients--patients who haven't been seen by a health care provider in a clinic within the past 2 years, including those scheduled in response to a consult request--as well as specifying how to determine the desired date for established patients' follow-up medical appointments--patients who have been seen within the past two years. [Footnote 19] * Requires VAMCs to track new patients waiting for medical appointments using the electronic wait list within VistA and to remind established patients of follow-up medical appointments using the recall/reminder software within VistA, which enables clinics to create a list of established patients who need follow-up medical appointments more than 3 or 4 months in the future. Additionally, VHA has a separate directive that establishes policy on the provision of telephone service related to clinical care, including facilitating telephone access for medical appointment management. [Footnote 20] Officials at the VHA central office, VISN, and VAMC all have oversight responsibilities for the implementation of VHA's scheduling policy. In the VHA central office, the Director of Systems Redesign, through the Office of the Deputy Undersecretary for Health for Operations and Management, is responsible for the oversight and implementation of medical appointment scheduling requirements. This oversight includes measurement and monitoring of ongoing performance. Each VISN director, or designee, is responsible for oversight of enrollment and medical appointment scheduling for eligible veterans.[Footnote 21] Each VAMC director, or designee, is responsible for ensuring that clinics' scheduling of medical appointments complies with VHA's scheduling policy, including clinics in affiliated CBOCs and ambulatory care centers. In addition, the VAMC director is responsible for ensuring that any staff who can schedule medical appointments in the VistA scheduling system has completed VHA scheduler training.[Footnote 22] Starting in fiscal year 2007, VHA required every VAMC to annually self- certify compliance with VHA's scheduling policy.[Footnote 23] This certification is signed by the VAMC director and also encompasses scheduling compliance in affiliated CBOCs and ambulatory care centers. For fiscal year 2011, the certification required VAMCs to self-certify compliance, partial compliance, or noncompliance with more than 30 individual aspects of VHA's scheduling policy as well as overall compliance, partial compliance, or noncompliance with VHA's scheduling policy as a whole. According to officials, VHA's central office does not penalize noncompliance with the certification and expects oversight to be managed locally. VHA's central office uses this certification of compliance as a tool for VAMCs to identify and improve performance on important aspects of the policy. VistA Information Technology System: VistA is the single integrated health information system used throughout VHA in all of its health care settings. There are many different VistA applications for clinical, administrative, and financial functions, including VHA's electronic medical record, known as the Computerized Patient Record System, and the scheduling system. As we reported in May 2010, the VistA scheduling system is more than 25 years old and inefficient in facilitating care coordination between different sites.[Footnote 24] In 2000, VHA began an initiative to modernize the scheduling system, but VA terminated the project in 2009. We also reported that VA's efforts to successfully replace the scheduling system were hindered by weaknesses in its project management processes and lack of effective oversight. Wait Time Measurement and Performance: In 1995, VHA established a goal of scheduling primary and specialty care medical appointments within 30 days to ensure veterans' timely access to care.[Footnote 25] In fiscal year 2011, VHA shortened the wait time goal to 14 days for both primary and specialty care medical appointments based on improved performance reported in previous years. Specifically, VA's reported wait times for fiscal year 2010 showed that nearly all primary care and specialty care medical appointments were scheduled within 30 days of desired date. In fiscal year 2012, VHA added a goal of completing primary care medical appointments within 7 days of the desired date. To facilitate accountability for achieving its wait time goals, VHA includes wait time measures--referred to as performance measures--in its VISN and VAMC directors' performance contracts known as Network Director Performance Plans (NDPP) and Facility Director Performance Plans (FDPP), respectively.[Footnote 26] Wait time performance measures also are included in VA's budget submissions and performance reports to Congress and stakeholders; the performance reports are published annually in VA's Performance and Accountability Report (PAR).[Footnote 27] However, the medical appointment wait time performance measures included in the NDPPs and FDPPs differ from the measures that are reported in the PAR. (See table 1.) For example, in fiscal year 2012, VHA's wait time goal of 7 days for primary care medical appointments was reflected in the NDPP and FDPP performance measures, but the fiscal year 2012 PAR reported primary care wait time performance using a 14-day standard.[Footnote 28] The performance measures have also changed over time.[Footnote 29] At the time of our review, all of VHA's medical appointment wait time performance measures reflected the number of days elapsed from the patient's or provider's desired date, which is recorded in the VistA scheduling system by VAMCs' schedulers. According to VHA central office officials, VHA measures wait times based on desired date in order to capture the patient's experience waiting and to reflect the patient's or provider's wishes; which is not reflected by other available wait time measures. Table 1: Selected Fiscal Year 2012 VHA Medical Appointment Wait Time Performance Measures: Performance plan or report: Network Director's Performance Plans and Facility Director's Performance Plans (NDPP, FDPP)[A]; Wait time performance measures: Percent of patients waiting for a specialty care appointment longer than 14 days from the desired date[B]; Percent of primary care appointments completed within 7 days of the desired date[C]; Same day access with primary care provider (percent of requested same day primary care appointments completed within one day)[C]. Performance plan or report: VA Budget Submission and Performance and Accountability Report (PAR)[D]; Wait time performance measures: Percent of new patient primary care appointments completed within 14 days of the desired date[E]; Percent of established patient primary care appointments completed within 14 days of the desired date; Percent of new patient specialty care appointments completed within 14 days of the desired date[E]; Percent of established patient specialty care appointments completed within 14 days of the desired date. Source: VA. Notes: New and established patient appointments, and primary care and specialty care appointments refer to outpatient medical appointments. [A] The NDPP and FDPP are the network (VISN) and facility (VAMC) directors' contracts that include performance measures against which directors are rated at the end of the fiscal year, and are monitored throughout the year. [B] For fiscal year 2012, the specialty care medical appointment wait time measure represents the percent of patients waiting for at least one appointment longer than 14 days from the desired date at a given point in time. The measure was collected twice a month and VISN and VAMC directors were rated against the average of these scores at the end of the year. The plans also included additional performance measures specifically for mental health appointment timeliness that are outside the scope of this report. [C] For fiscal year 2012, the plans included five individual measures related to primary care and a sixth measure that was a composite of the five individual measures. Two of the five individual primary care measures pertained to primary care medical appointment wait times; these two wait time measures represent the percent of appointments completed in the specified time frame. The three other individual measures were percentage of total encounters that occur by telephone (telephone utilization), percentage of primary care appointments with patient's assigned primary care provider (continuity of care), and percentage of patients discharged from hospital who were contacted by their primary care provider within 2 days (post-hospital discharge contact). At the end of the year, VISN and VAMC directors were scored for each of the five individual primary care measures and rated against the single composite measure. They were rated as meeting the composite measure if they met targets for at least three of the five individual primary care measures. [D] VA prepares a congressional budget justification that provides details supporting the policy and funding decisions in the President's budget request that is submitted to Congress prior to the beginning of each fiscal year. VA also publishes an annual PAR, which contains performance targets and results achieved against those targets in the previous year. For fiscal year 2012, the four PAR wait time measures represent the percent of each type of appointment completed within 14 days of the desired date. The cumulative year-to-date scores were reported on the PAR. [E] Generally, for wait time measurement purposes, VA defines new patients as those who have not been seen in a particular clinic at that facility within the last 2 years. This includes appointments for newly enrolled patients as well as those scheduled in response to a consult request. [End of table] Reported Medical Appointment Wait Times Are Unreliable; VHA Uses Additional Methods to Monitor Patients' Access to Medical Appointments: Medical appointment wait times used for measuring and assessing performance toward VHA's wait time goals are unreliable due to problems with recording the appointment desired date in the VistA scheduling system. Acknowledging limitations of the wait time measures, VHA uses additional information to monitor patients' access to medical appointments. Reported Medical Appointment Wait Times Derived from Desired Date Are Unreliable: VHA measures its medical appointment wait times as the number of days that have elapsed from the patient's or provider's desired date. Consequently, the reliability of reported wait time performance is dependent on the consistency with which schedulers record the desired date in the VistA scheduling system. However, aspects of VHA's scheduling policy and training documents regarding how to determine and record the desired date are unclear and do not ensure replicable and reliable use of the desired date. In addition, we found that some schedulers at select VAMCs did not correctly implement other aspects of VHA's scheduling policy for recording the desired date. Unclear Desired Date Policy: Aspects of VHA's scheduling policy and related training documents on how to determine and record the desired date are unclear and do not ensure replicable and reliable recording of the desired date by the large number of staff across VHA who can schedule medical appointments in the VistA scheduling system.[Footnote 30] Specifically, VHA's scheduling policy and related scheduler training documents do not provide consistent guidance about when or whether the desired date should be based on the patient's or provider's preference. While the policy defines desired date as "the date on which the patient or provider wants the patient to be seen," it also instructs that the "the desired date needs to be defined by the patient" for new patient medical appointments, medical appointments scheduled in response to consult requests, and established patient follow-up medical appointments. When there is a conflict between the provider and patient desired date, the scheduler is instructed to contact the provider for a decision on the return time frame, but the policy and training documents do not clearly describe under what circumstances the provider's date should be used as the desired date. Further, providers may designate a desired appointment time frame for a follow- up medical appointment rather than a specific date; in such cases, the policy is unclear as to which date within the provider's designated time frame the scheduler should enter as the desired date. The scheduling policy and training do not provide sufficient guidance to ensure consistent use of desired date in these various scheduling scenarios. VHA central office officials responsible for developing VHA's scheduling policy and related training documents told us that the desired date is intentionally broad to account for all of the scheduling scenarios that may exist. However, leadership officials from the four VAMCs we visited and their corresponding VISNs reported problems with the unclear guidance on the desired date definition, and difficulties achieving consistent and correct use of the desired date by their schedulers. In addition, given the ambiguity in the scheduling policy and related training documents, there are different interpretations of the desired date between officials at different levels. For example, a VISN director stated that if a provider gives a desired time frame, the scheduler is to use the earliest date in that range as the desired date; whereas a provider in a specialty care clinic at the VAMC we visited within that VISN stated that the clinic uses the latest date in the range to meet the 14-day specialty care medical appointment scheduling goal. Additionally, when presented with various scheduling scenarios, schedulers at the VAMCs we visited determined and recorded the desired date differently. For example, when posed with the question "What date do you enter into the scheduling system as the desired date for an established patient follow-up medical appointment?", 12 schedulers said they would enter the patient's desired date, 4 said the provider's date, and the remaining 3 said they used the next available medical appointment date. When posed with the question "If the patient's stated desired date conflicts with the provider's designated desired date or time frame, what date do you enter as the desired date?", 1 scheduler said that the patient's desired date would be entered, while another said the desired date has to come from the provider. The variation in schedulers' interpretation of the desired date suggests confusion about its correct use in different scheduling scenarios. Errors in Recording Desired Date: Although unclear about when to use the patient's or provider's desired date, VHA's scheduling policy clearly instructs that, in all circumstances, the desired date should be defined without regard to schedule capacity, and should not be altered once established to reflect a medical appointment date the patient accepts because of lack of medical appointment availability on the desired date. However, we found that at least one scheduler from each of the VAMCs we visited did not correctly implement these aspects of the policy when recording the desired date in the VistA scheduling system for specific hypothetical scheduling situations.[Footnote 31] As summarized in table 2, we identified the following three types of errors, each of which would have resulted in desired dates that did not accurately reflect the patients' or providers' desired date, as well as potentially result in the reporting of more favorable wait times for those medical appointments.[Footnote 32] * Determined appointment availability prior to establishing desired date: Although VHA's scheduling policy requires schedulers to establish the desired date for a medical appointment without regard to the schedule capacity, four schedulers from three VAMCs determined the clinic's next available medical appointment dates before establishing a desired date. Therefore, reported wait times for these appointments may not have accurately reflected how long patients actually waited. * Altered original desired date based on appointment availability: Three schedulers from two VAMCs established a desired date that was recorded in the VistA scheduling system independent of schedule capacity, but later altered the desired date because of appointment availability. Specifically, two of the three schedulers altered the originally established desired date to match the agreed-upon appointment date, which would have incorrectly resulted in no wait time reported for the appointment. The third scheduler altered the established desired date when there was no appointment availability within 2 weeks of that date; which would have resulted in an incorrectly reported wait time that was shorter than the patient actually waited from his or her original desired date. * Recorded a new desired date when rescheduling appointment: Additionally, eight schedulers from three VAMCs incorrectly recorded a new desired date when rescheduling an appointment canceled by the clinic rather than keeping the original desired date as required by VHA's scheduling policy. Changing the desired date in this way would incorrectly decrease the reported wait times for the rescheduled appointments; veterans actually would wait longer than the reported wait times indicated. Table 2: Number of Schedulers at Each VAMC Visited Who Incorrectly Recorded the Medical Appointment Desired Date, by Error Type: Desired date recording error: Determined medical appointment availability prior to establishing desired date[A]; Number of schedulers who demonstrated error (number of schedulers interviewed at each VAMC): VAMC A (3): 1; VAMC B (7): 2; VAMC C (4): 1; VAMC D (5): 0; Total schedulers who demonstrated error: 4. Desired date recording error: Altered original desired date based on medical appointment availability[A]; Number of schedulers who demonstrated error (number of schedulers interviewed at each VAMC): VAMC A (3): 0; VAMC B (7): 2; VAMC C (4): 1; VAMC D (5): 0; Total schedulers who demonstrated error: 3. Desired date recording error: Recorded a new desired date when rescheduling a medical appointment[B]; Number of schedulers who demonstrated error (number of schedulers interviewed at each VAMC): VAMC A (3): 0; VAMC B (7): 3; VAMC C (4): 3; VAMC D (5): 2; Total schedulers who demonstrated error: 8. Desired date recording error: Total number of errors; Number of schedulers who demonstrated error (number of schedulers interviewed at each VAMC): VAMC A (3): 1; VAMC B (7): 7; VAMC C (4): 5; VAMC D (5): 2; Total schedulers who demonstrated error: 15. Desired date recording error: Total number of schedulers who demonstrated at least one error; Number of schedulers who demonstrated error (number of schedulers interviewed at each VAMC): VAMC A (3): 1; VAMC B (7): 6; VAMC C (4): 3; VAMC D (5): 2; Total schedulers who demonstrated error: 12. Source: GAO analysis of scheduler interviews. Notes: This table presents the results of our interviews with 19 schedulers from the four VAMCs visited. We used structured questions to test how schedulers would determine and enter the desired date for specific medical appointment types using hypothetical patients. We identified three types of desired date scheduling errors which, for actual patient medical appointments, would have resulted in a desired date that would not accurately reflect the patient or provider's desired date, as well as potentially result in reporting of more favorable wait times for those medical appointments. A scheduler could have demonstrated more than one error. [A] According to VHA's scheduling policy, when scheduling new patient medical appointments--including medical appointments in response to a consult request--and established patient follow-up medical appointments, the medical appointment desired date needs to be defined by the patient without regard to schedule capacity. Once the patient's desired date has been established, it must not be altered to reflect a medical appointment date patients agree to accept due to lack of medical appointment availability on the original desired date. [B] According to VHA's scheduling policy, when a medical appointment is canceled and rescheduled by the clinic, the scheduler must record as the desired date for the new medical appointment, the desired date for the original medical appointment. Three of the 19 schedulers did not respond to questions about rescheduling medical appointments. [End of table] During our site visits, staff at some clinics told us they change medical appointment desired dates to show clinic wait times within VHA's performance goals. A scheduler at one primary care clinic specifically stated that she changes the recorded desired date to the patient's agreed-upon appointment date in order to show shorter wait times for the clinic. A provider at a specialty care clinic at another VAMC said providers in that clinic change the desired dates of their follow-up appointments if a patient cannot be scheduled within the 14- day performance goal. In addition, the reported wait times, derived from desired date, for one of the specialty care clinics we visited were inconsistent with the VAMC's account of appointment scheduling backlogs and scheduling challenges, indicating reported wait time inaccuracies. At the time of our site visit, officials from this clinic indicated that long waits for new patient appointments had existed prior to our visit and told us that the next available appointment for a new patient was in 6 to 8 weeks. However, reported wait time data for the month we visited showed that the clinic completed all new patient appointments on the desired date, resulting in an unlikely high percentage of appointments with zero-day wait times that was inconsistent with information gathered during our site visit, raising questions about whether the desired date was recorded in accordance with VHA's scheduling policy. Furthermore, according to reported wait times for the VAMC, this clinic completed nearly all new patient appointments within 14 days of the desired date for the 2 months prior to our visit; and, similarly, in the 2 months after our visit, reported wait times for this clinic show completion of all new patient appointments within the 14-day time frame.[Footnote 33] VHA Officials Cited Importance of Desired Date in Capturing Patient Preference and Supplement Wait Time Measures with Other Information to Monitor Patient Access to Appointments: VHA central office officials told us that they recognized the potential reliability issues of using the desired date for measuring wait times, but stated that use of the desired date is the best approach for capturing patient experience and preference. Officials told us that there is no single industry standard for measuring how long patients wait for appointments and commonly used measures--such as capacity measures--do not account for patient preference or reflect how long the patient actually waited for an appointment.[Footnote 34] In addition, officials told us that the VistA scheduling system was not designed to capture data for management purposes, which has limited VHA's options for developing wait time measures. Over the years, VHA has tried using many different approaches to measuring wait times, such as capacity measures and using the date the appointment was created rather than the desired date to determine wait times. [Footnote 35] Although these measures were not officially used for performance accountability or reported on the PAR or NDPP and FDPP in fiscal year 2012, data on these measures are available to VISNs and VAMCs for performance monitoring. Officials told us that improving how wait times are measured is an ongoing effort, and they have conducted research to identify wait time measures that most closely correlate with patient satisfaction and positive outcomes. At the time of our review, VHA had not implemented changes to wait time performance measures based on the results of this research. In addition to measuring medical appointment wait times, VHA central office officials reported that VHA also uses other information to monitor patients' access to medical appointments and to assist VISNs and VAMCs in managing clinics. Patient Satisfaction Measures: VHA central office and VISN officials with whom we spoke identified patient satisfaction as another important indicator of patient access to medical appointments and VA has incorporated measures of self-reported patient satisfaction in its performance assessments.[Footnote 36] Specifically, the annual PAR includes a measure of overall patient satisfaction with VHA inpatient and outpatient healthcare in addition to the wait time measures derived from desired date. Separate measures related to patient satisfaction with obtaining outpatient care were also among the measures available for VISN and VAMC directors to include in their fiscal years 2011 and 2012 performance plans (NDPP and FDPP).[Footnote 37] VHA also makes the satisfaction measures available to VISNs and VAMCs for continuous performance monitoring as well as available to the public. One of the four VAMCs we visited included the satisfaction measures in their performance plan for fiscal year 2012, and officials cited monitoring these measures on a regular basis. Officials from one VISN also specifically cited comparing its VAMCs' patient satisfaction scores to reported wait times to identify inconsistencies. However, the director of another VAMC said he does not rely on the satisfaction measures to monitor access because the data are dated by the time the VAMC sees the results, and instead, he relies on the scheduling data derived from wait time measures. Clinic Management Information: In addition to wait time measures, VHA has other information available for VISNs and VAMCs to manage clinics and monitor and improve clinic access, such as no-show rates and consult lists. Several clinic officials reported monitoring no-show rates--the rate at which patients do not appear for their scheduled appointment--in order to reduce unused appointments, for example, by identifying and providing additional appointment reminders to patients with frequent no-shows. Officials from multiple specialty clinics said they monitor lists of consults--requests for specialty care appointments--to ensure they are acted upon in a timely manner. [Footnote 38] Although the time between when the provider requests a consult and when the specialty clinic reviews the consult can affect the total time a patient waits for a specialty appointment, this time is not reflected in current wait time performance measures. Inconsistent Implementation of VHA's Scheduling Policy and Other Problems Impede VAMCs' Ability to Schedule Timely Medical Appointments: The four VAMCs we reviewed did not consistently implement certain elements of VHA's scheduling policy, including oversight requirements, which may result in increased wait time or delays in scheduling medical appointments. VAMCs also described other problems with scheduling timely medical appointments, including outdated technology, gaps in staffing of schedulers and providers, and telephone access problems. Inconsistent Implementation of VHA's Scheduling Policy Hinders VAMCs' Ability to Schedule Timely Medical Appointments: The four VAMCs we visited did not consistently implement VHA's scheduling policy, which is intended to facilitate the creation of medical appointments that meet patients' needs with no undue waits or delays. This policy includes the use of the VistA scheduling system to schedule medical appointments, and the use of the electronic wait list to track new patients waiting for medical appointments. (See table 3 for information on the number of clinics we visited that did not implement selected elements of the VHA's scheduling policy.) Inconsistent implementation of VHA's scheduling policy can result in increased wait time or delays in obtaining medical appointments. Table 3: Number of Clinics That Did Not Implement Selected Elements of the VHA's Scheduling Policy, by VAMC Visited: Element of VHA's scheduling policy (number of clinic responses): Using VistA in scheduling of medical appointments (23); Number of clinics at each VAMC that did not implement the element of VHA's scheduling policy (clinics visited at each VAMC): VAMC A (4): 0; VAMC B (9): 1; VAMC C (5): 0; VAMC D (5): 0. Element of VHA's scheduling policy (number of clinic responses): Scheduling medical appointments while in direct communication with the patient (20); Number of clinics at each VAMC that did not implement the element of VHA's scheduling policy (clinics visited at each VAMC): VAMC A (4): 0; VAMC B (9): 1; VAMC C (5): 5; VAMC D (5): 0. Element of VHA's scheduling policy (number of clinic responses): Using the electronic wait list for patients who have not been seen before in the clinic and are waiting to be scheduled[A] (21); Number of clinics at each VAMC that did not implement the element of VHA's scheduling policy (clinics visited at each VAMC): VAMC A (4): 0; VAMC B (9): 1; VAMC C (5): 2; VAMC D (5): 1. Element of VHA's scheduling policy (number of clinic responses): Using recall/reminder software for medical appointments needed more than 3 to 4 months into future (22); Number of clinics at each VAMC that did not implement the element of VHA's scheduling policy (clinics visited at each VAMC): VAMC A (4): 0; VAMC B (9): 0; VAMC C (5): 5; VAMC D (5): 0. Element of VHA's scheduling policy (number of clinic responses): Keeping medical appointment schedules open at least 3 to 4 months into future (19); Number of clinics at each VAMC that did not implement the element of VHA's scheduling policy (clinics visited at each VAMC): VAMC A (4): 1; VAMC B (9): 1; VAMC C (5): 1; VAMC D (5): 1. Element of VHA's scheduling policy (number of clinic responses): Total number instances in which elements of VHA's scheduling policy were not implemented; Number of clinics at each VAMC that did not implement the element of VHA's scheduling policy (clinics visited at each VAMC): VAMC A (4): 1; VAMC B (9): 4; VAMC C (5): 13; VAMC D (5): 2. Element of VHA's scheduling policy (number of clinic responses): Total number of clinics that did not implement at least one element of VHA's scheduling policy; Number of clinics at each VAMC that did not implement the element of VHA's scheduling policy (clinics visited at each VAMC): VAMC A (4): 1; VAMC B (9): 2; VAMC C (5): 5; VAMC D (5): 2. Source: GAO analysis of interviews at 23 clinics. Notes: Except for the element "using VistA in scheduling of medical appointments," we did not report responses for all 23 clinics we visited because interview responses were incomplete for some elements of the VHA's scheduling policy as depicted in the table. [A] Officials from 12 clinics told us that they do not use the electronic wait list because their clinic did not have new patients waiting for appointments or their clinic schedules patients for appointments with long wait times. [End of table] Use of VistA Scheduling System: One of the clinics we visited did not use the VistA scheduling system to determine available medical appointment dates and times, and to schedule medical appointments, as required by VHA's scheduling policy. Officials noted that this clinic lacked a full-time staff person dedicated to scheduling, and therefore, the providers called their patients to schedule their own medical appointments. Clinic staff reported that providers recorded medical appointments on sheets of paper and gave those sheets to a scheduler, who maintained a paper calendar of all medical appointments; this scheduler later recorded the appointment into the VistA scheduling system. Failing to use VistA to schedule medical appointments could create additional backlogs or scheduling errors because the schedule in VistA may not accurately reflect providers' availability. According to one provider in this clinic, for example, "staff from other departments look in VistA [scheduling system] and it looks like the clinic is not booked, so they'll send their patients as walk-in appointments. However, the clinic is really fully booked and patients are waiting." Communication with Patients: Officials from six clinics across two different VAMCs reported that staff scheduled new patient or established patient follow-up medical appointments without speaking to patients, and then notified patients of the scheduled medical appointment by letter, if the appointment was at least a few weeks away. This method of scheduling--referred to as "blind" scheduling by one official --is not in accordance with VHA's scheduling policy and could result in missed medical appointments for patients who do not receive the letters, or are not available at the scheduled time because patients are not involved in the scheduling process. One scheduler noted that he sent medical appointment letters because he didn't have time to call all patients to schedule appointments as he performs scheduling duties for 27 different clinics. Furthermore, outdated or incorrect patient contact information is an impediment to scheduling appointments via letters; an official in one of the six clinics told us that the databases containing patient contact information used to send such letters often do not have veterans' correct or up-to-date contact information. Use of the Electronic Wait List: Officials in four clinics across three VAMCs that had backlogs of patients waiting for medical appointments stated that they do not use the electronic wait list, the official VHA wait list used to track patients with whom a clinic does not have an established relationship.[Footnote 39] Clinics that do not use the electronic wait list may be at risk of losing track of new patients waiting for medical appointments. For example, at one specialty clinic with a backlog of consult requests, medical appointments for new patients were backed up almost 3 months; VAMC officials reported tracking patients waiting for medical appointments by printing paper copies of the consult requests from the electronic medical record. A provider at this clinic expressed concern that the clinic manager "has a tall stack of unscreened consult referrals just sitting on her desk, and no one is addressing them." Use of Recall/reminder Software: Officials from one VAMC stated that it did not have the required recall/reminder software to facilitate reminders for patients who need to return to the clinic for follow-up medical appointments more than 3 to 4 months into the future; therefore, none of its clinics, including the five clinics that we visited, were able to use it as intended. [Footnote 40] Instead clinics at this VAMC use a work-around in the scheduling system to remind clerks to print and send letters reminding patients to call and schedule their follow-up medical appointments. However, this work-around is not automated and relies on schedulers to remember to generate a list of patients who need follow-up medical appointments, and print and send those letters. The VAMC is in the process of implementing recall/reminder software, according to officials. Medical Appointment Schedule Availability: One clinic in each of the four VAMCs visited did not keep their medical appointment schedules open 3 to 4 months into the future as required by VHA's scheduling policy.[Footnote 41] Instead, these four clinics allowed medical appointments to be booked only 1 to 2 months into the future. Limiting the future medical appointment schedule may limit patients' ability to schedule a follow-up medical appointment before leaving the clinic, as recommended by the policy, and also may result in additional work for clinic staff to send recall/reminder letters to patients for medical appointments less than 3 to 4 months away. Oversight of VHA's Scheduling Policy: The VAMCs we visited inconsistently implemented certain oversight requirements in VHA's scheduling policy--specifically, completion of training and certification of compliance. VAMC officials stressed the importance of scheduler training for ensuring correct implementation of VHA's scheduling policy; however, certain VAMCs did not ensure completion of the training by all staff who were required to complete it.[Footnote 42] Although all VAMCs we visited provided a list of staff who can schedule appointments, three VAMCs did not provide documentation that all staff on the list had successfully completed the required training. For example, officials from one VAMC stated that it maintained a list of staff who can schedule appointments, and a separate list of staff who had completed the training, but only in response to GAO's request for documentation did the VAMC identify staff with scheduling access who needed to complete the training. Further, three of the 19 schedulers we interviewed said they completed training other than the required VHA scheduler training. Completion of required VHA scheduler training and maintaining up-to-date documentation of schedulers' completion of the training is particularly important for ensuring consistent implementation of VHA's scheduling policy, given the high rates of scheduler turnover described by officials. All four of the VAMCs we visited completed the required self- certification of compliance with the VHA's scheduling policy for fiscal year 2011, three of which certified overall compliance, and one certified overall noncompliance.[Footnote 43] However, leadership officials from two VAMCs, including the only one of the four that certified overall noncompliance, were initially uncertain who completed the certification or the steps taken to complete it, indicating that VAMCs are not always using the self-certification process to identify and improve problems with compliance with VHA's scheduling policy. VAMCs Identified Other Problems with Scheduling Medical Appointments, Including Issues with Outdated Technology, Staffing Gaps, and Telephone Access: VAMCs identified several problems that can impede the timely scheduling of medical appointments, which also may impact their compliance with VHA's scheduling policy. Problems with Outdated VistA Scheduling System: VHA central office officials and officials from all of the VAMCs we visited said the VistA scheduling system is outdated and inefficient, which hinders the timely scheduling of medical appointments. In particular, officials said the scheduling system requires schedulers to use commands requiring many keystrokes and does not allow them to view multiple screens at once. Schedulers must open and close multiple screens to check a provider's or clinic's full availability when scheduling a medical appointment, which is time-consuming and can lead to errors. For example, providers have separate schedules within VistA to accommodate the various types of services they provide.[Footnote 44] Because the scheduling system cannot display multiple schedules on the same screen, schedulers have to enter and exit multiple screens to check a provider's full daily schedule when scheduling a medical appointment. If schedulers do not open all of the necessary screens, they may unknowingly create scheduling errors such as booking two medical appointments at the same time in different sections of a provider's schedule. Further, staff at one VAMC told us the problem of not being able to easily view a provider's full schedule can result in the failure to ensure that appointments are canceled when a provider requests it. This error could cause patients to come to the VAMC unnecessarily or a failure to reschedule canceled appointments in a timely way, both of which might lead to increased wait times for those patients. Officials from all the VAMCs we visited also noted that the VistA scheduling system is not easily adapted to meet clinic needs. For example, staff cannot create a provider schedule in the scheduling system that is longer than 8 hours. If a provider wants to extend his or her schedule on certain days, staff must create additional clinic schedules in the scheduling system for that provider, which can result in more delays and possible errors because schedulers have to check additional screens for medical appointment availability. Furthermore, officials told us that the scheduling system does not automatically interface with VHA's electronic medical record,[Footnote 45] which makes the scheduling process more time-consuming as schedulers alternate between the two software applications to ensure medical appointments are made in accordance with providers' guidance. VAMC officials described steps they take to ensure schedulers use VistA in accordance with the scheduling directive, including ongoing scheduler training and supervisory reviews of scheduler performance. However, as noted above, a lack of clarity in the desired date training documents and a lack of documentation of scheduler training at certain facilities may limit the effectiveness of these interventions. One VAMC provides schedulers with dual monitors to enable them to open multiple screens at once. Another VAMC told us they considered this solution in their primary care clinic, but found that limited physical space in the clinic did not accommodate additional monitors. In response to ongoing problems with the VistA scheduling system, VHA undertook an initiative to replace it in 2000, but VA abandoned the replacement due to weaknesses in project management and a lack of effective oversight. VA released a new request for information in December 2011 to gather information about vendors and possible software packages that could replace the current scheduling system. In September 2012, VHA told us that vendors' responses to the request for information indicated that VHA will be able to choose among several viable software packages. According to officials, VA's next step is to compare different vendors' software packages through the summer of 2013, and subsequently issue a request for vendor proposals.[Footnote 46] Problems with Scheduler Turnover and Provider Staffing Gaps: VHA central office officials and officials from all of the VAMCs we visited stated that shortages or turnover of schedulers also creates problems for the timely scheduling of medical appointments. Officials said that schedulers perform many important roles, including greeting patients, checking patients in and out of clinics, answering telephone calls, scheduling medical appointments for primary care, as well as specialty care consults, and performing other administrative support functions on behalf of the clinical staff. Officials explained, however, that high stress and a demanding workload as well as the entry-level pay grade of the scheduler position leads to high turnover. Further, officials told us that high-performing schedulers often are quickly promoted to other positions within VA. According to VHA officials, most scheduler positions are classified as a low grade within the government general schedule pay scale with little room for upward movement within the grade. Officials at two of the VAMCs we visited told us they are working to raise the pay level for schedulers; for example, one VAMC has begun to assess scheduler position descriptions to determine whether they can be reclassified to allow for more flexibility in determining scheduler salaries based on the variation in their assigned duties. Given the important role of schedulers in the scheduling process, officials said that even temporary staffing gaps or shortages can cause medical appointment delays or wait times. Staff with whom we spoke in several clinics said that when scheduler staffing is lacking, including when a scheduler is on short-term leave, it is difficult to cover all the scheduler's duties, and that such gaps can cause delays for patients. Further, we were told that scheduler staffing gaps resulted in inefficient use of clinical staff time. For example, at one specialty clinic that lacked its own scheduler, providers routinely scheduled their own medical appointments, which took away from time seeing patients, and also resulted in incorrect scheduling practices. Given the training needs associated with using the VistA scheduling system, following VHA's scheduling policy, and ensuring the correct use of desired date, high rates of scheduler turnover could contribute to inconsistent use of desired date in the scheduling process or other appointment scheduling problems. Officials at two VAMCs noted that scheduler staffing gaps are compounded by recent changes in their roles and responsibilities as VHA implements a new team-based model of primary care, which calls for one scheduler to be assigned to each primary care team. Officials told us that these changes generally increase the administrative demands placed on schedulers, as they are asked to respond to team duties while continuing to answer phones, greet patients, and register new patients, among other responsibilities. Officials from two VAMCs told us they had requested approval to hire additional staff to meet these added administrative needs. Scheduler staffing gaps may also create problems managing patient flow through clinics, which can impede scheduling of follow-up appointments, according to officials at two of the VAMCs we visited. Staff at these VAMCs told us that they sometimes do not have sufficient schedulers available to staff check-out desks, and staff at one VAMC added that as a result patients might "fall through the cracks," leaving follow-up medical appointments unscheduled unless the patient remembers to call in to schedule the appointment. In addition, when patients do not check out, schedulers are responsible for tracking patients needing follow-up medical appointments. This situation may be exacerbated in clinics that do not use the required recall/reminder software to facilitate the scheduling of follow-up medical appointments more than 3 to 4 months in the future, adding further to the backlog of patients in need of follow-up medical appointments. Officials from all of the VAMCs we visited told us that provider shortages also contribute to scheduling backlogs in certain locations and specialties. Recruitment and retention of providers was a particular challenge for VAMCs in rural areas, areas with high costs of living, and for certain provider specialties. All of the VAMCs we visited described gaps in provider staffing in certain specialty care clinics. Officials at all VAMCs also stated that a lack of salary competitiveness or the length of time to hire new providers into the VA system also contributed to gaps in provider staffing and scheduling backlogs. Gaps in provider staffing also can result from providers being on extended or unexpected leave, including vacation time, sick leave, or military deployments. These absences may result in longer wait times for patients. For example, officials at one VAMC told us that even a brief absence of one provider on leave can cause significant wait times, and that it is difficult to catch up and eliminate the backlog. Staff from some clinics described steps they take to reduce backlogs caused by gaps in provider staffing, including overbooking provider schedules and scheduling temporary Saturday hours. Officials at one VAMC told us that they employ a "floater" primary care physician to provide coverage for providers on leave, but an official at another clinic told us that they were unable to hire additional providers to meet the demand for medical appointments. Problems with Telephone Access: Officials at all of the VAMCs we visited told us that high call volumes and a lack of staff dedicated to answering the telephones impede the timely scheduling of medical appointments.[Footnote 47] Despite VHA's telephone policy requiring the provision of continuous telephone service for clinical care and medical appointment management, VAMC officials noted that schedulers are frequently overwhelmed by high call volumes and are unable to respond to calls in a timely way. In addition, officials at one VAMC told us that outdated telephone technology, and the lack of a dedicated VAMC-wide call center, limited their ability to improve their telephone responsiveness. VHA has reported that telephone access to VHA health services has historically been a frustrating experience for veterans, including dropped calls, multiple transfers, and long waits to reach a staff person able to resolve their inquiries.[Footnote 48] Further, patients at all of the VAMCs we visited registered complaints about the difficulty of reaching outpatient clinic staff by telephone and unreturned telephone calls. According to information on patient complaints provided by the four VAMCs we visited, patient complaints about unreturned telephone calls ranked among the top two categories of complaints in fiscal year 2012 at all four VAMCs.[Footnote 49] Further, staff at two of the VAMCs reported that their telephone calls to outpatient clinics within their own VAMC went unanswered, and one added that their inability to reach staff in their own clinics also was an obstacle to timely medical appointment scheduling. In January 2012, VHA distributed suggested best practices for improving telephone design, service, and access in its Telephone Systems Improvement Guide.[Footnote 50] This guide outlines steps VHA found to be effective means of improving telephone service and maintaining health care access, including regularly monitoring the purpose and volume of telephone calls; establishing dedicated staff to answering calls, especially at times of peak call volume; and training staff responsible for answering telephones in call centers. To address telephone issues, officials at one VAMC we visited told us they were developing a proposal to establish a call center with a new telephone system, to be staffed by schedulers dedicated to answering the telephones. Officials at a different VAMC stated that a scheduling supervisor periodically checks schedulers' telephones to ensure that voice mail messages are listened to and that calls are returned. VHA Is Implementing a Number of Initiatives to Improve Access to Medical Appointments: VHA is implementing several initiatives to improve veterans' access to medical appointments. Specifically, these initiatives focus on more patient-centered care; using technology to provide care, through means such as telehealth; and using care outside of VHA to reduce travel and wait times for veterans who are unable to receive certain types of outpatient care in a timely way through local VHA facilities. VHA officials told us they are monitoring the implementation of these initiatives; however, in some cases, more information is needed to determine their impact on timely access to care over time. Patient-Centered Care Initiatives: VHA's patient-centered medical home model for primary care, Patient Aligned Care Teams (PACT), is intended, in part, to improve access to medical appointments and care coordination through the use of interdisciplinary care teams and technology to communicate with patients. Implementation of PACT began in 2010, and is an ongoing effort, according to VHA officials. PACT differs from how primary care was previously delivered by assigning each patient to an interdisciplinary team. The PACT team is intended to be comprised of a primary care provider, registered nurse care manager, a clinical support staff member such as a licensed practical nurse, and a scheduler.[Footnote 51] These teams offer patients a centralized way to get questions answered by nurses or other clinical support staff and aim to reduce the need for face-to-face medical appointments, thereby enabling more efficient use of providers' time. For example, at one of the VAMCs we visited, patients are given a direct telephone number to contact their PACT team and leave a voice mail message to be returned by the team's registered nurse. Encouraging PACT teams' use of telephone communication and telephone appointments is intended to enable patients to more quickly obtain answers to some of their administrative and medical questions, such as requests for prescription refills, without having to schedule a face-to-face medical appointment. VHA officials told us that they expect PACT teams' use of telephone communication and telephone appointments will open up face-to-face medical appointment slots for patients who need them and might enable clinics to reduce backlogs and improve access to same-day primary care medical appointments. Officials at two VAMCs we visited told us that the transition to the PACT model has created some initial scheduling and staffing difficulties. For example, officials at these VAMCs noted that it is difficult for scheduling staff to respond to their PACT team duties in addition to meeting other responsibilities such as answering phones, checking in patients, registering new patients, and scheduling for more than one clinic. This is compounded by the fact that not all PACT teams have been assigned their own scheduler, as prescribed by the PACT model, so an individual scheduler is sometimes serving multiple PACT teams. Officials at these two VAMCs explained that they would need to hire more schedulers to meet the goal of assigning one to each PACT team. To measure the progress of PACT implementation and its impact on access to quality care, VHA is collecting data and tracking a series of measures in a monthly internal data report. Five of the PACT measures are (1) primary care medical appointments completed within 7 days of the desired date;[Footnote 52] (2) same day access with primary care provider, or the percentage of appointments completed within 1 day; (3) telephone utilization, or the percentage of total encounters that occur by telephone; (4) continuity of care, or the percentage of primary care appointments with the patient's assigned primary care provider; and (5) post-hospital discharge contact, or percentage of patients discharged from the hospital who were contacted by their primary care provider within 2 days.[Footnote 53] As described earlier, accurate measurement of medical appointment wait times--including the first two PACT measures--is dependent upon the correct recording of the desired date in the VistA scheduling system. In fiscal year 2012, PACT measures were also included in the NDPP and FDPP.[Footnote 54] Initiatives Using Technology: Part of VHA's goal of achieving improved access to medical appointments is the increased use of technology such as telehealth and secure messaging.[Footnote 55] Use of these tools is intended to improve communication between patients and providers and open up providers' schedules for needed face-to-face medical appointments, thereby improving access to face-to-face appointments. VHA telehealth includes: * home telehealth for chronic disease management such as diabetes; * real-time clinic-based video telehealth, in which patients at a local CBOC may connect with a VHA provider at a different location to receive services that are unavailable at the CBOC, such as mental health or speech pathology; * and store-and-forward telehealth, in which digital images such as x- rays or images of skin problems, are taken, stored, and sent to an expert for review and consultation. VHA officials told us that the use of telehealth can reduce both travel and wait times for medical appointments and help meet the needs of patients with chronic conditions. All VAMCs we visited told us they were using telehealth to improve access to care. Another initiative that uses technology to reduce unnecessary face-to- face medical appointments is VHA's My HealtheVet, a web-based program that enables veterans to create and maintain a web-based personal health record with secure access to health information; services such as prescription refill requests; and secure messaging. Secure messaging allows veterans to communicate electronically with their health care team. According to VHA, of the more than 8 million veterans enrolled in VHA, 1.4 million are registered in My HealtheVet as of August 2012, and more than 437,000 have created secure messaging accounts. A recent VA study reports that secure messaging may improve access, patient perceptions about access, and provides for better communication.[Footnote 56] Non-VA Care Initiative: VHA uses non-VA care to reduce wait times and backlogs and to provide veterans' access to specialists not available through VHA.[Footnote 57] Under a statutory requirement to help veterans receive care closer to home, VHA is piloting a new model of non-VA care known as Project ARCH (Access Received Closer to Home).[Footnote 58] Project ARCH is a five-site, 3-year pilot program administered by the VHA Office of Rural Health to provide health care services through contracts with local community providers.[Footnote 59] According to VHA officials, Project ARCH might help alleviate wait times for specialty care services with high demand, or for which there is a shortage of local providers. At the Montana Project ARCH pilot site, which we visited as part of our site visit to the Montana VAMC, staff from the VAMC and the Billings Clinic, a non-VA provider delivering services to veterans through Project ARCH, identified both benefits and obstacles for patients enrolled in Project ARCH. For example, though VAMC and Billings Clinic staff noted that Project ARCH reduced both travel and wait times for Montana veterans in need of orthopedic care, Billings Clinic staff also noted that difficulties in coordinating care for veterans moving between VHA and non-VA providers at times resulted in delays in providing care to those and other veterans. Additionally, problems with processing authorizations for certain services were among the concerns raised in an April 2012 evaluation of the Montana Project ARCH program.[Footnote 60] Project ARCH contractors must submit monthly reports, including information on medical appointment scheduling timeliness, wait times, and other topics. For example, the contractor for the Project ARCH Program in Montana is required to report on the extent to which it is meeting VHA's 14-day wait time goal for medical appointments-- according to VHA officials, the contractor must meet a 90 percent target. These wait times may not accurately reflect how long patients are waiting for a medical appointment, however, because the wait time is counted from the time the contractor receives the authorization from VA, rather than from the time the patient or provider requests a medical appointment. Conclusions: VHA officials have expressed an ongoing commitment to providing veterans with timely access to medical appointments and have reported continued improvements in achieving this goal. However, unreliable wait time measurement has resulted in a discrepancy between the positive wait time performance VA has reported and veterans' actual experiences. Ambiguity in what constitutes the medical appointment desired date--the date VHA uses as the basis for measuring wait time-- as well as manipulation of the desired date to meet goals have contributed to these inaccuracies. With more than 50,000 schedulers making approximately 80 million medical appointments in fiscal year 2011, establishing a clear definition of the desired date or finding and reporting another acceptable measure of wait time is key to understanding how long veterans are actually waiting for medical appointments. Without reliable measurement of how long patients are waiting for medical appointments, VHA is less equipped to identify and address factors that contribute to wait times, or gauge the success of its initiatives to improve access to timely medical appointments, including efforts to improve primary care medical appointments. More consistent adherence to VHA's scheduling policy and oversight of the scheduling process, as well as the allocation of staffing resources in accordance with clinics' demands for scheduling of medical appointments, would potentially reduce medical appointment wait times. Furthermore, persistent problems with telephone access must be resolved to assure veterans' ability to schedule timely medical appointments. Ultimately, VHA's ability to ensure and accurately monitor access to timely medical appointments is critical to ensuring quality health care to veterans, who may have medical conditions that worsen if access is delayed. Recommendations for Executive Action: To ensure reliable measurement of veterans' wait times for medical appointments, we recommend that the Secretary of VA direct the Under Secretary for Health to take actions to improve the reliability of wait time measures either by clarifying the scheduling policy to better define the desired date, or by identifying clearer wait time measures that are not subject to interpretation and prone to scheduler error. To better facilitate timely medical appointment scheduling and improve the efficiency and oversight of the scheduling process, we recommend that the Secretary of VA direct the Under Secretary for Health to take actions to ensure that VAMCs consistently and accurately implement VHA's scheduling policy, including use of the electronic wait list, as well as ensuring that all staff with access to the VistA scheduling system complete the required training. To improve timely medical appointment scheduling, we recommend that the Secretary of VA direct the Under Secretary for Health to develop a policy that requires VAMCs to routinely assess clinics' scheduling needs and resources to ensure that the allocation of staffing resources is responsive to the demand for scheduling medical appointments. To improve timely medical appointments and to address patient and staff complaints about telephone access, we recommend that the Secretary of VA direct the Under Secretary for Health to ensure that all VAMCs provide oversight of telephone access and implement best practices outlined in its telephone systems improvement guide. Agency Comments and Our Evaluation: In reviewing a draft of this report, VA generally agreed with our conclusions and concurred with our recommendations. (VA's comments are reprinted in appendix I.) In summary, VA stated that VHA officials have closely followed our review and proactively taken steps in response to our findings. Specifically, VHA is revising and improving directives, policies, training, clinic management tools, and oversight related to scheduling practices. VA further stated that VHA is committed to routinely assessing clinics' scheduling needs and resources and developing practices and guidelines to ensure adequate staffing resources for scheduling medical appointments. VA described its plans to address each recommendation as follows: * In response to our recommendation that VA take actions to improve the reliability of wait time measures, VA concurred and stated that VHA will revise its scheduling policy to implement more reliable wait time measures and new processes to better define desired date with a targeted completion date of November 1, 2013. * In response to our recommendation that VA take actions to ensure that VAMCs consistently and accurately implement VHA's scheduling policy and ensure that all staff complete required training, VA concurred and stated that the revised scheduling policy will include improvements and standardization of the use of the electronic wait list. Additionally, VHA will require VISNs to update each VAMC's scheduler master list and verify that all schedulers on the list have completed required training, and will require schedulers to complete a standardized training update on the revised scheduling policy. The targeted completion date for these activities is November 1, 2013. * In response to our recommendation that VA develop a policy that requires VAMCs to routinely assess clinics' scheduling needs and resources, VA concurred and stated that VHA will ask VAMCs to routinely assess clinics' availability and ensure staff is distributed to meet access standards in clinics. However, VA has not specified requirements for VAMCs to complete these assessments nor has the agency provided a timeline for this process. Because schedulers are key to ensuring timely appointment scheduling, we believe that VA should establish a targeted completion date for requiring these assessments in policy or guidance. * In response to our recommendation that VA ensure that all VAMCs provide oversight of telephone access and implement best practices outlined in its telephone improvement guide, VA concurred and stated that VHA will require each VISN director to assess current phone service and develop strategic improvement telephone service plans to improve service. Additionally, VHA will identify a process to monitor performance on a quarterly basis for at least 1 year after the assessment. The targeted completion date for the telephone service assessments and plans is March 30, 2013. As arranged with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 28 days after its issuance date. At that time, we will send copies of this report to appropriate congressional committees, the Secretary of Veterans Affairs and other interested parties. In addition, the report is available at no charge on the GAO website at [hyperlink, http://www.gao.gov]. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or draperd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Signed by: Debra A. Draper: Director, Health Care: List of Requesters: The Honorable Jeff Miller: Chairman: Committee on Veterans' Affairs: House of Representatives: The Honorable Karen Bass: House of Representatives: The Honorable Shelley Berkley: House of Representatives: The Honorable Howard L. Berman: House of Representatives: The Honorable Brian P. Bilbray: House of Representatives: The Honorable Mary Bono Mack: House of Representatives: The Honorable Ken S. Calvert: House of Representatives: The Honorable John Campbell: House of Representatives: The Honorable Lois Capps: House of Representatives: The Honorable Judy Chu: House of Representatives: The Honorable Jim Costa: House of Representatives: The Honorable Susan Davis: House of Representatives: The Honorable David Dreier: House of Representatives: The Honorable Elton Gallegly: House of Representatives: The Honorable Joe Heck: House of Representatives: The Honorable Duncan D. Hunter: House of Representatives: The Honorable Darrell Issa: House of Representatives: The Honorable Jerry Lewis: House of Representatives: The Honorable Kevin McCarthy: House of Representatives: The Honorable Howard P. McKeon: House of Representatives: The Honorable Gary Miller: House of Representatives: The Honorable Grace Napolitano: House of Representatives: The Honorable Laura Richardson: House of Representatives: The Honorable Dana Rohrabacher: House of Representatives: The Honorable Lucille Roybal-Allard: House of Representatives: The Honorable Ed Royce: House of Representatives: The Honorable Loretta Sanchez: House of Representatives: The Honorable Adam Schiff: House of Representatives: The Honorable Brad Sherman: House of Representatives: The Honorable Henry A. Waxman: House of Representatives: [End of section] Appendix I: Comments from the Department of Veterans Affairs: Department of Veterans Affairs: Washington DC 20420: December 11, 2012: Ms. Debra A. Draper: Director, Health Care: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Ms. Draper: The Department of Veterans Affairs (VA) has reviewed the Government Accountability Office's (GAO) draft report, "VA Health Care: Reliability of Reported Outpatient Medical Appointment Wait Times and Scheduling Oversight Need Improvement" (GAO-13-130). VA generally agrees with GAO's conclusions and concurs with GAO's recommendations to the Department. The enclosure specifically addresses GAO's recommendations and provides general comments to the draft report. VA appreciates the opportunity to comment on your draft report. Sincerely, Signed by: John R. Gingrich: Chief of Staff: Enclosure: [End of letter] Enclosure: Department of Veterans Affairs (VA) Comments to Government Accountability Office (GAO) Draft Report "VA Health Care: Reliability of Reported Outpatient Medical Appointment Wait Times and Scheduling Oversight Need Improvement" (GA0-13-130): Recommendation 1: To ensure reliable measurement of veterans' wait times for medical appointments, we recommend that the Secretary of VA direct the Under Secretary of Health to take actions to improve the reliability of wait time measures either by clarifying the scheduling policy to better define the desired date, or by identifying clearer wait time measures that are not subject to interpretation and prone to scheduler error. VA Comment: Concur. The Veterans Health Administration (VHA) will revise its scheduling policy to implement more reliable wait time measures and new processes to better define desired date. Changes currently under review include: a. Measure "new patient wait time" as the time between the appointment create date and the appointment completed date. b. Prospectively measure "established patient wait time' from the patient desired date to the future scheduled appointment date. c. Use the patient and provider agreed upon date established in the exam room and documented in the Computerized Patient Record System electronic order to determine desired date for established patients returning for future visits. Pilot studies have found this process is not prone to interpretation or scheduler error. The response to Recommendation 2 describes plans for scheduler training to ensure policy revisions are correctly implemented. Targeted Completion Date: November 1, 2013. Recommendation 2: To better facilitate timely medical appointment scheduling and improve the efficiency and oversight of the scheduling process, we recommend that the Secretary of VA direct the Under Secretary of Health to take actions to ensure that VAMCs consistently and accurately implement VHA's scheduling policy, including use of the electronic wait list, as well as ensuring that all staff with access to the VistA scheduling system complete the required training. VA Comment: Concur. VHA's revised scheduling policy will include improvements and standardization of the use of electronic wait lists (EWL). In regard to training, VHA will require Veterans Integrated Service Networks (VISN) to verify and update each Facility Scheduler Master List. VHA will then require all schedulers to complete a standardized training update on the new procedures no later than (NLT) April 1, 2013. VA's Talent Management System (TMS) will be used to verify that all schedulers noted on the Master List of Schedulers have completed the mandated training. Training will include instructions about use of EWLs and the new scheduling requirements in the revised scheduling policy. Targeted Completion Date: November 1, 2013. Recommendation 3: To improve timely medical appointment scheduling, we recommend that the Secretary of VA direct the Under Secretary of Health to develop a policy that requires VAMCs to routinely assess clinics' scheduling needs and resources to ensure that the allocation of staffing resources is responsive to the demand for scheduling medical appointments. VA Comment: Concur. VHA will ask Medical Centers to routinely assess clinics' availability and ensure staff is distributed to meet access standards in our clinics. VHA will also revise and implement improved clinic management tools, such as an Access Index Report of key clinic operational metrics. Once the tools are developed, VHA will conduct training to assist clinical managers in making operational management decisions. Recommendation 4: To improve timely medical appointments and to address patient and staff complaints about telephone access, we recommend that the Secretary of VA direct the Under Secretary of Health to ensure that all VAMCs provide oversight of telephone access and implement best practices outlined in its telephone systems improvement guide. VA Comment: Concur. The DUSHOM will require each VISN Director to assess current phone service and develop strategic improvement telephone service plans (including milestones and timelines for implementation} to improve service. VISN and VA Medical Center (VAMC) leadership will use best practices, including those outlined in the VHA telephone systems improvement guide, to develop and implement these strategic improvement telephone service plans. VAMCs will also follow the policy related to providing telephone service for clinical care as outlined in VHA Directive 2007-033, Telephone Service for Clinical Care. The DUSHOM will identify a process to monitor performance on a quarterly basis for at least one year after the assessment. Targeted Completion Date: Assessments and plans including milestones and timelines to be completed NLT March 30, 2013, with monitoring of the assessments and implementation of improvements to be monitored for one year after the assessment. General Comments: The Veterans Health Administration (VHA) has identified that updates to our scheduling practices are needed. VHA officials have also closely followed the Government Accountability Office (GAO) review and proactively taken steps in response to GAO's findings. In our own efforts and in response to GAO, VHA is revising and improving directives, policies, training, clinic management tools, and oversight related to scheduling practices. VHA is unequivocally committed to providing the best care possible for Veterans and will act rapidly on all findings that may improve Veterans' access to health care across the system. VHA wants to emphasize that our updates to scheduling practices are more than a one-time effort. We are committed to routinely assessing clinics' scheduling needs and resources and developing practices and guidelines to ensure there are adequate staffing resources to be responsive to the demand for scheduling medical appointments. [End of section] Appendix II: GAO Contact and Staff Acknowledgments: GAO Contact: Debra A. Draper, (202) 512-7114 or draperd@gao.gov: Staff Acknowledgments: In addition to the contact named above, Bonnie Anderson, Assistant Director; Rebecca Abela; Jennie Apter; Rich Lipinski; Sara Rudow; and Ann Tynan made key contributions to this report. [End of section] Related GAO Products: VA Mental Health: Number of Veterans Receiving Care, Barriers Faces, and Efforts to Increase Access. [hyperlink, http://www.gao.gov/products/GAO-12-12]. Washington, D.C.: October 14, 2011. Information Technology: Department of Veterans Affairs Faces Ongoing Management Challenges. [hyperlink, http://www.gao.gov/products/GAO-11-663T]. Washington, D.C.: May 11, 2011. Information Technology: Management Improvements Are Essential to VA's Second Effort to Replace Its Outpatient Scheduling System. [hyperlink, http://www.gao.gov/products/GAO-10-579]. Washington, D.C.: May 27, 2010. VA Health Care: Access for Chattanooga-Area Veterans Needs Improvement. [hyperlink, http://www.gao.gov/products/GAO-04-162]. Washington, D.C.: January 30, 2004. VA Health Care: More National Action Needed to Reduce Waiting Times, but Some Clinics Have Made Progress. [hyperlink, http://www.gao.gov/products/GAO-01-953]. Washington, D.C.: August 31, 2001. Veterans' Health Care: VA Needs Better Data on Extent and Causes of Waiting Times. [hyperlink, http://www.gao.gov/products/GAO/HEHS-00-90]. Washington, D.C.: May 31, 2000. [End of section] Footnotes: [1] Generally a CBOC or ambulatory care center is defined as a fixed health care site that is geographically distinct or separate from its parent VAMC. All CBOCs and ambulatory care centers generally provide primary care, and some may provide specialty care; services can vary by individual CBOC or ambulatory care center. [2] Outpatient clinics offer services to patients that do not require a hospital stay. Primary care addresses patients' routine health needs and specialty care is focused on a specific specialty service such as orthopedics or dermatology. A "clinic" may be defined as an entity for dividing provider workload and scheduling different types of patient care appointments. A particular area of care, such as primary care or specialty care may have multiple clinics that vary in purpose and size. A VAMC can provide care in each area at the VAMC or its affiliated CBOCs and ambulatory care centers; for example, primary care could be provided through multiple primary care clinics at a VAMC's different locations. Throughout the report we refer to a specific area of care at a specific location as a "clinic." [3] Throughout the report we will use the term "medical appointments" to refer to outpatient medical appointments. [4] GAO, VA Health Care: More National Action Needed to Reduce Waiting Times, but Some Clinics Have Made Progress, [hyperlink, http://www.gao.gov/products/GAO-01-953] (Washington, D.C.: Aug. 31, 2001). [5] Department of Veterans Affairs, Office of Inspector General, Audit of the Veterans Health Administration's Outpatient Waiting Times, Report No. 07-00616-199 (Washington, D.C.: Sept. 10, 2007). [6] Department of Veterans Affairs, Office of Inspector General, Veterans Health Administration: Review of Veterans' Access to Mental Health Care, Report No. 12-00900-168 (Washington, D.C.: Apr. 23, 2012). [7] According to VHA's scheduling policy, the desired appointment date, referred to as the "desired date," is the date on which the patient or provider wants the patient to be seen. [8] VA assigns each VAMC a complexity score derived from multiple variables to measure facility complexity arrayed along four categories, namely patient population served, clinical services offered, education and research complexity, and administrative complexity. [9] We visited the highest volume CBOC for three VAMCs and the second highest volume for the fourth VAMC. [10] CBOCs and ambulatory care centers provide outpatient primary care and may provide specialty care services. Oversight for facility functions--including scheduling--occurs at the VAMC level. CBOCs and ambulatory care centers generally are geographically distinct from, but operate under the supervision of a parent VAMC, which maintains administrative responsibility. [11] During our site visits, we visited only outpatient clinics. Some CBOCs did not have specialty care clinics. From this point forward, we use VAMC to refer collectively to the VAMC and all of its affiliated CBOCs and ambulatory care centers. [12] Each of VA's 21 VISNs is responsible for managing and overseeing medical facilities within a defined geographic area. [13] VHA outpatient medical appointment scheduling policy is documented in VHA Directive 2010-027, VHA Outpatient Scheduling Processes and Procedures (June 9, 2010). We refer to the directive as "VHA's scheduling policy" from this point forward. [14] We refer to clerical or administrative support staff with scheduling responsibilities as "schedulers." We did not complete a scheduler interview for 4 of the 14 specialty clinics that we visited because either the clinic did not have a scheduler or it would have caused delays in patient care to take a scheduler off duty for an interview. [15] VHA Support Services Center maintains an internal VHA website that allows central office-, VISN-, and VAMC-level staff to access reports on a variety of topics including medical appointment data. [16] The electronic wait list is a type of computer software application designed for recording, tracking, and reporting veterans waiting for medical appointments. [17] VHA uses non-VA care to reduce wait times and backlogs and to provide veterans access to specialists not available through VHA. [18] From this point forward, the VistA medical appointment scheduling system will be referred to as the VistA scheduling system. [19] Consults--generally requests for specialty care appointments--are most often communicated electronically through an application in the electronic medical record within VistA. [20] VHA Directive 2007-033, Telephone Service for Clinical Care (Oct. 11, 2007). [21] To obtain VHA healthcare services, veterans generally must enroll with VHA and register at a specific VAMC. [22] Specifically, VAMCs are required to maintain a list of all staff who can schedule medical appointments in the VistA scheduling system and VAMC directors are required to ensure successful completion of required training by all staff on the list. Schedulers are not to be allowed to schedule medical appointments in the VistA scheduling system without proof of their successful completion of this training. [23] For fiscal year 2011, the most recent certification available at the time of our review, VHA Systems Redesign collected VAMC directors' certification through a web-based template. As part of the certification, VAMC directors certify that they have completed, using VISN-approved processes and procedures, a yearly standardized audit of schedulers on the timeliness and appropriateness of scheduling actions and the accuracy of desired dates. [24] GAO, Information Technology: Management Improvements Are Essential to VA's Second Effort to Replace Its Outpatient Scheduling System, [hyperlink, http://www.gao.gov/products/GAO-10-579] (Washington, D.C.: May 27, 2010). [25] VHA also has a goal of scheduling compensation and pension examinations within 30 days. Compensation and pension examinations may be provided to veterans to establish a claim for disability compensation; appointment wait times for these appointments are outside the scope of this report. [26] Directors' performance contracts include measures against which directors are rated at the end of the fiscal year, which determine their performance pay. The contracts include system-wide performance measures, as well as individualized performance measures that are selected based on specific problems or needs of the respective VISN or VAMC. [27] VA prepares a congressional budget justification that provides details supporting the policy and funding decisions in the President's budget request submitted to Congress prior to the beginning of each fiscal year. The budget justification articulates what VA plans to achieve with the resources requested; it includes performance measures by program area. VA also publishes an annual PAR, which contains performance targets and results achieved against those targets in the previous year. [28] VHA officials told us the department is working to better coordinate consistency of the performance measures. [29] For example, the fiscal year 2011 NDPP and FDPP included a measure "percent of patients waiting for a primary care appointment longer than 14 days from the desired date" instead of the primary care measures included in fiscal year 2012.The fiscal year 2010 NDPP and FDPP included the measure "percent of patients waiting for a primary care appointment longer than 30 days from the desired date." In addition, the fiscal year 2011 PAR included three rather than four wait time measures that did not break out the new and established patients for primary and specialty care; the fiscal year 2012 PAR included separate measures for new and established patients. The fiscal year 2010 PAR also included three measures, one of which measured the "percent of new patient appointments completed within 30 days of the appointment create date." [30] According to a VHA official, there were more than 50,000 staff across VHA who could schedule appointments at the time of our review. [31] Because the policy and training documents are unclear about when the desired date is defined by the patient or defined by the provider, we only identified errors related to aspects of the policy and training regarding how to determine and record the desired date that hold true despite the ambiguity. [32] For reporting on the PAR, VHA measures medical appointment wait times as the number of days between the desired date and appointment date. [33] This is based on a measure similar to the performance measure "percent of new patient specialty care medical appointments completed within 14 days of the desired date," reported on the PAR. [34] Clinic capacity is the supply of available future appointments. Capacity measures typically count the number of days between the day the measure is taken and the day the first or third next available appointment occurs. [35] For example, in the 2010 PAR, VA reported a wait time performance measure for new patient medical appointments based on days from the date the appointment was created. The appointment create date is automatically generated in the VistA scheduling system and therefore not prone to scheduler error. VHA officials told us that wait time measures based on create date do not reflect patient preference and therefore can incorrectly characterize wait times, particularly for established patient follow-up appointments which may be scheduled months before they are completed. [36] Since 2002, VHA has measured veterans' perceived access through a monthly Survey of Health Experiences of Patients --a survey of satisfaction with inpatient and outpatient care similar to the Department of Health and Human Services Consumer Assessment of Healthcare Providers Survey. [37] Specifically, those measures were: Getting Needed Care--combines responses from questions regarding how much of a problem, if any, patients had with various aspects of getting needed care; and Getting Care Quickly--combines responses from questions regarding how often patients received various types of care in a timely manner. [38] Consults are most often communicated electronically through an application in the electronic medical record within VistA. The electronic medical record is separate from the scheduling system. [39] According to VHA's scheduling policy, the electronic wait list is used to keep track of patients with whom the provider does not yet have an established relationship and who cannot be scheduled for appointments in target time frames. No other wait list formats (such as paper or electronic spreadsheets) are to be used for tracking requests for medical appointments. [40] Patients are entered into the recall/reminder software for the date they are to return to the clinic--which should be identified by the provider--and VistA automatically generates correspondence to the patient (post card or letter) a week or 2 prior to that date to remind the patient to call the clinic and schedule a medical appointment. [41] VHA's scheduling policy states that for clinics to most efficiently operate, "schedules must be open and available for the patient to make [medical] appointments at least three to four months into the future. Permissions may be given to schedulers to make appointments beyond these limits when doing so is appropriate and consistent with patient or provider requests. Blocking the scheduling of future [medical] appointments by limiting the maximum days into the future an appointment can be scheduled is inappropriate and is disallowed." [42] VAMCs are required to maintain a list of all staff who can schedule medical appointments in the VistA scheduling system, and are required to ensure successful completion of required VHA scheduler training by all staff on that list. Schedulers are not to be allowed to schedule medical appointments in the VistA scheduling system without proof of their successful completion of this training. [43] Of the 144 VAMCs that completed the certification for fiscal year 2011, 109 certified overall compliance, 27 certified partial compliance, and 8 certified noncompliance. [44] For example, a physical therapist may have a separate schedule for amputee clinic, general physical therapy, or other types of services. [45] The electronic medical record is a component of the VistA system that includes patient health information and enables providers to record notes, such as when the provider would like to see the patient for a follow-up appointment, and place orders for procedures, x-rays, and laboratory tests, among other things. [46] In October 2012, VA announced a contest seeking proposals for a new medical appointment scheduling system from commercial software developers. The contest is intended to reduce risks in the future procurement and implementation of a new scheduling system. [47] VHA's policy on telephone service for clinical care, VHA Directive 2007-033, establishes VHA's policy of providing telephone access for appointment management and continuous access to health care advice. The telephone directive also establishes recommended benchmarks for telephone service at VA facilities. VAMCs differ in how they manage the telephones; for example, some VAMCs establish VAMC- wide call centers to answer and direct incoming calls. VAMC telephone systems generally serve the VAMC including its affiliated CBOCs and ambulatory care centers. [48] Veterans Health Administration, Telephone Systems Improvement Guide, Second Edition (December 2011). [49] Each VAMC has a patient advocate who accepts and addresses patient complaints. The patient advocate records complaints in the patient advocate tracking system and tracks complaints in various categories, including "phone calls not returned, letters not answered." Two of the four VAMCs provided information on patient complaints for fiscal year 2012 from October 1, 2011, through May 31, 2012. One VAMC provided information for fiscal year 2012 through June 25, 2012, and the other provided information for fiscal year 2012 through August 31, 2012. [50] Veterans Health Administration, Telephone Systems Improvement Guide. [51] VHA officials noted that even in primary care clinics in which PACT implementation has begun, some PACT teams are not yet fully staffed in accordance with the model. [52] In contrast to the PACT 7-day wait time measure reported in the NDPP and FDPP, VA reported primary care wait time performance as the completion of appointments within 14 days of desired date in its fiscal year 2012 PAR. [53] There are additional measures in the monthly internal data report. [54] VISN and VAMC directors were scored for each of the five PACT measures listed and rated against a sixth composite PACT measure. They were rated as meeting the composite PACT measure if they met targets for three of the five individual PACT measures. [55] Telehealth is the delivery of health care services using telecommunications technology. Using technology such as videoconferencing, telehealth changes the location where health care services are delivered. Secure messaging is VHA's web-based message service that allows patients to communicate nonemergency health- related information with their health care team. [56] Kim Nazi, Department of Veterans Affairs Experiences with System- wide Transformation Activities that Foster Continuous Learning and Improvement, Institute of Medicine Consensus Study on the Learning Healthcare System in America (May 2012). [57] Non-VA care is medical care paid for by VA but provided to veterans outside of VA. Non-VA care may be offered on a temporary basis to a veteran when medical services are not available due to a lack of available VA specialists, long wait times, or when VA care is only available at extraordinary distances from a veteran's home. VAMCs do not track wait times for patients using non-VA care. [58] See Pub. L. No. 110-387, § 403, 122 Stat. 4110, 4124 (2008). Veterans are eligible to participate in the program if they reside in a location where a pilot site is located and if they are enrolled in VA health care when the program starts, and meet any of the following criteria: live more than (1) 60 miles driving distance from the nearest VA health care facility providing primary care services, if the veteran is seeking such services; (2) 120 miles driving distance from the nearest VA health care facility providing acute hospital care, if the veteran is seeking such care; or (3) 240 miles driving distance from the nearest VA health care facility providing tertiary care, if the veteran is seeking such care. Nonenrolled veterans who are eligible to enroll in VA health care because they served in a combat theater after November 11, 1998, are also eligible to participate in the program. Health care delivery contracts for services covered under Project ARCH were awarded to Humana Veterans, a health services support contractor, in four pilot sites, and to a health care provider in the fifth pilot site. [59] Project ARCH services are currently being piloted at five sites, including Northern Maine; Farmville, Virginia; Pratt, Kansas; Flagstaff, Arizona; and Billings, Montana. [60] VA is required to evaluate the program and prepare an annual report to Congress for each of the 3 years of the pilot. VHA engaged a contractor to conduct site visits and provide VHA with quarterly progress reports on Project ARCH implementation. The first progress report for Montana VAMC's Project ARCH program was produced in April 2012. [End of section] GAO’s Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select “E-mail Updates.” Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, DC 20548. [End of document]