Skip to main content

Veterans' Disability Benefits: Improvements Could Further Enhance Quality Assurance Efforts

GAO-15-50 Published: Nov 19, 2014. Publicly Released: Nov 19, 2014.
Jump To:
Skip to Highlights

Highlights

What GAO Found

The Veterans Benefits Administration (VBA)—within the Department of Veterans Affairs—measures and reports the accuracy of its disability compensation claim decisions in two ways: (1) by claim and (2) by disabling condition, though its approach has limitations. When calculating accuracy rates for either measure through its Systematic Technical Accuracy Review (STAR), VBA does not always follow generally accepted statistical practices, resulting in imprecise performance information. For example, VBA does not adjust its accuracy estimates to reflect that it samples the same number of claims for review from each regional office—despite their varying workloads—and thus produces imprecise estimates of national and regional accuracy. Further, VBA reviews about 39 percent (over 5,000) more claims nationwide than is necessary to achieve its desired precision in reported accuracy rates, thereby diverting limited resources from other important quality assurance activities, such as targeted reviews of error-prone cases. In addition to issues with its statistical practices, VBA's process for selecting claims for STAR review creates an underrepresentation of claims that are moved between regional offices, which may inflate accuracy estimates because these claims have had historically lower accuracy rates. Finally, VBA has not clearly explained in public reports the differences in how its two accuracy measures are calculated or their associated limitations, as suggested by best practices for federal performance reporting.

VBA has taken steps to enhance and coordinate its other quality assurance activities, but GAO found shortcomings in how VBA is implementing and evaluating these activities. To improve local accuracy, VBA created regional office quality review teams (QRTs) with staff dedicated primarily to performing local accuracy reviews. QRTs assess individual claims processor performance and conduct special reviews to forestall certain types of errors. In addition, VBA began using questionnaires for assessing decision-making consistency, which are more efficient to administer than VBA's prior approach to conducting consistency studies. VBA also coordinates quality assurance efforts by disseminating national accuracy and consistency results, trends, and related guidance to regional offices for use in training claims processors. Further, VBA uses STAR results to inform other quality assurance activities, such as focusing certain QRT reviews on commonly made errors. However, GAO identified implementation shortcomings that may detract from the effectiveness of VBA's quality assurance activities. For example, contrary to accepted practices for ensuring the clarity and validity of questionnaires, VBA did not pre-test its consistency questionnaires to ensure the clarity of questions or validity of the expected results, although VBA officials indicated that they plan to do so for future questionnaires. In contrast with federal internal control standards that call for capturing and distributing information in a form that allows people to efficiently perform their duties, staff in the four regional offices that we visited had trouble finding the guidance they needed to do their work, which could affect the accuracy as well as the speed with which staff decide claims. Federal standards also call for knowing the value of efforts such as quality assurance activities and monitoring their performance over time; however, VBA has not evaluated the effect of its special QRT reviews or certain consistency studies on improving targeted accuracy rates, and lacks clear plans to do so.

Why GAO Did This Study

With a backlog of disability compensation claims, VBA faces difficulties in improving the accuracy and consistency of the claim decisions made by staff in its 57 regional offices. To help achieve its goal of 98 percent accuracy by fiscal year 2015, VBA recently implemented a new way of measuring accuracy and changed several quality assurance activities to assess the accuracy and consistency of decisions and to provide feedback and training to claims processors. GAO was asked to examine VBA's quality assurance activities.

This report evaluates (1) the extent to which VBA effectively measures and reports the accuracy of its disability compensation claim decisions and (2) whether VBA's other quality assurance activities are coordinated and effective. GAO analyzed VBA claims and STAR accuracy data from fiscal year 2013 (the most recent fiscal year for which complete data are available); reviewed relevant federal laws, VBA guidance, and other documents relevant to quality assurance activities; and interviewed VBA staff from headquarters and four VBA regional offices (selected to achieve variety in geography, workload, and accuracy rates), as well as veteran service organization officials.

Recommendations

GAO is making eight recommendations to VA to improve its measurement and reporting of accuracy, review the multiple sources of policy guidance available to claims processors, enhance local data systems, and evaluate the effectiveness of quality assurance activities. VA concurred with all of GAO's recommendations.

Recommendations for Executive Action

Agency Affected Recommendation Status
Department of Veterans Affairs To help improve the quality of VBA's disability compensation claim decisions, the Secretary of Veterans Affairs should direct the Under Secretary for Benefits to take steps to ensure that redistributed claims and those moved between regional offices are not underrepresented in the STAR sample.
Closed – Implemented
In February 2015, VBA changed its sample selection methodology to ensure that redistributed claims are not underrepresented by selecting claims on the basis of regional office completion regardless of whether the claim was redistributed.
Department of Veterans Affairs To help improve the quality of VBA's disability compensation claim decisions, the Secretary of Veterans Affairs should direct the Under Secretary for Benefits to review the multiple sources of policy guidance VBA provides to determine ways to consolidate them or otherwise improve their availability and accessibility for use by staff in regional offices.
Closed – Implemented
According to VA's December 2015 update, VBA has consolidated its policy and procedural guidance into a single source available to regional office staff via the agency's intranet. Officials stated that the M21-1 Adjudication Procedures Manual has improved search capabilities and is the authoritative source for claims processing guidance. The manual's content is also available to external stakeholders through a separate link, KnowVA (www.knowva.ebenefits.va.gov), and will be updated simultaneously.
Department of Veterans Affairs To help improve the quality of VBA's disability compensation claim decisions, the Secretary of Veterans Affairs should direct the Under Secretary for Benefits to increase transparency in explaining how the claim-based and issue-based accuracy rates are calculated as well as their key limitations when publicly reporting these metrics.
Closed – Implemented
VBA has created an abstract that describes its sample design, how it calculates accuracy estimates, and assessment criteria for claim-based and issue-based errors. This abstract is publicly available on VBA's performance website. VBA reported in December 2015 that this abstract now includes an explanation of why the issue-based accuracy estimate is higher than the claim-based accuracy estimate, to help users make meaningful comparisons between the two estimates.
Department of Veterans Affairs To help improve the quality of VBA's disability compensation claim decisions, the Secretary of Veterans Affairs should direct the Under Secretary for Benefits to take steps to ensure that any future upgrades to local data systems allow QRTs to pause the claims process when errors are detected and enable QRTs to better track error trends.
Closed – Not Implemented
In April 2019, VBA reported that according to data from its Quality Management System (QMS) system, 81.4 percent of individual quality reviews do not result in a critical error (i.e., an error that affects benefit payments). As a result, VBA determined that it was not cost effective to add functionality to the Veterans Benefit Management System (VBMS) to pause the claims process when an individual quality review identifies an error. Instead, VA reported that the agency is focused on ensuring that individual quality reviews are conducted in a timely way, and that claims processors are notified of errors prior to the Veteran being provided incorrect benefits or information. Further, VA reported that the QMS system already includes a feature to notify employees within 2 hours if errors are found through a quality review, and that supervisors and quality review staff are also alerted to errors and are required to track each error until it is corrected. As a result, the agency determined that additional functionality to pause claims is not necessary for addressing errors. VA is focusing on alternative approaches to further ensuring timely correction of errors prior to veterans receiving incorrect benefit information.
Department of Veterans Affairs To help improve the quality of VBA's disability compensation claim decisions, the Secretary of Veterans Affairs should direct the Under Secretary for Benefits to take additional steps to evaluate the effectiveness of quality assurance activities to identify opportunities to improve or better target these activities.
Closed – Implemented
As of September 2018, VA reported that VBA is using the Quality Management System (QMS) to capture data from all types of quality reviews at various stages of the claims process, to include local regional office individual quality reviews, in-process reviews, national STAR reviews and consistency studies. According to VA's QMS user guide, QMS provides a single unified source of claims quality information that can be used for multiple purposes, including analysis and process improvements. For example, VBA reported that in fiscal year 2018, the agency's consistency studies were designed to target error trends identified in QMS data and other sources. In addition, the agency reported using data on deferrals (i.e., claims that have to be re-worked due to errors) from its electronic claims processing system, Veterans Benefit Management System (VBMS) to target in-process reviews and develop training to help claims processors with multiple deferrals improve their performance. According to VA, QMS has been functional since July 2017 and has captured over 130,000 quality reviews.
Department of Veterans Affairs To help improve the quality of VBA's disability compensation claim decisions, the Secretary of Veterans Affairs should direct the Under Secretary for Benefits to leverage appropriate expertise to help VBA weigh its accuracy estimates to reflect the sample design for reviewed claims.
Closed – Implemented
VA reported in July 2015 that VBA finalized a statistical program to reflect its approach to selecting claims by regional office, which vary in size. VBA now weights its accuracy estimates to account for this.
Department of Veterans Affairs To help improve the quality of VBA's disability compensation claim decisions, the Secretary of Veterans Affairs should direct the Under Secretary for Benefits to leverage appropriate expertise to help VBA determine and report the confidence intervals associated with its reported accuracy estimates.
Closed – Implemented
VA reported that VBA finalized a statistical program to calculate weighted claim-based accuracy estimates, and has modified its Monday Morning Workload Report (MMWR) to report margin of error associated with regional office and national claim-based accuracy estimates. In February 2016, VBA reported claim-based and issue-based accuracy estimates in VA?s 2017 President?s Budget Submission and in VA?s Annual Performance Plan and Report, and included margins of error for these estimates. In addition, in April 2016, VBA updated its Performance website with additional information about its accuracy estimates. The website now states that VBA?s quality assurance sample design is designed to yield estimates within a 5 percent margin of error and a confidence level of 95 percent.
Department of Veterans Affairs To help improve the quality of VBA's disability compensation claim decisions, the Secretary of Veterans Affairs should direct the Under Secretary for Benefits to leverage appropriate expertise to help VBA re-examine its approach to calculating the regional office sample size for STAR.
Closed – Implemented
Effective February 2015, VBA changed its approach for calculating the regional office sample size for its Systematic Technical Accuracy Review (STAR).

Full Report

GAO Contacts

Office of Public Affairs

Topics

Best practicesClaims processingEmployee trainingDisability benefitsInternal controlsPerformance measuresQuality assuranceReporting requirementsStandardsVeterans benefitsVeterans disability compensationPolicies and procedures