Social Security Administration: More Effort Needed to Assess Consistency of Disability Decisions

SSA no longer conducts the biennial case review, but planned to take several steps to expand its assessment of the areas and causes of inconsistency in decisions between the state and federal adjudication levels as it implemented major changes to its disability decision making process. SSA began implementing its new Disability Service Improvement (DSI) process in August 2006 in its Boston Region. The agency has evaluated the Boston results in preparation for a national rollout decision. Several of the DSI process changes, if adopted, could provide the agency with information on decision inconsistency in line with efforts undertaken as part of its biennial review. For example, state and federal decision makers plan to use standardized forms to consistently document the rationale for their decisions, addressing areas that have contributed to errors and inconsistency. The forms and feedback loops would have allowed decision makers to learn why their decisions were reversed or remanded. Once in full use, SSA's Office of Quality Performance (OQP) planned to collect and analyze the content of the forms to identify problem areas and provide monthly reports to help managers improve the quality and consistency of decision making. OQP has also performed analyses of decisions made under DSI at the new federal level to identify factors associated with the allowance of cases which were denied at the prior state level. SSA also indicated that the Disability Review Board envisioned under DSI would be in a good position to provide feedback regarding decisions made by adjudicators at the hearings and prior levels. Although the agency has discontinued DSI and the associated pilot, it told us that it had other plans underway to conduct in-depth multilevel case analyses to identify inconsistencies between adjudication levels. As of September 2008, the agency has not conducted any new multi-level analyses or provided any information or results on decisional inconsistency that may have resulted from the DSI pilot.

In moving forward with the SSA Commissioner's new approach to disability decision making, the agency planned to develop, but has not implemented, a more sophisticated approach to analyzing the significant case characteristics that contribute to inconsistency in decision making between adjudication levels. For example, SSA planned to develop a sophisticated predictive model (using data mining techniques) to identify error-prone cases for review by the Decision Review Board envisioned under its Disability Service Improvement (DSI) initiative that would replace SSA's Appeals Council. During the early implementation in the Boston region, the DRB planned to review all Adjudicative Law Judge (ALJ) decisions to develop and validate the model as it looks for the trends and causes of error-prone decisions. To further support its new DSI initiative, SSA planned to change other aspects of its quality assurance system to promote the quality and consistency of decisions, such as the creation of a central quality assurance review to eliminate the potential for differences among regional reviews and the use of standardized electronic tools for decision writing with feedback loops and quality assurance analysis to identify and address inaccurate and inconsistent decisions. SSA has also told us that it had plans underway to conduct new in-depth multi-level case analyses to identify inconsistencies across adjudication levels. However, as of September 2008, SSA has not begun conducting any new multi-level analyses or provided any information on decisional inconsistency identified during DSI initiative, which was discontinued before being fully implemented.

Under the prior SSA Commissioner, a new approach to decision making, the Disability Service Improvement (DSI) initiative, was tested. Under DSI, a new Decision Review Board (DRB) would review a sample of Adjudicative Law Judges' (ALJ) decisions--a sample selected using a predictive model for error-prone cases. In reviewing each case, the DRB should have had the ALJ's rationales for his case decision. It was not clear if the DRB would also have the complete case file, including the ALJ's hearing tape, for review purposes. The DRB would have also used the available information to look for trends and causes of inconsistent and error-prone cases. In addition to the DRB, SSA told us that it had plans under way to conduct new in-depth multi-level case analyses to identify inconsistencies across adjudication levels. However, as of September 2008, the agency has not conducted any new multi-level analysis or provided any information on decisional inconsistency identified from the DSI initiative, which was never fully implemented and has been discontinued.

SSA tested a new approach to decision making that included several steps that were intended to allow adjudicators and reviewers an opportunity to more fully evaluate disability cases and understand the causes of inconsistent decisions. Under the new approach, referred to as the Disability Service Improvement (DSI) initiative, state and federal decision makers used a standard decision form to document the rationale for a decision and the reversal of a prior decision. SSA also planned to provide feedback loops to allow prior decision makers to learn why their decisions were reversed. In addition, its Office of Quality Performance planned to collect and analyze the content of the forms to identify problem areas and provide monthly reports to help managers improve the quality and consistency of decision making. Furthermore, the Decision Review Board envisioned under DSI planned to review ALJ decisions to identify trends and causes of errors and inconsistencies. SSA tested its new approach in 2006 in its Boston Region and planned to evaluate the results prior to rolling out changes to its other regions. SSA's Office of Quality Performance also planned to conduct some quality review of federal decisions and evaluate the data for feedback to both federal and state levels. In addition, the agency had new plans to conduct in-depth multi-level cases analysis to identify decision inconsistencies. However, as of September 2008, the agency has not begun conducting any new multi-level analysis or provided any information on decisional inconsistency identified from the DSI initiative, which was never fully implemented and has been discontinued.

As part of its new decision making approach, the Disability Service Improvement (DSI) initiative, the agency planned to collect, assess, and distribute information on the consistency of decisions between adjudication levels. SSA planned to have feedback loops, which would allow its decision makers to understand why their decisions were reversed. It also planned to implement a quality assurance review of the decision rationales, which would feed into monthly internal reports identifying problem areas to help management improve the quality and consistency of decision making. SSA also planned to have additional in-line and end-of-line reviews, but to date had not clarified how it would report the findings of these reviews. Also unclear was the extent to which the agency would keep external stakeholders informed about its progress in addressing consistency in decisions. However, the agency used a proactive and collaborative approach to share information and included internal and external stakeholders in its development and implementation of its new decision making process, which began implementation in one region in August 2006. In 2007, the agency's Office of Quality Performance developed an intranet site to share its quality review analysis with internal stakeholders and planned to provide information to state decision makers via various management meetings. The agency also had plans underway to create a new in-depth, multi-level case analysis to identify inconsistencies between adjudication levels, but had not made it clear how the results of such analysis would be shared with internal and external stakeholders. The DSI initiative was discontinued under the new Commission and SSA has not indicated to us that it has distributed to internal or external stakeholders new information on reasons for inconsistent disability decisions that it may have gleaned from the DSI effort.

SSA planned to improve its collection of management information (MI) to monitor and improve its new decision-making approach, referred to as the Disability Service Improvement (DSI) initiative. As part of this effort, SSA consulted with internal stakeholders and incorporated their needs into its new MI strategy. SSA planned to use information from quality reviews and its electronic disability system, to capture information about decision making and enhance its guidance and training. SSA, however, has not provided any details on how it would accomplish this task. Furthermore, in preparing to implement its new approach, SSA provided training on how to use the new writing tools (along with other aspects of the new process) and planned to use on-line guidance for decision makers to help improve the quality and consistency of decisions. Furthermore, SSA's Office of Quality Performance planned to analyze information from the new on-line writing tools and provide feedback to both state and federal decision makers. However, as of September 2008, SSA has not provided any indication that efforts undertaken as part of DSI have provided any information on decisional inconsistency and has discontinued the DSI initiative.