This is the accessible text file for GAO report number GAO-05-100R entitled 'SSA's Disability Programs: Improvements Could Increase the Usefulness of Electronic Data for Program Oversight' which was released on December 10, 2004. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: Washington, DC 20548: December 10, 2004: The Honorable Jo Anne B. Barnhart: Commissioner, Social Security Administration: Subject: SSA's Disability Programs: Improvements Could Increase the Usefulness of Electronic Data for Program Oversight: Dear Ms. Barnhart: In 2003, we added the federal government's disability programs to our high-risk list in part because of difficulties agencies faced in managing these programs and the expected growth in the rolls as baby boomers reach their disability-prone years. The Social Security Administration (SSA) manages the federal government's two largest disability programs, the Disability Insurance (DI) program and the Supplemental Security Income (SSI) program, which together paid out $91 billion in federal benefits to 11.4 million individuals with disabilities in 2003. To help address management difficulties and prepare for expected growth in the rolls, SSA must have reliable administrative data from its disability decision-making process to adequately understand the population it serves and the possible effect of proposed program changes on this population. However, in a prior study, we identified potential problems with the reliability of SSA's electronic administrative data.[Footnote 1] This report examines (1) the extent to which SSA collects useful and reliable electronic administrative data in order to effectively manage its DI and SSI programs and (2) whether ongoing and planned changes to SSA's computer systems and internal controls will address any weaknesses that we identified. To determine the adequacy of SSA's electronic administrative data for management of its disability programs, we reviewed SSA, Office of the Inspector General (OIG), and our prior reports regarding these data; interviewed users of these data; and performed a literature search on criteria for establishing information databases. To identify the nature and extent of data problems, we interviewed staff responsible for managing and using the data to assess the controls and processes in the disability system, conducted data reliability tests on selected variables in the disability records for calendar year 2003, and reviewed relevant reports by OIG and SSA contractors for their assessment of the data quality. To determine the extent to which SSA has plans or efforts under way to address these problems, we interviewed SSA officials and reviewed SSA documents and OIG reports. We conducted our work between December 2003 and September 2004 in accordance with generally accepted government auditing standards. Results in Brief: While most types of information collected by SSA are useful for program management purposes, the agency lacks sufficient internal controls to ensure that these data are reliable. In its electronic records, SSA collects information related to the claimant, process, and decision-- all of which are important for informing aspects of program management. For example, SSA electronically collects information on the claimant's impairment that is used to identify beneficiaries who should be reviewed for medical improvement and continuing eligibility for disability benefits. However, SSA does not collect some information that would enhance program oversight. For example, without an indicator on claimant's race, SSA lacks a complete basis for assessing the equity of disability decisions. At the same time, SSA collects other information in its electronic disability record that the agency considers of limited value. For example, codes entered for the claimant's prior occupation and industry are based on outdated lists, and the agency acknowledges that the value of that information is therefore limited. Although the types of information SSA collects in its electronic records generally contribute to the management of the disability programs, the accuracy of the records is unknown. For example, inadequate data entry controls permit the entry of invalid impairment codes--codes that do not reflect a meaningful impairment. In addition, inadequate data entry controls have resulted in missing data for some information, such as the claimant's educational level, which is a critical factor in the disability decision and is therefore important in a study of the disability decision-making process. Most important, because SSA's policy does not require that the electronic record be verified against the information in the case file, the agency does not know the extent to which codes that appear valid reflect the claimant's actual information, such as the claimant's actual impairment or education level. While SSA's ongoing and planned changes to computer systems and internal controls may reduce the chance for some inaccuracies, SSA's plans do not provide adequate assurance of the accuracy of its electronic administrative data. SSA's ongoing transition from a paper- based disability system to an electronic one should reduce some of the inaccuracies we identified by limiting the amount of data re-entered at various levels in the process. However, SSA's planned changes will not address data entry problems found by GAO that could be prevented with additional data entry controls. SSA's current plans also do not include an internal control strategy for ensuring that electronic data match the information in the case file, nor do they provide for corrective action when inaccuracies are found. Finally, although SSA has proposed far-reaching changes to its disability decision-making process and is currently reassessing the processes for ensuring the quality of its disability decisions, the agency has not yet made any plans for evaluating the types of information it currently collects and whether other types of information would improve program management and oversight. To improve the value of SSA's electronic administrative data for managing its disability programs, we recommend that the agency establish a cost-effective internal control strategy to ensure that those data are reliable and that they accurately reflect the information in the case file. In addition, the agency should take steps to review the usefulness of the types of information collected and consider whether additional types of information could improve program oversight. In commenting on the draft of this report, SSA generally agreed with our conclusions and recommendations. SSA acknowledged that the report made valid points about steps SSA can take to increase the usefulness of electronic data for program oversight, and that the agency would consider how best to incorporate improvements in its data collection activities. Background: DI and SSI are the two largest federal programs providing cash assistance to people with disabilities. Established in 1956, DI provides monthly payments to workers with disabilities (and their dependents) under the age of 65 who have enough work experience to qualify for disability benefits. Created in 1972, SSI is a means-tested income assistance program that provides monthly payments to adults or children who are blind or who have other disabilities and whose income and assets fall bellow a certain level. To be considered eligible for either program as an adult, a person must be unable to perform any substantial gainful activity because of a medically determinable physical or mental impairment that is expected to result in death or that has lasted or can be expected to last for a continuous period of at least 12 months. The disability determination process is complex, involving more than one office and decision maker. The process begins at a field office, where an SSA representative determines whether a claimant meets the programs' nonmedical eligibility criteria. Claims meeting these criteria are forwarded to a state's disability determination service (DDS) office to determine if a claimant meets the agency's medical eligibility criteria. At the DDS, the disability examiner and the medical or psychological consultants work as a team to analyze a claimant's documentation, gather additional evidence as appropriate, and approve or deny the claim. A denied claimant may ask the DDS for a reconsideration of its finding, at which point a different DDS team will review the claim.[Footnote 2] If the claim is denied again, the claimant may appeal the determination to SSA's Office of Hearings and Appeals (OHA), where an administrative law judge (ALJ) will review it. The ALJ usually conducts a hearing in which the claimant and others may testify and present new evidence. In making the disability decision, the ALJ uses information from the hearing and from the state DDS, including the findings of the DDS's medical consultant. A claimant whose appeal is denied may request a review by SSA's Appeals Council and, if the claim is denied again, may file suit in federal court. At each step of the disability determination process, information collected is entered into the claimant's case file as well as into an electronic database. SSA field office staff create the case file, which is the legal record for the disability determination and contains all documents related to the case. If the claimant meets the nonmedical eligibility criteria, the field office staff forwards the case file to the appropriate state DDS office. DDS staff add the information collected at the DDS level and the initial disability determination to the case file and enter selected data from the case file into an electronic record known as the Form 831. Once the DDS submits these 831 records to SSA, the records become part of SSA's 831 file.[Footnote 3] If the claimant asks for a reconsideration, another DDS team will place the new information in the case file and enter selected data from the case file into a new 831 record reflecting the reconsideration decision, which will also become part of SSA's 831 file. If the claimant appeals the reconsideration by requesting a hearing by an administrative law judge at the Office of Hearings and Appeals, the ALJ conducts a new review of the claimant's file and any additional evidence the claimant submitted since the DDS determination. The ALJ may also hear testimony from medical or vocational experts and the claimant regarding the claimant's medical condition and ability to work. As with the DDS, OHA staff place the new information in the case file and enter selected data from the case file into an electronic record. These electronic records become part of a nationwide electronic database referred to as the Case Control System (CCS) database. It has been well established that adequate internal controls over electronic data systems are necessary to effectively manage programs, such as SSA's disability programs. In 1998, the Social Security Advisory Board noted that it is essential for policy makers have accurate, balanced, and objective information to help them determine the extent to which the program is meeting the long-standing objectives of social adequacy and individual equity, the nature and extent of changes that may be needed, and the impact of proposals for change. SSA stated in its 2003 Performance and Accountability Report that the agency was committed to providing clear and reliable data to those who use the data for managing, decision making, and oversight of SSA's programs. Finally, GAO's standards state that adequate internal controls should be established to provide reasonable assurance that agency objectives are being achieved. These internal control standards define the minimum level of quality acceptable for internal controls in the government and include controls over information processing and management review of performance. SSA is implementing significant changes to its disability determination process, and considering numerous others, in order to improve speed and efficiency. Significant changes currently under way include the Electronic Disability System (eDib)--which SSA considers to be a foundation for other initiatives currently under consideration. With eDib, all components involved in disability claims adjudication and review--including SSA field offices, DDSs, OHA, and its Office of Quality Assurance--will use an electronic disability folder that replaces the paper case file. The electronic folder will contain an electronic copy of everything that was formerly included in the paper case file, such as electronic images of medical evidence and other documents related to the disability determination.[Footnote 4] The electronic folder will also contain electronic images of the 831 or CCS records.[Footnote 5] When the process is fully implemented, each component will be able to electronically view and process claims using the electronic disability folder. Other initiatives under consideration include centralizing medical expertise in regional expert review teams to expedite claims where the claimant is clearly disabled, requiring the DDSs to more fully document decisions and provide in-line quality review, eliminating the DDS reconsideration and Appeals Council decisional levels, and establishing end-of-line quality review at OHA. While Most Types of Information Collected Are Useful, Unknown Accuracy Detracts from Their Value: Although, for the most part, the types of information SSA collects in its electronic records contribute to the management of the disability programs, the accuracy of the records is unknown. In its electronic records, SSA collects information related to the claimant, process, and decision, which informs aspects of program management. However, SSA does not collect some information that would enhance program oversight. At the same time, SSA enters codes in its electronic disability records for the claimant's prior occupation and industry, even though these codes are based on outdated lists of occupations and industries, and the agency acknowledges that the value of that information is therefore limited. While SSA has some internal controls to help capture reliable data, these controls are not sufficient to ensure those data are reliable. SSA's policy does not require that the electronic record be verified against the information in the case file, and inadequate data entry controls have also resulted in some inaccurate and missing data. SSA Captures Important Information for Program Management, but Additional Information Could Improve Program Oversight: In its electronic disability records, SSA collects important information about the claimant, the process, and the decision. For example, SSA electronically collects information on individuals who apply for disability benefits, such as the applicant's name, Social Security number, gender, date of birth, state of residence, and type of benefits requested. It also collects information on the various events in the disability determination process, such as the application and decision dates, as well as the office processing the application and the adjudicative level involved. Finally, it collects information on the disability decision, such as the date the disability began, the primary impairment used in the medical determination of eligibility, and the regulation basis code for the disability decision that indicates the specific DI or SSI program requirements and whether they were met. This electronic disability information is analyzed and used to manage the DI and SSI programs. For example, SSA uses the data to monitor the time it takes the agency to process a claim and to determine the number of cases allowed and denied at each adjudicative level. From the 831 and CCS databases, SSA selects (randomly or based on particular characteristics) cases for review to help the agency ensure the accuracy of disability determinations. For each selected case, staff from SSA's Office of Quality Assurance obtain and review the paper case file to ensure that the DDS or ALJ disability decision is accurate.[Footnote 6] SSA also uses the data to determine when beneficiaries should be reviewed for continuing disability and what type of review is cost-effective. The disability data may also be used to study specific aspects of its disability determination process, and other issues, such as the prevalence or handling of specific illnesses, like chronic fatigue syndrome. The electronic disability data are also used to inform the Congress and the general public about the administration of the DI and SSI programs. For example, SSA uses these data to prepare the Annual Statistical Supplement, which is a major data resource on SSA-administered programs that serves as a foundation for various research and policy analyses and helps SSA to respond to requests for program data from congressional committees, government agencies at all levels, and the research community. Further, SSA uses these data to provide the Congress with an annual report on the status of the Supplemental Security Income program, including 25-year projections of program participation prepared by SSA's Office of the Chief Actuary. SSA's Office of the Chief Actuary uses these data to analyze demographic trends in mortality and morbidity rates, in order to project program growth and financial status. SSA shares its electronic disability data with other entities for research and analysis purposes. For example, the Disability Research Institute, which is funded through a cooperative agreement between SSA and the College of Applied Life Studies at the University of Illinois at Urbana-Champaign, uses SSA's data to conduct in-depth research to directly inform disability policy decision makers. In addition, SSA's administrative and program data have also been used by outside researchers and academics to study relationships between and among various characteristics of people with disabilities, including their work histories, impairments, and outcomes.[Footnote 7] SSA's databases have the advantage of being large enough to allow researchers to make statistically meaningful distinctions among subgroups of disabled individuals. While SSA has many uses for the information it collects, there is some information not being collected that could enhance program oversight. For example, after identifying unexplained racial differences in decisions made at the OHA level, we recommended that SSA collect information on race so that the agency can better monitor the equity of its decisions.[Footnote 8] In this same study, we also recommended that SSA collect information on the source of medical evidence (such as specialist, primary physician, emergency physician) and type of medical evidence (such as blood test, x-ray, observation, opinion), which could be used to analyze and better understand the role the agency and claimant representatives play in developing evidence for the disability determination. An SSA contractor also found the agency's electronic medical information lacking for other purposes. In 2001, PricewaterhouseCoopers (PwC) reported on the quality of electronic disability data used to target cases for continuing disability reviews (CDRs)--i.e., reviews of medical improvement of beneficiaries to determine their continuing eligibility for disability benefits. In its report, PwC stated that although SSA collects a large amount of medical evidence for SSA disability decisions, very little of this medical information is captured in SSA's electronic disability databases. Specifically, PwC noted that only 8 of the 162 variables captured in the 831 record relate directly to medical condition. PwC stated that despite the importance of medical information to any decision on continuing eligibility, a relatively small number of medical variables are available in SSA's 831 record for use in determining the most cost- effective method to target cases for CDRs.[Footnote 9] In addition to medical information, SSA lacks electronic data on claimants' vocational and functional capacity, which are critical factors in the disability decision and are therefore important in a study of the disability decision-making process.[Footnote 10] Nearly 60 percent of disability decisions cannot be made based solely on medical considerations and must rely on these additional factors. Nevertheless, in the CCS database, we found that no variable is related to vocational and functional capacity. For the DDS level, we found one variable in the 831 database that provides limited vocational and functional information.[Footnote 11] The 831 database has no other variables that capture additional factors often considered in making disability decisions (such as mental conditions, fine motor skills, and stamina). Moreover, for DDS decisions made in 2003, we found that the one variable related to vocational and functional factors is captured in only 10 percent of the cases in which such factors had to be considered for the disability determination.[Footnote 12] In the remaining 90 percent of these cases, DDS staff did not capture any vocational or functional factors used in the decision. Without this type of information, SSA lacks electronic data on critical factors considered in the disability decision and important to any study of the disability decision-making process. Mathematica--a contractor hired by SSA to evaluate new strategies for promoting employment among people with disabilities--also noted that important vocational and functional information was not being collected electronically by SSA. Specifically, Mathematica reported that certain characteristics (such as household composition, occupation, industry, and the presence of functional limitations) are, according to the research community, important predictors of employment, but they are not available in SSA's electronic databases. While SSA does not collect many types of information that contractors and others believe would be useful for managing the agency's programs, SSA may be collecting other data that are not particularly useful. SSA officials told us that they believe information on the claimant's prior occupation and industry could be useful, but because the lists of industries and occupations used by SSA are largely outdated, the agency considers these variables to be of limited value. Nevertheless, some DDS staff members still collect information on the claimant's prior occupation and industry.[Footnote 13] SSA Does Not Have Adequate Internal Controls to Ensure That the Electronic Data Accurately Reflect the Facts of the Case: While SSA emphasizes the importance of prompt and accurate disability determinations, the agency has not put sufficient emphasis on creating a positive control environment that would ensure a reasonable level of quality for its electronic disability data. SSA officials told us that although the DDSs and OHA are required to ensure that disability determinations are accurate, there is no specific requirement that electronic records provided to SSA be accurate. Regulations specify only that the DDSs must "provide the organizational structure, qualified personnel, medical consultant services, and a quality assurance function sufficient to ensure that disability determinations are made accurately and promptly." SSA officials told us that while some DDS offices may verify the accuracy of electronic data, SSA does not require or suggest that the DDSs follow a specific strategy or format for doing so. Some DDS and OHA staff and supervisors told us that they were unaware of how SSA used the 831 and CCS records or why it was important that they be accurate. If SSA required verification on at least a sample of cases, it would indicate to DDS and OHA staff the importance of accuracy in the electronic data. SSA has some electronic controls in place to help ensure the quality of data. These front-end data entry controls help ensure data quality by limiting what can be keyed into certain variables. For example, some variables accept only a numeric entry, and others, such as the Social Security number, must be filled in or the record will be rejected. Many date variables require a month/date/year entry, and certain dates must fall within a prescribed range. For example, the application date must be on or after a person's date of birth. In addition to these front-end data entry controls, SSA also reviews the 831 file by checking for unacceptable entries for certain variables. If SSA finds an increasing trend of unacceptable entries being keyed into those variables, the agency may issue additional guidance to staff in the field. On the basis of the results of this review, SSA officials told us that while the data may not be 100 percent accurate, they believe the data are sufficiently accurate. However, while such controls may reduce data entry errors, we believe the data entry controls are inadequate. In reviewing SSA's electronic records for disability decisions made by the DDSs and OHA in 2003, we found that even though information keyed into the record had not been rejected by the data entry controls, the record contained information that was clearly wrong because it did not agree with other information in the same record.[Footnote 14] This type of comparison was possible for only a few variables--impairment code, body system, the code indicating the basis for the disability decision, and the disability decision itself--but these are important variables that contain information crucial to SSA's management of its disability programs. Although our analysis identified only a small number of errors, the errors were in key variables. For example, GAO found that: * More than 4 percent of DDS records for claims processed in 2003 contained codes for the body system and impairment that were incompatible, indicating that either the impairment code or the body system code was incorrect. * For 211 claims allowed by the DDSs and 1,541 claims allowed by OHA, the impairment code indicated an unknown impairment for which there was no medical evidence. * For 239 claims allowed by the DDSs and 519 claims allowed by OHA, the impairment code indicated an unknown impairment for which there was insufficient medical evidence. * For nearly 4 percent of OHA's electronic disability records for allowed claims, the impairment codes did not appear in the OHA list of impairment codes. * OHA's records indicated that 2,367 applicants, who applied for both the DI and SSI programs, met SSA's medical criteria for one program but not the other--a situation not allowed under SSA regulations, which require that the same medical criteria apply to both programs. Nevertheless, for each of these applicants, OHA recorded different decision codes, primary impairment codes, or regulation basis codes for the two programs. Inadequate controls have also resulted in missing data. In reviewing the 831 and CCS records, we found that for many of the variables, leaving the variable blank is considered a valid entry. Therefore, when variables are not filled in, it's not possible to determine if staff had the information but failed to key it in, or if staff did not have the information and correctly left it blank. Although for most variables we were unable to determine that the variable should have been filled in, the education variable was an exception. For cases in which the adjudicator is required to consider educational level in making the disability determination, the information should have been obtained and keyed into the record.[Footnote 15] Yet, for 25 percent of such cases, that variable was left blank. This raises the question of how much of the missing data for other variables was available and should have been keyed into the record. Other organizations have also found weaknesses in SSA's data entry controls. In its 2001 study for SSA, PwC reported that poor data quality could present a significant barrier to efficient CDR case selection. As a result, it recommended that the agency improve system data edits, stating that such data entry controls would improve the integrity of disability programs, reduce inaccuracies and inconsistencies in data collection and reporting, and reduce the risk of error or erroneous data collection and reporting. PwC estimated that between 2001 and 2005, poor and missing disability data would cause SSA to improperly target CDRs and thereby incur additional administrative and program costs. An OIG report in 2000 reported that missing or invalid diagnostic codes had contributed to SSA's failing to perform mandatory reviews of certain disability cases and prevented the agency from accumulating and disseminating more accurate disability statistics. Even when variables have been filled in and accepted by existing data entry controls, SSA does not know the extent to which the data reflect the information in the case file because the agency's policy does not require that the electronic record be verified against the information in the case file. SSA's case files contain all documents related to the claim (such as medical records) and represent the legal record of the claim. Although we did not perform a comparison of electronic records against actual case files, a study by SSA's OIG in 2000 provides an example of the type of inaccuracies that may exist in disability records, even when the entries have been accepted by data entry controls. The OIG reviewed case files for a sample of 132 disability beneficiaries whose electronic records contained questionable impairment codes. It found 63 electronic records that contained an impairment code indicating, "diagnosis established--no predetermined list code of medical nature applicable," although the case file contained enough information to determine the proper impairment code. At the same time, the OIG sample was quite small and not intended to be representative. It is not cost-effective to check all, or even a large percentage of, electronic records against case files, and perfect reliability is not possible. However, without comparing a more representative sample of electronic records against case files, SSA cannot reliably estimate the true extent of inaccuracies in the electronic data. SSA's Ongoing and Planned Changes May Reduce the Chance for Some Inaccuracies, but the Agency Has No Plans for Improving the Accuracy or Usefulness of Electronic Data: SSA's ongoing and planned changes to computer systems and internal controls may reduce the chance for some inaccuracies. However, these planned changes will not address data entry problems found by GAO that could be prevented with additional data entry controls. SSA's current plans also do not include an internal control strategy for ensuring that electronic data match the information in the case file, nor do they provide for corrective action when inaccuracies are found. Finally, although SSA has proposed far-reaching changes to its disability decision-making process and is currently reassessing the processes for ensuring the quality of its disability decisions, the agency has not yet made any specific plans for evaluating the types of information it currently collects and whether other types of information would improve program management and oversight. SSA's Ongoing and Planned Changes May Reduce the Chance for Some Inaccuracies: Although most of SSA's ongoing and planned changes do not involve aspects of the electronic data systems touched on in this report, a few will alter the way information is transmitted and, in this way, could reduce the number of chances for unintentional errors. Under eDib, SSA field office staff keying in the disability history for an applicant will now use the Electronic Data Collect System (EDCS). Once the field office staff have keyed the Social Security number into the EDCS record, EDCS automatically verifies that there is a disability application in existing SSA databases. Information such as the application date, disability allegations, and disability onset date are automatically propagated into the EDCS record. This will improve the accuracy of the information keyed into the EDCS record, which goes into the electronic disability folder used at all levels of the disability determination process. Some additional data entry controls that are part of the new OHA case tracking software (the Case Processing Management System--CPMS) that was implemented this year as part of eDib may also reduce data entry errors. SSA's Plans Do Not Include Strategies for Ensuring Data Accuracy or Evaluating Types of Information Needed: SSA's current plans do not include an overall internal control strategy for ensuring the accuracy of electronic disability data. While SSA has included some front-end data entry controls, it does not address problems with electronic data we found that could be prevented with additional controls. SSA also has no plans to improve its system for the back-end review of electronic disability records, so that the agency can estimate the accuracy of its electronic disability records and provide feedback to staff on accuracy. Further, the agency has no specific plans to evaluate the types of information collected during the disability determination process. Most of the changes currently under way--such as creating a shared electronic folder for each disability case, scanning evidence for inclusion in the electronic folder, providing new software systems, and improving interfaces between systems--were intended to improve the speed and efficiency of the disability claims process and not necessarily to improve the reliability or usefulness of the data in SSA's 831 and CCS databases. To accelerate the rollout of eDib, SSA carried over the requirements from the old systems to the new systems. Although the agency did introduce some additional front-end data entry controls to reduce the entry of invalid data, the agency did not add sufficient controls to address the problem with missing and invalid data we found. An OHA official told us that even the additional data entry controls available in the new CPMS software are not adequate to prevent data entry errors we identified in this study. For example, under eDib, SSA's data entry control still allows any four digits for the impairment variable in the CPMS, as well as the 831 file, rather than limiting data entry to one of the 240 acceptable impairment codes. These officials said that limited resources have prevented the agency from establishing and maintaining additional or more exacting controls- -such as a current list of acceptable impairment codes. Moreover, SSA officials told us that additional data entry controls would require staff to determine the correct entry when errors were identified, which could slow case processing. SSA also has no plans to improve its system for the back-end review of electronic disability records, so that the agency can estimate the accuracy of its electronic disability records and provide feedback to staff on accuracy. SSA already has a process in place for reviewing a random sample of case files as part of its quality assurance for disability decisions, which allows the agency to make estimations of the accuracy of decisions at both the DDS and the OHA levels. In the course of this quality assurance review, staff are supposed to verify the electronic 831 record against the information in the case file and note errors.[Footnote 16] If a significant error is found, staff are directed to fax a corrected 831 form to a central location so that the correction can be made in the 831 file. However, SSA officials told us that corrections don't always end up in the 831 file, and the agency does not track input errors because such errors are low on the priority list. Because there is no tracking system that would collect statistical data on the number or type of errors, SSA is unable to determine the level of accuracy for this electronic data. A tracking system that provided feedback to the field would also send a message to staff that the items being tracked are important--a basic element in any quality assurance system. Without the statistical data and feedback to staff, the reviews for data quality affect only those records reviewed and do not affect the possible quality of the rest of the records. Finally, the back-end review covers only 831 records, not CCS records. SSA has not yet made concrete plans for evaluating the types of information it currently collects and whether other types of information would improve program management and oversight. Users of administrative data that we interviewed indicated that they have not been surveyed as to their data needs, and some indicated that their requests for additional data collection have not yet been implemented. However, the agency may take such steps after other priorities are accomplished. An SSA official stated that SSA's first priority was to implement eDib, in order to modernize its core business process of taking and adjudicating claims. The official also stated that after eDib is fully implemented, SSA will assess whether the information collected is sufficient. In more recent interviews, high-level SSA officials validated the importance to the agency of reliable and useful data for program management purposes, and indicated a strong interest in developing a comprehensive information management plan that would allow the agency to systematically and regularly assess its information needs and make appropriate adjustments to its computer systems. According to agency officials, this effort would include a review of best practices in information management. However, the ideas expressed were still being formulated, and the agency has not yet taken any concrete actions. Conclusion: SSA lacks a comprehensive strategic plan for collecting useful and reliable data for effective oversight and program planning. Despite the fact that the reliability of its data has been an ongoing issue, the agency currently has no specific plans to implement additional front- end data entry controls or for a tracking and feedback system for the back-end verification of the electronic records. In addition, the agency has no specific plans for evaluating whether the types of information it currently collects continue to be useful for program oversight and whether other types of information would contribute to oversight. While stretched resources and other priorities may have prevented SSA from addressing these issues to date, SSA plans to make more changes to its disability processes, which will create additional opportunities and a sense of urgency for improving the usefulness and reliability of its electronic disability data. For example, in September 2003, SSA's Commissioner proposed significant changes to its disability decision- making process--such as the proposed use of centralized medical expertise and full documentation of initial-level decisions--that will, if implemented, translate into new requirements for collecting administrative data to monitor and assess these changes. Moreover, as SSA continues to transition from a paper-based case-processing system to an electronic one, SSA will have additional opportunities to institute changes that would improve upon the usefulness and reliability of the data that it routinely collects. Finally, SSA has been considering fundamental changes to its quality assurance processes, at both the front and back ends of its disability determination process. Combined, these changes, each significant and interrelated, create a compelling case for SSA to broadly assess and improve the nature and reliability of the data it uses to manage its disability programs. Recommendations for Executive Action: To improve the value of SSA's electronic administrative data for managing its disability programs, we recommend that the Commissioner of Social Security develop a comprehensive strategic plan to ensure the reliability and usefulness of the data the agency collects. In doing so, the agency should take the following steps: 1. Establish a cost-effective internal control strategy for ensuring the reliability of data in the electronic disability records that would include both front-end controls on data entry and a tracking and feedback system for back-end verification of the electronic records. 2. Take steps to review the usefulness of the types of information collected and consider whether additional types of information could improve program oversight. This effort could include a survey of users of electronic disability data. Agency Comments and Our Evaluation: We provided a draft of this report to SSA for comment. SSA generally agreed with our conclusions and recommendations, acknowledging that the report made valid points about steps SSA can take to increase the usefulness of electronic data for program oversight and that the agency would consider how best to incorporate improvements in its data collection activities. SSA agreed in theory with our first recommendation and noted that the agency is continuing to examine future software enhancements to improve front-end controls on data entry and back-end review. SSA also noted that its back-end review of cases may provide some check on data entry. We agree and modified our report and first recommendation to further clarify this. However, SSA expressed concern regarding the cost and technical feasibility of implementing this recommendation. We agree that the agency needs to balance enhanced internal controls with operational costs and productivity, but we believe some of the specific concerns raised by SSA are unwarranted. For example, as discussed in our response to SSA's comment in appendix I, we believe that the internal controls proposed by GAO would not require additional "structured" data or a data collection system "robust enough to catalog and aggregate the information." In its response to our second recommendation, SSA agreed that the disability program would be well served by reviewing all the information that SSA could potentially collect on a disability claim and determining which of these data elements might be beneficial for program oversight and policy development. SSA also stated that the agency is drafting a statement of work to contract for an assessment of data needs for managing the disability programs, which will identify possible sources and methods for obtaining data and make recommendations for improving existing data and data retrieval. The agency further observed that it would have to consider the potential adverse effect of increased electronic data collection on productivity and timeliness. SSA provided additional general comments, which we have included (along with our responses to them) in appendix I and addressed in the body of our report where appropriate. SSA also provided technical comments that we have incorporated in the report as appropriate. We are sending copies of this report to the appropriate congressional committees and other interested parties. We will also make copies available to others on request. In addition, the report will be available at no charge on GAO's Web site at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact me or Michele Grgich, Assistant Director, at (202) 512-7215. You may also reach us by e-mail at firstname.lastname@example.org or email@example.com. Other major contributors to this assignment were Ann T. Walker, Jill D. Yost, and Corinna Nicolaou. Sincerely yours, Signed by: Robert E. Robertson: Director, Education, Workforce, and Income Security Issues: [End of section] Appendix I: Comments from the Social Security Administration: Note: GAO comments supplementing those in the report text appear at the end of this appendix. SOCIAL SECURITY: The Commissioner: November 4, 2004: Mr. Robert E. Robertson: Director, Education, Workforce and Income Security Issues: U.S. Government Accountability Office: Room 5-T-57: 441 G Street, NW: Washington, D.C. 20548: Dear Mr. Robertson: Thank you for the opportunity to review and comment on the draft report "SSA's Disability Programs: Improvements Could Increase the Usefulness of Electronic Data for Program Oversight" (GAO-05-100R). Our comments on the report are enclosed. If you have any questions, please have your staff contact Candace Skurnik, Director, Audit Management and Liaison Staff at (410) 965- 4636. Sincerely, Signed by: Jo Anne B. Barnhart: Enclosure: SOCIAL SECURITY ADMINISTRATION BALTIMORE MD 21235-0001: COMMENTS ON THE GOVENMENT ACCOUNTABILITY OFFICE (GAO) DRAFT REPORT "SSA'S DISABILITY PROGRAMS: IMPROVEMENTS COULD INCREASE THE USEFULNESS OF ELECTRONIC DATA FOR PROGRAM OVERSIGHT" (GAO-05-100R): We appreciate the opportunity to comment on the GAO draft report concerning recommended improvements to the Social Security Administration's (SSA) disability programs which would increase the usefulness of electronic data for program oversight. We recognize the importance of the Disability Insurance (DI) and Supplemental Security Income (SSI) programs to the American public and the need for reliable administrative data from our disability decision-making process to adequately understand the population we serve. The report makes valid points about steps SSA can take to increase the usefulness of electronic data for program oversight, and we will consider how best to incorporate such improvements in our data collection activities under electronic disability (eDib) and the Commissioner's new disability approach. However, since these initiatives might be resource intensive, we will need to balance operational cost and productivity concerns as we pursue enhanced data collection controls. In addition, we are drafting a "statement of work" (SOW) to contract for an assessment of data needs for managing the disability programs. The SOW will require the contractor to identify possible sources and methods for obtaining data and make recommendations for improving existing data and data retrieval. We also appreciate GAO's acknowledgement that most types of information collected by SSA are useful for program purposes. We know data quality has improved greatly over the years as evidenced by the findings in PricewaterhouseCoopers' (PwC) February 2001 report entitled "Disability Quality Review" and in our validations of data that are done periodically during the year. The report states there is no review of coding to ensure the inputs are correct. However, in conducting the quality reviews (e.g., pre- effectuation review), disability examiners do check the coding and make pen and ink changes and corrections on the SSA-831. The changes they make are prepared for systems input. Our Disability Quality Branches review several thousand cases a year. However, the Agency does not keep track of these types of errors for statistical purposes. We disagree with the statements that SSA's current plans do not include an overall internal control strategy for ensuring the accuracy of electronic disability data or current plans do not provide for a back- end review to ensure that electronic data match the information in the case file. As the electronic folder is developed and the eDib is rolled-out nationwide, software enhancements addressing the data entry and data collection needs are being addressed. So far, limited resources have prevented the Agency from establishing and maintaining additional or more exacting controls. As to specific points, the first paragraph on page 2 of the report notes that SSA lacks an indicator on claimant's race. We recognize that this longstanding issue has prevented us from conducting certain types of data analyses. Therefore, in September 2003, SSA convened a workgroup to develop a plan to collect race/ethnicity information during the initial claims process. The first paragraph on page 6 notes that limited medical information contained in the agency's 831 records is available for determining the most cost effective method for targeting cases for continuing disability reviews (CDR). While we agree with this assessment, it should be noted that SSA relies on other internal Agency data such as information contained in the Master Beneficiary Record System and Supplemental Security Record System in the CDR process. In addition, it should be noted that since the 2001 PwC study was referenced in the report, SSA has systematically incorporated the use of additional electronic data in the CDR process including medical information obtained from Medicare claims records and individual responses from the Agency's CDR mailer forms. The second to last paragraph on page 8 references the 2001 PwC study which reported that poor data quality could present a significant barrier to efficient CDR case selection. While we agree with the results of that study for the time period it was written, it should be noted that the completeness and overall reliability of the Agency's electronic data has improved significantly since that time. Data capture policies and practices have strengthened, and the number of cases allowed prior to implementation of many current system data edits have steadily diminished. As noted in the study, for example, the proportion of entitled DI beneficiaries with missing or invalid primary diagnosis codes in 831 records dropped from about 20 percent in 1989 to about six percent in 1999. Significant further data quality improvements have occurred since the time the PwC study reported estimates of potential CDR targeting inefficiencies due to poor data quality. We offer the following comments on the two GAO draft report recommendations: Recommendation 1: Establish a cost-effective internal control strategy for ensuring the reliability of data in the electronic disability records that would include both front-end controls on data entry and back-end reviews of data entry to ensure that the record accurately reflects the information in the case file. For example, staff performing back-end quality reviews of cases could verify the accuracy of these randomly sampled electronic records by comparing them with the case files. Response: We agree in theory and we are continuing to examine future software enhancements to improve front-end controls on data entry, obtain useful data, and improve back-end reviews. However, we are concerned about the costs required to create such a data collection and maintenance structure and need to determine cost-effectiveness. In order to attain this goal, we first would need to support the addition of most data into the electronic record as 'structured' data. In these early phases of transition to an electronic disability process, much of the material and documents added to the electronic file are 'tiff images (as an example, the SSA-831). As long as data is unstructured, we will have difficulty aggregating and reporting based on these data. In a broader sense, the key is the creation of a "cost effective" method of front-end data capture, and back-end reviews or audits. We need to remain cognizant of the fact that Social Security disability determinations are built up of data that is often not 'proof-certain.' As an example, in claim "A," we receive five pieces of medical evidence: Claim "A" MER #: MER 1; Primary Diagnosis: Diabetes; Secondary Diagnosis: Arthritis. Claim "A" MER #: MER 2; Primary Diagnosis: Diabetic Nephropathy. Claim "A" MER #: MER 3; Primary Diagnosis: DJD; Secondary Diagnosis: Diabetes Mellitus. Claim "A" MER #: MER 4; Primary Diagnosis: Type 11 Diabetes; Secondary Diagnosis: Osteoarthritis. Claim "A" MER #: MER 5; Primary Diagnosis: Arthritis, knees, bilateral; Secondary Diagnosis: Renal Disease. [End of table] What should the 'official' primary and secondary diagnoses be in the above case scenario? Here a decision-maker will have to select a diagnosis from a set of diagnoses, code them in a way that captures different names for the same types of impairments, and make priority selections. After all this is done, we would need to have a data collection system robust enough to catalog and aggregate the information, along with a system for either manual or automated edits. These are all important, but they are neither easy nor inexpensive in start up or ongoing costs. Finally, such a post-review would require a comparison of the paper documents to electronic documents. Once SSA adopts an electronic folder, we will forego the paper or source documents. Thus, it is difficult to see how we can agree to a post-review. Recommendation 2: Take steps to review the usefulness of the types of information collected and consider whether additional types of information could improve program oversight. This effort could include a survey of users of electronic disability data. Response: In general, we agree that the disability program would be well-served by reviewing all the information that we could potentially collect on a disability claim and determining which of these data elements might potentially be beneficial for program oversight and policy development. We agree that sufficient internal controls over electronic data systems are necessary to effectively manage programs. Some information currently collected may be of limited value and there is additional data the Agency could collect (e.g., household composition, occupation, industry, and the presence of functional limitations) that are important predictors of employment. The collection of these additional characteristics would be useful in meeting the Agency's strategic objective to increase employment for people with disabilities. With the rapid evolution of the electronic disability system, the proposed changes to SSA's policies regarding the development of internal controls to ensure that data are reliable and useful come at an appropriate time. We would note that efforts are underway to ensure the accuracy of electronic data. With respect to the specific recommendation to develop a front-end validation process, the Agency has already established a strategy for ensuring electronic claims data is accurate and complete. In terms of a back-end quality control process, the Agency will continue to rely on its existing end-of-line quality review procedure for determining the accuracy of electronic records. We agree that consideration should be given to the relevancy of data collected for disability program purposes. However, we will also give consideration to the potential adverse effect increased electronic data collection may have on productivity and timeliness. GAO Comments: 1. We acknowledge that the Disability Quality Branches' (DQB) back-end review of cases may provide some check on data entry, and we have adjusted our recommendation and text in our report to clarify this. However, SSA officials told us that tracking of input errors is low on DQB's priority list, and DQB changes don't always end up in the 831 file. Further, as SSA indicated, it does not track input errors for statistical purposes; therefore SSA does not know the level of data accuracy. A tracking system that also provides feedback to the field would also send a message to staff that the items being tracked are important--a basic element in any quality assurance system. Without the statistical data and feedback to staff, the reviews for data quality affect only those records reviewed, and do not affect the possible quality of the rest of the records. Finally, the back-end review by DQB covers only 831 records (i.e., DDS decisions), not CCS records (i.e., ALJ decisions). 2. Our report recognizes that as part of the implementation of eDib, SSA is making software enhancements that may address some data entry problems. However, we believe that the eDib enhancements fall short of an effective overall strategy. Our previous comment explains why we believe SSA's back-end reviews do not sufficiently ensure that electronic data match the information in the case file. 3. We amended the report to include information about the work group. 4. We have incorporated into our report the additional sources of information used in the CDR targeting process. 5. We recognize that the agency has improved data capture policies and practices over the years, but in our study, we examined electronic disability data for 2003, the most recent year available, and found problems with that data. In addition, without an effective system for verifying the electronic record against the case file and a tracking system for errors, neither SSA nor we can determine the extent to which the electronic disability data are accurate. Further, PricewaterhouseCoopers' projection that SSA would incur hundreds of millions of dollars in CDR administrative and program costs between 2001 and 2005 because of poor and missing electronic disability data was made despite its acknowledgment that SSA's electronic disability data had improved. In order to not overstate PwC's findings, we adjusted the text of our report to say that the agency would incur "additional administrative and program costs" because of problems with the quality of its electronic disability data. 6. We do not agree that additional "structured" data are necessary before data quality can improve. The 831 record is an electronic version of the SSA-831 form, in a structured data format. For cases processed prior to the use of the electronic case folder, a printout of the completed SSA-831 form was placed in the paper case file to document the final decision. Under eDib, which uses an electronic case folder, an electronic image ("tiff" image) of the SSA-831 form is created and placed in the electronic case folder (along with other electronic images such as medical records) to document the final decision. Regardless of this 831 image, the original 831 record still exists as structured data and becomes part of a national database known as the "831 file." SSA is still able to aggregate and report on these data. Therefore, it is possible for SSA to implement our recommendation that it compare the information in the electronic 831 record against the information in the case file or the electronic case folder, and correct the electronic 831 record as needed. Even after the agency eliminates paper source documents and relies totally on electronic documents in the case folder, SSA will still be able to compare the electronic 831 record against the electronic images of medical records and other documents related to the disability determination that are in the case folder. 7. We agree that it may be difficult to determine the primary impairment. However, our concern is that once a disability examiner makes the decision, the 831 record should accurately reflect the disability decision. For example, if the primary impairment is determined to be diabetes, then the 831 record should contain the four- digit impairment code for diabetes and the appropriate body system code for that primary impairment. SSA could develop data entry controls that would allow only acceptable impairment codes and the corresponding body system code. The agency would have to update those listings only when changes in policy dictated changes in impairment codes or body system codes. This type of front-end data entry control should not require a "data collection system robust enough to catalog and aggregate the information." [End of section] FOOTNOTES  GAO, SSA Disability Decision Making: Additional Steps Needed to Ensure Accuracy and Fairness of Decisions at the Hearings Level, GAO- 04-14, Washington, D.C.: Nov. 12, 2003.  While most claimants may request a reconsideration, at the time of our study, SSA was testing an initiative that eliminates the reconsideration step from the DDS decision-making process.  SSA's database of 831 records is called the National Disability Determinations Service System (NDDSS) but is generally referred to as the "831 file." Therefore, in this report, we use the term "831 file" or "831 database" to refer to the NDDSS.  Electronic images are like electronic photocopies of the documents. They can be retrieved or viewed in their entirety, but the elements that constitute the image--such as words or numbers--are not in a format that allows them to be identified using electronic search mechanisms or readily retrieved and manipulated for reporting or analytical purposes.  Although these electronic images cannot be manipulated, data keyed into the 831 and CCS databases are in a format that allows for easy identification, retrieval, and manipulation for reporting or analytical purposes.  Once eDib is fully implemented, instead of reviewing a paper case file, quality assurance staff will review electronic documents in the electronic disability folder, including medical records and other information related to the disability determination.  Outside researchers and academics using SSA disability data include Kajal Lahiri, Denton R. Vaughan, and Bernard Wixon, "Modeling Social Security's Sequential Disability Determination Using Matched SIPP Data," Social Security Bulletin, 58, 3-42; Congressional Research Service Report to the Congress (1992), "Status of the Disability Programs of the Social Security Administration," 92-691 EPW, the Library of Congress, Washington, D.C.; and Eddy Bresnitz, Howard Frumkin, Lawrence Goldstein, David Neumark, Michael Hodgson, and Carolyn Needleman, "Occupational Impairment and Disability among Applicants for Social Security Disability Benefits in Pennsylvania," American Journal of Public Health 84 (November 1994), 1786-1790.  GAO-04-14. SSA officials told us that the agency formed a work group in September 2003 to develop a plan for collecting race/ethnicity information.  In targeting cases for CDRs, SSA also uses information contained in the Master Beneficiary Record system and the Supplemental Security Record system, as well as information from Medicare claim forms and individual responses to prior CDR mailer forms.  Vocational factors are age, education and work experience. A claimant's residual functional capacity is the individual's maximum sustained capability for sedentary, light, medium, heavy, or very heavy work.  The vocational rule number variable indicates certain vocational and functional patterns that include functional capacity, age, work history, and education level.  SSA uses a five-step disability determination process whereby the agency considers vocation and functional capacity in steps 4 and 5, i.e., when it has determined that the claimant has a severe, medically determinable physical or mental impairment, but the impairment is not severe enough for the individual to be determined eligible based on medical factors alone. In steps 4 and 5, SSA uses these additional factors to determine whether the claimant's impairment(s) prevents the performance of his or her past work or other work in the national economy. Federal regulations contain rules (vocational rule numbers) that direct a finding of disabled or not disabled based on vocational factors (such as age, education, and work experience) in combination with the individual's residual functional capacity.  In 1992, when the DDS workload was heavy, SSA informed the DDSs that the variables for industry, occupation, and years in occupation did not need to be filled in. Although this was intended to be a temporary measure, SSA's incentive for re-instituting the policy diminished as the job classifications grew more outdated over time.  We reviewed 3.9 million 831 records for disability decisions made in 2003, of which 31 percent (1.2 million) were allowed. We also reviewed over 400,000 CCS records for disability decisions made in 2003, of which 70 percent (nearly 282,000) were allowed.  Over 50 percent of all claims processed by DDSs in 2003 required that educational level be considered in making the disability determination.  Under eDib, such a back-end review would involve matching the 831 or CCS record against the electronic images of medical records and other documents related to the disability determination.