This is the accessible text file for GAO report number GAO-05-100R
entitled 'SSA's Disability Programs: Improvements Could Increase the
Usefulness of Electronic Data for Program Oversight' which was released
on December 10, 2004.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
United States Government Accountability Office:
Washington, DC 20548:
December 10, 2004:
The Honorable Jo Anne B. Barnhart:
Commissioner, Social Security Administration:
Subject: SSA's Disability Programs: Improvements Could Increase the
Usefulness of Electronic Data for Program Oversight:
Dear Ms. Barnhart:
In 2003, we added the federal government's disability programs to our
high-risk list in part because of difficulties agencies faced in
managing these programs and the expected growth in the rolls as baby
boomers reach their disability-prone years. The Social Security
Administration (SSA) manages the federal government's two largest
disability programs, the Disability Insurance (DI) program and the
Supplemental Security Income (SSI) program, which together paid out $91
billion in federal benefits to 11.4 million individuals with
disabilities in 2003. To help address management difficulties and
prepare for expected growth in the rolls, SSA must have reliable
administrative data from its disability decision-making process to
adequately understand the population it serves and the possible effect
of proposed program changes on this population. However, in a prior
study, we identified potential problems with the reliability of SSA's
electronic administrative data.[Footnote 1]
This report examines (1) the extent to which SSA collects useful and
reliable electronic administrative data in order to effectively manage
its DI and SSI programs and (2) whether ongoing and planned changes to
SSA's computer systems and internal controls will address any
weaknesses that we identified. To determine the adequacy of SSA's
electronic administrative data for management of its disability
programs, we reviewed SSA, Office of the Inspector General (OIG), and
our prior reports regarding these data; interviewed users of these
data; and performed a literature search on criteria for establishing
information databases. To identify the nature and extent of data
problems, we interviewed staff responsible for managing and using the
data to assess the controls and processes in the disability system,
conducted data reliability tests on selected variables in the
disability records for calendar year 2003, and reviewed relevant
reports by OIG and SSA contractors for their assessment of the data
quality. To determine the extent to which SSA has plans or efforts
under way to address these problems, we interviewed SSA officials and
reviewed SSA documents and OIG reports. We conducted our work between
December 2003 and September 2004 in accordance with generally accepted
government auditing standards.
Results in Brief:
While most types of information collected by SSA are useful for program
management purposes, the agency lacks sufficient internal controls to
ensure that these data are reliable. In its electronic records, SSA
collects information related to the claimant, process, and decision--
all of which are important for informing aspects of program management.
For example, SSA electronically collects information on the claimant's
impairment that is used to identify beneficiaries who should be
reviewed for medical improvement and continuing eligibility for
disability benefits. However, SSA does not collect some information
that would enhance program oversight. For example, without an indicator
on claimant's race, SSA lacks a complete basis for assessing the equity
of disability decisions. At the same time, SSA collects other
information in its electronic disability record that the agency
considers of limited value. For example, codes entered for the
claimant's prior occupation and industry are based on outdated lists,
and the agency acknowledges that the value of that information is
therefore limited. Although the types of information SSA collects in
its electronic records generally contribute to the management of the
disability programs, the accuracy of the records is unknown. For
example, inadequate data entry controls permit the entry of invalid
impairment codes--codes that do not reflect a meaningful impairment. In
addition, inadequate data entry controls have resulted in missing data
for some information, such as the claimant's educational level, which
is a critical factor in the disability decision and is therefore
important in a study of the disability decision-making process. Most
important, because SSA's policy does not require that the electronic
record be verified against the information in the case file, the agency
does not know the extent to which codes that appear valid reflect the
claimant's actual information, such as the claimant's actual impairment
or education level.
While SSA's ongoing and planned changes to computer systems and
internal controls may reduce the chance for some inaccuracies, SSA's
plans do not provide adequate assurance of the accuracy of its
electronic administrative data. SSA's ongoing transition from a paper-
based disability system to an electronic one should reduce some of the
inaccuracies we identified by limiting the amount of data re-entered at
various levels in the process. However, SSA's planned changes will not
address data entry problems found by GAO that could be prevented with
additional data entry controls. SSA's current plans also do not include
an internal control strategy for ensuring that electronic data match
the information in the case file, nor do they provide for corrective
action when inaccuracies are found. Finally, although SSA has proposed
far-reaching changes to its disability decision-making process and is
currently reassessing the processes for ensuring the quality of its
disability decisions, the agency has not yet made any plans for
evaluating the types of information it currently collects and whether
other types of information would improve program management and
oversight.
To improve the value of SSA's electronic administrative data for
managing its disability programs, we recommend that the agency
establish a cost-effective internal control strategy to ensure that
those data are reliable and that they accurately reflect the
information in the case file. In addition, the agency should take steps
to review the usefulness of the types of information collected and
consider whether additional types of information could improve program
oversight.
In commenting on the draft of this report, SSA generally agreed with
our conclusions and recommendations. SSA acknowledged that the report
made valid points about steps SSA can take to increase the usefulness
of electronic data for program oversight, and that the agency would
consider how best to incorporate improvements in its data collection
activities.
Background:
DI and SSI are the two largest federal programs providing cash
assistance to people with disabilities. Established in 1956, DI
provides monthly payments to workers with disabilities (and their
dependents) under the age of 65 who have enough work experience to
qualify for disability benefits. Created in 1972, SSI is a means-tested
income assistance program that provides monthly payments to adults or
children who are blind or who have other disabilities and whose income
and assets fall bellow a certain level. To be considered eligible for
either program as an adult, a person must be unable to perform any
substantial gainful activity because of a medically determinable
physical or mental impairment that is expected to result in death or
that has lasted or can be expected to last for a continuous period of
at least 12 months.
The disability determination process is complex, involving more than
one office and decision maker. The process begins at a field office,
where an SSA representative determines whether a claimant meets the
programs' nonmedical eligibility criteria. Claims meeting these
criteria are forwarded to a state's disability determination service
(DDS) office to determine if a claimant meets the agency's medical
eligibility criteria. At the DDS, the disability examiner and the
medical or psychological consultants work as a team to analyze a
claimant's documentation, gather additional evidence as appropriate,
and approve or deny the claim. A denied claimant may ask the DDS for a
reconsideration of its finding, at which point a different DDS team
will review the claim.[Footnote 2] If the claim is denied again, the
claimant may appeal the determination to SSA's Office of Hearings and
Appeals (OHA), where an administrative law judge (ALJ) will review it.
The ALJ usually conducts a hearing in which the claimant and others may
testify and present new evidence. In making the disability decision,
the ALJ uses information from the hearing and from the state DDS,
including the findings of the DDS's medical consultant. A claimant
whose appeal is denied may request a review by SSA's Appeals Council
and, if the claim is denied again, may file suit in federal court.
At each step of the disability determination process, information
collected is entered into the claimant's case file as well as into an
electronic database. SSA field office staff create the case file, which
is the legal record for the disability determination and contains all
documents related to the case. If the claimant meets the nonmedical
eligibility criteria, the field office staff forwards the case file to
the appropriate state DDS office. DDS staff add the information
collected at the DDS level and the initial disability determination to
the case file and enter selected data from the case file into an
electronic record known as the Form 831. Once the DDS submits these 831
records to SSA, the records become part of SSA's 831 file.[Footnote 3]
If the claimant asks for a reconsideration, another DDS team will place
the new information in the case file and enter selected data from the
case file into a new 831 record reflecting the reconsideration
decision, which will also become part of SSA's 831 file. If the
claimant appeals the reconsideration by requesting a hearing by an
administrative law judge at the Office of Hearings and Appeals, the ALJ
conducts a new review of the claimant's file and any additional
evidence the claimant submitted since the DDS determination. The ALJ
may also hear testimony from medical or vocational experts and the
claimant regarding the claimant's medical condition and ability to
work. As with the DDS, OHA staff place the new information in the case
file and enter selected data from the case file into an electronic
record. These electronic records become part of a nationwide electronic
database referred to as the Case Control System (CCS) database.
It has been well established that adequate internal controls over
electronic data systems are necessary to effectively manage programs,
such as SSA's disability programs. In 1998, the Social Security
Advisory Board noted that it is essential for policy makers have
accurate, balanced, and objective information to help them determine
the extent to which the program is meeting the long-standing objectives
of social adequacy and individual equity, the nature and extent of
changes that may be needed, and the impact of proposals for change. SSA
stated in its 2003 Performance and Accountability Report that the
agency was committed to providing clear and reliable data to those who
use the data for managing, decision making, and oversight of SSA's
programs. Finally, GAO's standards state that adequate internal
controls should be established to provide reasonable assurance that
agency objectives are being achieved. These internal control standards
define the minimum level of quality acceptable for internal controls in
the government and include controls over information processing and
management review of performance.
SSA is implementing significant changes to its disability determination
process, and considering numerous others, in order to improve speed and
efficiency. Significant changes currently under way include the
Electronic Disability System (eDib)--which SSA considers to be a
foundation for other initiatives currently under consideration. With
eDib, all components involved in disability claims adjudication and
review--including SSA field offices, DDSs, OHA, and its Office of
Quality Assurance--will use an electronic disability folder that
replaces the paper case file. The electronic folder will contain an
electronic copy of everything that was formerly included in the paper
case file, such as electronic images of medical evidence and other
documents related to the disability determination.[Footnote 4] The
electronic folder will also contain electronic images of the 831 or CCS
records.[Footnote 5] When the process is fully implemented, each
component will be able to electronically view and process claims using
the electronic disability folder. Other initiatives under consideration
include centralizing medical expertise in regional expert review teams
to expedite claims where the claimant is clearly disabled, requiring
the DDSs to more fully document decisions and provide in-line quality
review, eliminating the DDS reconsideration and Appeals Council
decisional levels, and establishing end-of-line quality review at OHA.
While Most Types of Information Collected Are Useful, Unknown Accuracy
Detracts from Their Value:
Although, for the most part, the types of information SSA collects in
its electronic records contribute to the management of the disability
programs, the accuracy of the records is unknown. In its electronic
records, SSA collects information related to the claimant, process, and
decision, which informs aspects of program management. However, SSA
does not collect some information that would enhance program oversight.
At the same time, SSA enters codes in its electronic disability records
for the claimant's prior occupation and industry, even though these
codes are based on outdated lists of occupations and industries, and
the agency acknowledges that the value of that information is therefore
limited. While SSA has some internal controls to help capture reliable
data, these controls are not sufficient to ensure those data are
reliable. SSA's policy does not require that the electronic record be
verified against the information in the case file, and inadequate data
entry controls have also resulted in some inaccurate and missing data.
SSA Captures Important Information for Program Management, but
Additional Information Could Improve Program Oversight:
In its electronic disability records, SSA collects important
information about the claimant, the process, and the decision. For
example, SSA electronically collects information on individuals who
apply for disability benefits, such as the applicant's name, Social
Security number, gender, date of birth, state of residence, and type of
benefits requested. It also collects information on the various events
in the disability determination process, such as the application and
decision dates, as well as the office processing the application and
the adjudicative level involved. Finally, it collects information on
the disability decision, such as the date the disability began, the
primary impairment used in the medical determination of eligibility,
and the regulation basis code for the disability decision that
indicates the specific DI or SSI program requirements and whether they
were met.
This electronic disability information is analyzed and used to manage
the DI and SSI programs. For example, SSA uses the data to monitor the
time it takes the agency to process a claim and to determine the number
of cases allowed and denied at each adjudicative level. From the 831
and CCS databases, SSA selects (randomly or based on particular
characteristics) cases for review to help the agency ensure the
accuracy of disability determinations. For each selected case, staff
from SSA's Office of Quality Assurance obtain and review the paper case
file to ensure that the DDS or ALJ disability decision is
accurate.[Footnote 6] SSA also uses the data to determine when
beneficiaries should be reviewed for continuing disability and what
type of review is cost-effective. The disability data may also be used
to study specific aspects of its disability determination process, and
other issues, such as the prevalence or handling of specific illnesses,
like chronic fatigue syndrome.
The electronic disability data are also used to inform the Congress and
the general public about the administration of the DI and SSI programs.
For example, SSA uses these data to prepare the Annual Statistical
Supplement, which is a major data resource on SSA-administered programs
that serves as a foundation for various research and policy analyses
and helps SSA to respond to requests for program data from
congressional committees, government agencies at all levels, and the
research community. Further, SSA uses these data to provide the
Congress with an annual report on the status of the Supplemental
Security Income program, including 25-year projections of program
participation prepared by SSA's Office of the Chief Actuary. SSA's
Office of the Chief Actuary uses these data to analyze demographic
trends in mortality and morbidity rates, in order to project program
growth and financial status.
SSA shares its electronic disability data with other entities for
research and analysis purposes. For example, the Disability Research
Institute, which is funded through a cooperative agreement between SSA
and the College of Applied Life Studies at the University of Illinois
at Urbana-Champaign, uses SSA's data to conduct in-depth research to
directly inform disability policy decision makers. In addition, SSA's
administrative and program data have also been used by outside
researchers and academics to study relationships between and among
various characteristics of people with disabilities, including their
work histories, impairments, and outcomes.[Footnote 7] SSA's databases
have the advantage of being large enough to allow researchers to make
statistically meaningful distinctions among subgroups of disabled
individuals.
While SSA has many uses for the information it collects, there is some
information not being collected that could enhance program oversight.
For example, after identifying unexplained racial differences in
decisions made at the OHA level, we recommended that SSA collect
information on race so that the agency can better monitor the equity of
its decisions.[Footnote 8] In this same study, we also recommended that
SSA collect information on the source of medical evidence (such as
specialist, primary physician, emergency physician) and type of medical
evidence (such as blood test, x-ray, observation, opinion), which could
be used to analyze and better understand the role the agency and
claimant representatives play in developing evidence for the disability
determination.
An SSA contractor also found the agency's electronic medical
information lacking for other purposes. In 2001, PricewaterhouseCoopers
(PwC) reported on the quality of electronic disability data used to
target cases for continuing disability reviews (CDRs)--i.e., reviews of
medical improvement of beneficiaries to determine their continuing
eligibility for disability benefits. In its report, PwC stated that
although SSA collects a large amount of medical evidence for SSA
disability decisions, very little of this medical information is
captured in SSA's electronic disability databases. Specifically, PwC
noted that only 8 of the 162 variables captured in the 831 record
relate directly to medical condition. PwC stated that despite the
importance of medical information to any decision on continuing
eligibility, a relatively small number of medical variables are
available in SSA's 831 record for use in determining the most cost-
effective method to target cases for CDRs.[Footnote 9]
In addition to medical information, SSA lacks electronic data on
claimants' vocational and functional capacity, which are critical
factors in the disability decision and are therefore important in a
study of the disability decision-making process.[Footnote 10] Nearly 60
percent of disability decisions cannot be made based solely on medical
considerations and must rely on these additional factors. Nevertheless,
in the CCS database, we found that no variable is related to vocational
and functional capacity. For the DDS level, we found one variable in
the 831 database that provides limited vocational and functional
information.[Footnote 11] The 831 database has no other variables that
capture additional factors often considered in making disability
decisions (such as mental conditions, fine motor skills, and stamina).
Moreover, for DDS decisions made in 2003, we found that the one
variable related to vocational and functional factors is captured in
only 10 percent of the cases in which such factors had to be considered
for the disability determination.[Footnote 12] In the remaining 90
percent of these cases, DDS staff did not capture any vocational or
functional factors used in the decision. Without this type of
information, SSA lacks electronic data on critical factors considered
in the disability decision and important to any study of the disability
decision-making process.
Mathematica--a contractor hired by SSA to evaluate new strategies for
promoting employment among people with disabilities--also noted that
important vocational and functional information was not being collected
electronically by SSA. Specifically, Mathematica reported that certain
characteristics (such as household composition, occupation, industry,
and the presence of functional limitations) are, according to the
research community, important predictors of employment, but they are
not available in SSA's electronic databases.
While SSA does not collect many types of information that contractors
and others believe would be useful for managing the agency's programs,
SSA may be collecting other data that are not particularly useful. SSA
officials told us that they believe information on the claimant's prior
occupation and industry could be useful, but because the lists of
industries and occupations used by SSA are largely outdated, the agency
considers these variables to be of limited value. Nevertheless, some
DDS staff members still collect information on the claimant's prior
occupation and industry.[Footnote 13]
SSA Does Not Have Adequate Internal Controls to Ensure That the
Electronic Data Accurately Reflect the Facts of the Case:
While SSA emphasizes the importance of prompt and accurate disability
determinations, the agency has not put sufficient emphasis on creating
a positive control environment that would ensure a reasonable level of
quality for its electronic disability data. SSA officials told us that
although the DDSs and OHA are required to ensure that disability
determinations are accurate, there is no specific requirement that
electronic records provided to SSA be accurate. Regulations specify
only that the DDSs must "provide the organizational structure,
qualified personnel, medical consultant services, and a quality
assurance function sufficient to ensure that disability determinations
are made accurately and promptly." SSA officials told us that while
some DDS offices may verify the accuracy of electronic data, SSA does
not require or suggest that the DDSs follow a specific strategy or
format for doing so. Some DDS and OHA staff and supervisors told us
that they were unaware of how SSA used the 831 and CCS records or why
it was important that they be accurate. If SSA required verification on
at least a sample of cases, it would indicate to DDS and OHA staff the
importance of accuracy in the electronic data.
SSA has some electronic controls in place to help ensure the quality of
data. These front-end data entry controls help ensure data quality by
limiting what can be keyed into certain variables. For example, some
variables accept only a numeric entry, and others, such as the Social
Security number, must be filled in or the record will be rejected. Many
date variables require a month/date/year entry, and certain dates must
fall within a prescribed range. For example, the application date must
be on or after a person's date of birth. In addition to these front-end
data entry controls, SSA also reviews the 831 file by checking for
unacceptable entries for certain variables. If SSA finds an increasing
trend of unacceptable entries being keyed into those variables, the
agency may issue additional guidance to staff in the field. On the
basis of the results of this review, SSA officials told us that while
the data may not be 100 percent accurate, they believe the data are
sufficiently accurate.
However, while such controls may reduce data entry errors, we believe
the data entry controls are inadequate. In reviewing SSA's electronic
records for disability decisions made by the DDSs and OHA in 2003, we
found that even though information keyed into the record had not been
rejected by the data entry controls, the record contained information
that was clearly wrong because it did not agree with other information
in the same record.[Footnote 14] This type of comparison was possible
for only a few variables--impairment code, body system, the code
indicating the basis for the disability decision, and the disability
decision itself--but these are important variables that contain
information crucial to SSA's management of its disability programs.
Although our analysis identified only a small number of errors, the
errors were in key variables. For example, GAO found that:
* More than 4 percent of DDS records for claims processed in 2003
contained codes for the body system and impairment that were
incompatible, indicating that either the impairment code or the body
system code was incorrect.
* For 211 claims allowed by the DDSs and 1,541 claims allowed by OHA,
the impairment code indicated an unknown impairment for which there was
no medical evidence.
* For 239 claims allowed by the DDSs and 519 claims allowed by OHA, the
impairment code indicated an unknown impairment for which there was
insufficient medical evidence.
* For nearly 4 percent of OHA's electronic disability records for
allowed claims, the impairment codes did not appear in the OHA list of
impairment codes.
* OHA's records indicated that 2,367 applicants, who applied for both
the DI and SSI programs, met SSA's medical criteria for one program but
not the other--a situation not allowed under SSA regulations, which
require that the same medical criteria apply to both programs.
Nevertheless, for each of these applicants, OHA recorded different
decision codes, primary impairment codes, or regulation basis codes for
the two programs.
Inadequate controls have also resulted in missing data. In reviewing
the 831 and CCS records, we found that for many of the variables,
leaving the variable blank is considered a valid entry. Therefore, when
variables are not filled in, it's not possible to determine if staff
had the information but failed to key it in, or if staff did not have
the information and correctly left it blank. Although for most
variables we were unable to determine that the variable should have
been filled in, the education variable was an exception. For cases in
which the adjudicator is required to consider educational level in
making the disability determination, the information should have been
obtained and keyed into the record.[Footnote 15] Yet, for 25 percent of
such cases, that variable was left blank. This raises the question of
how much of the missing data for other variables was available and
should have been keyed into the record.
Other organizations have also found weaknesses in SSA's data entry
controls. In its 2001 study for SSA, PwC reported that poor data
quality could present a significant barrier to efficient CDR case
selection. As a result, it recommended that the agency improve system
data edits, stating that such data entry controls would improve the
integrity of disability programs, reduce inaccuracies and
inconsistencies in data collection and reporting, and reduce the risk
of error or erroneous data collection and reporting. PwC estimated that
between 2001 and 2005, poor and missing disability data would cause SSA
to improperly target CDRs and thereby incur additional administrative
and program costs. An OIG report in 2000 reported that missing or
invalid diagnostic codes had contributed to SSA's failing to perform
mandatory reviews of certain disability cases and prevented the agency
from accumulating and disseminating more accurate disability
statistics.
Even when variables have been filled in and accepted by existing data
entry controls, SSA does not know the extent to which the data reflect
the information in the case file because the agency's policy does not
require that the electronic record be verified against the information
in the case file. SSA's case files contain all documents related to the
claim (such as medical records) and represent the legal record of the
claim. Although we did not perform a comparison of electronic records
against actual case files, a study by SSA's OIG in 2000 provides an
example of the type of inaccuracies that may exist in disability
records, even when the entries have been accepted by data entry
controls. The OIG reviewed case files for a sample of 132 disability
beneficiaries whose electronic records contained questionable
impairment codes. It found 63 electronic records that contained an
impairment code indicating, "diagnosis established--no predetermined
list code of medical nature applicable," although the case file
contained enough information to determine the proper impairment code.
At the same time, the OIG sample was quite small and not intended to be
representative. It is not cost-effective to check all, or even a large
percentage of, electronic records against case files, and perfect
reliability is not possible. However, without comparing a more
representative sample of electronic records against case files, SSA
cannot reliably estimate the true extent of inaccuracies in the
electronic data.
SSA's Ongoing and Planned Changes May Reduce the Chance for Some
Inaccuracies, but the Agency Has No Plans for Improving the Accuracy or
Usefulness of Electronic Data:
SSA's ongoing and planned changes to computer systems and internal
controls may reduce the chance for some inaccuracies. However, these
planned changes will not address data entry problems found by GAO that
could be prevented with additional data entry controls. SSA's current
plans also do not include an internal control strategy for ensuring
that electronic data match the information in the case file, nor do
they provide for corrective action when inaccuracies are found.
Finally, although SSA has proposed far-reaching changes to its
disability decision-making process and is currently reassessing the
processes for ensuring the quality of its disability decisions, the
agency has not yet made any specific plans for evaluating the types of
information it currently collects and whether other types of
information would improve program management and oversight.
SSA's Ongoing and Planned Changes May Reduce the Chance for Some
Inaccuracies:
Although most of SSA's ongoing and planned changes do not involve
aspects of the electronic data systems touched on in this report, a few
will alter the way information is transmitted and, in this way, could
reduce the number of chances for unintentional errors. Under eDib, SSA
field office staff keying in the disability history for an applicant
will now use the Electronic Data Collect System (EDCS). Once the field
office staff have keyed the Social Security number into the EDCS
record, EDCS automatically verifies that there is a disability
application in existing SSA databases. Information such as the
application date, disability allegations, and disability onset date are
automatically propagated into the EDCS record. This will improve the
accuracy of the information keyed into the EDCS record, which goes into
the electronic disability folder used at all levels of the disability
determination process. Some additional data entry controls that are
part of the new OHA case tracking software (the Case Processing
Management System--CPMS) that was implemented this year as part of eDib
may also reduce data entry errors.
SSA's Plans Do Not Include Strategies for Ensuring Data Accuracy or
Evaluating Types of Information Needed:
SSA's current plans do not include an overall internal control strategy
for ensuring the accuracy of electronic disability data. While SSA has
included some front-end data entry controls, it does not address
problems with electronic data we found that could be prevented with
additional controls. SSA also has no plans to improve its system for
the back-end review of electronic disability records, so that the
agency can estimate the accuracy of its electronic disability records
and provide feedback to staff on accuracy. Further, the agency has no
specific plans to evaluate the types of information collected during
the disability determination process.
Most of the changes currently under way--such as creating a shared
electronic folder for each disability case, scanning evidence for
inclusion in the electronic folder, providing new software systems, and
improving interfaces between systems--were intended to improve the
speed and efficiency of the disability claims process and not
necessarily to improve the reliability or usefulness of the data in
SSA's 831 and CCS databases. To accelerate the rollout of eDib, SSA
carried over the requirements from the old systems to the new systems.
Although the agency did introduce some additional front-end data entry
controls to reduce the entry of invalid data, the agency did not add
sufficient controls to address the problem with missing and invalid
data we found. An OHA official told us that even the additional data
entry controls available in the new CPMS software are not adequate to
prevent data entry errors we identified in this study. For example,
under eDib, SSA's data entry control still allows any four digits for
the impairment variable in the CPMS, as well as the 831 file, rather
than limiting data entry to one of the 240 acceptable impairment codes.
These officials said that limited resources have prevented the agency
from establishing and maintaining additional or more exacting controls-
-such as a current list of acceptable impairment codes. Moreover, SSA
officials told us that additional data entry controls would require
staff to determine the correct entry when errors were identified, which
could slow case processing.
SSA also has no plans to improve its system for the back-end review of
electronic disability records, so that the agency can estimate the
accuracy of its electronic disability records and provide feedback to
staff on accuracy. SSA already has a process in place for reviewing a
random sample of case files as part of its quality assurance for
disability decisions, which allows the agency to make estimations of
the accuracy of decisions at both the DDS and the OHA levels. In the
course of this quality assurance review, staff are supposed to verify
the electronic 831 record against the information in the case file and
note errors.[Footnote 16] If a significant error is found, staff are
directed to fax a corrected 831 form to a central location so that the
correction can be made in the 831 file. However, SSA officials told us
that corrections don't always end up in the 831 file, and the agency
does not track input errors because such errors are low on the priority
list. Because there is no tracking system that would collect
statistical data on the number or type of errors, SSA is unable to
determine the level of accuracy for this electronic data. A tracking
system that provided feedback to the field would also send a message to
staff that the items being tracked are important--a basic element in
any quality assurance system. Without the statistical data and feedback
to staff, the reviews for data quality affect only those records
reviewed and do not affect the possible quality of the rest of the
records. Finally, the back-end review covers only 831 records, not CCS
records.
SSA has not yet made concrete plans for evaluating the types of
information it currently collects and whether other types of
information would improve program management and oversight. Users of
administrative data that we interviewed indicated that they have not
been surveyed as to their data needs, and some indicated that their
requests for additional data collection have not yet been implemented.
However, the agency may take such steps after other priorities are
accomplished. An SSA official stated that SSA's first priority was to
implement eDib, in order to modernize its core business process of
taking and adjudicating claims. The official also stated that after
eDib is fully implemented, SSA will assess whether the information
collected is sufficient.
In more recent interviews, high-level SSA officials validated the
importance to the agency of reliable and useful data for program
management purposes, and indicated a strong interest in developing a
comprehensive information management plan that would allow the agency
to systematically and regularly assess its information needs and make
appropriate adjustments to its computer systems. According to agency
officials, this effort would include a review of best practices in
information management. However, the ideas expressed were still being
formulated, and the agency has not yet taken any concrete actions.
Conclusion:
SSA lacks a comprehensive strategic plan for collecting useful and
reliable data for effective oversight and program planning. Despite the
fact that the reliability of its data has been an ongoing issue, the
agency currently has no specific plans to implement additional front-
end data entry controls or for a tracking and feedback system for the
back-end verification of the electronic records. In addition, the
agency has no specific plans for evaluating whether the types of
information it currently collects continue to be useful for program
oversight and whether other types of information would contribute to
oversight.
While stretched resources and other priorities may have prevented SSA
from addressing these issues to date, SSA plans to make more changes to
its disability processes, which will create additional opportunities
and a sense of urgency for improving the usefulness and reliability of
its electronic disability data. For example, in September 2003, SSA's
Commissioner proposed significant changes to its disability decision-
making process--such as the proposed use of centralized medical
expertise and full documentation of initial-level decisions--that will,
if implemented, translate into new requirements for collecting
administrative data to monitor and assess these changes. Moreover, as
SSA continues to transition from a paper-based case-processing system
to an electronic one, SSA will have additional opportunities to
institute changes that would improve upon the usefulness and
reliability of the data that it routinely collects. Finally, SSA has
been considering fundamental changes to its quality assurance
processes, at both the front and back ends of its disability
determination process. Combined, these changes, each significant and
interrelated, create a compelling case for SSA to broadly assess and
improve the nature and reliability of the data it uses to manage its
disability programs.
Recommendations for Executive Action:
To improve the value of SSA's electronic administrative data for
managing its disability programs, we recommend that the Commissioner of
Social Security develop a comprehensive strategic plan to ensure the
reliability and usefulness of the data the agency collects. In doing
so, the agency should take the following steps:
1. Establish a cost-effective internal control strategy for ensuring
the reliability of data in the electronic disability records that would
include both front-end controls on data entry and a tracking and
feedback system for back-end verification of the electronic records.
2. Take steps to review the usefulness of the types of information
collected and consider whether additional types of information could
improve program oversight. This effort could include a survey of users
of electronic disability data.
Agency Comments and Our Evaluation:
We provided a draft of this report to SSA for comment. SSA generally
agreed with our conclusions and recommendations, acknowledging that the
report made valid points about steps SSA can take to increase the
usefulness of electronic data for program oversight and that the agency
would consider how best to incorporate improvements in its data
collection activities.
SSA agreed in theory with our first recommendation and noted that the
agency is continuing to examine future software enhancements to improve
front-end controls on data entry and back-end review. SSA also noted
that its back-end review of cases may provide some check on data entry.
We agree and modified our report and first recommendation to further
clarify this. However, SSA expressed concern regarding the cost and
technical feasibility of implementing this recommendation. We agree
that the agency needs to balance enhanced internal controls with
operational costs and productivity, but we believe some of the specific
concerns raised by SSA are unwarranted. For example, as discussed in
our response to SSA's comment in appendix I, we believe that the
internal controls proposed by GAO would not require additional
"structured" data or a data collection system "robust enough to catalog
and aggregate the information."
In its response to our second recommendation, SSA agreed that the
disability program would be well served by reviewing all the
information that SSA could potentially collect on a disability claim
and determining which of these data elements might be beneficial for
program oversight and policy development. SSA also stated that the
agency is drafting a statement of work to contract for an assessment of
data needs for managing the disability programs, which will identify
possible sources and methods for obtaining data and make
recommendations for improving existing data and data retrieval. The
agency further observed that it would have to consider the potential
adverse effect of increased electronic data collection on productivity
and timeliness.
SSA provided additional general comments, which we have included (along
with our responses to them) in appendix I and addressed in the body of
our report where appropriate. SSA also provided technical comments that
we have incorporated in the report as appropriate.
We are sending copies of this report to the appropriate congressional
committees and other interested parties. We will also make copies
available to others on request. In addition, the report will be
available at no charge on GAO's Web site at http://www.gao.gov.
If you or your staff have any questions concerning this report, please
contact me or Michele Grgich, Assistant Director, at (202) 512-7215.
You may also reach us by e-mail at robertsonr@gao.gov or
grgichm@gao.gov. Other major contributors to this assignment were Ann
T. Walker, Jill D. Yost, and Corinna Nicolaou.
Sincerely yours,
Signed by:
Robert E. Robertson:
Director, Education, Workforce, and Income Security Issues:
[End of section]
Appendix I: Comments from the Social Security Administration:
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
SOCIAL SECURITY:
The Commissioner:
November 4, 2004:
Mr. Robert E. Robertson:
Director, Education, Workforce and Income Security Issues:
U.S. Government Accountability Office:
Room 5-T-57:
441 G Street, NW:
Washington, D.C. 20548:
Dear Mr. Robertson:
Thank you for the opportunity to review and comment on the draft report
"SSA's Disability Programs: Improvements Could Increase the Usefulness
of Electronic Data for Program Oversight" (GAO-05-100R). Our comments
on the report are enclosed.
If you have any questions, please have your staff contact Candace
Skurnik, Director, Audit Management and Liaison Staff at (410) 965-
4636.
Sincerely,
Signed by:
Jo Anne B. Barnhart:
Enclosure:
SOCIAL SECURITY ADMINISTRATION BALTIMORE MD 21235-0001:
COMMENTS ON THE GOVENMENT ACCOUNTABILITY OFFICE (GAO) DRAFT REPORT
"SSA'S DISABILITY PROGRAMS: IMPROVEMENTS COULD INCREASE THE USEFULNESS
OF ELECTRONIC DATA FOR PROGRAM OVERSIGHT" (GAO-05-100R):
We appreciate the opportunity to comment on the GAO draft report
concerning recommended improvements to the Social Security
Administration's (SSA) disability programs which would increase the
usefulness of electronic data for program oversight. We recognize the
importance of the Disability Insurance (DI) and Supplemental Security
Income (SSI) programs to the American public and the need for reliable
administrative data from our disability decision-making process to
adequately understand the population we serve.
The report makes valid points about steps SSA can take to increase the
usefulness of electronic data for program oversight, and we will
consider how best to incorporate such improvements in our data
collection activities under electronic disability (eDib) and the
Commissioner's new disability approach. However, since these
initiatives might be resource intensive, we will need to balance
operational cost and productivity concerns as we pursue enhanced data
collection controls. In addition, we are drafting a "statement of work"
(SOW) to contract for an assessment of data needs for managing the
disability programs. The SOW will require the contractor to identify
possible sources and methods for obtaining data and make
recommendations for improving existing data and data retrieval.
We also appreciate GAO's acknowledgement that most types of information
collected by SSA are useful for program purposes. We know data quality
has improved greatly over the years as evidenced by the findings in
PricewaterhouseCoopers' (PwC) February 2001 report entitled
"Disability Quality Review" and in our validations of data that are
done periodically during the year.
The report states there is no review of coding to ensure the inputs are
correct. However, in conducting the quality reviews (e.g., pre-
effectuation review), disability examiners do check the coding and make
pen and ink changes and corrections on the SSA-831. The changes they
make are prepared for systems input. Our Disability Quality Branches
review several thousand cases a year. However, the Agency does not keep
track of these types of errors for statistical purposes.
We disagree with the statements that SSA's current plans do not include
an overall internal control strategy for ensuring the accuracy of
electronic disability data or current plans do not provide for a back-
end review to ensure that electronic data match the information in the
case file. As the electronic folder is developed and the eDib is
rolled-out nationwide, software enhancements addressing the data entry
and data collection needs are being addressed. So far, limited
resources have prevented the Agency from establishing and maintaining
additional or more exacting controls.
As to specific points, the first paragraph on page 2 of the report
notes that SSA lacks an indicator on claimant's race. We recognize that
this longstanding issue has prevented us from conducting certain types
of data analyses. Therefore, in September 2003, SSA convened a
workgroup to develop a plan to collect race/ethnicity information
during the initial claims process.
The first paragraph on page 6 notes that limited medical information
contained in the agency's 831 records is available for determining the
most cost effective method for targeting cases for continuing
disability reviews (CDR). While we agree with this assessment, it
should be noted that SSA relies on other internal Agency data such as
information contained in the Master Beneficiary Record System and
Supplemental Security Record System in the CDR process. In addition, it
should be noted that since the 2001 PwC study was referenced in the
report, SSA has systematically incorporated the use of additional
electronic data in the CDR process including medical information
obtained from Medicare claims records and individual responses from the
Agency's CDR mailer forms.
The second to last paragraph on page 8 references the 2001 PwC study
which reported that poor data quality could present a significant
barrier to efficient CDR case selection. While we agree with the
results of that study for the time period it was written, it should be
noted that the completeness and overall reliability of the Agency's
electronic data has improved significantly since that time. Data
capture policies and practices have strengthened, and the number of
cases allowed prior to implementation of many current system data edits
have steadily diminished. As noted in the study, for example, the
proportion of entitled DI beneficiaries with missing or invalid primary
diagnosis codes in 831 records dropped from about 20 percent in 1989 to
about six percent in 1999. Significant further data quality
improvements have occurred since the time the PwC study reported
estimates of potential CDR targeting inefficiencies due to poor data
quality.
We offer the following comments on the two GAO draft report
recommendations:
Recommendation 1:
Establish a cost-effective internal control strategy for ensuring the
reliability of data in the electronic disability records that would
include both front-end controls on data entry and back-end reviews of
data entry to ensure that the record accurately reflects the
information in the case file. For example, staff performing back-end
quality reviews of cases could verify the accuracy of these randomly
sampled electronic records by comparing them with the case files.
Response:
We agree in theory and we are continuing to examine future software
enhancements to improve front-end controls on data entry, obtain useful
data, and improve back-end reviews. However, we are concerned about the
costs required to create such a data collection and maintenance
structure and need to determine cost-effectiveness.
In order to attain this goal, we first would need to support the
addition of most data into the electronic record as 'structured' data.
In these early phases of transition to an electronic disability
process, much of the material and documents added to the electronic
file are 'tiff images (as an example, the SSA-831). As long as data is
unstructured, we will have difficulty aggregating and reporting based
on these data.
In a broader sense, the key is the creation of a "cost effective"
method of front-end data capture, and back-end reviews or audits. We
need to remain cognizant of the fact that Social Security disability
determinations are built up of data that is often not 'proof-certain.'
As an example, in claim "A," we receive five pieces of medical
evidence:
Claim "A" MER #: MER 1;
Primary Diagnosis: Diabetes;
Secondary Diagnosis: Arthritis.
Claim "A" MER #: MER 2;
Primary Diagnosis: Diabetic Nephropathy.
Claim "A" MER #: MER 3;
Primary Diagnosis: DJD;
Secondary Diagnosis: Diabetes Mellitus.
Claim "A" MER #: MER 4;
Primary Diagnosis: Type 11 Diabetes;
Secondary Diagnosis: Osteoarthritis.
Claim "A" MER #: MER 5;
Primary Diagnosis: Arthritis, knees, bilateral;
Secondary Diagnosis: Renal Disease.
[End of table]
What should the 'official' primary and secondary diagnoses be in the
above case scenario? Here a decision-maker will have to select a
diagnosis from a set of diagnoses, code them in a way that captures
different names for the same types of impairments, and make priority
selections. After all this is done, we would need to have a data
collection system robust enough to catalog and aggregate the
information, along with a system for either manual or automated edits.
These are all important, but they are neither easy nor inexpensive in
start up or ongoing costs.
Finally, such a post-review would require a comparison of the paper
documents to electronic documents. Once SSA adopts an electronic
folder, we will forego the paper or source documents. Thus, it is
difficult to see how we can agree to a post-review.
Recommendation 2:
Take steps to review the usefulness of the types of information
collected and consider whether additional types of information could
improve program oversight. This effort could include a survey of users
of electronic disability data.
Response:
In general, we agree that the disability program would be well-served
by reviewing all the information that we could potentially collect on a
disability claim and determining which of these data elements might
potentially be beneficial for program oversight and policy development.
We agree that sufficient internal controls over electronic data systems
are necessary to effectively manage programs. Some information
currently collected may be of limited value and there is additional
data the Agency could collect (e.g., household composition, occupation,
industry, and the presence of functional limitations) that are
important predictors of employment. The collection of these additional
characteristics would be useful in meeting the Agency's strategic
objective to increase employment for people with disabilities. With the
rapid evolution of the electronic disability system, the proposed
changes to SSA's policies regarding the development of internal
controls to ensure that data are reliable and useful come at an
appropriate time.
We would note that efforts are underway to ensure the accuracy of
electronic data. With respect to the specific recommendation to develop
a front-end validation process, the Agency has already established a
strategy for ensuring electronic claims data is accurate and complete.
In terms of a back-end quality control process, the Agency will
continue to rely on its existing end-of-line quality review procedure
for determining the accuracy of electronic records. We agree that
consideration should be given to the relevancy of data collected for
disability program purposes. However, we will also give consideration
to the potential adverse effect increased electronic data collection
may have on productivity and timeliness.
GAO Comments:
1. We acknowledge that the Disability Quality Branches' (DQB) back-end
review of cases may provide some check on data entry, and we have
adjusted our recommendation and text in our report to clarify this.
However, SSA officials told us that tracking of input errors is low on
DQB's priority list, and DQB changes don't always end up in the 831
file. Further, as SSA indicated, it does not track input errors for
statistical purposes; therefore SSA does not know the level of data
accuracy. A tracking system that also provides feedback to the field
would also send a message to staff that the items being tracked are
important--a basic element in any quality assurance system. Without the
statistical data and feedback to staff, the reviews for data quality
affect only those records reviewed, and do not affect the possible
quality of the rest of the records. Finally, the back-end review by DQB
covers only 831 records (i.e., DDS decisions), not CCS records (i.e.,
ALJ decisions).
2. Our report recognizes that as part of the implementation of eDib,
SSA is making software enhancements that may address some data entry
problems. However, we believe that the eDib enhancements fall short of
an effective overall strategy. Our previous comment explains why we
believe SSA's back-end reviews do not sufficiently ensure that
electronic data match the information in the case file.
3. We amended the report to include information about the work group.
4. We have incorporated into our report the additional sources of
information used in the CDR targeting process.
5. We recognize that the agency has improved data capture policies and
practices over the years, but in our study, we examined electronic
disability data for 2003, the most recent year available, and found
problems with that data. In addition, without an effective system for
verifying the electronic record against the case file and a tracking
system for errors, neither SSA nor we can determine the extent to which
the electronic disability data are accurate. Further,
PricewaterhouseCoopers' projection that SSA would incur hundreds of
millions of dollars in CDR administrative and program costs between
2001 and 2005 because of poor and missing electronic disability data
was made despite its acknowledgment that SSA's electronic disability
data had improved. In order to not overstate PwC's findings, we
adjusted the text of our report to say that the agency would incur
"additional administrative and program costs" because of problems with
the quality of its electronic disability data.
6. We do not agree that additional "structured" data are necessary
before data quality can improve. The 831 record is an electronic
version of the SSA-831 form, in a structured data format. For cases
processed prior to the use of the electronic case folder, a printout of
the completed SSA-831 form was placed in the paper case file to
document the final decision. Under eDib, which uses an electronic case
folder, an electronic image ("tiff" image) of the SSA-831 form is
created and placed in the electronic case folder (along with other
electronic images such as medical records) to document the final
decision. Regardless of this 831 image, the original 831 record still
exists as structured data and becomes part of a national database known
as the "831 file." SSA is still able to aggregate and report on these
data. Therefore, it is possible for SSA to implement our recommendation
that it compare the information in the electronic 831 record against
the information in the case file or the electronic case folder, and
correct the electronic 831 record as needed. Even after the agency
eliminates paper source documents and relies totally on electronic
documents in the case folder, SSA will still be able to compare the
electronic 831 record against the electronic images of medical records
and other documents related to the disability determination that are in
the case folder.
7. We agree that it may be difficult to determine the primary
impairment. However, our concern is that once a disability examiner
makes the decision, the 831 record should accurately reflect the
disability decision. For example, if the primary impairment is
determined to be diabetes, then the 831 record should contain the four-
digit impairment code for diabetes and the appropriate body system code
for that primary impairment. SSA could develop data entry controls that
would allow only acceptable impairment codes and the corresponding body
system code. The agency would have to update those listings only when
changes in policy dictated changes in impairment codes or body system
codes. This type of front-end data entry control should not require a
"data collection system robust enough to catalog and aggregate the
information."
[End of section]
FOOTNOTES
[1] GAO, SSA Disability Decision Making: Additional Steps Needed to
Ensure Accuracy and Fairness of Decisions at the Hearings Level, GAO-
04-14, Washington, D.C.: Nov. 12, 2003.
[2] While most claimants may request a reconsideration, at the time of
our study, SSA was testing an initiative that eliminates the
reconsideration step from the DDS decision-making process.
[3] SSA's database of 831 records is called the National Disability
Determinations Service System (NDDSS) but is generally referred to as
the "831 file." Therefore, in this report, we use the term "831 file"
or "831 database" to refer to the NDDSS.
[4] Electronic images are like electronic photocopies of the documents.
They can be retrieved or viewed in their entirety, but the elements
that constitute the image--such as words or numbers--are not in a
format that allows them to be identified using electronic search
mechanisms or readily retrieved and manipulated for reporting or
analytical purposes.
[5] Although these electronic images cannot be manipulated, data keyed
into the 831 and CCS databases are in a format that allows for easy
identification, retrieval, and manipulation for reporting or analytical
purposes.
[6] Once eDib is fully implemented, instead of reviewing a paper case
file, quality assurance staff will review electronic documents in the
electronic disability folder, including medical records and other
information related to the disability determination.
[7] Outside researchers and academics using SSA disability data include
Kajal Lahiri, Denton R. Vaughan, and Bernard Wixon, "Modeling Social
Security's Sequential Disability Determination Using Matched SIPP
Data," Social Security Bulletin, 58, 3-42; Congressional Research
Service Report to the Congress (1992), "Status of the Disability
Programs of the Social Security Administration," 92-691 EPW, the
Library of Congress, Washington, D.C.; and Eddy Bresnitz, Howard
Frumkin, Lawrence Goldstein, David Neumark, Michael Hodgson, and
Carolyn Needleman, "Occupational Impairment and Disability among
Applicants for Social Security Disability Benefits in Pennsylvania,"
American Journal of Public Health 84 (November 1994), 1786-1790.
[8] GAO-04-14. SSA officials told us that the agency formed a work
group in September 2003 to develop a plan for collecting race/ethnicity
information.
[9] In targeting cases for CDRs, SSA also uses information contained in
the Master Beneficiary Record system and the Supplemental Security
Record system, as well as information from Medicare claim forms and
individual responses to prior CDR mailer forms.
[10] Vocational factors are age, education and work experience. A
claimant's residual functional capacity is the individual's maximum
sustained capability for sedentary, light, medium, heavy, or very heavy
work.
[11] The vocational rule number variable indicates certain vocational
and functional patterns that include functional capacity, age, work
history, and education level.
[12] SSA uses a five-step disability determination process whereby the
agency considers vocation and functional capacity in steps 4 and 5,
i.e., when it has determined that the claimant has a severe, medically
determinable physical or mental impairment, but the impairment is not
severe enough for the individual to be determined eligible based on
medical factors alone. In steps 4 and 5, SSA uses these additional
factors to determine whether the claimant's impairment(s) prevents the
performance of his or her past work or other work in the national
economy. Federal regulations contain rules (vocational rule numbers)
that direct a finding of disabled or not disabled based on vocational
factors (such as age, education, and work experience) in combination
with the individual's residual functional capacity.
[13] In 1992, when the DDS workload was heavy, SSA informed the DDSs
that the variables for industry, occupation, and years in occupation
did not need to be filled in. Although this was intended to be a
temporary measure, SSA's incentive for re-instituting the policy
diminished as the job classifications grew more outdated over time.
[14] We reviewed 3.9 million 831 records for disability decisions made
in 2003, of which 31 percent (1.2 million) were allowed. We also
reviewed over 400,000 CCS records for disability decisions made in
2003, of which 70 percent (nearly 282,000) were allowed.
[15] Over 50 percent of all claims processed by DDSs in 2003 required
that educational level be considered in making the disability
determination.
[16] Under eDib, such a back-end review would involve matching the 831
or CCS record against the electronic images of medical records and
other documents related to the disability determination.