Immigration Benefits System: U.S. Citizenship and Immigration Services Can Improve Program Management

Highlights

What GAO Found

U.S. Citizenship and Immigration Services (USCIS) created a reliable updated estimate to project the Transformation Program's cost, but has experienced program management challenges. In particular, the program's cost estimate was well-documented and substantially comprehensive, accurate, and credible. However, among other things, software development and systems integration and testing for USCIS's Electronic Immigration System (USCIS ELIS) have not consistently been managed in line with the program's policies and guidance or with leading practices.

Regarding software development, the Transformation Program has produced some software increments, but is not consistently following its own guidance and leading practices. The software development model (Agile) adopted by the USCIS Transformation Program in 2012 includes practices aimed at continuous, incremental release of segments of software. Important practices for Agile defined in program policies, guidance, and leading practices include ensuring that the software meets expectations prior to being deployed, teams adhere to development principles, and development outcomes are defined. The table below lists the program's status in addressing eight key Agile development practices. For example, the program has committed to a specific framework for software development, referred to as Scrum, but has deviated from the underlying practices and principles of this framework.

Table: Implementation of Key Agile Practices for USCIS ELIS

Practice	Rating
Completing planning for software releases prior to initiating development and ensuring software meets business expectations prior to deployment	◐
Adhering to the principles of the framework adopted for implementing Agile software development	◐
Defining and consistently executing appropriate roles and responsibilities for individuals responsible for development activities	◐
Identifying users of the system and involving them in release planning activities	◐
Writing user stories that identify user roles, include estimates of complexity, take no longer than one sprint to complete, and describe business value	◐
Prioritizing user stories to maximize the value of each development cycle	●
Setting outcomes for Agile software development	○
Monitoring and reporting on program performance through the collection of reliable metrics	◐

Source: GAO analysis of USCIS documentation. I GAO-16-467.

Note: ● yes ◐ partial ○ noc

The Transformation Program has established an environment that allows for effective systems integration and testing and has planned for and performed some system testing. However, the program needs to improve its approach to system testing to help ensure that USCIS ELIS meets its intended goals and is consistent with agency guidance and leading practices. Among other things, the program needs to improve testing of the software code that comprises USCIS ELIS and ensure its approaches to interoperability and end user testing, respectively, meet leading practices. Collectively, these limitations have contributed to issues with USCIS ELIS after new software is released into production.

Why GAO Did This Study

Each year, USCIS processes millions of applications for persons seeking to study, work, visit, or live in the United States, and for persons seeking to become a U.S. citizen. In 2006, USCIS began the Transformation Program to enable electronic adjudication and case management tools that would allow users to apply and track their applications online. In 2012, to address performance concerns, USCIS changed its acquisition strategy to improve system development.

In May 2015, GAO reported that USCIS expected the program to cost up to $3.1 billion and be fully operational by March 2019. This includes more than $475 million that was invested in the initial version of the program's key case management component, USCIS ELIS, which has since been decommissioned.

This report evaluates the extent to which the program is using information technology program management leading practices. To perform this work, GAO identified agency policy and guidance and leading practices in, among other things, cost estimation, Agile software development, and systems integration and testing, and compared these with practices being used by the program.

Recommendations

GAO is making 12 recommendations to improve Transformation Program management, including ensuring alignment among policy, guidance, and leading practices in areas such as Agile software development and systems integration and testing. DHS concurred with the recommendations.

Recommendations for Executive Action

Agency Affected	Recommendation	Status
Department of Homeland Security	To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of the Department of Homeland Security (DHS) should direct the Director of USCIS to direct the USCIS Chief Information Officer (CIO), in coordination with the DHS CIO and the Chief of the Office of Transformation Coordination (OTC), to review and update, as needed, existing policies and guidance and consider additional controls to complete planning for software releases prior to initiating development and ensure software meets business expectations prior to deployment.	Closed – Implemented As of April 2020, USCIS has demonstrated that it has taken sufficient steps to close this recommendation. In January 2020, USCIS updated its management instruction governing planning and deploying software releases. USCIS policy requires programs to document its Agile processes in a USCIS Service Hub, an online repository of information for USCIS systems. For example, the USCIS Service Hub for the Transformation Program documents a roadmap with prioritized capabilities and projected implementation dates. The program has also modified its deployment approach, relying on Team Managed Deployments (TMDs). USCIS provided TMD documentation for releases that are required by the updated policy, including the readiness review approval memo for release 14. The program also works with an independent verification and validation team to help assess outcomes called for by the USCIS management instruction. These actions should help to ensure that the program completes planning for software releases prior to initiating development and that the software meets business expectations prior to deployment.
Department of Homeland Security	To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to consistently implement the principles of the framework adopted for Agile software development.	Closed – Implemented As of August 2019, the Department of Homeland Security (DHS) has demonstrated that it has taken sufficient steps to close this recommendation. In May 2017, USCIS provided updated policy governing the development of software releases, dated April 2017, along with release planning artifacts specific to USCIS ELIS. The updated policy included an appendix devoted to generally accepted agency practices and applying Agile principles in the agency. The program was in breach status as of October 2016. As a result, in December 2016, most new software development activities were placed in a strategic pause. This breach condition was remediated in July 2018 with the approval of a new program baseline. With this new baseline, the strategic pause was lifted and development restrictions were removed. In May 2019, USCIS again updated policy governing the development of software releases along with the associated addendum on Applying Lean-Agile-DevSecOps Principles at USCIS. The Transformation program also works with an independent verification and validation team to help assess program outcomes called for by the USCIS management instruction governing planning and deploying software releases. The outcome of this effort is a monthly report that assesses the Transformation program's adherence to the management instruction. USCIS provided monthly reports for May 2019, June 2019 and July 2019 as evidence of additional controls in place to consistently implement the principles of the framework adopted for Agile software development.
Department of Homeland Security	To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to define and consistently execute appropriate roles and responsibilities for individuals responsible for development activities consistent with its selected development framework.	Closed – Implemented As of August 2019, the Department of Homeland Security (DHS) has demonstrated that it has taken sufficient steps to close this recommendation. In May 2017, USCIS provided updated policy governing the development of software releases, dated April 2017, along with release planning artifacts specific to USCIS ELIS. The updated policy included an appendix devoted to generally accepted agency practices and applying Agile principles in the agency. The updated policy and release documentation defined some roles and responsibilities that were previously only described by USCIS in its informal November 2014 management model, such as the authority and responsibility of a product owner. The program was in breach status as of October 2016. As a result, in December 2016, most new software development activities were placed in a strategic pause. This breach condition was remediated in July 2018 with the approval of a new program baseline. With this new baseline, the strategic pause was lifted and development restrictions were removed. In May 2019, USCIS again updated policy governing the development of software releases along with the associated addendum on Applying Lean-Agile-DevSecOps Principles at USCIS. In addition, the Transformation program works with an independent verification and validation team to help assess program outcomes called for by the USCIS management instruction governing planning and deploying software releases. The outcome of this effort is a monthly report that assesses the Transformation program's adherence to its policy. USCIS provided the monthly reports for May 2019, June 2019 and July 2019 as evidence of additional controls in place to consistently execute roles and responsibilities for individuals defined in policy. USCIS also includes a slide on team composition as part of in the Quarterly Program Review presentation that is prepared for the DHS Acquisition Review Board. Additional roles are further captured in the Service Hub, a repository for current information on the program and individual releases of software.
Department of Homeland Security	To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to identify all system users and involve them in release planning activities.	Closed – Implemented As of June 2020, USCIS has demonstrated that it has taken sufficient steps to close this recommendation. As part of the program's transition from the Office of Transformation Coordination (OTC) to the Office of Information Technology (OIT), the program modified its development approach to focus on portfolios instead of functionality. This shift allowed for increased end user engagement throughout the development process to include daily interaction between the ELIS Portfolio Managers and subject matter experts from other offices, such as Service Center Operations and the Field Operations Directorate. The experts were in turn responsible for developing acceptance criteria for user stories and active in daily user story demos and monthly ELIS program demos. The program also employed Beta Testing and Usability Testing to further elicit feedback from the end user community. These actions should help ensure the program identifies and involves system users in release planning and subsequent activities.
Department of Homeland Security	To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to write user stories that identify user roles, include estimates of complexity, take no longer than one sprint to complete, and describe business value.	Closed – Implemented As of June 2020, USCIS has demonstrated that it has taken sufficient steps to close this recommendation.. In January 2020, USCIS updated its policy governing the development of software releases along with release planning artifacts specific to USCIS ELIS and an IV&V assessment. Although policy no longer specifies the user story format, it does require programs to document Agile processes in a USCIS Service Hub, including a roadmap with prioritized features and projected implementation dates. According to the USCIS Service Hub, the program is now employing a Kanban approach to Agile software development, a methodology that does not utilize increments of development or necessarily rely on estimates of complexity the same way a Scrum approach does. As such, the program no longer expects user stories to be completed in less than one sprint or include estimates of complexity. The agency provided a series of backlogs that captured user stories for select software releases. These user stories identified user roles and traced back to the prioritized features for the release. In addition, IV&V assessments of the program tracked user story quality as part of assessing whether value was continuously discovered and aligned to the mission. This assessment also tracked the length of time to complete each user story as part of assessing team work flows. These actions should help ensure the program writes and executes user stories in order to deliver value.
Department of Homeland Security	To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to establish outcomes for Agile software development.	Closed – Implemented In April 2017, the USCIS Chief Information Officer (CIO) issued a Management Instruction on Agile development that defined ten outcomes that all USCIS IT programs are required to achieve. These ten outcomes are that (1) programs and projects frequently deliver valuable product; (2) value is continuously discovered and aligned to mission; (3) work flows in small batches and is validated; (4) quality is built in; (5) the organization continuously learns and improves; (6) teams collaborate across groups and roles to improve flow and remove delays; (7) security, accessibility and other compliance constraints are embedded and verifiable; (8) consistent and repeatable processes are used across build, deploy, and test; (9) the entire system is deployable at any time to any environment; and (10) the system has high reliability, availability, and serviceability. In addition, the appendix to this revised instruction identified a series of generally accepted practices that programs may or may not choose to adopt and how each practice can help the program to achieve one or more of the defined outcomes. USCIS also provided a series of quality assurance review products demonstrating that the agency is monitoring the relative success of the program against each of the ten outcomes identified in the USCIS instruction.. As a result, we consider this recommendation to be closed as implemented.
Department of Homeland Security	To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to monitor program performance and report to appropriate entities through the collection of reliable metrics.	Closed – Implemented As of April 2020, USCIS has demonstrated that it has taken sufficient steps to close this recommendation. In January 2020, USCIS updated its policy governing planning and deploying software releases. USCIS policy defined outcomes that all USCIS IT programs are required to achieve. USCIS also provided a series of independent verification and validation reports demonstrating that the agency monitored the relative success of the program against each of the outcomes identified in the USCIS policy. DHS also published a set of Agile core metrics to monitor program performance beyond just cost, schedule, and key performance parameters. For example, the Agile core metrics include story points completed relative to story points planned to be completed. The department also updated its Acquisition Review Board presentation slides to require that Agile programs submit these core metrics. These actions should help to monitor program performance and ensure appropriate reporting.
Department of Homeland Security	To help manage the USCIS ELIS system, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update existing policies and guidance and consider additional controls to conduct unit and integration, and functional acceptance tests, and code inspection consistent with stated program goals.	Closed – Implemented As of June 2020, USCIS has demonstrated that it has taken sufficient steps to close this recommendation. In May 2017, USCIS provided artifacts from internal systems in place to monitor software development performance. These metrics monitored aspects of testing, such as code quality and code coverage. As part of the program's approval for a re-baseline in July 2018, the program developed an updated Test and Evaluation Master Plan. The updated plan set new testing expectations for the program as agreed upon with its stakeholders in DHS and USCIS. In January 2020, USCIS also updated its policy governing the development of software releases and release planning artifacts specific to USCIS ELIS and an IV&V assessment. Among other things, the IV&V assessment monitors code quality, including unit and integration test coverage and functional acceptance, through tracking of closed user stories, ticket resolutions, and user satisfaction. Similar metrics are also reported In ARB briefings along with performance progress compared to the prior reporting period. These actions should help ensure the program conducts testing and code inspection consistent with stated program goals.
Department of Homeland Security	To help manage the USCIS ELIS system, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update existing policies and guidance and consider additional controls to develop complete test plans and cases for interoperability and end user testing, as defined in the USCIS Transformation Program Test and Evaluation Master Plan, and document the results.	Closed – Implemented As of June 2020, USCIS demonstrated that it has taken sufficient steps to close this recommendation. As part of the program's approval for a re-baseline in July 2018, the program developed an updated Test and Evaluation Master Plan. The updated plan set new testing expectations for the program as agreed upon with its stakeholders in DHS and USCIS. Specifically, the new plan replaced End User Testing with Beta and Usability testing. This approach allows the program to test the functionality of the system in a production environment with a subset of users prior to a full deployment to all users, rather than having end users interact with the system in a non-production environment. In a written response, the program added that Transformation team members are also continuously working with interface partners to help them resolve issues impacting the performance and reliability of transactions/message exchange between their system and the ELIS platform - in particular, critical system interfaces. In response to interface concerns, the program has worked with interface partners to reduce the impacts to ELIS digital case processing, deployed tools and utilities to perform active logging and monitoring, and held regular meetings with interface partners. These actions should help ensure the program conducts testing and code inspection consistent with stated program goals.
Department of Homeland Security	To help improve oversight of selected Transformation Program contracts, the Secretary of DHS should direct the Director of USCIS to direct the Chief of the USCIS Contracting Office, in coordination with the appropriate contracting officer, to consider inconsistencies between policy and leading practices in contract administration and, as needed, institute controls to clearly define measures against which to analyze differences between services expected and those delivered.	Closed – Implemented As of March 2017, USCIS has demonstrated that it has taken sufficient steps to close this recommendation. For example, USCIS demonstrated that it has provided training to both contracting officers (COs) and contracting officer representatives (CORs) to further clarify the intention of a Quality Assurance Surveillance Plan, when such a plan is expected, and what such a plan should document. This training included a discussion of performance measures. Further, the program office demonstrated that it included missing performance criteria (e.g. productivity, standards adherence, and test quality and test coverage) in its monthly scorecard and is now assessing Flexible Agile Development Services (FADS) contractors against all performance criteria identified in the contract. Also, in January 2017, USCIS provided two Quality Assurance Surveillance Plans for its new FADS contracts, referred to as FADS II, signed by the COR. These surveillance plans clearly defined measures against which to analyze differences between services expected and those delivered. In addition to defining performance measures and expectations for all of the performance criteria defined in the contract, the surveillance plans also set the expectation that all contract deliverables will be delivered on time. USCIS further articulated in the Quality Assurance Surveillance Plans the ratings that will be used to evaluate the extent to which contractor deliverables meet government expectations and the process for performing those evaluations. In addition, the plans included templates for performance reports and contract discrepancy reports that are to be completed by the COR. In July 2016, we had reported that these two items were not being properly maintained in the COR file. Including the templates will assist USCIS in improving the consistency and documentation of its monitoring of contractor performance. Finally, in March 2017, USCIS provided the signed FADS II contracts along with the associated performance work statements, confirming the contract awards and the applicability of the Quality Assurance Surveillance Plans.
Department of Homeland Security	To help improve oversight of selected Transformation Program contracts, the Secretary of DHS should direct the Director of USCIS to direct the Chief of the USCIS Contracting Office, in coordination with the appropriate contracting officer, to consider inconsistencies between policy and leading practices in contract administration and, as needed, institute controls to ensure contracting officer's representatives are maintaining complete contract files.	Closed – Implemented As of April 2017, USCIS has demonstrated that it has taken sufficient steps to close this recommendation. For example, in March 2017, USCIS provided evidence that it completed an internal review of all USCIS Transformation Program contracting officer representatives (COR) files. Based on our review of the inspection results, the additional training for contracting officer representatives (COR) and contracting officers on the proper maintenance of COR files, and the COR checklist previously provided by USCIS (see recommendation 10), we determined that USCIS had taken sufficient steps to close this recommendation. These actions should help to ensure that CORs maintain complete contract files.
Department of Homeland Security	To help improve oversight of selected Transformation Program contracts, the Secretary of DHS should direct the Director of USCIS to direct the Chief of the USCIS Contracting Office, in coordination with the appropriate contracting officer, to consider inconsistencies between policy and leading practices in contract administration and, as needed, institute controls to ensure quality assurance surveillance plans are developed when appropriate.	Closed – Implemented As of March 2017, USCIS has demonstrated that it has taken sufficient steps to close this recommendation. For example, in February 2017, USCIS provided evidence that it completed an internal review of all 46 contracts that it identified as performance based acquisitions. Based on our review of the inspection results, the additional Quality Assurance Surveillance Plan training for contracting officer representatives (COR) and contracting officers, and the COR checklist previously provided by USCIS (see recommendation 10), we determined that USCIS had taken sufficient steps to close this recommendation. These actions should help to ensure that quality assurance surveillance plans are developed when appropriate.

See All 12 Recommendations

Full Report

Highlights Page (1 page)

Full Report (128 pages)

Accessible PDF (135 pages)

GAO Contacts

Carol R. Cha

chac@gao.gov

Media Inquiries

Sarah Kaczmarek

Managing Director

Office of Public Affairs

media@gao.gov

Public Inquiries

Public Affairs

PublicAffairs@gao.gov

Topics

Information Management

ImmigrationImmigration servicesImmigration benefitsImmigrantsSoftware developmentInformation technologyImmigration information systemsIntegrated softwareQuality assuranceOperational testingProduct evaluationProgram evaluationSoftwareProgram managementIT acquisitions