Immigration Benefits System: U.S. Citizenship and Immigration Services Can Improve Program Management
Highlights
What GAO Found
U.S. Citizenship and Immigration Services (USCIS) created a reliable updated estimate to project the Transformation Program's cost, but has experienced program management challenges. In particular, the program's cost estimate was well-documented and substantially comprehensive, accurate, and credible. However, among other things, software development and systems integration and testing for USCIS's Electronic Immigration System (USCIS ELIS) have not consistently been managed in line with the program's policies and guidance or with leading practices.
Regarding software development, the Transformation Program has produced some software increments, but is not consistently following its own guidance and leading practices. The software development model (Agile) adopted by the USCIS Transformation Program in 2012 includes practices aimed at continuous, incremental release of segments of software. Important practices for Agile defined in program policies, guidance, and leading practices include ensuring that the software meets expectations prior to being deployed, teams adhere to development principles, and development outcomes are defined. The table below lists the program's status in addressing eight key Agile development practices. For example, the program has committed to a specific framework for software development, referred to as Scrum, but has deviated from the underlying practices and principles of this framework.
Table: Implementation of Key Agile Practices for USCIS ELIS
|
Practice |
Rating |
|
Completing planning for software releases prior to initiating development and ensuring software meets business expectations prior to deployment |
◐ |
|
Adhering to the principles of the framework adopted for implementing Agile software development |
◐ |
|
Defining and consistently executing appropriate roles and responsibilities for individuals responsible for development activities |
◐ |
|
Identifying users of the system and involving them in release planning activities |
◐ |
|
Writing user stories that identify user roles, include estimates of complexity, take no longer than one sprint to complete, and describe business value |
◐ |
|
Prioritizing user stories to maximize the value of each development cycle |
● |
|
Setting outcomes for Agile software development |
○ |
|
Monitoring and reporting on program performance through the collection of reliable metrics |
◐ |
Source: GAO analysis of USCIS documentation. I GAO-16-467.
Note: ● yes ◐ partial ○ noc
The Transformation Program has established an environment that allows for effective systems integration and testing and has planned for and performed some system testing. However, the program needs to improve its approach to system testing to help ensure that USCIS ELIS meets its intended goals and is consistent with agency guidance and leading practices. Among other things, the program needs to improve testing of the software code that comprises USCIS ELIS and ensure its approaches to interoperability and end user testing, respectively, meet leading practices. Collectively, these limitations have contributed to issues with USCIS ELIS after new software is released into production.
Why GAO Did This Study
Each year, USCIS processes millions of applications for persons seeking to study, work, visit, or live in the United States, and for persons seeking to become a U.S. citizen. In 2006, USCIS began the Transformation Program to enable electronic adjudication and case management tools that would allow users to apply and track their applications online. In 2012, to address performance concerns, USCIS changed its acquisition strategy to improve system development.
In May 2015, GAO reported that USCIS expected the program to cost up to $3.1 billion and be fully operational by March 2019. This includes more than $475 million that was invested in the initial version of the program's key case management component, USCIS ELIS, which has since been decommissioned.
This report evaluates the extent to which the program is using information technology program management leading practices. To perform this work, GAO identified agency policy and guidance and leading practices in, among other things, cost estimation, Agile software development, and systems integration and testing, and compared these with practices being used by the program.
Recommendations
GAO is making 12 recommendations to improve Transformation Program management, including ensuring alignment among policy, guidance, and leading practices in areas such as Agile software development and systems integration and testing. DHS concurred with the recommendations.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Homeland Security | To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of the Department of Homeland Security (DHS) should direct the Director of USCIS to direct the USCIS Chief Information Officer (CIO), in coordination with the DHS CIO and the Chief of the Office of Transformation Coordination (OTC), to review and update, as needed, existing policies and guidance and consider additional controls to complete planning for software releases prior to initiating development and ensure software meets business expectations prior to deployment. | As of April 2020, USCIS has demonstrated that it has taken sufficient steps to close this recommendation. In January 2020, USCIS updated its management instruction governing planning and deploying software releases. USCIS policy requires programs to document its Agile processes in a USCIS Service Hub, an online repository of information for USCIS systems. For example, the USCIS Service Hub for the Transformation Program documents a roadmap with prioritized capabilities and projected implementation dates. The program has also modified its deployment approach, relying on Team Managed Deployments (TMDs). USCIS provided TMD documentation for releases that are required by the updated policy,... including the readiness review approval memo for release 14. The program also works with an independent verification and validation team to help assess outcomes called for by the USCIS management instruction. These actions should help to ensure that the program completes planning for software releases prior to initiating development and that the software meets business expectations prior to deployment.
View More |
| Department of Homeland Security | To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to consistently implement the principles of the framework adopted for Agile software development. | As of August 2019, the Department of Homeland Security (DHS) has demonstrated that it has taken sufficient steps to close this recommendation. In May 2017, USCIS provided updated policy governing the development of software releases, dated April 2017, along with release planning artifacts specific to USCIS ELIS. The updated policy included an appendix devoted to generally accepted agency practices and applying Agile principles in the agency. The program was in breach status as of October 2016. As a result, in December 2016, most new software development activities were placed in a strategic pause. This breach condition was remediated in July 2018 with the approval of a new program...
|
| Department of Homeland Security | To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to define and consistently execute appropriate roles and responsibilities for individuals responsible for development activities consistent with its selected development framework. | As of August 2019, the Department of Homeland Security (DHS) has demonstrated that it has taken sufficient steps to close this recommendation. In May 2017, USCIS provided updated policy governing the development of software releases, dated April 2017, along with release planning artifacts specific to USCIS ELIS. The updated policy included an appendix devoted to generally accepted agency practices and applying Agile principles in the agency. The updated policy and release documentation defined some roles and responsibilities that were previously only described by USCIS in its informal November 2014 management model, such as the authority and responsibility of a product owner. The program...
|
| Department of Homeland Security | To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to identify all system users and involve them in release planning activities. | As of June 2020, USCIS has demonstrated that it has taken sufficient steps to close this recommendation. As part of the program's transition from the Office of Transformation Coordination (OTC) to the Office of Information Technology (OIT), the program modified its development approach to focus on portfolios instead of functionality. This shift allowed for increased end user engagement throughout the development process to include daily interaction between the ELIS Portfolio Managers and subject matter experts from other offices, such as Service Center Operations and the Field Operations Directorate. The experts were in turn responsible for developing acceptance criteria for user stories...
|
| Department of Homeland Security | To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to write user stories that identify user roles, include estimates of complexity, take no longer than one sprint to complete, and describe business value. | As of June 2020, USCIS has demonstrated that it has taken sufficient steps to close this recommendation.. In January 2020, USCIS updated its policy governing the development of software releases along with release planning artifacts specific to USCIS ELIS and an IV&V assessment. Although policy no longer specifies the user story format, it does require programs to document Agile processes in a USCIS Service Hub, including a roadmap with prioritized features and projected implementation dates. According to the USCIS Service Hub, the program is now employing a Kanban approach to Agile software development, a methodology that does not utilize increments of development or necessarily rely on...
|
| Department of Homeland Security | To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to establish outcomes for Agile software development. | In April 2017, the USCIS Chief Information Officer (CIO) issued a Management Instruction on Agile development that defined ten outcomes that all USCIS IT programs are required to achieve. These ten outcomes are that (1) programs and projects frequently deliver valuable product; (2) value is continuously discovered and aligned to mission; (3) work flows in small batches and is validated; (4) quality is built in; (5) the organization continuously learns and improves; (6) teams collaborate across groups and roles to improve flow and remove delays; (7) security, accessibility and other compliance constraints are embedded and verifiable; (8) consistent and repeatable processes are used across...
|
| Department of Homeland Security | To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to monitor program performance and report to appropriate entities through the collection of reliable metrics. | As of April 2020, USCIS has demonstrated that it has taken sufficient steps to close this recommendation. In January 2020, USCIS updated its policy governing planning and deploying software releases. USCIS policy defined outcomes that all USCIS IT programs are required to achieve. USCIS also provided a series of independent verification and validation reports demonstrating that the agency monitored the relative success of the program against each of the outcomes identified in the USCIS policy. DHS also published a set of Agile core metrics to monitor program performance beyond just cost, schedule, and key performance parameters. For example, the Agile core metrics include story points...
|
| Department of Homeland Security | To help manage the USCIS ELIS system, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update existing policies and guidance and consider additional controls to conduct unit and integration, and functional acceptance tests, and code inspection consistent with stated program goals. | As of June 2020, USCIS has demonstrated that it has taken sufficient steps to close this recommendation. In May 2017, USCIS provided artifacts from internal systems in place to monitor software development performance. These metrics monitored aspects of testing, such as code quality and code coverage. As part of the program's approval for a re-baseline in July 2018, the program developed an updated Test and Evaluation Master Plan. The updated plan set new testing expectations for the program as agreed upon with its stakeholders in DHS and USCIS. In January 2020, USCIS also updated its policy governing the development of software releases and release planning artifacts specific to USCIS...
|
| Department of Homeland Security | To help manage the USCIS ELIS system, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update existing policies and guidance and consider additional controls to develop complete test plans and cases for interoperability and end user testing, as defined in the USCIS Transformation Program Test and Evaluation Master Plan, and document the results. | As of June 2020, USCIS demonstrated that it has taken sufficient steps to close this recommendation. As part of the program's approval for a re-baseline in July 2018, the program developed an updated Test and Evaluation Master Plan. The updated plan set new testing expectations for the program as agreed upon with its stakeholders in DHS and USCIS. Specifically, the new plan replaced End User Testing with Beta and Usability testing. This approach allows the program to test the functionality of the system in a production environment with a subset of users prior to a full deployment to all users, rather than having end users interact with the system in a non-production environment. In a...
|
| Department of Homeland Security | To help improve oversight of selected Transformation Program contracts, the Secretary of DHS should direct the Director of USCIS to direct the Chief of the USCIS Contracting Office, in coordination with the appropriate contracting officer, to consider inconsistencies between policy and leading practices in contract administration and, as needed, institute controls to clearly define measures against which to analyze differences between services expected and those delivered. | As of March 2017, USCIS has demonstrated that it has taken sufficient steps to close this recommendation. For example, USCIS demonstrated that it has provided training to both contracting officers (COs) and contracting officer representatives (CORs) to further clarify the intention of a Quality Assurance Surveillance Plan, when such a plan is expected, and what such a plan should document. This training included a discussion of performance measures. Further, the program office demonstrated that it included missing performance criteria (e.g. productivity, standards adherence, and test quality and test coverage) in its monthly scorecard and is now assessing Flexible Agile Development...
|
| Department of Homeland Security | To help improve oversight of selected Transformation Program contracts, the Secretary of DHS should direct the Director of USCIS to direct the Chief of the USCIS Contracting Office, in coordination with the appropriate contracting officer, to consider inconsistencies between policy and leading practices in contract administration and, as needed, institute controls to ensure contracting officer's representatives are maintaining complete contract files. | As of April 2017, USCIS has demonstrated that it has taken sufficient steps to close this recommendation. For example, in March 2017, USCIS provided evidence that it completed an internal review of all USCIS Transformation Program contracting officer representatives (COR) files. Based on our review of the inspection results, the additional training for contracting officer representatives (COR) and contracting officers on the proper maintenance of COR files, and the COR checklist previously provided by USCIS (see recommendation 10), we determined that USCIS had taken sufficient steps to close this recommendation. These actions should help to ensure that CORs maintain complete contract...
|
| Department of Homeland Security | To help improve oversight of selected Transformation Program contracts, the Secretary of DHS should direct the Director of USCIS to direct the Chief of the USCIS Contracting Office, in coordination with the appropriate contracting officer, to consider inconsistencies between policy and leading practices in contract administration and, as needed, institute controls to ensure quality assurance surveillance plans are developed when appropriate. | As of March 2017, USCIS has demonstrated that it has taken sufficient steps to close this recommendation. For example, in February 2017, USCIS provided evidence that it completed an internal review of all 46 contracts that it identified as performance based acquisitions. Based on our review of the inspection results, the additional Quality Assurance Surveillance Plan training for contracting officer representatives (COR) and contracting officers, and the COR checklist previously provided by USCIS (see recommendation 10), we determined that USCIS had taken sufficient steps to close this recommendation. These actions should help to ensure that quality assurance surveillance plans are...
|