Agile Software Development: DHS Has Made Significant Progress in Implementing Leading Practices, but Needs to Take Additional Actions
GAO-20-213
Published: Jun 01, 2020. Publicly Released: Jun 01, 2020.
Fast Facts
Many of the Department of Homeland Security’s IT acquisitions have taken longer than planned or failed to deliver desired results.
In April 2016, DHS started transitioning to Agile software development to help improve its IT acquisitions. Agile focuses on collaborative processes and workflows to quickly and frequently deliver working software.
DHS has made significant progress implementing leading practices during this transition but needs to take additional steps. For example, it needs to ensure all staff are trained in this new approach.
We recommended that DHS fully implement leading practices in its transition to Agile software development.
Homeland Security building
Highlights
What GAO Found
The Department of Homeland Security (DHS) has taken steps to implement selected leading practices in its transition from waterfall, an approach that historically delivered useable software years after program initiation, to Agile software development, which is focused on incremental and rapid delivery of working software in small segments. As shown below, this quick, iterative approach is to deliver results faster and collect user feedback continuously.
Comparison of Agile and Waterfall Methods for Developing Software
DHS has fully addressed one of three leading practice areas for organization change management and partially addressed the other two. Collectively, these practices advise an organization to plan for, implement, and measure the impact when undertaking a significant change. The department has fully defined plans for transitioning to Agile development. DHS has partially addressed implementation—the department completed 134 activities but deferred roughly 34 percent of planned activities to a later date. These deferred activities are in progress or have not been started. With respect to the third practice, DHS clarified expected outcomes for the transition, such as reduced risk of large, expensive IT failures. However, these outcomes are not tied to target measures. Without these, DHS will not know if the transition is achieving its desired results.
DHS has also addressed four of the nine leading practices for adopting Agile software development. For example, the department has modified its acquisition policies to support Agile development methods. However, it needs to take additional steps to, among other things, ensure all staff are appropriately trained and establish expectations for tracking software code quality. By fully addressing leading practices, DHS can reduce the risk of continued problems in developing and acquiring current, as well as, future IT systems.
Why GAO Did This Study
Many of DHS's major IT acquisition programs have taken longer than expected to develop or failed to deliver the desired value. In April 2016, to help improve the department's IT acquisition and management, DHS identified Agile software development as the preferred approach for all of its IT programs and projects.
GAO was asked to examine DHS's adoption of Agile software development. The objective of this review was to assess the extent to which DHS has addressed selected leading practices for its transition to the use of Agile software development.
GAO identified leading practices for planning, implementing, and measuring organizational change that apply to DHS's transition to Agile through its review of guidance published by the Project Management Institute and GAO. GAO also reviewed work it performed to develop leading practices for Agile software development adoption. GAO analyzed DHS documentation, such as policies, guidance, plans, and working group artifacts and assessed them against the selected leading practices. GAO also reviewed the implementation of selected practices within individual IT projects. Finally, GAO interviewed DHS officials to discuss any practices that were not fully implemented.
Recommendations
GAO is making 10 recommendations to DHS to implement selected leading practices for its transition to Agile software development. DHS agreed with GAO's recommendations and described actions taken and planned to address them.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Homeland Security | The Secretary should ensure that the Director of Strategic Technology Management (STM), in collaboration with other members of the Information Technology Program Management Center of Excellence (ITPM COE), identifies the skills and resources needed to complete the work intended for the upcoming fiscal year, including the availability of supplementary staff, such as subject matter experts. (Recommendation 1) |
In September 2022, DHS demonstrated that it had taken sufficient steps to close this recommendation. In its 180-day letter provided in response to our report, DHS stated that during the first quarter of each fiscal year, DHS Office of the Chief Information Officer (OCIO) staff host a planning session to review ongoing and upcoming tasks related to 18 action plans and the Information Technology Program Management Center of Excellence (ITPM COE) reviews its charter annually to ensure it is properly aligned to the ITPM COE's scope and objectives. During these planning sessions, DHS develops criteria for completing each of the outstanding tasks and identifies potential leads. Following this planning, DHS OCIO works with members of the ITPM COE to ensure personnel who have the necessary skills and subject matter expertise address each task. As the vehicle for completing these tasks, ITPM COE defines workloads within existing resources. DHS subsequently provided presentation slides from the planning session held for fiscal year (FY) 2020 and a recap session to revisit accomplishments achieved in FY 2020. These slides addressed the fiscal year core strategies for each office that makes up the ITPM COE and any new tasks. The documentation also demonstrated that the ITPM COE completed some of the tasks, but the majority of tasks were still in progress or not achieved. In October 2021, DHS provided the ITPM COE planning slides for FY 2021, an update on the status of Agile adoption action items, and a list of outstanding action plan tasks. This documentation demonstrated that DHS made improved progress towards achieving the planned goals of the ITPM COE during FY 2021. Further, in September 2022, DHS demonstrated that it continued to make progress in addressing planned tasks. Specifically, DHS provided documentation indicating that the ITPM COE addressed all of the tasks planned for FY 2022.
|
| Department of Homeland Security | The Secretary should ensure that the Executive Steering Committee overseeing the activities of the ITPM COE establishes target measures for the department's desired outcomes of its transition to Agile development. (Recommendation 2) |
In September 2022, DHS demonstrated that it had taken sufficient steps to close this recommendation. In September 2021, DHS demonstrated that it had finalized its Agile Core Metrics and required projects to report progress relative to those metrics. DHS also provided a mapping between its Agile Core Metrics and its desired outcomes of its transition to Agile development. In September 2022, DHS demonstrated that it is collecting data on its Agile core metrics, had established targets for those metrics, and continued to map those metrics to its desired outcomes.
|
| Department of Homeland Security | The Secretary should ensure that the DHS Chief Information Officer (CIO) defines a process and associated set of controls to ensure that Agile programs and projects are reporting a set of core required performance metrics for monitoring and measuring Agile adoption. (Recommendation 3) |
As of July 2022, DHS has demonstrated that it has taken sufficient steps to close this recommendation. In July 2021 DHS finalized an updated set of Agile core metrics for programs and projects to report. In March 2022, the department also updated its program health assessment process to ensure that programs and projects consistently report the Agile core metrics on a monthly basis. DHS supplemented this update by developing a quality assurance plan to make sure that the program health assessment process is executed as intended. By establishing a set of control to ensure that Agile programs and projects report a set of core required performance metrics, DHS is better positioned to begin assessing the impact of Agile adoption.
|
| Department of Homeland Security |
Priority Rec.
The Secretary should ensure that the ITPM COE, in coordination with the CIO, begins measuring results associated with the transition to Agile and the success of the transition based on its impact on the department. (Recommendation 4) |
In September 2022, DHS demonstrated that it had taken sufficient steps to close this recommendation. In July 2021, DHS's Acting Chief Technology Officer approved an updated Agile Software Delivery Core Metrics Guidebook. The guidebook explains that programs must report monthly on six Agile core metrics (e.g., availability, cycle time, and unit test coverage) in DHS's Investment Evaluation Submission and Tracking system. In addition, in August 2021, DHS noted that these metrics are included as part of its Program Health Assessments for major and standard IT programs across the department. DHS also stated that the Office of the Chief Information Officer has informed programs that non-compliance will result in an adverse Program Health Assessment score. In addition, September 2021, DHS provided a mapping between its Agile Core Metrics and the department's desired outcomes of its transition to Agile development. Lastly, in September 2022, DHS demonstrated that it is collecting data on its Agile core metrics, had established targets for those metrics, and continued to map those metrics to its desired outcomes.
|
| Department of Homeland Security | The Secretary should ensure that the CIO, in collaboration with the Chief Procurement Officer, through the Homeland Security Acquisition Institute, establish Agile training requirements for senior stakeholders. (Recommendation 5) |
As of August 2024, DHS has not yet demonstrated that it has fully addressed this recommendation. In September 2022, DHS stated that a detailed catalog of Agile courses is available in the DHS Learning Management System. DHS added that the availability of these courses has been communicated to all learning management leads across DHS. However, DHS stated that, after consultation with the DHS Chief Learning Officer's Council, the department has not made the courses mandatory. Nevertheless, DHS stated that its Agile Center of Excellence will continue to highlight and recommend the courses for introductory or non-acquisition personnel and senior executives in accordance with these recommendations (recommendations 5 and 7). In August 2024, DHS stated that there is broad acceptance of Agile development at the department. Nevertheless, DHS stated that it is incorporating basic Agile training into its customer experience training and provided documentation about its customer experience courses and course attendance. In addition, DHS stated that it plans to initiate the revision of enterprise-wide policy to reflect that component heads should strongly encourage customer experience and training for all personnel affiliated with IT acquisition program management, support, or oversight. Further, DHS's Customer Experience Directorate has initiated a new learning management system that will allow it to track the component, directorate, and office of staff who register for and complete training. The Directorate also plans to build a training dashboard that will allow it to identify targeted audiences that attended training or need to be trained. However, as of August 2024, the department does not require Agile training for senior stakeholders. GAO continues to believe that this recommendation is warranted for the reasons described in the report. Accordingly, we will continue to monitor DHS's efforts to address this recommendation.
|
| Department of Homeland Security | The Secretary should ensure that the Chief Human Capital Officer, in collaboration with the CIO, consider modifications to the current employee recognition and performance management governance to ensure that teamwork and team performance of Agile programs and projects are incentivized. (Recommendation 6) |
In April 2022, DHS demonstrated that it has taken sufficient steps to fully implement this recommendation. In its 180-day letter provided in response to our report, DHS stated that it launched the OCIO Employee Awards Program on January 28, 2020. DHS stated that this program added guidance to incentivize teamwork, team performance, and IT programs (including Agile). In an October 2021 update, OCIO stated that OCIO's Human Capital Management Division updated the OCIO Awards and Recognition Program to incentivize teamwork, team performance, and Agile programs. While supporting documentation was limited to OCIO personnel and did not address compensation for staff outside of OCIO, DHS subsequently provided additional information documenting how teamwork and team performance of Agile programs and projects can be recognized for staff outside of OCIO. For example, in April 2022, DHS identified guidance for its Honorary Awards. These awards can recognize, among other things, cross component cooperation and teamwork. In addition, DHS provided a DHS Directive, Instruction, and guidance for annual performance ratings documenting that teamwork and cooperation is one element considered in annual performance ratings. By establishing an incentives and rewards structure that recognizes team performance, DHS is better positioned to improve team productivity and output.
|
| Department of Homeland Security | The Secretary should ensure that the CIO, in collaboration with the Chief Procurement Officer, through the Homeland Security Acquisition Institute, establish Agile training requirements for staff outside of the acquisition workforce but assigned to Agile programs. (Recommendation 7) |
As of August 2024, DHS has not yet demonstrated that it has fully addressed this recommendation. In September 2022, DHS stated that a detailed catalog of Agile courses is available in the DHS Learning Management System. DHS added that the availability of these courses has been communicated to all learning management leads across DHS. However, DHS stated that, after consultation with the DHS Chief Learning Officer's Council, the department has not made the courses mandatory. Nevertheless, DHS stated that its Agile Center of Excellence will continue to highlight and recommend the courses for introductory or non-acquisition personnel and senior executives in accordance with these recommendations (recommendations 5 and 7). In August 2024, DHS stated that there is broad acceptance of Agile development at the department. Nevertheless, DHS stated that it is incorporating basic Agile training into its customer experience training and provided documentation about its customer experience courses and course attendance. In addition, DHS stated that it plans to initiate the revision of enterprise-wide policy to reflect that component heads should strongly encourage customer experience and training for all personnel affiliated with IT acquisition program management, support, or oversight. Further, DHS's Customer Experience Directorate has initiated a new learning management system that will allow it to track the component, directorate, and office of staff who register for and complete training. The Directorate also plans to build a training dashboard that will allow it to identify targeted audiences that attended training or need to be trained. However, as of August 2024, the department does not require Agile training for staff outside of the acquisition workforce but assigned to Agile programs. GAO continues to believe that this recommendation is warranted for the reasons described in the report. Accordingly, we will continue to monitor DHS's efforts to address this recommendation.
|
| Department of Homeland Security | The Secretary should ensure that the CIO, upon establishing a set of core performance metrics, tracks and monitors the pace of Agile team development. (Recommendation 8) |
In September 2022, DHS demonstrated that it has taken sufficient steps to close this recommendation. In its 180-day letter provided in response to our report, DHS stated that the Agile Core Metrics were scheduled to be published to INVEST in the first quarter of FY 2021. DHS added that OCIO would commence gathering and analyzing the data submitted to INVEST in order to track and monitor the pace of Agile team development and anticipated completing the actions necessary to implement this recommendation by June 30, 2021. In July 2021, in response to our recommendation, DHS updated the Agile Core Metrics guidebook. Among other things, the guidebook requires Agile projects to track and report cycle time, defined as the time between when work is started on a story and when that story is deployed to production. In September 2022, DHS demonstrated that programs are reporting cycle time as part of their Agile Core Metrics. In addition, the department demonstrated that it is monitoring program compliance with the requirement that they report their Agile Core Metrics.
|
| Department of Homeland Security | The Secretary should ensure that the CIO, in collaboration with the Executive Director of the Office of Program Accountability and Risk Management (PARM), update or develop new guidance on Agile methodologies to describe how Agile teams can estimate the relative complexity of user stories. (Recommendation 9) |
As of October 2021, DHS has demonstrated that it has taken sufficient steps to close this recommendation. In December 2020, DHS updated its Agile Guidebook. The guidebook included a definition for relative complexity as it pertains to user stories. However, the guidebook still did not provide techniques on how Agile teams can estimate complexity of user stories and measure program performance. In July 2021, in response to our recommendation, DHS again updated the Agile Guidebook with a number of techniques to estimate relative complexity, such as triangulation and affinity estimation. The guidance also directed the reader to sources for additional information on the topic. By taking these steps to describe the process for estimating relative complexity, the department can increase confidence that Agile teams will effectively commit to an appropriate amount of work during a given iteration.
|
| Department of Homeland Security | The Secretary should ensure that the CIO, upon establishing a set of core performance metrics, sets expectations for automated testing and code quality, and tracks and monitors against those expectations. (Recommendation 10) |
In September 2022, DHS demonstrated that it has taken sufficient steps to close this recommendation. In its 180-day letter provided in response to our report, DHS stated that the Agile Core Metrics were scheduled to be published to INVEST in the first quarter of FY 2021. DHS added that OCIO would commence gathering and analyzing the data submitted to INVEST in order to track and monitor the pace of Agile team development and anticipated completing the actions necessary to implement this recommendation by June 30, 2021. In July 2021, in response to our recommendation, DHS updated the Agile Core Metrics guidebook. Among other things, the guidebook requires Agile projects to track and report change failure rate and unit test coverage, both metrics associated with code quality. In September 2022, DHS demonstrated that programs were reporting change failure rate and unit test coverage as part of their Agile Core Metrics. In addition, the department demonstrated that it was monitoring program compliance with the requirement that they report their Agile Core Metrics.
|
Full Report
GAO Contacts
Topics
Best practicesChange managementIT acquisitionsInformation technologyOrganizational changeSoftware developmentSystems acquisitionSoftwareHomeland securityIT projects