Coast Guard:

Non-Homeland Security Performance Measures Are Generally Sound, but Opportunities for Improvement Exist

GAO-06-816: Published: Aug 16, 2006. Publicly Released: Sep 18, 2006.

Additional Materials:

Contact:

Stephen L. Caldwell
(202) 512-9610
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

Using performance measures, the Coast Guard explains how well its programs are performing. To do so, it reports one "primary" measure for each program (such as percent of mariners rescued) and maintains data on other, "secondary" measures (such as percent of property saved). Concerns have been raised about whether measures for non-homeland security programs accurately reflect performance, that is, they did not rise or fall as resources were added or reduced. For the six non-homeland security programs, GAO used established criteria to assess the soundness of the primary measures--that is, whether measures cover key activities; are clearly stated; and are objective, measurable, and quantifiable--and the reliability of data used to calculate them. GAO also used these criteria to assess the soundness of 23 selected secondary measures. Finally, through interviews and report review, GAO assessed challenges in using measures to link resources to results.

While some opportunities for improvement exist, the primary measures for the Coast Guard's six non-homeland security programs are generally sound, and the data used to calculate them are generally reliable. All six measures cover key program activities and are objective, measurable, and quantifiable, but three are not completely clear--that is, they do not consistently provide clear and specific descriptions of the data, events, or geographic areas they include. Also, the processes used to enter and review the Coast Guard's own internal data are likely to produce reliable data; however, neither the Department of Homeland Security (DHS) nor the Coast Guard have policies or procedures for reviewing or verifying data from external sources, such as other federal agencies. Currently, the review processes vary from source to source, and for the primary measure covering marine environmental protection (which concerns oil and chemical spills), the processes are insufficient. Of the 23 secondary performance measures GAO assessed, 9 are generally sound, with weaknesses existing in the remaining 14. These weaknesses include (1) a lack of measurable performance targets, (2) a lack of agencywide criteria or guidance to ensure objectivity, and (3) unclear descriptions of the measures. Two main challenges exist with using primary measures to link resources to results. In one case, the challenge is comprehensiveness--that is, although each primary measure captures a major segment of program activity, no one measure captures all program activities and thereby accounts for all program resources. The other challenge involves external factors, some of which are outside the Coast Guard's control, that affect performance. For example, weather conditions can affect the amount of ice that must be cleared, the number of aids to navigation that need repair, or mariners that must be rescued. As a result, linking resources and results is difficult, and although the Coast Guard has a range of ongoing initiatives to do so, it is still too early to assess the agency's ability to successfully provide this link.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: To improve the quality of program performance reporting and to more efficiently and effectively assess progress toward achieving the goals or objectives stated in agency plans, the Secretary of Homeland Security should direct the Commandant of the Coast Guard to develop and implement a policy to review external data provided by third parties that is used in calculating performance measures to, at a minimum, be familiar with the internal controls external parties use to determine the reliability of their data.

    Agency Affected: Department of Homeland Security: United States Coast Guard

    Status: Closed - Implemented

    Comments: In our review, we found that while the Coast Guard had controls in place to ensure the timeliness, completeness, accuracy, and consistency of original data that Coast Guard personnel collect and enter into its data systems, the agency did not have sufficient controls in place to verify or review the completeness and accuracy of data obtained from external sources that it uses to calculate some primary performance measures such as those for marine environmental protection. For this measure, the "5-year average annual number of oil spills greater than 100 gallons and chemical discharges per 100 million tons shipped," the Coast Guard used Army Corps of Engineers data on the amount of oil and chemicals shipped annually in the United States. We found that the Coast Guard did not review the Corps' data for completeness or accuracy, nor did it review the data reliability procedures the Corps used to test its own data for completeness or accuracy. As a result, we recommended that the Coast Guard implement a policy to review external data provided by third parties used in calculating performance measures and that the Coast Guard be familiar with the internal controls external parties use to determine the reliability of their data. In May 2010, the Coast Guard reported that for the measures that require Army Corps of Engineers' data, the Coast Guard practice is to contact the Corps and review the Corps' internal controls to determine the reliability of their data. In addition, the Coast Guard reported that a formal review process is now in place for all performance measures and performance measure data being used within the Department of Homeland Security (DHS). Whenever introducing new performance measures or making changes to existing measures, Coast Guard officials must complete and submit to DHS a Performance Measure Definition Form that requires program managers to obtain and consider information regarding the source of the data, how the data is collected and maintained, the processes used to determine the reliability and completeness of the data, and make an overall determination of the data's reliability based on this information. This information is further reviewed by the Office of Management and Budget. These actions are consistent with our recommendation.

    Recommendation: To improve the quality of program performance reporting and to more efficiently and effectively assess progress toward achieving the goals or objectives stated in agency plans, the Secretary of Homeland Security should direct the Commandant of the Coast Guard to refine certain Coast Guard primary and secondary performance measures by establishing agencywide criteria or guidance to help ensure the objectivity and consistency of the search and rescue program's "percent of property saved" secondary performance measure.

    Agency Affected: Department of Homeland Security: United States Coast Guard

    Status: Closed - Implemented

    Comments: In our review, we found that one of the search and rescue program's secondary performance measures that we analyzed, "percent of property saved," did not have criteria or guidance for agency personnel to objectively and consistently determine the value of saved property. In addition, we found that Coast Guard units do not consistently record property values across the agency. For example, some units did not record property values at all, other units recorded property values only when the actual value can be determined, and other units estimate property values using a $1,000-per-foot-of-vessel-length rule of thumb. As a result, we recommended that the Coast Guard establish agencywide criteria or guidance to help ensure the objectivity and consistency of the search and rescue program's "percent of property saved" secondary performance measure. In October 2009 the Coast Guard replaced the previous performance measure with a new measure called "percent of property in danger of loss saved." To ensure the consistency and objectivity of data inputs for the measure, the Coast Guard included a Vessel Value Estimator tool in the input screens of its information system for Coast Guard officials to use in estimating the value of a vessel. The tool has seven pleasure vessel types and eleven commercial vessel types that can be selected for estimation. With the tool, Coast Guard officials can select a vessel type and enter the vessel's length, for which the tool then provides an estimated value output based on a per foot value of vessel types for specific length ranges developed from comparative prices of representative vessel costs. The revised measure and the inclusion of the estimator tool is consistent with our recommendation for an objective and consistent performance measure.

    Recommendation: To improve the quality of program performance reporting and to more efficiently and effectively assess progress toward achieving the goals or objectives stated in agency plans, the Secretary of Homeland Security should direct the Commandant of the Coast Guard to refine certain Coast Guard primary and secondary performance measures by developing measurable performance targets to facilitate assessments of whether program and agency goals and objectives are being achieved for the 11 living marine resources secondary measures and the 1 marine environmental protection secondary measure, "Tokyo and Paris memorandums of understanding port state control reports," that lack annual targets.

    Agency Affected: Department of Homeland Security: United States Coast Guard

    Status: Closed - Not Implemented

    Comments: In our review, we found that 12 secondary measures (11 living marine resources measures and 1 marine environmental protection) did not have annual targets to assess whether program and agency goals and objectives are being achieved. During our review, Coast Guard officials told us that these measures do not have targets because the focus of the program is on the primary performance measures, and not the inputs and outputs reflected in these secondary measures. However, we found that without any quantifiable, numeric targets, it is difficult for the Coast Guard to know the extent to which program and agency goals and objectives are being achieved. As a result, we recommended that the Coast Guard develop measurable performance targets to facilitate assessments of whether program and agencies goals and objectives are being achieved. As of June 2010, the Coast Guard reported that it still does not explicitly set targets for its secondary (Tier II) measures for its living marine resources measures nor set specific targets for its marine environmental protection mission in externally produced reports.

    Recommendation: To improve the quality of program performance reporting and to more efficiently and effectively assess progress toward achieving the goals or objectives stated in agency plans, the Secretary of Homeland Security should direct the Commandant of the Coast Guard to refine certain Coast Guard primary and secondary performance measures by further clarifying the ice operations primary measure by clearly and consistently describing the geographic area and number of waterways included in the measure; the living marine resources primary measure by clearly and consistently reporting the scope of the measure; and the search and rescue primary measure and the search and rescue "percent of lives saved after Coast Guard notification" secondary measure by reporting those incidents or data that are not included in the measures.

    Agency Affected: Department of Homeland Security: United States Coast Guard

    Status: Closed - Not Implemented

    Comments: In our review we found that the primary performance measures for the ice operations, living marine resources, and search and rescue programs did not consistently provide clear and specific descriptions of the data, events, or geographic areas they include. The measure for ice operations ("annual number of waterway closure days") did not reflect the annual number of closure days for all waterways across the United States, but rather reflects only the annual number of closure days in the Great Lakes region, although the Coast Guard breaks ice in many East Coast ports and waterways. While this limitation is included in accompanying text in some documents, the description of the limitation is inconsistent across department and agency publications. For the measure for living marine resources ("percent of fishermen in compliance with regulations"), the Coast Guard reports the results of checking entities most likely to be in violation of fishery regulations, rather than checking vessels at random. We found this measure for living marine resources was also not consistently and clearly defined in all department and agency publications. For example, in the Department of Homeland Security's fiscal year 2005 Performance and Accountability Report and the Coast Guard's Budget-in-Brief, this measure was described as an observed compliance rate, but the DHS fiscal year 2007 budget request did not clarify that this measure represents an observed compliance rate rather than the compliance rate of all fishermen in those areas patrolled by the Coast Guard. For the search and rescue primary measure ("percent of mariners in imminent danger who are rescued") and secondary measure ("percent of lives saved after Coast Guard notification"), while they reflect the programs priority of saving lives, they excluded those incidents in which 11 or more lives were saved or lost because those incidents were considered to be data anomalies. As a result of these findings, we recommended that the Coast Guard refine and clarify these measures. As of May 2010, the Coast Guard reported taking steps to refine these measures. For the ice operations measure, the Coast Guard reported conducting studies to explore program mission performance and is in the process of redefining the geographic area and waterways included in the measure to be more inclusive of all domestic ice operations. For the living marine resources measure, the Coast Guard reported that it now has a centralized office of Performance Management and Assessment to aid in ensuring consistency in reporting the performance measures in both internal and external documents. For the search and rescue primary and secondary measures, the Coast Guard reported that it is making changes to its data collection systems to identify incidents that include 11 or more lives at risk. However, these changes are not expected to be in place before September 2010. Because these efforts are largely still in process, it cannot be determined whether these efforts will fully implement the recommendation.

    Recommendation: To improve the quality of program performance reporting and to more efficiently and effectively assess progress toward achieving the goals or objectives stated in agency plans, the Secretary of Homeland Security should direct the Commandant of the Coast Guard to report additional information--besides the one primary measure--in appropriate agency publications or documents where doing so would help provide greater context or perspective on the relationship between resources expended and program results achieved.

    Agency Affected: Department of Homeland Security: United States Coast Guard

    Status: Closed - Implemented

    Comments: During our review, we found that while the primary measures for the Coast Guard's six non-homeland security programs were generally sound and use reliable data, challenges existed with using the primary measures to assess the link between resources expended and results achieved. Specifically, the challenges involved the difficulty of fully reflecting an entire program such as ice operations or marine environmental protection in a single performance measure and the ability to account for the many factors, other than resources, that can affect program results. As a result, we recommended that the Coast Guard report additional information besides the one primary performance measure in appropriate venues to better inform stakeholders and decision makers about the relationship between resources expended and results achieved. For example, we suggested that reporting supplemental information on such things as the percentage of aids to navigation available and in need of maintenance, the annual number of search and rescue cases, and icebreaking activities beyond the Great Lakes region would provide additional information on the annual levels of activity that external decision makers, such as Congress, might find helpful. In May 2010, the Coast Guard reported that it has expanded its suite of measures to help provide additional information in publications such as its annual performance reports. For example, for its aids to navigation mission, the Coast Guard now reports the availability of federal aids to navigation. For its ice operations mission, the Coast Guard includes additional descriptions of ice operations activities in areas beyond the Great Lakes region. For its search and rescue mission, the Coast Guard includes a more in depth discussion of the number of cases and the number of lives saved and comparison of these numbers to previous years.

    Jul 24, 2014

    Jul 16, 2014

    Jun 27, 2014

    Jun 24, 2014

    Jun 23, 2014

    Jun 18, 2014

    Jun 16, 2014

    Jun 11, 2014

    Looking for more? Browse all our products here