National Partnership Offers Benefits, but Faces Considerable Challenges
GAO-06-392: Published: Mar 24, 2006. Publicly Released: Mar 24, 2006.
In 1997, the National Security Agency and the National Institute of Standards and Technology formed the National Information Assurance Partnership (NIAP) to boost federal agencies' and consumers' confidence in information security products manufactured by vendors. To facilitate this goal, NIAP developed a national program that requires accredited laboratories to independently evaluate and validate the security of these products for use in national security systems. These systems are those under control of the U.S. government that contain classified information or involve intelligence activities. GAO was asked to identify (1) the governmentwide benefits and challenges of the NIAP evaluation process on national security systems, and (2) the potential benefits and challenges of expanding the requirement of NIAP to non-national security systems, including sensitive but unclassified systems.
While NIAP process participants--vendors, laboratories, and federal agencies--indicated that the process offers benefits for use in national security systems, its effectiveness has not been measured or documented, and considerable challenges to acquiring and using NIAP-evaluated products exist. Specific benefits included independent testing and evaluation of products and accreditation of the performing laboratories, the discovery and correction of product flaws, and improvements to vendor development processes. However, process participants also face several challenges, including difficulty in matching agencies' needs with the availability of NIAP-evaluated products, vendors' lack of awareness regarding the evaluation process, and a lack of performance measures and difficulty in documenting the effectiveness of the NIAP evaluation process. Collectively, these challenges hinder the effective use of the NIAP evaluation process by vendors and agencies. Expanding the requirement of the NIAP evaluation process to non-national security systems is likely to yield similar benefits and challenges as those experienced by current process participants. For example, a current benefit--independent testing and evaluation of IT products--gives agencies confidence that validated features of a product will perform as claimed by the vendor. However, federal policy already allows agencies with non-national security systems to consider acquiring NIAP-evaluated products for those systems, and requiring that they do so may further exacerbate current resource constraints related to the evaluation and validation of products. In the absence of such a requirement, agencies seeking information assurance (measures that defend and protect information and information systems by ensuring their confidentiality, integrity, authenticity, availability, and utility) for their non-national security systems have other federal guidance and standards available to them.
- Review Pending
- Closed - implemented
- Closed - not implemented
Recommendations for Executive Action
Recommendation: To assist the NIAP in documenting the effectiveness of the NIAP evaluation process, the Secretary of Defense should direct the Director of the National Security Agency, in coordination with NIST under the provisions of the NIAP partnership, to coordinate with vendors, laboratories, and various industry associations that have knowledge of the evaluation process to develop awareness training workshops for program participants.
Agency Affected: Department of Defense
Status: Closed - Implemented
Comments: In 2006, we reported that the National Information Assurance Partnership program participants faced a number of challenges. Specifically, we reported that software vendors' were not knowledgeable of the evaluation process used by the Common Criteria testing laboratories. Accordingly, we recommended that the National Security Agency (NSA) coordinate with vendors, laboratories, and various industry associations that have knowledge of the evaluation process to develop awareness training workshops for program participants. In 2010, we verified that NSA, in response to our recommendation, coordinated with laboratories, and that the laboratories are offering training to participants of the program via their websites. As a result of the training, evaluations have a greater likelihood of being completed more efficiently since the vendors are already familiar with the evaluation process and the extensive documentation requirements necessary to complete the evaluation.
Recommendation: To assist the NIAP in documenting the effectiveness of the NIAP evaluation process, the Secretary of Defense should direct the Director of the National Security Agency, in coordination with NIST under the provisions of the NIAP partnership, to consider collecting, analyzing, and reporting metrics on the effectiveness of NIAP tests and evaluations. Such metrics could include summary information on the number of findings, flaws, and associated fixes.
Agency Affected: Department of Defense
Status: Closed - Not Implemented
Comments: In 2007, Deputy Assistant Secretary of Defense for Information and Identity Assurance in response to this recommendation, stated that in order to continue the program within its constrained budget, National Information Assurance Partnership (NIAP) personnel focused their efforts on re-engineering the validation oversight process for all NIAP evaluations. In addition, the letter stated that NIAP personnel focused on establishing a fee-for-service schedule in order to recoup validation costs from vendors for each evaluation. The re-engineering of the validation process along with the fee-for-service efforts necessitated a complete revision in the way NIAP was originally proposing to gather metrics on vulnerabilities from the NIAP labs. Furthermore, the letter stated that NIAP hopes to develop and institute a uniform methodology for gathering metrics that will coincide with the implementation of the fee-for-service strategy expected in the second or third quarter of FY 2008. As of August 2009, the NIAP Director provided documentation that stated that the overall strategy for the program is currently undergoing a major revision and plans are underway to overhaul the entire program. In addition, the documentation stated that NIAP personnel have re-established their efforts to implement a fee-for-service program and are revising their previous plans for gathering metrics. The new plan for gathering metrics will be pursued in collaboration with IT vendors and with the commercial NIAP laboratories and is expected to take several years to implement. As of August 2010, NIAP stated that it was working with the National Voluntary Lab Accreditation Program (NVLAP) to include more transparency of results of the tests and evaluations to be shared with NIAP and other federal agencies. However, metrics are not currently being collected on the effectiveness of tests and evaluations. NIAP responded that the metrics currently being collected relate to the evolution of the NIAP program.