B-238597.2, Jul 5, 1990, 69 Comp.Gen. 562

B-238597.2, Jul 5, 1990, 69 Comp.Gen. 562

PROCUREMENT - Bid Protests - GAO authority 1. Rule that General Accounting Office (GAO) generally will not review protests of agency refusal to exercise a contract option is inapplicable where agency uses the exercise of contract options in parallel development contracts to select one contractor to continue the effort, because, under such circumstances, the agency's actions do not constitute contract administration but are, in fact, a form of limited competition properly subject to review by GAO. PROCUREMENT - Bid Protests - GAO procedures - Protest timeliness - 10-day rule 2. Agency challenge to timeliness of protest is denied where protester diligently pursues information that forms the basis of its protest, and files a timely protest upon receipt of such information. PROCUREMENT - Competitive Negotiation - Offers - Evaluation errors - Evaluation criteria - Application 3. Protest that agency abandoned evaluation criteria in solicitation and that contracting officer lacked a reasonable basis for selection decision is sustained where performance testing of protester's prototype equipment contributed significantly to selection decision under evaluation scheme, and such testing was conducted using test equipment that did not comply with the specification requirement; where the faulty operation of the test equipment was clearly related to operation of the prototype equipment; and where valid tests were never completed.

Attorneys

Mine Safety Appliances Company:

Mine Safety Appliances Company (MSA) protests the selection by the Department of the Navy of National Draeger, Inc., to proceed with the Navy's ongoing development of a new generation firefighter's breathing apparatus (FFBA). MSA argues that the Navy's selection of National Draeger-- by exercising the option in National Draeger's existing contract, and by not exercising the option in MSA's corresponding contract -- is in violation of the evaluation criteria in the solicitation, and the stated testing procedures in the contract.

We sustain the protest.

BACKGROUND

On February 12, 1988, the Navy issued request for proposals (RFP) No. N61331-88-R-0016, for the development of a new generation of FFBA equipment. /1/ After reviewing and evaluating the five best and final offers, the Navy awarded parallel development contracts to the two highest -rated offerors: MSA and National Draeger.

Both contracts required the contractor to fabricate and deliver to the Navy one working prototype FFBA; sufficient expendable oxygen supply for 20 single hour-long tests of the prototype device; and a mock-up version of the equipment. The RFP required the Navy, upon receipt of the prototype and supporting equipment, to conduct two types of performance tests on each contractor's proposed device: one measuring the machine's breathing performance, the other its suitability for shipboard use (size, weight, fit, etc.). Under the evaluation scheme, 80 percent of the total performance score is derived from the breathing performance tests, while 20 percent of the score is derived from the tests to measure shipboard suitability.

The evaluation scheme next anticipated combining the equipment performance scores with the scores given the technical and management proposals, and the cost proposal, during the initial preaward evaluation. Based on the combined score for each contractor, the RFP provided for selection of one of the contractors to proceed with the option quantities found in both contracts. /2/ The first option quantity calls for fabrication and delivery of additional prototype models and associated spares to an independent contractor for environmental testing; the second and third option provisions require delivery of additional prototypes and accompanying technical data for further testing. In addition, the RFP states that, at the conclusion of this contract, the government will award to the contractor selected to perform the option quantities a sole-source contract for the production of 1,000 FFBA units and expendable packages for fleet-wide testing.

On or about May 30, 1989, both MSA and National Draeger delivered their respective prototype units to the Navy for testing and evaluation. The testing of the proposed equipment was to be conducted in accordance with the FFBA specification and the test plan incorporated in the RFP. Breathing performance testing of the MSA equipment was scheduled for September 27 at a Bureau of Mines facility using test equipment provided by that agency. After 2 days of testing, hampered by performance problems with the test equipment, tests were suspended by the contracting officer. Even though testing was suspended, the evaluation team rated the proposed equipment, giving the National Draeger FFBA a total technical/management score of 10, on a scale of 1 to 10, while the MSA device was given a score of 8.83. /3/

Despite the small difference between the technical ratings assigned to the two devices, the narrative summary of the evaluation results prepared by the head of the evaluation team and sent to the contracting officer concluded that MSA should not be selected for award (option exercised) under any circumstance. The contracting officer, noting the inconsistency between the evaluation scores and the narrative, requested and received a written explanation of the disparity and, on December 8, exercised National Draeger's option to proceed with further development of the FFBA. This protest challenges the agency decision to exercise National Draeger's option, and the decision not to exercise the option in MSA's contract.

PROCEDURAL ISSUES

Jurisdiction

The Navy argues that our Office lacks authority to review the merits of the protest, claiming this dispute involves matters of contract administration beyond the reach of our bid protest jurisdiction-- i.e., an agency's exercise of a contract option, and an agency's decision to refrain from exercising a contract option. We do not agree.

In our view, the Navy here was conducting a limited competition between MSA and National Draeger to decide which contractor would be selected to proceed with the FFBA procurement. For example, to select between the prototypes built by MSA and National Draeger, the Navy convened a technical evaluation panel to score the performance of the two proposed devices. After testing and evaluation was completed, the evaluation panel presented its findings to the contracting officer to support a decision to select one of the two contractors to proceed with further FFBA development. Rather than make a new award, however, the Navy procurement was structured to permit the agency to choose a successful offeror through the mere exercise of an existing contract option. In such cases, we have recognized that the agency is, in fact, conducting a competition, and accordingly, we have held that our rule against reviewing an agency's refusal to exercise a contract option is inapplicable. See Westinghouse Elec. Corp., 57 Comp.Gen. 328 (1978), 78-1 CPD Para. 181.

Timeliness

The Navy also argues that MSA's protest is untimely, because it was not filed within 10 days of the date MSA knew, or should have known, of the basis for its protest. See 4 C.F.R. Sec. 21.2(a)(2) (1990). According to the Navy, once MSA learned that the Navy had exercised the National Draeger option, MSA should have known that it would not be selected for award because of the language in the contract stating that only one of the options would be exercised.

As mentioned above, the Navy suspended testing of MSA's FFBA after the second day of attempted testing at the Bureau of Mines. By letter dated October 20, the contracting officer notified MSA that testing was suspended indefinitely, and told MSA that it could raise any questions by telephone. Despite this statement, the Navy provided no further information to MSA regarding the selection decision, even though the Navy admits MSA made several telephonic attempts to learn the status of the contract and when testing would resume.

On December 8, the Navy exercised the National Draeger option. Afterwards, as late as December 20, Navy contracting personnel refused to answer MSA's questions about the status of the competition. On December 22, the contracting officer advised MSA by telephone that the Navy had exercised the National Draeger option. However, according to MSA's December 26 letter to the Navy asking for a meeting, the contracting officer indicated during the December 22 conversation that he did not know if testing would be resumed, or if MSA would also be getting an award. response to MSA's written request for a meeting, the Navy offered five possible dates, of which January 29, 1990, was the earliest. At the meeting between MSA and the Navy's contracting officials on that date, the Navy officials explained the status of the procurement and the Navy's decision not to award to MSA. This protest was filed within 10 working days thereafter.

While we will dismiss a protest as untimely where the protester fails to diligently pursue information that forms the basis of its protest, see, e.g., Illumination Control Sys., Inc., B-237196, Dec. 12, 1989, 89-2 CPD Para. 546, we also generally will resolve doubts about timeliness in favor of the protester. Northwest Digital Sys., B-232959.2, Mar. 2, 1989, 89-1 CPD Para. 221. Here, it appears that MSA was actively pursuing information from the Navy that would form the basis for a protest. note that MSA's contemporaneous description of the information provided by the contracting officer regarding the exercise of the Draeger option-- i.e., that the contracting officer could not say whether testing of MSA's FFBA would be resumed, or whether MSA's option would be exercised-- has not been challenged by the Navy, even though MSA's December 26 letter describing these events was included as an attachment with MSA's initial protest. In our view, in light of the Navy's refusal to provide MSA with such information, MSA reasonably believed that it need not file a protest at that time. Under these circumstances, we find that MSA's protest was timely filed.

ANALYSIS

As stated above, the RFP required performance testing and evaluation of each contractor's proposed FFBA before one contractor would be selected to proceed with further development and testing. Under the evaluation scheme, the scores awarded during preaward evaluation for the technical and management proposals, and the cost proposal, were to be combined with the scores awarded for performance testing, with the performance score accounting for nearly half the total evaluation points available. Within the parameter of performance testing, 80 percent of the performance score is derived from the breathing performance tests, while 20 percent of the performance score is derived from shipboard suitability tests.

The protester argues that the Navy improperly abandoned the evaluation scheme set forth in the solicitation by failing to conduct certain tests in accordance with the requirements of the FFBA specification and the test plan incorporated in the RFP. In particular, the protester contends that the Navy could not rationally evaluate MSA's equipment because the test equipment used by the Navy failed to operate in accordance with the specifications in the test plan, thus making an objective review of MSA's FFBA impossible. We agree.

In considering protests against an agency's evaluation of proposals, we will not evaluate the proposals anew in order to make our own determinations as to their acceptability or relative merits. Technical Servs. Corp., B-216408.2, June 5, 1985, 85-1 CPD Para. 640. However, we will examine the record to determine whether the evaluation was fair, reasonable and consistent with the evaluation criteria. Paper Corp. of the United States, B-229785, Apr. 20, 1988, 88-1 CPD Para. 388. We also will review the documentation supporting the source selection decision to determine whether that decision was adequately supported and rationally related to the evaluation factors as required by Federal Acquisition Regulation (FAR) Sec. 15.612(d)(2). Programmatics, Inc., et al., B-228916.2; B-228916.3, Jan. 14, 1988, 88-1 CPD Para. 35.

The FFBA specification incorporated in the RFP required that the breathing performance testing of the unit be conducted on machinery designed to mimic human breathing. In particular, the specification at paragraph 4.2.2 required, in relevant part, that:

"Gas expired from the testing machinery to the FFBA shall be maintained at between 93 (degrees fahrenheit) and 97 (degrees fahrenheit), dry bulb, and saturated with water. ..."

However, during the breathing tests on the MSA unit, the testing machinery was unable to operate at the specified temperature and humidity. According to the Navy, the Bureau of Mines testing equipment was unable to perform in accordance with the specification in part because MSA's device removed water vapor from the system faster than the test equipment could replenish itself, and in part because the testing equipment's heating element failed. After two days of testing- during which the testing equipment never once temperature and humidity. According to the Navy, the Bureau of Mines testing equipment was unable to perform in accordance with the specification in part because MSA's device removed water vapor from the system faster than the test equipment could replenish itself, and in part because the testing equipment's heating element failed. After two days of testing- during which the testing equipment never once operated at the temperature and humidity levels in the specification-- the Navy suspended indefinitely further testing of MSA's device.

In considering this protest, we reviewed the evaluation documents generated by the agency, including the individual evaluators' scores for the parameters to be rated during breathing testing, and the supporting evaluation worksheets. Based on the performance tests conducted with the testing equipment described above, the evaluation team scored breathing performance in nine areas. These areas and their respective weights in the evaluation scheme are set forth below:

Breathing Performance (80 percent of total performance testing):

1. Inhalation Pressure 20% 2. Gas Temperature 20%

3. Duration 15%

4. Exhalation Pressure15%

5. Position 10%

6. Immersion 5%

7. Surface Temperature 5%

8. Gas Composition 5%

9. Face Seal 5%

Our review of the performance evaluation indicates that MSA's scores on six of the nine parameters listed above, comprising 85 percent of the available evaluation points for breathing performance, were either adversely affected by the failure of the testing equipment to operate in accordance with the specification, or were related to tests not performed. /4/

Specifically, the problems with the testing equipment adversely affected the evaluators' ability to rate MSA's equipment in the areas of inhalation pressure, gas temperature, duration, exhalation pressure, and gas composition. The points associated with these parameters comprised 75 percent of the breathing performance score. Further, scoring of two additional parameters was dependent on particular tests that were never performed-- the position and immersion tests, comprising an additional 15 percent of the breathing performance score. /5/

Comments found on the individual evaluators' worksheets reveal that many evaluators believed an objective rating could not be given to MSA's FFBA in several of the nine areas shown above, due to the problems with the test equipment. In other areas, evaluators resorted to guessing-- in the absence of objective testing data-- about MSA's most likely score for a given parameter, based on the evaluator's familiarity with the equipment. In making his selection, the contracting officer adopted the evaluation team's point scores. Our review indicates that the breathing performance of MSA's FFBA was given a numerical score despite the fact that 85 percent of the evaluated points, not counting the immersion test, related to parameters that could not be objectively measured because the required tests were invalid or not performed. Under these circumstances, the contracting officer's decision to award on the basis of this inadequate testing clearly was unreasonable.

The Navy first argues that MSA was not prejudiced by the inability of the testing equipment to operate at the temperature and humidity levels in the specification, because MSA itself "presumably" advocated testing its device at lower temperature and humidity levels in a September 11 letter to the contracting officer. We do not agree with the Navy's interpretation of MSA's letter. The letter in question states that MSA's research indicates that the Navy's temperature and humidity requirements, among other things, are inconsistent with the characteristics of human breath. Nonetheless, MSA was still required to produce an FFBA for testing in accord with the specifications. MSA's letter neither encouraged nor agreed to the agency's abandonment of the evaluation scheme in testing prototype FFBAs.

Further, the operation of the testing equipment at the temperature and humidity levels in the specification was of great importance for the successful operation of MSA's device. MSA's prototype FFBA was designed to operate with a replaceable canister filled with potassium superoxide (KO2). The KO2 in MSA's unit reacts with the moisture and gas content of the user's exhaled breath to free oxygen for inhalation while removing carbon dioxide. As a result, the failure of the test equipment to operate as specified clearly hindered the evaluation of MSA's device, designed to use the moisture in exhaled breath to operate. On the other hand, National Draeger's device did not utilize KO2 and was less sensitive to the lower moisture content of the testing machine's exhaled breath.

The Navy also argues that MSA was not prejudiced by the Navy's failure to perform all the required tests, because the evaluators rating MSA's device gave it the "benefit of the doubt." According to the Navy, this "benefit" took the form of giving MSA a score of 4-- the rating suggested in the evaluation guidelines for a device that meets, but does not exceed, requirements-- in instances where MSA's FFBA did not operate in accordance with the specification. The Navy's argument here has no merit. The evaluators rated each FFBA on a scale of 1 to 10 for each of the 9 breathing performance parameters. Without actually performing the tests as required, the choice of a rating of 4 rather than a rating of 10 is arbitrary.

The Navy also contends that, regardless of the testing deficiencies discussed above, sufficient tests were conducted for the technical evaluators to determine that MSA's FFBA was technically unacceptable. Specifically, the Navy argues that the bailment mechanism-- the ripcord and lever arrangement chosen by MSA to pull the canister up to the manifold in the device to begin artificial breathing-- did not work well, was subject to wear, and in one instance, fell apart. Further, the Navy claims the KO2 canister did not firmly seal against the manifold due to the type of bailment mechanism chosen by MSA. As a result, the Navy claims a complete redesign of the canister, the seal, or the bailment mechanism would be required to overcome the problems with MSA's prototype, and that given these problems, it is absurd for MSA to argue that its device should be fully tested.

As a preliminary matter, MSA challenges the Navy's conclusion that the mechanics of its FFBA device-- i.e., the cannister, seal and bailment mechanism-- require a complete redesign to overcome problems seen in shipboard suitability testing. MSA argues that Navy test subjects using the device lacked proper training, and that the mechanical parts of the device subject to wear could be treated and strengthened during further development. The Navy responds that strengthening the device would only add weight to the unit, and that MSA's device already exceeds the weight requirements in the specification.

Although the Navy enumerates various defects in the bailment mechanism, we do not, at this point, accept the Navy's blanket conclusion that these problems require a complete redesign of MSA's device. The Navy here has awarded dual research and development contracts to develop new breathing capabilities in the devices it uses to provide oxygen to firefighters combating shipboard fires. The mechanics of connecting the breathing device to its oxygen source (in MSA's case, the KO2 canister), do not involve advanced technology-- i.e., on its own, the bailment mechanism would not have been the subject of a research and development contract. Further, MSA has successfully produced the predecessor to this device for nearly 40 years.

Moreover, we find unconvincing the Navy's position that the mechanical parts of the prototype bailment mechanism could not be treated and strengthened for production. The Navy anticipates further development and testing of any device selected at this point, and National Draeger's unit is also deficient in certain areas. Further, the Navy's contention that strengthening the parts used in MSA's bailment mechanism would only add unacceptable weight to the device is unpersuasive, given that National Draeger's unit exceeds the specified weight requirements by several pounds more than does MSA's device.

Although the Navy terms MSA's device "technically unacceptable," due to the bailment mechanism chosen and the perceived lack of reliability of that bailment mechanism, the record indicates that the evaluation scheme here was comparative. Nothing in the RFP indicates that failure of a prototype to perform in certain areas would render that device technically unacceptable. See Paper Corp. of United States, B-229785, supra. Here, under the scheme for evaluating the prototypes (which will be subject to further development and testing), the operation of the bailment mechanism and the resulting seal were scored as part of the expendable replenishment subfactor of the shipboard suitability factor. /6/ The expendable replenishment subfactor was valued at 5 percent of the total performance score-- indicating its relative importance in the evaluation. Terming MSA's FFBA "technically unacceptable" for its problems in this area was inconsistent with the weight given this subfactor in the evaluation scheme. See TRW, Inc., 68 Comp.Gen. 511 (1989), 89-1 CPD Para. 584; Kaufman Lasman Assocs., Inc., 68 Comp.Gen. 34 (1988), 88-2 CPD Para. 381.

As mentioned earlier, the scores awarded MSA and National Draeger were close: 8.83 for MSA, on a scale of 1 to 10, and 10 for National Draeger. However, 85 percent of the available evaluation points for breathing performance-- worth ultimately 30 to 35 percent of the total evaluated score-- is related to tests that either did not comply with the FFBA specification, or were not performed at all. Since proper performance testing could have changed the Navy's award decision, we sustain the protest.

By separate letter to the Secretary of the Navy, we are recommending that the prototype FFBAs already submitted be subjected to proper performance testing on working test equipment, or that the two contractors be permitted to resubmit prototypes for such testing. We also recommend that the evaluation be conducted on the basis stated in the RFP, or that new criteria be established and published so that both offerors are competing equally. If, based on this evaluation, MSA is in line for award, we recommend that the Navy terminate the contract awarded to National Draeger and exercise the option in MSA's contract to proceed with development of the FFBA. We also find MSA entitled to recover the costs of filing and pursuing its protest, including attorneys' fees.

The protest is sustained.

/1/ The previous generation of FFBA equipment, called the Oxygen Breathing Apparatus, has been provided to the Navy by MSA for many years.

/2/ Both contracts state that the Navy anticipates exercising only one contractor's option.

/3/ Despite the apparently perfect score given the National Draeger unit, neither unit was rated as highly as these scores would indicate. After the evaluators assigned initial ratings to each unit, the Navy "normalized" the scores by awarding the maximum number of points available to the device receiving the highest score, and making a corresponding proportionate adjustment to the score of the other device.

/4/ An additional 5 percent of the available evaluation points for breathing performance were related to an immersion test not performed for reasons discussed below. We do not here question the Navy's decision not to perform the immersion test.

/5/ For the position test, evaluators extrapolated from other scores, many of which were also guesses, about how MSA's FFBA would perform in different positions-- i.e., on its side, at an angle, etc. According to the agency, MSA was given the benefit of the doubt because it was assumed there would be no deterioration in performance associated with operating the device in other positions.

For the immersion test, the Navy evaluators expressed concerns about whether an immersion test could be safely performed on MSA's FFBA given the presence of a gap in the unit's seal located at the top of the potassium superoxide canister. Based on our review of the record, since the Navy reasonably concluded it could not safely perform the immersion test, we will not question the Navy's decision not to perform this test and to award a zero score-- worth 5 percent of the evaluated points available-- for this parameter of the breathing performance evaluation.

/6/ Problems with the bailment mechanism and the seal in MSA's FFBA could also be reflected in the breathing performance scores if the seal was so inadequate as to cause deterioration in the performance of the FFBA. However, since breathing performance of the device was not properly tested, we have no objective basis to measure the effect of such a problem.