Raytheon Company

DOCUMENT FOR PUBLIC RELEASE
The decision issued on the date below was subject to a GAO Protective Order. This redacted version has been approved for public release.

Decision

Matter of: Raytheon Company

File: B-404998

Date: July 25, 2011

DIGEST

1. Protest is sustained where discussions were conducted, but the protester was not provided with an opportunity to address the significant weaknesses identified in its proposal, even though one of the awardees had been provided the opportunity to revise its technical proposal in a significant way.

2. Protest is sustained where the agency assessed a weakness under the experience factor (which required the submission of five references) whenever an offeror's reference did not have one of six areas of the experience required by the solicitation, even though the solicitation only required that offerors' references "collectively" demonstrate experience in the required areas.

3. Protest challenging agency's use of adjectival ratings is denied where the ratings were consistent with the evaluation factor weights set forth in the solicitation.

Raytheon Company, of Dulles, Virginia, protests the decision of the Department of Commerce, U.S. Patent and Trademark Office (PTO) not to award Raytheon one of the multiple contracts awarded under request for proposals (RFP) No. DOC52PAPT1000015, for software development and integration services. Contracts were awarded to the following firms: Amentra, Inc., of McLean, Virginia; Computer Sciences Corporation (CSC), of Chantilly, Virginia; Science Applications International Corporation (SAIC), of McLean, Virginia; Pragmatics, Inc., of McLean, Virginia; and Unisys Corporation, of Reston, Virginia. The protester contends that the agency failed to provide it with an opportunity for meaningful discussions, unreasonably evaluated its experience, and used an evaluation ratings scheme that was inconsistent with the terms of the RFP.

We sustain the protest.

BACKGROUND

The RFP was issued on April 7, 2010, and was amended five times. The solicitation sought proposals to provide the following services in support of the PTO Office of the Chief Information Officer: software development and integration, development testing, configuration management support, production support and software maintenance and transition, and program management support. RFP amend. 4, sect. C.1.3.

The solicitation anticipated award of multiple indefinite-delivery/indefinite-quantity, labor-hour contracts, with a 1-year base period and four 1'year options. Offerors were advised that they would be evaluated on the basis of price, and the following non-price factors: (1) experience, (2) past performance, and (3) sample tasks. Id. sect. M.3.1.a. The RFP stated that "[t]he Experience factor and the Past Performance factor are of equal value," and that "[t]he Experience factor and the Past Performance factor are more important than the Sample Tasks factor." Id. sect. M.3.1.b. For purposes of award, the non-price factors were "significantly more important" than price. Id. sect. M.3.1.c.

PTO received 16 proposals by the closing date of June 17. As relevant here, the agency convened a technical evaluation team (TET) to evaluate the offerors' technical proposals, including the experience factor. The TET members prepared individual evaluation comments and requests for clarification for each proposal. The individual comments and requests for clarification were provided to the contracting officer (CO). Based on the individual evaluator comments and concerns, the agency sent questions to a number of offerors, including Raytheon and Amentra. As discussed further below, the agency's questions for Amentra requested that the offeror provide additional details concerning its proposal. See Agency Report (AR), Tab 10, Amentra Question Nos. 1-3.[1]

After receiving offerors' responses to the questions, the TET prepared a consensus evaluation report, and assigned ratings under the experience and past performance factors. The TET then assigned for each offeror an "initial technical rating," which combined the ratings for the experience and past performance factors. On December 1, the agency established a competitive range consisting of the nine most highly-rated offerors, which were those offerors whose initial technical rating was acceptable-plus, or higher and whose prices were fair and reasonable. CO Statement para. 13. In January 2011, the offerors in the competitive range were asked to provide oral presentations for the sample task evaluation factor. The TET reviewed the oral presentations and prepared consensus ratings for each offeror. The TET then prepared a final evaluation report, which established an "overall technical rating" for each offeror that combined its initial technical rating (which was a combination of the experience and past performance factor ratings) and the sample task rating. Following the oral presentations, the CO requested that the competitive range offerors submit final proposals, but limited revisions to the offerors' prices. The final evaluation ratings for each of the competitive range offerors was as follows:

	INITIAL TECHNICAL RATING		SAMPLE TASK RATING	OVERALL TECHNICAL RATING	AVERAGE PRICE (on-site/ off-site)
	Experience	Past Performance
AMENTRA	ACCEPTABLE+		EXCELLENT	GOOD	$132.61/ $139.92
	Acceptable	Good
CSC	GOOD+		GOOD	GOOD	$69.06/ $72.81
	Good	Excellent
SAIC	GOOD		ACCEPTABLE	GOOD	$85.93/ $92.76
	Good	Good
PRAGMATICS	ACCEPTABLE+		EXCELLENT	GOOD	$76.79/ $78.57
	Acceptable	Good
UNISYS	GOOD-		EXCELLENT	GOOD	$83.92/ $87.28
	Acceptable	Excellent
RAYTHEON	ACCEPTABLE+		GOOD	ACCEPTABLE	$105.92 $111.76
	Acceptable	Good
OFFEROR 7	ACCEPTABLE+		GOOD	ACCEPTABLE	$72.76/ $80.75
	Acceptable	Good
OFFEROR 8	ACCEPTABLE+		ACCEPTABLE	ACCEPTABLE	$84.02/ $98.84
	Acceptable	Good
OFFEROR 9	ACCEPTABLE+		ACCEPTABLE	ACCEPTABLE	$92.44/ $108.06
	Acceptable	Good

AR, Tab 14, Initial Technical Evaluation, at 4; Tab 27, Source Selection Recommendation (SSR),at 5-6. [2]

The CO prepared an award recommendation, which was accepted by the source selection official. AR, Tab 27, Source Selection Decision, at 1. In her recommendation, the CO concluded that all offerors with an overall technical rating of good or better should receive a contract award. AR, SSR, at 7. The CO identified strengths for each awardee with an overall rating of good, and concluded that these offerors' strengths merited award as compared to the other offerors, whose proposals received overall technical ratings of acceptable. Id. at 7-13. The CO further concluded that the prices proposed by the awardees were fair and reasonable, and that the price proposed by the highest-priced offeror, Amentra, merited award as compared to any of the competitive range offerors whose proposals had overall ratings of acceptable--each of which had a lower price than Amentra. Id. at 12. On March 22, the agency awarded contracts to Amentra, CSC, SAIC, Pragmatics, and Unisys; the contracts had a combined ceiling value of $532 million. AR, Tab 29, Award Notice, at 1.

Raytheon requested a debriefing, which the agency provided in writing on March 28, and in a meeting on April 13. This protest followed.

FAILURE TO PROVIDE RAYTHEON WITH MEANINGFUL DISCUSSIONS

Raytheon argues that PTO was required to provide it with an opportunity for meaningful discussions, but failed to do so. Specifically, the protester argues that the agency identified various significant weaknesses in its proposal, yet did not mention any of them during discussions, as it was required to do. Moreover, Raytheon argues that the agency's pre-competitive range exchanges with one of the awardees, Amentra, constituted discussions. Because PTO conducted discussions, Raytheon argues, the agency was obligated to provide the protester with an opportunity to address the significant weaknesses found in its proposal. For the reasons discussed below, we agree with the protester.

Federal Acquisition Regulation (FAR) sect. 15.306 describes a range of exchanges that may take place between an agency and an offeror during negotiated procurements. Clarifications are "limited exchanges" between the agency and offerors that may allow offerors to clarify certain aspects of proposals or to resolve minor or clerical mistakes. FAR sect. 15.306(a)(2). Discussions, on the other hand, occur when an agency indicates to an offeror aspects of its proposal that could be altered or explained to materially enhance the proposal's potential for award or to obtain information from the offeror that is necessary to determine the proposal's acceptability. See FAR sect. 15.306(d)(3); Nu-Way, Inc., B-296435.5, B-296435.10, Sept. 28, 2005, 2005 CPD para. 195 at 7. When conducting discussions, agencies must identify, at a minimum, "deficiencies, significant weaknesses, and adverse past performance information to which the offeror has not yet had an opportunity to respond." FAR sect. 15.306(d)(3). When an agency conducts discussions with one offeror, it must conduct discussions with all other offerors in the competitive range, FAR sect. 15.306(d)(1), and those discussions must be meaningful; that is, the discussions must identify deficiencies and significant weaknesses in each offeror's proposal. FAR sect. 15.306(d)(3); Spherix, Inc., B-294572, B-294572.2, Dec. 1, 2004, 2005 CPD para. 3 at 13-14.

In examining whether exchanges between an agency and an offeror are clarifications or discussions, our Office looks to the actions of the parties, and not merely the characterization of the exchanges by the agency. Gulf Copper Ship Repair, Inc., B-293706.5, Sept. 10, 2004, 2005 CPD para. 108 at 6. In this regard, we have found that the "acid test" for deciding whether an agency has engaged in discussions is whether the agency has provided an opportunity for proposals to be revised or modified. See, e.g., Priority One Servs., Inc., B-288836, B-288836.2, Dec. 17, 2001, 2002 CPD para. 79 at 5.

As set forth below, we conclude that: (1) PTO identified significant weaknesses in Raytheon's proposal; (2) the agency provided one of the awardees, Amentra, with an opportunity to respond to questions, which constituted discussions, and amounted to unequal treatment as compared to Raytheon; and (3) the agency's request for revised price proposals from all offerors also constituted discussions.

Significant Weaknesses in Raytheon's Proposal

We first address whether the concerns identified by PTO under the experience factor for Raytheon's proposal were weaknesses, significant weaknesses, or deficiencies. Raytheon argues that the agency identified numerous significant weaknesses in its proposal, and that if it had been provided an opportunity to address them during discussions, it could have improved its proposal and received an award. PTO argues that none of the weaknesses were significant, and therefore none required discussions.

As discussed above, the FAR requires agencies to identify during discussions "deficiencies, significant weaknesses, and adverse past performance information to which the offeror has not yet had an opportunity to respond." FAR sect. 15.306(d)(3). The FAR defines the types of potential concerns as follows:

"Deficiency" is a material failure of a proposal to meet a Government requirement or a combination of significant weaknesses in a proposal that increases the risk of unsuccessful contract performance to an unacceptable level.

"Weakness" means a flaw in the proposal that increases the risk of unsuccessful contract performance.

A "Significant weakness" in the proposal is a flaw that appreciably increases the risk of unsuccessful contract performance.

RFP sect. 15.001.

As our Office has consistently held, agencies are not required to afford offerors all'encompassing discussions or to discuss every aspect of a proposal that receives less than the maximum score, and are not required to advise an offeror of a minor weakness that is not considered significant, even where the weakness subsequently becomes a determinative factor in choosing between two closely ranked proposals. Apptis, Inc., B-403249, B-403249.3, Sept. 30, 2010, 2010 CPD para. 237 at 4. In determining whether a concern identified by an agency was a weakness, significant weakness, or deficiency, our Office does not rely solely on the label or term used by the agency, but instead looks also to the context of the evaluation. See AT&T Corp., B-299542.3, B-299542.4, Nov. 6, 2007, 2008 CPD para. 65 at 11; Alliant Techsystems, Inc.; Olin Corp., B-260215.4, B-260215.5, Aug. 4, 1995, 95-2 CPD para. 79 at 7-8.

Here, the record shows that, in evaluating Raytheon's proposal under the experience factor, PTO assessed weaknesses based on a 5-point scale as follows: category 1, no impact to program success; category 2, negligible impact to program success; category 3, limited impact to program success; category 4, program success could be jeopardized; and category 5, program success in doubt. The agency also assigned an overall risk rating for each weakness of low, medium or, high. AR, Tab 15, Raytheon Consensus Evaluation, at 1-7.

In its consensus evaluation, the agency identified 31 "weaknesses" for Raytheon's experience, including 12 weaknesses which were labeled category 4, "program success could be jeopardized." Id. Of those 12 weaknesses, 10 were rated as "medium risk." Id.

Examples of the category 4, medium-risk weaknesses identified by the agency include the following:

[Raytheon] demonstrates a weakness in Unit and Integration Testing in their experience on the National Oceanic and Atmospheric Administration (NOAA) . . . projects in meeting the RFP Requirements C.12.8.3 (Track Unit Testing within a tool), C.12.10.2 (Follow Defect Management Plan), and C.12.10.3 (Use Defect Management Tools) which increases the risk to the [PTO] as it negatively impacts identifying, tracking, and resolving defects in a timely manner to avoid introduction into the production environment.

* * * * *

[Raytheon's] Unit and Integration Testing write-up for Contract Reference 5 is lacking detailed descriptions of the duties performed which exhibits limited confidence that the Offeror has relevant experience in this area.

* * * * *

The Offeror did not provide information about the ability to operate and install all operational scripts and configuration files from the [configuration management (CM)] repository (C.12.9.4). The Offeror provides no information regarding documentation/instructions on building from the CM repository (C.12.9.5). This lack of experience negatively impacts [PTO's] ability to recreate builds, including disaster recovery, and make sure code, scripts, and configuration files in production match those in CM for audit purposes/disaster recovery.

AR, Raytheon Consensus Evaluation, Tab 15, Comment Nos. J18, J20, J23.

The agency contends that none of these weaknesses were significant, and that no discussions were required. However, given that category 4 weaknesses were said to jeopardize program success, and based on our review of the comments themselves, we think that category 4 weaknesses were significant weaknesses. Moreover, we note that the TET Leader admits that "the term 'significant weakness' was not used in the technical evaluations of the offerors' proposals." AR, Tab 42, Decl. of TET Leader, June 7, 2011 para. 13. Thus, the agency's failure to use the term "significant weakness" in the evaluation documentation is not controlling for purposes of determining whether the concerns were, in fact, significant weaknesses.[3]

In sum, we find that the agency identified significant weaknesses in Raytheon's proposal. As discussed below, we also conclude that the agency conducted discussions with the offerors, and therefore should have permitted Raytheon an opportunity to revise its proposal to address these significant weaknesses.

Exchanges with Amentra Were Discussions

Next, we address whether PTO was required to conduct discussions with Raytheon concerning the significant weaknesses in its proposal. We address three areas where the record shows that the agency engaged in discussions with Amentra.

As discussed above, the agency asked certain offerors to address questions concerning their proposals prior to establishing the competitive range. Raytheon was asked to address one clarification question regarding its proposed prices; the agency did not ask Raytheon to address any questions regarding its technical proposal. See AR, Tab 10, Raytheon Question No. 1. The protester argues that the agency asked one of the awardees, Amentra, to address questions concerning its technical proposal under the experience factor, which permitted that offeror to submit proposal revisions that constituted discussions.

PTO primarily argues that, regardless of whether the pre-competitive range questions asked of Amentra were requests for clarification or discussions, discussions did not in fact occur because the agency did not consider Amentra's responses to the questions. The CO states that for all responses that exceeded the scope of a request for clarifications, the agency acted as follows:

Upon receipt of the responses, it became evident that some of the responses received were attempts to supplement their technical proposals which were required to be limited to the five contract references consisting of two pages each. Therefore, any supplemental information regarding an offerors' technical proposal was not considered. The TET did not alter its technical evaluation ratings as a result of the clarification responses.

CO Statement para. 10; see also AR, Tab 42, Decl. of TET Leader, June 7, 2011, paras. 7-8; Supp. AR (SAR) at 3-4.

Based on our review, however, we conclude that the record is not consistent with the agency's description above. Instead, the record shows that Amentra was provided an opportunity to revise its proposal, and that the agency considered the responses in its evaluation.

First, the agency's initial evaluation identified a concern with Amentra's approach to mentoring customer personnel under the experience factor. As discussed above, TET evaluators prepared initial evaluation comments, and those comments were the basis of the pre-competitive range questions. Evaluator 1 identified a request for more information concerning Amentra's contract reference for the U.S. Mint: "Provide details on the Amentra Gartner-and-industry recognized signature mentoring model."[4] AR, Tab 39, Amentra Initial Evaluation, Comment No. 5.

Evaluator 2 identified three strengths concerning one of Amentra's contract references:

[Amentra] used a mentoring model to transition knowledge to the customer and sub contractor throughout the contract.

[Amentra] used a mentoring model to transition knowledge right from the start. They trained customer in some of the technologies, best practices as well as on the build process including Ant and Cruise Control.[5]

[Amentra] provided documentation and mentoring of the staff including [deleted] and troubleshooting to transfer the support to other groups as specified in Section C.12.11 in the RFP.

Id., Comment Nos. 35, 41, 46.

Following these evaluations, the agency asked Amentra to address the following question: "Please provide further clarification on the Amentra Gartner-and-industry recognized signature mentoring model discussed on Pages I.3, I.5, I.7, I.9, and I.11." AR, Tab 10, Amentra Question No. 2.

In answer, Amentra prepared a 1-page response, which provided specific details about the offeror's approach to mentoring, which the offeror stated "has been featured by industry analysts Gartner and Forrester as well as periodicals such as CIO Magazine." AR, Tab 11, Amentra Response to Question No. 2. The offeror described its mentoring approach as "[deleted]," and provided details concerning the [deleted] "high-level steps" that comprise its approach. Id.

After receipt of the offerors' responses to questions, the TET prepared a consensus evaluation. The TET's evaluation of Amentra's proposal identified a strength based on its mentoring approach, as follows:

[Amentra] provided documentation and mentoring of the staff including [deleted] and troubleshooting during phase out to transfer the support to other groups which exceeds RFP Section C.12.11 requirements and the experience will benefit [PTO]. They trained customer in some of the technologies, best practices as well as on the build process including Ant and Cruise Control. Beneficial to the [PTO] to mentor, train and develop employee skills regarding new technology developed by the awardee.

[Amentra] provided further clarification on the . . . Gartner-and industry recognized signature mentoring model in its response to [Amentra]-Experience-02.

AR, Tab 41, Amentra Consensus Evaluation, at 6.

In selecting Amentra's proposal for award, the selection decision also cited Amentra's approach to mentoring, as follows:

Amentra uses a Gartner-recognized Signature Mentoring Model for Training and Knowledge retention. That has increased benefit to the [PTO] by providing the government with knowledge retention. That reduces the cost of support for new task order transition, and reduces maintenance costs.

* * * * *

Amentra was acquired by Red Hat and its Open Source Solutions are recognized by Gartner. . . . Combining this with their signature mentoring model, provides a strong [PTO] partner as the [PTO] modernizes its information technology.

AR, Tab 27, SSR, at 8-9.

Thus, despite the use of the word "clarification," we think that the request that Amentra "provide further clarification on the Amentra Gartner-and-industry recognized signature mentoring model" clearly requested a substantive revision of the offeror's proposal with regard to the level of detail concerning mentoring, and thus constituted discussions. AR, Tab 10, Amentra Question No. 2.

Notwithstanding the nature of the question, PTO contends that it disregarded Amentra's response because it exceeded the scope of the question. CO Statementpara. 10; Supp. AR (SAR) at 3-4. Further, the agency states that the strengths cited in its evaluation of Amentra's proposal concerning mentoring did not arise from the offeror's response to the concerns raised by evaluator 1, but were instead based on strengths recognized independently by evaluator 2. SAR at 5-6. The agency states that it was the consensus judgment of the TET that the strengths identified by evaluator 2 should "carry forward" to the consensus judgment. SAR at 7. We do not find these arguments persuasive.

In reviewing protest allegations, we do not limit our review to contemporaneous evidence, but consider all the information provided, including the parties' arguments, explanations, and documentation prepared in response to protest contentions. Systems Research and Applications Corp.; Booz Allen Hamilton, Inc., B-299818 et al., Sept. 6, 2007, 2008 CPD para. 28 at 12. While we consider the entire record, including the parties' later explanations and arguments, we accord greater weight to contemporaneous evaluation and source selection material than to arguments and documentation prepared in response to protest contentions. Technology Concepts & Design, Inc., B-403949.2, B-403949.3, Mar. 25, 2011, 2011 CPD para. 78 at 9.

Here, there is no contemporaneous evidence in the record that the agency disregarded Amentra's response to the agency's questions. To the contrary, the record plainly shows that the agency cited Amentra's response in the final TET consensus report. Although evaluator 2 cited strengths for Amentra's "mentoring model" and for "mentoring of the staff," the TET consensus report and SSR both specifically cite the "Gartner-recognized Signature Mentoring Model," which was the subject of question No. 2, and Amentra's response to that question. More significantly, the TET consensus report states, in describing the strength in Amentra's proposal, that that firm "provided further clarification on the . . . Gartner-and industry recognized signature mentoring model in its response to Offeror B-Experience-02." AR, Tab 41, Amentra Consensus Evaluation, at 6.

On this record, we conclude that the agency relied on Amentra's response to question No. 2 in evaluating its proposal. We further conclude that the agency's request for information from Amentra regarding the Amentra Gartner-and-industry recognized signature mentoring model constituted discussions, and not clarifications.

Second, Raytheon argues that PTO provided Amentra an opportunity for discussions concerning its experience performing a contract for GEICO that permitted the awardee to revise its proposal and resulted in a strength in its evaluation.

Evaluator 3 identified the following "deficiency" concerning Amentra's reference for a contract with GEICO regarding software development and integration: "Offeror uses terms such as 'assisted', 'Supported', 'leveraged'. It is not clear the extent of authority and involvement that the Offeror had with these efforts." AR, Tab 39, Amentra Initial Evaluation, Comment No. 56.

In contrast, evaluator 2 noted strengths for Amentra's proposal concerning software development and integration with regard to two of its past performance references, as follows:

[Amentra] has experience in upgrading an existing system to a Service Oriented Architecture [SOA]. The offeror leveraged [deleted], [and] [deleted] to design a service tier. The expertise is specified in Section C.8.2 of the RFP.

[Amentra] has additional experience upgrading outdated legacy systems to a SOA based architecture using opens standards and web services as specified in Section C.8.2.

Id., Comment Nos. 36 (GEICO reference), 42 (Federal Home Loan Bank reference).

During the pre-competitive range exchanges, PTO asked Amentra to address the following question:

AR, Tab 10

For Contract Reference . . . GEICO, what was the Offeror's role with GEICO software development and integration? Did you conduct the analysis, design, development, integration, and maintenance of the new and existing systems?

, Amentra Question No. 1.

In response to the question, Amentra prepared a 1-page response which provided specific details about the offeror's role as "an organizational catalyst for enterprise modernization and the realization of an enterprise-wide [SOA] for GEICO's technical and business architecture." AR, Tab 11, Amentra Response to Question No. 1. As relevant here, the offeror stated that "[t]hroughout the years, Amentra has performed all facets of the [system development life cycle] including analysis, design, development, integration, and maintenance of the new and existing systems using a modified Agile approach."[6] Id.

In the consensus evaluation the TET concluded that Amentra's proposal merited the following strength:

[Amentra] has demonstrated a strength in Software Development and Integration in the Business Process Mapping Services, SOA and Java Development Services, Discovery.com Dynamic Website Platform, FMS4.0 projects by demonstrating past experience in Agile Software Development which exceeds the requirements in the RFP and has increased benefit to the [PTO]. Agile experience aligns with [Office of Management and Budget] guidance to build [information technology (IT)] modernization in smaller, more modular programs as it has a higher success rate than waterfall and allows products to be demonstrated to the customer sooner for feedback.

AR, Tab 41, Amentra Consensus Evaluation, at 2.

We think that the question asked by PTO to Amentra clearly asked the offeror to provide a substantive response by asking the offeror to explain the following question: "[W]hat was the Offeror's role with GEICO software development and integration?" and thus constituted discussions. AR, Tab 10, Amentra Question No. 1.

Again, however, the agency contends that the Amentra response was not considered by the evaluators. The agency states that although evaluator 3 identified a weakness concerning the GEICO contract, evaluator 2 identified a strength regarding the GEICO contract, which was the basis of the consensus evaluation, rather than the offeror's response to question No. 1.

We think the consensus evaluation addressed an aspect of Amentra's proposal that was not reflected in the initial comments by evaluator 2. As the protester notes, none of the individual evaluator comments cited experience with Agile as a strength --in fact, one evaluator's comment questioned whether Amentra's proposal adequately explained its experience with Agile. See AR, Tab 39, Amentra Initial Evaluation, Comment No. 11. Aside from the agency's post-protest statement that the TET disregarded Amentra's response to the pre-competitive range questions, the record does not show how the agency reached the conclusion that Amentra's proposal merited the strength reflected in consensus evaluation. Specifically, while certain aspects of the strength identified for Amentra in the consensus evaluation clearly relate to evaluator 2's initial comments, the agency does not show how, in the absence of Amentra's response to question No. 1, it concluded that Amentra's experience with Agile software development was a strength.

On this record, we do not find persuasive the agency's explanation that the strengths cited in the TET consensus report and the SSR did not rely on Amentra's response to question 1. We further conclude that the agency's request for information from Amentra regarding its GEICO experience constituted discussions, and not clarifications.

Third, the protester argues that PTO conducted discussions with Amentra to address a risk identified by the agency in the initial evaluation under the experience factor. Evaluator 3 identified the following "risk" regarding Amentra's proposal under the program management support [subfactor], concerning the offeror's reference for a contract with the U.S. Mint:

Offeror describes the results in vague terms ("The as-is business processes were slow, inefficient, manual, and costly to move foods from idea conception to the actual delivery to customers from both a business and IT perspective. Through the use of task orders, Amentra is leading the Mint on a path toward a complete business process ownership by implementing a strategic [business process management (BPM)] program.") There should be quantifiable impacts/results of their effort.

AR, Tab 39, Amentra Initial Evaluation, Comment No. 51.

The agency asked Amentra to address this matter during the pre-competitive range exchanges by quoting the evaluated risk and requesting, "[p]lease identify the quantifiable impacts/results of your efforts." AR, Tab 10, Amentra Question No. 3. In response to the question, Amentra prepared a 1-page response which provided specific details about the task orders performed by the offeror, and the offeror's BPM initiatives and their results. AR, Tab 11, Amentra Response to Question No. 3.

The risk cited by Evaluator 3 was not contained in the TET consensus report. See AR, Tab 41, Amentra Consensus Evaluation. In the SSR, the agency stated that Amentra's proposal merited strengths for "demonstrat[ing] strong business analysis capabilities," which provides "a strong benefit to the [PTO] as it will result in building systems that meet business needs." AR, Tab 27, SSR, at 9.

As with the previous two questions, we think that the question to Amentra that it "identify the quantifiable impacts/results," asked the offeror to provide a substantive response that allowed the company to revise its proposal. AR, Tab 39, Amentra Initial Evaluation, Comment No. 51.

With regard to the effect of Amentra's response, Raytheon argues that the risk cited by evaluator 3 was addressed in the offeror's response to question No. 3, and the risk was not included in the TET consensus report. The protester further argues that the agency assigned a new strength based on Amentra's response to question No. 3, concerning business analysis capabilities, which had not been identified by the TET evaluators prior to the pre-competitive range exchanges. See AR, Tab 27, SSR, at 9.

PTO contends that the strength for Amentra identified by the protester was assessed as a result of information provided for the first time during Amentra's oral presentation. See SAR at 8-9. With regard to the risk identified by evaluator 3 regarding the level of detail in Amentra's proposal concerning BPM, the agency implies that the TET's evaluation did not rely on Amentra's response to question 3, and instead relied on the past performance reference information and the awardee's oral presentation. See SAR at 8. The agency, however, does not cite any information from the past performance references that addressed the concern raised in question 3. Moreover, as the protester notes, the oral presentations took place in January 2011--after the weakness was removed from the consensus evaluation report on December 1, 2010. See CO Statement para. 14. The agency provides no other explanation as to how, in the absence of Amentra's response to question No. 3, Amentra's proposal did not merit the identified risk.

On this record, we do not find persuasive PTO's explanation as to how it concluded that Amentra had addressed the agency's concerns regarding the level of detail in Amentra's proposal regarding its BPM initiatives. For this reason, we conclude that the agency relied in part on Amentra's response to question 3, and conclude that the agency engaged in discussions with Amentra regarding this evaluated risk.

Request for Revised Proposals Constituted Discussions

As set forth above, PTO identified significant weaknesses in Raytheon's proposal, and conducted discussions with Amentra prior to establishing the competitive range, without providing the protester an equal opportunity for meaningful discussions. Moreover, as a matter of law the record shows that the agency also conducted discussions with offerors in the competitive range by requesting revised proposals.

Specifically, the CO requested that the competitive range offerors submit final proposals, but limited revisions to the offerors' prices. Where an agency provides the opportunity to offerors to revise their price proposals, this constitutes discussions. See Price Waterhouse, B-254492.2, Feb. 16, 1994, 94-1 CPD para. 168 at 11; Dyncorp, B-245289, B'245289.2, Dec. 23, 1991, 91-2 CPD para. 575 at 11. As indicated above, whenever an agency conducts discussions, those discussions must be meaningful; that is, the discussions must identify deficiencies and significant weaknesses in each offeror's proposal. FAR sect. 15.306(d)(3); Spherix, Inc., supra. Here, Raytheon was not provided with an opportunity to address the significant weaknesses identified in its proposal and the discussions were therefore not meaningful, even though one of the awardees was allowed to revise its technical proposal in a significant way. We sustain the protest on this basis.

EVALUATION OF RAYTHEON'S EXPERIENCE

Next, Raytheon argues that PTO's evaluation of its proposal under the experience factor was unreasonable, based on the agency's improper interpretation of the solicitation. For the reasons discussed below, we agree, and sustain the protest in this area as well.

Agencies are required to evaluate proposals based solely on the factors identified in the solicitation, and must adequately document the bases for their evaluation conclusions. Intercon Assocs., Inc., B-298282, B-298282.2, Aug. 10, 2006, 2006 CPDpara. 121 at 5. While agencies properly may apply evaluation considerations that are not expressly outlined in the RFP where those considerations are reasonably and logically encompassed within the stated evaluation criteria, there must be a clear nexus between the stated criteria and the unstated consideration. Global Analytic Info. Tech. Servs., Inc., B-298840.2, Feb. 6, 2007, 2007 CPD para. 57 at 4. Although we will not substitute our judgment for that of the agency, we will question the agency's conclusions where they are inconsistent with the solicitation criteria, undocumented, or not reasonably based. Sonetronics, Inc., B-289459.2, Mar. 18, 2002, 2002 CPD para. 48 at 3.

Here, the RFP advised offerors that their proposals should address the experience factor as follows:

a.The Government is seeking five (5) reference contracts for the offeror who is proposing as the contractor that will:

1. emonstrate the Offeror's ability to perform the [software development and integration (SD&I)] activities specified in C.12 of this solicitation; and
2. Demonstrate the Offeror's experience performing SD&I work that is relevant to [PTO's] SD&I requirements.

* * * * *

c.Specifically, the Government expects to see contract references that:

1. Collectively demonstrate experience in the following areas:
1. Program Management Support
2. Software Development and Integration
3. Unit and Integration Testing
4. Configuration Management Support
5. Software Maintenance
6. Contract Transition

RFP amend. 4, sect. L.9.2.1.1. Section M of the solicitation stated that "the evaluation of the Offeror's experience will be a subjective assessment of the offeror's ability to perform and demonstrated experience" in the six areas of experience, cited above. RFP amend. 4, sect. M.3.2.1.

Raytheon submitted five contract references for the experience factor. The TET consensus evaluation identified 31 weaknesses based on the protester's references. See AR, Tab 15, Raytheon Consensus Evaluation. The protester contends that 28 of these weaknesses, including 9 of the 11 weaknesses identified in the agency's debriefing, were based on the agency's view that offerors were required to submit references that reflected experience in each of the six areas of experience identified in sections L and M of the RFP. The protester argues, however, that the RFP stated that offerors were required to provide references that "[c]ollectively demonstrate experience" in the six areas. RFP amend. 4, sect. L.9.2.1.1.c.1 (emphasis added). The protester argues that although its five references did not each, individually, demonstrate experience in all six areas of experience identified in the RFP, at least one of the references demonstrated experience in each of the six areas. Put differently, the protester contends, the five references "collectively" demonstrated the requisite experience.

In its report on the protest, PTO prepared a chart addressing eight of the nine weaknesses, which were identified in the debriefing, and which the protester argued were assessed based on the agency's improper interpretation. The agency acknowledged that it assessed a weakness for a contract reference if it did not reflect one of the areas of experience--regardless of whether another contract reference demonstrated experience in this area. AR at 24-26.

Where, as here, a dispute exists as to the actual meaning of a particular solicitation provision, our Office will resolve the matter by reading the solicitation as a whole and in a manner that gives effect to all its provisions; to be reasonable, an interpretation of a solicitation must be consistent with such a reading. The Boeing Co., B-311344 et al., June 18, 2008, 2008 CPD para. 114 at 34.

In support of its interpretation, PTO agency notes that section C of the RFP advised offerors of all of the areas of work that would be required under the solicitation, and stated that offerors would be required to demonstrate their ability to perform the work required. AR at 21, citing RFP amend. 4, sections C.1.3, L.9.2.1.1.a. The agency argues that these provisions, when read as a whole, should have informed offerors that each of the five required references was required to demonstrate experience in all six of the areas identified in the RFP. AR at 22. The agency thus argues that the protester's interpretation of the term "collectively" is inconsistent with the requirements of the RFP, as a whole.

We think that the agency's interpretation of the RFP is unreasonable. Nothing in the express language of the solicitation supports the agency's position that each reference had to demonstrate experience in all six areas. Moreover, an interpretation that each reference had to address all six areas would render superfluous the term "collectively," as the agency's interpretation would effectively substitute the word "individually" for the word "collectively." In this regard, the word "collective" is commonly understood to refer to "a number of persons or things considered as one group or whole." See Merriam-Webster Dictionary, available at: http://www.merriamwebster.com.

Moreover, in a decision addressing a similar protest, our Office held that where a solicitation required offerors to demonstrate experience in a number of areas, the agency's evaluation was unreasonable where it concluded that offerors were required to demonstrate all areas of experience within the same reference. See Consolidated Eng'g Servs., Inc., B-311313, June 10, 2008, 2008 CPD para. 146 at 7-8. We concluded that the agency's interpretation of the RFP as requiring an offeror to demonstrate all of the areas of experience in a single reference was not reasonably related to the RFP's general requirement to demonstrate experience in each area. Id. at 8.

Here, the agency cannot demonstrate that giving effect to the plain meaning of the word "collectively" in the section L proposal instructions defeats or undermines the plain meaning of section M or any other part of the solicitation. On this record, we conclude that the agency assessed numerous weaknesses based on an unreasonable interpretation of the RFP, and sustain the protest on this basis.[7]

ADJECTIVAL EVALUATION RATINGS

Finally, Raytheon argues that PTO's evaluation scheme resulted in ratings that were inconsistent with the terms of the solicitation. As discussed below, we conclude that the agency's evaluation scheme was consistent with the RFP's terms.

In reviewing protests objecting to an agency's technical evaluation, our role is limited to ensuring that the evaluation was reasonable and consistent with the terms of the solicitation. CMI Mgmt., Inc., B-402172, B-402172.2, Jan. 26, 2010, 2010 CPDpara. 65 at 2. Agencies' use of evaluation ratings for offerors' proposals, whether numeric, color or adjectival, are but guides to, and not substitutes for, intelligent decision making; they do not mandate automatic selection of a particular proposal. See Jacobs COGEMA, LLC, B-290125.2, B-290125.3, Dec. 18, 2002, 2003 CPD para. 16 at 31.

As discussed above, the RFP stated that the experience and past performance factors were of equal importance, and were more important than the sample task factor. RFP sect. M.3.1.b. The RFP did not specify the ratings scheme that the agency would use to evaluate offerors' proposals.

For the experience, past performance, and sample task factors, the agency used a 5'point rating scale of excellent, good, acceptable, marginal, and unacceptable. AR, Tab 7, Source Selection Plan, attach. 3; Tab 14, Initial TET Report, at 3. The agency then created a composite "initial technical rating" that combined the experience and past performance factors, using an 8-point scale, as follows:

Excellent	The Experience and Past Performance factors have a rating of Excellent
Good+	The Experience factor has a rating of Excellent and the Past Performance factor has a rating of Good OR The Past Performance Factor has a rating of Excellent and the Experience Factor has a rating of Good.
Good	All factors have a rating of Good.
Good-	The Experience factor has a rating of Excellent and the Past Performance factor has a rating of Acceptable OR The Past Performance Factor has a rating of Excellent and the Experience Factor has a rating of Acceptable.
Acceptable+	The Experience factor has a rating of Good and the Past Performance factor has a rating of Acceptable OR The Past Performance Factor has a rating of Good and the Experience Factor has a rating of Acceptable
Acceptable	The Experience and Past Performance factors have a rating of Acceptable
Marginal	The Experience or Past Performance factor has a rating of Marginal
Unacceptable	The Experience or Past Performance factor has a rating of Unacceptable

AR, Tab 14, Initial TET Report, at 4.

The agency combined the initial technical rating (which used the 8-point scale and combined the experience and past performance ratings), with the sample task factor ratings (which used the 5-point scale), to produce the final technical rating. AR, Tab 23, Final TET Report, at 5. The final technical rating also used a 5-point scale of excellent, good, acceptable, marginal, and unacceptable. Id.

Raytheon argues that PTO's use of the 8-point scale for the "initial technical rating," was inconsistent with the terms of the solicitation, which stated that the experience and past performance ratings were equally weighted. In this regard, the protester contends that when an offeror, such as Raytheon, received ratings of good and acceptable under the past performance and experience factors, the 8-point scale skewed the combined rating to a rating of acceptable-plus, which was closer to an acceptable rating than a good rating. The protester contends that this approach violated the RFP's requirement to weigh each factor equally.

We think that the agency's evaluation scheme was consistent with the evaluation weights set forth in the RFP. Although the 8-point scale used the same adjectival terms as the 5-point scale (e.g., excellent, good, and acceptable), it is not the case, as Raytheon contends, that the two scales are directly comparable. In this regard, it is not the case that an acceptable-plus rating on the 8-point scale is more closely comparable to an "acceptable" rating on the 5-point scale, than to a "good" rating on the 5-point scale. Instead, the ratings on the 8-point scale are merely reflections of the agency's attempt to rank all of the eight possible permutations of the ratings.[8] As relevant to the protester, a rating combination of good and acceptable, was ranked lower than a combination of good and excellent, good and good, or excellent and acceptable. In our view, this rating scheme does not run afoul of the RFP's evaluation scheme.

Raytheon also argues that PTO's use of the 5-point scale for the "overall technical rating," which combined the initial technical rating and the sample task rating, was inconsistent with the terms of the solicitation, which stated that the experience and past performance ratings were more important than the sample task rating. The protester argues that the combination of its rating of acceptable-plus on the 8-point scale for the initial technical evaluation, and its rating of good for the sample task, should not have resulted in an overall technical rating of acceptable because both of the individual ratings were higher than the overall technical rating of acceptable.

Again, however, Raytheon mistakenly assumes that the 5-point and 8-point scales are directly comparable. The protester's rating of acceptable-plus on the 8-point scale was the 4th-highest rating, i.e., in the bottom half of the rating scale. In contrast, Raytheon's acceptable rating on the 5-point scale for the final rating was in the precise middle of the 5-point scale. For this reason, we do not think it is accurate to state that Raytheon's proposal was rated lower as a result of the combination of the two ratings. In sum, we think that although the agency's evaluation ratings scheme was somewhat confusing, it did not result in an evaluation that was inconsistent with the terms of the solicitation.

CONCLUSION AND RECOMMENDATION

For the reasons discussed above, we conclude that PTO failed to meet its obligation to conduct meaningful discussions with Raytheon, and that the agency also unreasonably evaluated the protester's experience. We further conclude that the protester was prejudiced because a reevaluation of the protester's experience and an opportunity for meaningful discussions could permit the protester to improve its score to at least the level at which other offerors were awarded contracts, i.e., the overall score of good. We further note that one other offeror, Amentra, received a contract award with an overall score of good and a higher proposed price than Raytheon.

We recommend that PTO reevaluate the proposals consistent with the RFP and this decision, conduct meaningful discussions with the competitive range offerors, obtain revised proposals, and make a new award decision. We also recommend that Raytheon be reimbursed the costs of filing and pursuing this protest, including reasonable attorney fees. Bid Protest Regulations, 4 C.F.R. sect. 21.8(d)(1) (2011). Raytheon should submit its certified claim for costs, detailing the time expended and cost incurred, directly to the contracting agency within 60 days after receipt of this decision. 4 C.F.R. sect. 21.8(f)(1).

The protest is sustained.

Lynn H. Gibson
General Counsel