SDS International, Inc., B-291183; B-291183.2, December 2, 2002
B-291183,B-291183.2: Dec 2, 2002
Was unreasonable where it was based in material part on an unsupported reported strength in the awardee's proposal. Was for aircrew training and courseware development at the 57th Wing United States Air Force Weapons School (USAFWS). Past performance and mission capability were of equal importance in the evaluation scheme. " and "will be evaluated by examining four (4) references determined by the Government to be the most recent and relevant in relation to this requirement. " although "more or less may be evaluated as appropriate." /1/ The RFP further explained that for this purpose "[r]ecency . . . is defined as performance occurring in the last five (5) years". That "[r]elevancy . . . is defined as courseware development and aircrew training related to Factor 2.
SDS International, Inc., B-291183; B-291183.2, December 2, 2002
SDS International, Inc. protests the award of a contract to CBD Training, Inc. under request for proposals (RFP) No. F44650-02-R-0009, issued by the Department of the Air Force, Air Combat Command Contracting Squadron, Langley Air Force Base Virginia, for aircrew training and courseware development.
We sustain the protests.
The RFP, issued April 3, 2002 as a small business set-aside, was for aircrew training and courseware development at the 57th Wing United States Air Force Weapons School (USAFWS), Nellis Air Force Base, Nevada, under a fixed-price contract for a 30-day phase-in period and a base year with 4 option years. In addition to the training and courseware development requirements, the statement of work (SOW) required the contractor to maintain the USAFWS's academic database and make it accessible from the Internet. The SOW explained that "[t]he database shall provide an [I]nternet[-]based Learning Management System (LMS)/Courseware Management System [(CMS)] that identifies and tracks student deficiencies, provides graduate critiques, and service as a 'one stop shopping' site for all instructor and student academic/courseware needs." The SOW further stated that "all unclassified material shall be accessible from the weapons school unclassified LAN [local area network], while all classified and unclassified material shall be located on a classified server that can be accessed via a SIPRNET (Secret Internet Protocol Router Network) web page with password controls," and required the task be accomplished "with no additional workload" on the USAFWS LAN and staff. See RFP, SOW Paras. 1.7.7, 188.8.131.52, 184.108.40.206.
The RFP provided for award on a "best value" basis considering as evaluation factors past performance, mission capability, risk, and price. Past performance and mission capability were of equal importance in the evaluation scheme, followed by risk, with price being of least importance. RFP at 45. Under the mission capability factor, the RFP listed three subfactors in descending order of importance: (1) training and workload management, (2) courseware development (CWD) approach and instructional systems development (ISD) management plan, and (3) phase-in. RFP at 48.
The RFP stated "[p]ast performance shall be evaluated as a measure of the Government's confidence in the offeror's ability to successfully perform based on previous and current contract efforts," and "will be evaluated by examining four (4) references determined by the Government to be the most recent and relevant in relation to this requirement," although "more or less may be evaluated as appropriate." /1/ The RFP further explained that for this purpose "[r]ecency . . . is defined as performance occurring in the last five (5) years"; that "[r]elevancy . . . is defined as courseware development and aircrew training related to Factor 2, Mission Capability, experiences, especially those provided to the military at multiple locations"; and that "[e]xperience with training contracts, which utilize the Air Force's model of [ISD] . . .will also be viewed favorably." Completed Contractor Performance Assessment Reports (CPAR) were said to be the primary source for evaluating past performance. RFP at 40, 46.
The Air Force received four proposals, including SDS's and CBD's, by the May 23 closing date. A source selection evaluation team (SSET) rated proposals utilizing an adjectival scale for rating past performance, a color-coded scale for rating mission capability, and a graduated risk scale for rating risk. /2/ Following the initial evaluation, all proposals were included in the competitive range.
The Air Force conducted discussions with the offerors. In the discussions with SDS and CBD the agency expressed concern that their proposals did not adequately address how they would provide Web access capability in accordance with SOW Para. 1.7.7. Both offerors responded to these concerns. SDS provided further details about how it would furnish this capability. CBD, which did not address this requirement in its initial proposal, responded with details about how this capability would be provided, and by proposing a "[DELETED]."
Final proposal revisions were submitted by May 30. SDS and CBD submitted the most highly rated final proposal revisions as follows:
. Past Performance Price
. Mission Capability/Risk . Subfactor Subfactor Subfactor . No. 1 No. 2 No. 3 SDS Very Good/ Blue Green Blue $15,964,852 . Significant . Confidence Low Low Low
CBD Exceptional/High Blue Green Green $17,916,218 . Confidence Low Low Low
Agency Report, Tab 14, Source Selection Decision, at 2.
The source selection authority (SSA), after discussions with "the SSET Chair, contracting officer, and various Source Selection Advisors," decided to increase CBD's rating to blue for the second subfactor of the mission capability factor, determining that "[CBD's] proposal [in this area] exceed[ed] the stated requirements in a beneficial way." Id. at 2-3.
In his source selection decision, the SSA concluded that the technical strengths associated with CBD's more highly-rated proposal were worth the higher price. For example, the SSA noted that CBD received an exceptional rating for past performance because it had four relevant contracts with CPARs, in which the majority of the performance ratings were "exceptional" and included a statement that the rater "definitely would award" CBD another contract. On the other hand, the SSA noted that SDS, as the incumbent contractor, received very good ratings based on two CPARs it received for the current contract that reflected a majority of very good ratings and a statement that the rater "probably would award" SDS another contract. Id. at 3-4. Although the record is not entirely clear on this point, it appears that the other seven contracts that SDS had offered as evidence of its experience were determined either not relevant or recent enough, and were not considered in the very good rating.
The source selection decision also referenced as "extremely beneficial" CBD's proposal of [DELETED] and a [DELETED], both of which were identified strengths under the first subfactor of the mission capability factor. The decision acknowledged that SDS's proposal also has strengths under this subfactor (as indicated by its blue rating), which ensure that the "school is well managed," but mitigated these strengths by observing that SDS's approach and staffing "does little more than otherwise maintain the status quo in terms of overall training approach." Id. at 4, 6.
Finally, the source selection decision references CBD's "visionary approach of using the [DELETED]" as a particular strength to justify the selection of the higher-priced proposal. Id. at 6.
The award was made to CBD on August 13. This protest followed. SDS contends that the source selection decision was irrational because the SSA unjustifiably increased CBD's rating under the second subfactor of the mission capability factor in justifying the award to the significantly higher-priced offeror, even though CBD's proposal assertedly did not provide any additional benefit with regard to the provision of Internet and Web services as compared to CBD's proposal.
Where an agency chooses between a higher-priced, higher-rated proposal, and a lower-priced, lower-rated one, we will review the agency's source selection decision to determine whether it is consistent with the stated evaluation criteria, and whether the agency's price/technical tradeoff is reasonable. A source selection decision based on inconsistent or inaccurate information concerning the technical evaluation or the relative merits and contents of the offerors' technical proposals is not reasonable. New Breed Leasing Corp., B-259328, Mar. 24, 1995, 96-2 CPD Para. 84 at 4.
As indicated above, before making the source selection decision, the SSET chair and others briefed the SSA. The SSET rated CBD's proposal green under the second subfactor of the mission capability factor. The briefing charts identified three strengths in CBD's proposal under this subfactor: /3/
Agency Report, Tab 13, SSA Briefing Slides, at 24. In justifying raising CBD's rating from green to blue under this subfactor, the SSA recognized and discussed all three of these strengths. Based on our review, it appears that the strengths relating to CBD's improved [DELETED] were reasonably based. /4/ With regard to the third strength of an [DELETED], the SSA found:
CBD . . . also proposed an [DELETED] to determine the appropriate USAFWS [LMS/CMS]. The last few years we have seen technology advance at an incredible pace, and too few of those benefits have been applied to government processes. The approach to provide an [DELETED] that would use Weapons School Instructor inputs to determine a custom approach is viewed to be of significant value. . . . The personnel proposed were deemed appropriate and highly desirable for the level of effort required. This approach was viewed positively and definitely exceeded the minimum performance requirements in a way beneficial to the Air Force.
Id. at 4-5.
As detailed below, the SSA's discussion of the [DELETED] strength does not comport to the contents of CBD's proposal. This is important here because of the three strengths discussed by the SSA to justify raising CBD's rating to blue under this subfactor, the [DELETED] strength was the only one that was "viewed to be of significant value." Thus, CBD's blue rating for this subfactor and the source selection decision identifying this as a "visionary approach" in justifying the award are not, on this record, reasonably based.
The change in this rating was based on CBD's response to the agency's discussion concern about CBD's proposal's failure to discuss the migration of courseware and curriculum to a Web environment as required by SOW Para. 1.7.7. In its response, CBD stated in pertinent part:
We had assumed that all [Weapons School] Database requirements in the current contract had been fully met and that the current [Weapons School] LAN was web-accessible. Discussions revealed that our assumption was incorrect. To correct this shortcoming, following contract award we will institute an aggressive approach to fully meet the contract requirements as shown below. In addition to providing the hardware and software outlined in our proposal, we [DELETED]. Lacking specific knowledge of the current state of the [Weapons School] Academic Database, we anticipate that the web-based system will require considerable effort to establish and to provide ongoing maintenance/update support to meet all USAFWS requirements without burdening USAF personnel.
. . . . .
CBD will maintain the USAFWS Academic Database to provide unclassified material over the [Weapons School] LAN and both classified and unclassified material on a database that can be accessed over the Internet. A [LMS/CMS] provides student tracking, graduate critiques, and a site for all instructor and student academic/courseware needs will be provided by modifying the existing [Microsoft] Access-based database. We will also provide web interface via a SIPRNET web page using Active Server Pages and file download capabilities that are SCORM compliant. We will use [DELETED] approved by the USAFWS LAN Manager. The system will also record pertinent tracking information, such as web log-on, user actions, and log-off data, while fully meeting government security standards. . . . At contract start, CBD will acquire the services of a [DELETED] and provide the Web access.
Agency Report, Tab 10, CBD Discussions, at 1-2.
Contrary to the source selection decision, nowhere in CBD's proposal does it offer to provide an [DELETED] to develop a custom approach to address technological weaknesses in the current database to meet the current and future requirements, nor does CBD agree that it would seek input from USAFWS instructors for this purpose. In our view, a fairer reading of CBD's proposal to engage an [DELETED] is that CBD recognized that it had no knowledge of the USAFWS's current academic database and LAN, and that [DELETED] was only to assist the [DELETED] CBD to "meet the contract requirements." /5/ We find no support for the SSA's conclusion that CBD offered a "custom approach to address technological weaknesses in the current database." /6/ In fact, CBD's approach (quoted above) for the most part either acknowledges, restates, or generically responds to the SOW requirements with no discussion or promise of a customized approach to satisfying these requirements, except for some reference to employing [DELETED] software ([DELETED]). /7/ See Tr. at 80-88.
At the hearing at which testimony from the SSA was elicited concerning this issue, the SSA stated that he did not read the proposals before making his source selection decision, but reviewed and relied upon information provided by the SSET. /8/ Tr. at 31-32, 57, 64, 98. The SSA testified as to his understanding of CBD's approach to satisfying the SOW requirements concerning migration of courseware and curriculum to a Web environment. He testified that he understood that CBD's proposal was based on "leveraging [the] newest technologies" in order to make the database Web accessible, and "to look at whatever the most current Internet techniques or technologies are, as opposed to trying to modify an existing database to meet that need." Tr. at 16-17, 20, 25, 95, 97, 99, 175-76. However, he could not identify where in CBD's proposal these alleged benefits are discussed, did not explain how these benefits would necessarily accrue from the actual contents of CBD's proposals (particularly given CBD's admission that it was not familiar with the existing database), and was unable to detail the differences between CBD's and SDS's approaches, other than stating his perception that CBD was proposing to take a "fresh look" at the agency's database requirements. See, e.g., Tr. at 86-88, 91-98.
Based on the foregoing, we find that the source selection decision contained statements and information that were not supported by the record yet were material to the decision.
The Air Force argues that even accepting that the blue rating for CBD's proposal was unjustified, the award should be upheld because of the other strengths identified by the SSA in making the cost/technical tradeoff decision. The Air Force argues that these documented strengths demonstrate that SDS was not prejudiced by any misevaluation under the second subfactor. Agency Post-Hearing Comments at 7-10. We are unpersuaded by this argument in the circumstances of this case.
While we consider the entire record, including the parties' later explanations and arguments, we accord greater weight to contemporaneous evaluation and source selection material than to arguments and documentation prepared in response to protest contentions. Boeing Sikorsky Aircraft Support, B-277263.2, B-277263.3, Sept. 29, 1997, 97-2 CPD Para. 91 at 15.
Here, the record does not evidence that the SSA's misstatement concerning the benefits attributable to CBD's proposal was immaterial and would not have affected the source selection decision. As noted, the decision erroneously described and highlighted the perceived contribution and benefits of CBD's proposed [DELETED], and, based on this description, characterized this perceived benefit to be of "significant value" and a "visionary approach" in the cost/technical tradeoff. Thus, a key part of the source selection decision justifying the award to CBD, notwithstanding SDS's significantly lower price, was based on information not supported by the record. /9/ Under these circumstances, we find no basis to find the cost/technical tradeoff reasonable, and sustain SDS's protest on this ground. See OneSource Energy Servs., Inc., B-283445, Nov. 19, 1999, 2000 CPD Para. 109 at 10 (source selection decision based upon erroneous portrayal of protester's past performance was unreasonable); New Breed Leasing Corp., supra (source selection decision was unreasonable where it was based upon uncorrected errors in protester's ratings and an inaccurate portrayal of the protester's past performance, and it included conclusions about the proposals' technical differences that were inconsistent with the evaluation record and proposals).
SDS also challenges the past performance evaluation. Based on our review of the record, including the hearing testimony, we find no basis to find that the Air Force's ratings of CBD's past performance as exceptional and SDS's as very good were unreasonable. For example, SDS's claim that its performance should have been considered exceptional on the incumbent contract, apparently the sole contract considered in its past performance evaluation, has no merit, considering that the CPARs for this contract contain a preponderance of very good ratings for SDS.
SDS complains that its other contracts submitted in its past performance proposal should also have been considered and that if they had been SDS would have received an exceptional rating. Where a solicitation contemplates the evaluation of an offeror's past performance, the agency has the discretion to determine the scope
of the offeror's performance history to be considered, provided all proposals are evaluated on the same basis and the evaluation is consistent with the terms of the RFP. Systems Mgmt., Inc.; Qualimetrics, Inc., B-287032.5; B-287032.6, Nov. 19, 2001, 2002 CPD Para. 29 at 4-5. To the extent that these contracts were not considered, it was because the agency for the most part reasonably determined that they did not involve aircrew training or because SDS had been performing under these contracts for only a short period of time. Nevertheless, even if these contracts (most of which did not have CPARs) had been considered, we find no basis to believe that this would have elevated SDS's past performance rating to an exceptional level, given its very good rating based on the CPARs on the most relevant incumbent contract. /10/
We recommend that the Air Force reevaluate the proposals, reopen discussions and obtain revised final proposals if appropriate, and make a new source selection. If CBD is not the successful offeror, its contract should be terminated and a new award made. We also recommend that the agency reimburse SDS the reasonable costs of filing and pursuing the protest, including attorney's fees. 4 C.F.R. Sec. 21.8(d)(1) (2002).
SDS's certified claim for costs, detailing the time spent and the costs incurred, must be submitted to the agency within 60 days of receiving this decision. 4 C.F.R. Sec. 21.8(f)(1).
The protests are sustained.
Anthony H. Gamboa General Counsel
1. Contrary to the protester's argument, in our view, this provision does not require the agency to consider four contract references where it does not consider them relevant.
2. The possible past performance ratings were exceptional/high confidence, very good/significant confidence, satisfactory/confidence, neutral/unknown confidence, marginal/little confidence, and unsatisfactory/no confidence. The possible color ratings were blue/exceptional, green/acceptable, yellow/marginal, and red/unacceptable. The possible risk ratings were high, moderate, and low. RFP at 47, 49.
3. One strength was identified for SDS's proposal for this subfactor.
4. In contrast, SDS did not propose a [DELETED], but proposed that its [DELETED] devote far less than half of his time to LMS/CMS management, in addition to his primary function of updating the database. Agency Report, Tab 10, SDS's Response to Discussions, at 5; Tab 18, SDS's Proposal, at 35-37.
5. CBD only proposed to engage an [DELETED]. There may have been no need for SDS to propose [DELETED] for this purpose because the record reflects that SDS "built and maintains the USAFWS academic database using the SDS[-]established Academix(tm) database" and because its expansion to the Internet would be based on this "robust database," as it was doing under another contract. See Agency Report, Tab 10, SDS's Discussions, at 5; Tab 18, SDS's Proposal, at 36; Hearing Transcript (Tr.) at 90-96.
6. Neither the identity nor qualifications of the proposed [DELETED] were identified. Tr. at 60, 81-82.
7. Ironically, the benefits attributed to, but not stated in, CBD's proposal concerning its proposed [DELETED] were actually discussed in SDS's proposal, as indicated by the following SDS response to the agency's discussions:
SDS has a solid background in ADL [Advance Distributed Learning] design and SCORM standards. Under a contract with the US Army FORSCOM, we are currently migrating ISD-based courseware toward interactive learning modules using Academix as the base engine. We will draw on this expertise and experience should [Weapons School] courseware needs change in the future. In our approach to the [Weapons School], we will work with the [Weapons School] to identify courseware for interactive Web applications. When identified, we will determine the level of interactivity expected and the requirements to configure the content to be SCORM compliant. Once the interactive learning modules are completed, the courseware can be made available on the appropriate server, and maintained concurrently with the [Weapons School] formal curricula.
Agency Report, Tab 10, SDS's Discussions, at 5. Moreover, from our review, it appears that SDS provided more specific details in its initial proposal and discussion responses than did CBD about how it intends to comply with the SOW Para. 1.7.7 requirements. While CBD suggests that SDS's failure to offer an [DELETED] showed that firm's insensitivity to these requirements, there is no evidence (and the agency does not argue) that converting the Academix database for Web applications would require [DELETED], given CBD's experience with this database and migrating it for Web applications.
8. Other than noting that CBD's proposal of an[DELETED], Agency Report, Tab 13, SSA Briefing, at 24 and that the "[DELETED]," Agency Report, Tab 12, Final Proposal Analysis Report, at 28, the SSET's evaluation documentation does not elaborate on what this [DELETED] would contribute to CBD's proposal. The agency did not call any SSET witness to testify on this matter.
9. In its Post-Hearing Comments (at 4), the Air Force essentially admits that this was a material part of the source selection decision, since the SSA could rely upon this perceived strength to justify the award to CBD, notwithstanding the associated price premium.
10. SDS raised a number of other protest issues, not specifically discussed here, which we have carefully reviewed and find lack merit; for example, that discussions should have been conducted with SDS regarding its past performance, that the agency should not have considered the past performance of CBD's large business subcontractor in the evaluation, that CBD may violate the subcontracting limitation, and that there was not an adequate price evaluation of CBD's proposal.
* DOCUMENT FOR PUBLIC RELEASE
The decision issued on the date below was subject to a GAO Protective Order. This redacted version has been approved for public release.