Systems Plus, Inc.; CANN Softtech, Inc.; Dfuse Technologies, Inc.; Red Oak Solutions, LLC; White Oak Solutions, LLC; ShorePoint, Inc; JSSA, Inc.; Knowledge Management, Inc.; 2050 Technology, LLC; JarWare, LLC; iDoxSolutions, Inc.; cFocus Software, Inc.; SOFITC JV, LLC; Spatial Front, Inc.; ImpactOne JV, LLC; Technology Solutions Provider, Inc.; A1FedImpact, LLC; Saliense Consulting, LLC; Xfinion, Inc.; Hendall, Inc.; Syneren Technologies Corp.; iVision, Inc., d/b/a iVision Consulting, Inc.; CWS FMTI JV, LLC; Astor & Sanders Corporation; Computer World Services Corporation; DevTech Systems, Inc.; Criterion Systems, LLC; Cyquent, Inc.; Audacious Inquiry; ICS-TSPi, LLC; SRG-TSPi, LLC; Horizon Industries, Ltd.; MASAI Technologies Corporation; CTIS, Inc.; JCS Solutions, LLC; TSC-ITG JV, LLC; Karsun Solutions, LLC; Neev-KS Technologies, LLC; ASSYST, Inc.; Platinum Business Services, LLC; IS CIO JV; Inserso Corporation; Credence Dynamo Solutions, LLC; Sky Solutions, LLC; Blue Grove Solutions, LLC; Ennoble First-Macro Solutions, LLC; OCT Consulting, LLC; Swain Online, Inc., d/b/a Swain Techs; Katmai Management Services, LLC; Capital Data Partners JV, LLC; Network Management Resources, Inc., d/b/a NMR Consulting; mPower, Inc.; ADG Tech Consulting, LLC; USmax Corporation; Rip Ripple Effect Communications, Inc., d/b/a Ripple Effect; MicroTechnologies, LLC; A Square Group, LLC; eKuber Ventures, Inc.; The Electric On-Ramp, Inc.; MiamiTSPi, LLC; Decision Point Corporation; AgilisTEK, LLC; OM Partners JV 2, LLC; A-Tek, Inc., Revised on July 13, 2023

DOCUMENT FOR PUBLIC RELEASE
This corrected public version of the decision issued on the date below adds the protester “Inserso Corporation” to page 52, footnote 34. This redacted version has been approved for public release.

Decision

Matter of: Systems Plus, Inc.; CANN Softtech, Inc.; Dfuse Technologies, Inc.; Red Oak Solutions, LLC; White Oak Solutions, LLC; ShorePoint, Inc; JSSA, Inc.; Knowledge Management, Inc.; 2050 Technology, LLC; JarWare, LLC; iDoxSolutions, Inc.; cFocus Software, Inc.; SOFITC JV, LLC; Spatial Front, Inc.; ImpactOne JV, LLC; Technology Solutions Provider, Inc.; A1FedImpact, LLC; Saliense Consulting, LLC; Xfinion, Inc.; Hendall, Inc.; Syneren Technologies Corp.; iVision, Inc., d/b/a iVision Consulting, Inc.; CWS FMTI JV, LLC; Astor & Sanders Corporation; Computer World Services Corporation; DevTech Systems, Inc.; Criterion Systems, LLC; Cyquent, Inc.; Audacious Inquiry; ICS-TSPi, LLC; SRG-TSPi, LLC; Horizon Industries, Ltd.; MASAI Technologies Corporation; CTIS, Inc.; JCS Solutions, LLC; TSC-ITG JV, LLC; Karsun Solutions, LLC; Neev-KS Technologies, LLC; ASSYST, Inc.; Platinum Business Services, LLC; IS CIO JV; Inserso Corporation; Credence Dynamo Solutions, LLC; Sky Solutions, LLC; Blue Grove Solutions, LLC; Ennoble First-Macro Solutions, LLC; OCT Consulting, LLC; Swain Online, Inc., d/b/a Swain Techs; Katmai Management Services, LLC; Capital Data Partners JV, LLC; Network Management Resources, Inc., d/b/a NMR Consulting; mPower, Inc.; ADG Tech Consulting, LLC; USmax Corporation; Rip Ripple Effect Communications, Inc., d/b/a Ripple Effect; MicroTechnologies, LLC; A Square Group, LLC; eKuber Ventures, Inc.; The Electric On‑Ramp, Inc.; MiamiTSPi, LLC; Decision Point Corporation; AgilisTEK, LLC; OM Partners JV 2, LLC; A-Tek, Inc., Revised on July 13, 2023

File: B-419956.184; B-419956.185; B-419956.186; B-419956.187; B-419956.188; B‑419956.189; B-419956.190; B-419956.191; B-419956.192; B-419956.193; B‑419956.194; B-419956.195; B-419956.196; B-419956.197; B-419956.198; B‑419956.199; B-419956.201; B-419956.202; B-419956.203; B-419956.205; B‑419956.207; B-419956.208; B-419956.209; B-419956.210; B-419956.213; B‑419956.214; B-419956.215; B-419956.216; B-419956.219; B-419956.220; B‑419956.227; B-419956.230; B-419956.231; B-419956.233; B‑419956.234; B‑419956.241; B-419956.242; B-419956.248; B-419956.252; B‑419956.259; B‑419956.260; B-419956.261; B-419956.264; B-419956.268; B‑419956.269; B‑419956.270; B-419956.271; B-419956.272; B-419956.273; B‑419956.274; B‑419956.277; B-419956.278; B-419956.279; B-419956.280; B‑419956.281; B‑419956.282; B-419956.283; B-419956.284; B-419956.286; B‑419956.289; B‑419956.290; B-419956.292; B-419956.293; B-419956.294; B‑419956.295; B‑419956.296; B-419956.298; B-419956.306; B-419956.309; B‑419956.310; B‑419956.311; B-419956.313; B-419956.314; B-419956.315; B‑419956.316; B‑419956.317; B-419956.319; B-419956.320; B-419956.321; B‑419956.322; B‑419956.323; B-419956.324; B-419956.325; B-419956.326; B‑419956.327; B‑419956.328; B-419956.329; B-419956.330; B-419956.333; B‑419956.334; B‑419956.336; B-419956.337; B-419956.340; B-419956.341; B‑419956.342; B‑419956.345; B-419956.346; B-419956.347

Date: June 29, 2023

DIGEST

1. Protests challenging the agency’s exclusion of proposals from phase 1 of the competition are sustained where neither the record provided by the agency nor the agency’s responses to the protests show that the evaluations and exclusion decisions were reasonable. Additionally, the agency’s initial explanations of the record were incomplete and misleading, as shown by the significant revisions made to the agency’s responses to some--but not all--of the protesters’ allegations.

2. Protests challenging the agency’s exclusion of proposals from phase 1 of a multi-phase procurement based on what the protesters contend are undisclosed and unduly restrictive evaluation criteria are dismissed as untimely where the terms of the solicitation clearly disclosed the basis on which offerors would be evaluated.

3. Protesters whose proposals were not advanced past phase 1 of the competition are not interested parties to argue that the agency failed to follow the solicitation’s award criteria, as the solicitation made clear that only proposals that advanced past phase 1 of the competition were eligible for award.

4. Protest challenging the agency’s evaluation of a protester’s proposed self-score is sustained where the agency did not meaningfully respond to the protester’s arguments.

Sixty-four firms[1] protest the exclusion of their proposals from the competition conducted by the Department of Health and Human Services (HHS), National Institutes of Health (NIH), under request for proposals (RFP) No. 75N98121R00001, which was issued for the award of multiple indefinite-delivery, indefinite-quantity (IDIQ) government-wide acquisition contracts for information technology services, known as Chief Information Officer-Solutions and Partners (CIO-SP4). The protesters raise various arguments alleging that the agency improperly failed to advance their proposals from phase 1 to phase 2 of the competition.

We sustain the protests.[2]

BACKGROUND

NIH issued the initial solicitation on May 25, 2021, seeking proposals to provide information technology (IT) solutions and services in the areas of health, biomedical, scientific, administrative, operational, managerial, and information systems requirements. Agency Report (AR)[3], Tab Q.4, RFP amend. 16 at 7.[4] The purpose of the CIO-SP4 contracts is to “provide government agencies a mechanism for quick ordering of IT solutions and services at fair and reasonable prices, to give qualified small businesses a greater opportunity to participate in these requirements, and give government agencies a mechanism to help meet their socio-economic contracting goals.” Id.

The RFP anticipated the award of multiple contracts, each of which will have a base period of 5 years and one 5-year option. Id. at 38. Each awarded contract will have a maximum ordering value of $50 billion. Id. at 50. The solicitation advised that the agency will award approximately 305 to 510 IDIQ contracts across multiple socio-economic categories, including the following: (1) other than small business (OTSB); (2) emerging large business (ELB);[5] (3) small business (SB); (4) woman-owned small business (WOSB); (5) veteran-owned small business (VOSB); (6) service‑disabled veteran-owned small business (SDVOSB); (7) historically underutilized business zone (HUBZone); (8) Small Business Administration 8a business (8a); (9) Indian economic enterprise (IEE); and (10) Indian small business economic enterprise (ISBEE). Id. at 143. The RFP informed offerors that their proposals would only compete with proposals in the same socio-economic category. Id. at 145-146. For example, an offeror proposing as a small business would compete only against other small business proposals. Offerors were permitted to submit proposals for consideration under more than one socio-economic category. Id. at 147.

The RFP provided that for each socio-economic category, the government estimates making a certain number of awards, but also explains that “[t]he government may deviate from these numbers.” Id. at 143. The solicitation estimated making awards as follows:

Id.

The RFP established a 3-phase evaluation of proposals. Id. at 173. As discussed in more detail below, the phase 1 competition required offerors to submit a self-scoring sheet that assigned points based on offerors’ representations concerning experience and other capabilities pursuant to certain criteria identified in the solicitation. Id. at 157‑158. Offerors were required to submit documentation to support the self-score points claimed. Id. at 152. The solicitation informed offerors that under the phase 1 evaluation, NIH would “validate the offerors’ completed self-scoring,” and “[o]nly the highest rated offerors will advance to phase 2 of the evaluation.” Id. at 174.

The self-scoring criteria contained in solicitation sections L.5.2.1 through L.5.2.4 for the phase 1 competition provided that offerors could claim points based on experience in the following areas: corporate experience; leading edge technology; federal multiple‑award contracts; and Executive Order (EO) 13779, which concerns experience with projects directly supporting historically black colleges and universities. Id. at 157. Offerors had to submit experience examples, which could be a contract, an order, or a collection of orders performed by that offeror, in each of these areas to claim the points. Id. The self-score value that could be claimed for each experience example submitted depended on the dollar value of the example, with larger dollar values generally meriting more points. Id. at 159-165.

The RFP permitted offerors to form contractor team arrangements (CTAs), as defined by Federal Acquisition Regulation (FAR) subpart 9.6, to submit proposals, including as mentor‑protégé joint ventures (MPJVs) approved by the Small Business Administration (SBA).[6] Id. at 147-148. For each of the self‑scoring experience areas, offerors were permitted to use the experience of each team member, subject to various limitations by the RFP. Id. at 159-164. For example, mentors in a MPJV could submit only two experience examples per task area identified in the RFP under the corporate experience criterion, L.5.2.1. Id. at 158.

The RFP required experience examples to be from the last 3 years prior to the date the solicitation was originally released, May 25, 2021. Id. at 159, 161, 163-164. The RFP stated that the dollar value of each experience example “is determined by the total dollars that were obligated (funded)” and this included “exercised options.” Id. at 158‑159. For an experience example that was a collection of orders placed under an IDIQ contract or blanket purchase agreement (BPA), the dollar value was the sum of all orders based on the application of the obligated dollar amount for each order. Id. at 158.

The self-scoring criteria contained in the solicitation at sections L.5.2.5 through L.5.2.12 provided that offerors could claim points based on their possession of certain certifications, systems, and clearances. Id. at 165-168. For example, an offeror could claim 300 points if they possessed a level 2 capability maturity model integration (CMMI) appraisal or higher, or 200 points if they had an approved purchasing system. Id. For these requirements, offerors were permitted to use the qualification of any team member, so long as the offeror identified which member possessed the qualification, and provided “how that member / affiliate would use it in the normal course of business for the offeror.” Id.

The initial due date for proposals was August 27, 2021. AR, Tab L.1, RFP amend. 11, Cover Letter at 1. In November 2021, we issued a decision in Computer World Services Corp.; CWS FMTI JV LLC, B‑419956.18 et al., Nov. 23, 2021, 2021 CPD ¶ 368, which sustained a challenge to the terms of the solicitation with regard to the consideration of proposals submitted by MPJVs. In response to our Office’s recommendation to amend the solicitation, the agency issued RFP amendments 12 through 16, which revised the terms for MPJV offerors to submit experience examples. The agency set an amended deadline for proposals of February 11, 2022, that applied to MPJV offerors that were affected by the revisions in amendments 12 through 16. AR, Tab P.1, RFP amend. 15, Cover Letter at 1.

NIH received proposals from 1,150 offerors, many of which competed for awards under more than one socio-economic category. AR, Tab BB, SSA Master Tracking Sheet; see AR, Tab X, Final Cutline Methodology Memorandum at 4, 8, 11, 14, 18, 21, 24, 26. The agency then established a “cutline” or “cut-off score” for each socio-economic category; offerors above the self-score cutlines were to advance to phase 2 of the competition, while offerors below the cutlines were to be eliminated from the competition. E.g., Contracting Officer’s Statement (COS) (B-419956.268) at 6.

Exclusion Notices and Current Protests

In September 2022, NIH notified offerors “who had submitted proposals that were self-scored below the cut-off” and would not be advancing to phase 2 of the competition. E.g., COS (B-419956.185 et al.) at 4. In response to approximately 120 protests filed with our Office from offerors challenging their exclusion from the competition, the agency advised that it would take the following corrective action: “1. Reassess the self-scoring cut-off line; and 2. Make a new determination on the highest rated offerors that proceed to Phase 2 of the procurement.” iDoxSolutions, Inc. et al., B‑419956.40 et al., Nov. 29, 2022 (unpublished decision) at 3. The agency also stated it reserved the right to “correct any additional errors or deficiencies, if any, in the procurement process/record that are discovered during the agency’s implementation of the [] corrective action plan.” Id. Our Office accordingly dismissed these protests as academic on November 29. Id. at 1.

After implementing the November 2022 round of corrective action, NIH again provided notification to offerors that would not be advancing to phase 2 of the competition in February 2023. E.g., COS (B-419956.185 et al.) at 5. Our Office again received multiple protests from unsuccessful offerors, and the agency again advised that it would take corrective action, as follows: “1. Reassess the source selection methodology; and 2. Make a new determination on the highest rated offerors that proceed to Phase 2 of the procurement.” Saliense Consulting, LLC, B‑419956.179, Mar. 14, 2023 (unpublished decision) at 1. In response to questions from our Office concerning how this corrective action would be meaningfully different from the prior rounds, the agency explained that the anticipated corrective action “aim[ed] to address inconsistencies that arose during implementation of the prior corrective action.” Id. at 1‑2. NIH further explained that it “was imperative to address these inconsistencies and reassess its source selection methodology documentation to ensure that all offerors are treated fairly in the evaluation and that the cutlines are properly supported and reasonable.” Id. at 2. Our Office accordingly dismissed these protests as well on March 14, 2023. Id. at 1.

After implementing the March 2023 round of corrective action, NIH provided notices to offerors whose proposals had been excluded from the competition. The agency’s pre‑award debriefings, for those offerors that requested a debriefing, included the offeror’s phase 1 self‑score, the agency’s phase 1 “validated” score, and the phase 1 “cutline” scores for the socio-economic categories under which the offeror submitted a proposal. E.g., AR, Tab DD.83.j, Credence Dynamo Solutions Pre‑Award Debriefing at 1. The instant protests were subsequently filed with our Office, starting with protests filed on March 21, 2023.[7]

DISCUSSION

This decision addresses 98 protests and supplemental protests filed by 64 offerors that challenge their exclusions by NIH from the competition based on the phase 1 evaluations. We address the protesters’ arguments in three parts. In part I of this decision, we address arguments that the agency failed to validate offerors’ proposed self‑scores, and did not establish reasonable self-score point cutlines to determine which proposals would advance past the phase 1 evaluation. In part II of this decision, we address arguments that the agency improperly departed from evaluation and award criteria by converting the phase 1 evaluation to a “down-selection” that was the final award decision. In part III of this decision, we address arguments raised by protesters challenging adjustments to their proposed self-scores.

For the reasons discussed below, we sustain the protests based on challenges addressed in part I of the decision, which concern the agency’s validation of self-scores and the use of those self-scores in determining which proposals would advance past phase 1 of the competition. We also sustain the protests based on challenges addressed in part III of the decision, with regard to arguments raised by one of the protesters who challenges the agency’s validation of its self-score.

Due to the large number of protesters raising similar arguments, we do not specifically identify which protester raised a particular argument, unless it is an argument unique to one or a small number of protesters. Additionally, while many of the protesters raised similar arguments, the manner in which they raised these arguments varied. Although we do not address every argument or variation of the arguments raised by the protesters, we have reviewed all of them and find that none provides a basis to sustain the protests, with the exception of those specifically identified.[8]

When a dispute exists as to a solicitation’s requirements, we begin by examining the plain language of the solicitation and read the solicitation as a whole and in a manner that gives effect to all provisions; to be reasonable, and therefore valid, an interpretation must be consistent with such a reading. Beechcraft Def. Co., LLC, B‑406170.2 et al., June 13, 2013, 2013 CPD ¶ 147 at 30. Our Office will find unreasonable an interpretation that requires reading certain provisions out of the solicitation. See C&S Corp., B-411725, Oct. 7, 2015, 2015 CPD ¶ 311 at 6-7. Similarly, an interpretation is not reasonable if it fails to give meaning to all of a solicitation’s provisions, renders any part of the solicitation absurd or surplus, or creates conflicts. Innovative Mgmt. Concepts, Inc., B-419834.2, B-419834.3, Sept. 20, 2021, 2021 CPD ¶ 319 at 15.

The evaluation of an offeror’s proposal is a matter within the agency’s discretion. National Gov’t Servs., Inc., B-401063.2 et al., Jan. 30, 2012, 2012 CPD ¶ 59 at 5. In reviewing protests challenging an agency’s evaluation of proposals, our Office does not reevaluate proposals or substitute our judgment for that of the agency, but rather examines the record to determine whether the agency’s judgment was reasonable and in accord with the stated evaluation criteria and applicable procurement laws and regulations. 22nd Century Techs., Inc., B-413210, B-413210.2, Sept. 2, 2016, 2016 CPD ¶ 306 at 8. An offeror’s disagreement with the agency’s evaluation judgment, without more, is insufficient to establish that the agency acted unreasonably. See Vectrus Sys. Corp., B-412581.3 et al., Dec. 21, 2016, 2017 CPD ¶ 10 at 3.

Additionally, in order for us to review an agency’s evaluation judgments, the agency must have adequate documentation to support those judgments. Ohio KePRO, Inc., B‑417836, B‑417836.2, Nov. 18, 2019, 2020 CPD ¶ 47 at 6-7. Where an agency fails to document or retain evaluation materials, it bears the risk that there may not be adequate supporting rationale in the record for us to conclude that its judgments were reasonable. Id.; see also Solers Inc., B-409079, B-409079.2, Jan. 27, 2014, 2014 CPD ¶ 74 at 9-10.

Competitive prejudice is an essential element of a viable protest. Coast to Coast Computer Prods., Inc., B‑419116, B‑419116.2, Dec. 18, 2020, 2020 CPD ¶ 370 at 10‑11. We will sustain a protest only where the protester demonstrates that, but for the agency’s improper actions, it would have had a substantial chance of receiving the award. Id. Where the record establishes no reasonable possibility of prejudice, we will not sustain a protest even if a defect in the procurement is found. Procentrix, Inc., B‑414629, B‑414629.2, Aug. 4, 2017, 2017 CPD ¶ 255 at 11‑12.

In reviewing an agency’s evaluation and award decisions, our Office generally accords lesser weight to post-hoc arguments or analyses made in response to protest allegations because we are concerned that new judgments made in the heat of an adversarial process may not represent the fair and considered judgment of the agency. Wolff & Mueller Gov’t Servs. GmbH & Co. KG, B-419181, B-419181.2, Dec. 28, 2020, 2021 CPD ¶ 12 at 4. While we accord greater weight to contemporaneous source selection materials as opposed to judgments made in response to protest contentions, post-protest explanations that provide a detailed rationale for contemporaneous conclusions, and simply fill in previously unrecorded details, will generally be considered in our review of the reasonableness of selection decisions--so long as those explanations are credible and consistent with the contemporaneous record. Strategi Consulting LLC; Signature Consulting Grp., LLC, B-416867, B-416867.4, Dec. 21, 2018, 2019 CPD ¶ 10 at 5.

Part I - Challenges to the Validation of Proposals and Establishment of Cutlines

Certain protesters argue that NIH’s phase 1 evaluation did not reasonably validate proposals as required by the solicitation, and set unreasonable self-scoring point cutlines to determine which proposals would advance to phases 2 and 3 of the competition. The protesters raise 10 primary arguments:[9] (1) the agency did not validate any offerors’ self-scores, or, alternatively, did not document the agency’s validation of self-scores; (2) the agency relied on unstated criteria and unreasonable methods to establish the cutlines; (3) the agency used unvalidated self-scoring points to establish the cutlines; (4) the agency failed to consider all of the solicitation’s evaluation factors in making the phase 1 determinations; (5) the agency’s establishment of cutlines converted certain self-scoring criteria into undisclosed mandatory requirements; (6) the agency’s additional adjustments to cutlines to reflect increased government-wide socio-economic goals were unreasonable; (7) small business offerors had a reasonable basis to expect a lower cutline; (8) the agency treated small and large businesses unequally; (9) the cutlines were improper competitive range determinations that did not follow the requirements of FAR part 15; and (10) the cutlines were improper de facto nonresponsibility determinations that required the agency to refer unsuccessful offerors’ proposals to the SBA for a certificate of competency determination. For the reasons discussed below, we sustain the protests concerning argument (1) with regard to the challenges to the documentation of the validation of proposals, and argument (3) with regard to the establishment of cutlines using unvalidated scores.

Background of Validation and Cutline Establishment

The solicitation provided that the phase 1 portion of the competition would assess offerors’ self-scores for experience and certifications. RFP at 157-68. NIH was to “validate” an offeror’s score “to determine whether the offeror advances to phase II or is eliminated from the competition.” Id. at 157; see also id. at 173. The RFP provided that “[o]nly the offerors who score the highest will advance to the next phase” of the competition, that is, phase 2. Id.

NIH’s response to the protests relies primarily on citations to four documents: (1) AR, Tab X, the “Final Cutline Methodology Phase 1 Memorandum,” which described the process by which proposed self-scores were validated and how the agency established the self-scoring point cutlines to determine which proposals would advance past phase 1; (2) AR, Tab BB, the “SSA Master Tracking Sheet,” which is a Microsoft Excel spreadsheet showing which proposals advanced through each of the three phases of the competition; (3) AR, Tab BB.1, the “Cutline Methodology - Listing of Offerors Self Scores Spreadsheet,” which is a Microsoft Excel spreadsheet showing the self-scores and calculations used by the agency to establish the cutlines for each socio-economic category; and (4) AR, Tab AA, the “Source Selection Decision Memorandum for Award,” (SSD) which described the source selection process over the course of the three phases of the evaluation and rationale for selecting the apparently successful offerors for award.[10]

NIH’s responses to the protests made representations to the protesters and to our Office regarding the phase 1 evaluation based on these four documents. In general, the agency stated that it validated all offerors’ proposals, and then established the phase 1 cutlines based on these validated self-scores. E.g., MOL (B419956.205 et al.) at 10-14; AR, Tab X, Final Cutline Methodology Memorandum at 3. As discussed below, the agency’s responses to supplemental protests filed by two offerors, Karsun Solutions, LLC and Neev-KS Technologies, made significant revisions to the agency’s initial explanation provided to the protesters. For the sake of clarity, we first describe the agency’s initial explanation of the process as related to all protesters. We then describe the revisions to that explanation provided only in response to the Karsun and Neev supplemental protests.[11]

Initial Agency Explanation

The agency’s final cutline methodology memorandum stated that the cutlines were established based on three primary steps: (1) validating offerors’ self-scoring points, (2) applying a “3-filter” mathematical analysis to the validated scores for proposals in each socio-economic category, and (3) making additional adjustments to each cutline “to ensure the agency met the increased small business goals mandated by Congress for Fiscal year 2023 through Fiscal Year 2025.” AR, Tab X, Final Cutline Methodology Memorandum at 3, 4. The agency stated that the final cutline results represented “the greatest number [of proposals] that will permit efficient competition among the most highly rated proposals.” Id. at 3.

In explaining step 1, the final cutline methodology memorandum stated that a “‘validated score’ is the offeror’s final score as validated by the Government, which includes any offeror’s self-score adjusted by the Government based on the validation process.” Id. at 3. With regard to adjustments in the validation process, the agency stated that “[i]f at any point, there was a discrepancy with the application of points then the evaluators documented any discrepancy and adjusted the offeror’s self-score as needed.” Id. The agency notes that although the source selection plan for the procurement provided for the validation of only those proposals whose self-scores were above a cutline, the agency explains that it “revised this approach as part of corrective action taken in November 2022” in response to protests filed with our Office, and that the corrective action involved “validating all proposals.” COS (B-419956.205 et al.) at 6 n.3.

As to step 2, the agency explained that the 3-filter analyses used three mathematical techniques: (1) “differentiation,” which “refers to a gap of at least 100 points or more in self-scores between Offerors” within a socio-economic category, and thereby identifies the highest score where the next-lowest score is at least 100 points lower; (2) “mean,” i.e., the average of all scores within a socio-economic category; and (3) “mode,” which identifies which particular score occurs the most frequently amongst all scores within a socio-economic category. AR, Tab X, Final Cutline Methodology Memorandum at 3. The agency then identified which of the three techniques produced a cutline score that resulted in the number of offerors moving to phase 2 that was closest to the anticipated maximum number of awards identified in the RFP. Id.

In this regard, the agency explained that “[t]he preferred method to be utilized to establish a cutline was based on the outcome of the analysis for each category. The method that produced a result/cutline closest to the maximum number of potential awards provided in Section L.2 of the solicitation determined the preferred method.” Id. In other words, for each category, the agency selected one of three methods based on which method produced a number of offerors moving to phase 2 that was closest to the number of maximum awards identified in the solicitation for each socio-economic category.

With regard to step 3, after identifying the preferred method and the resulting cutline, the agency made additional adjustments to some of the cutlines to reflect increases to government-wide small business and socio-economic business category contracting goals set forth in the Fiscal Year (FY) 2022 National Defense Authorization Act (NDAAs). Id. at 5-6. These adjustments were made to the SB, 8a, WOSB, HUBZone, VOSB, and SDVOSB categories, and resulted in lowering the cutlines for these categories, thereby increasing the number of proposals that would advance from phase 1 of the competition. AR, Tab X, Final Cutline Methodology Memorandum at 4‑24.

The source selection memorandum subsequently explained that the reference to the FY 2022 NDAA was an error because it was “based on a version of the FY2022 NDAA passed by the House of Representatives and that was not signed into law by the President.” AR, Tab AA, SSD at 7. The agency nonetheless concluded that the adjustments were appropriate for the following reasons:

[T]he agency’s cutline adjustments for the SB, 8a, WOSB, HUBZone and SDVOSB socioeconomic categories fulfills the Biden Administration’s goal to increase contracting opportunities for small business concerns. EO 13985, Advancing Racial Equity and Support for Underserved Communities through the Federal Government and Office of Management and Budget (OMB) Memorandum (M-22-03), Advancing Equity in Federal Procurement, mandates that agencies take certain actions related to potential barriers that underserved communities and individuals may face in taking advantage of agency procurement and contracting opportunities. In addition, the EO 14901, Further Advancing Racial Equity and Support for Underserved Communities Through the Federal Government, mandates a government-wide goal for federal procurement dollars awarded to small business concerns owned and controlled by socially and economically disadvantaged individuals of 15 [percent] in FY2025 and agencies are required to increase contracting opportunities for small business concerns.

Id.

To illustrate the agency’s application of the 3-filter analyses and subsequent adjustment, we address the small business socio-economic category. The RFP identified an anticipated target of 100 to 125 awards to small businesses. RFP at 143. The agency states that it sorted all 908 proposals based on their validated scores, then applied the 3‑filter to those scores.[12] AR, Tab X, Final Cutline Methodology Memorandum at 4.

The differentiation analysis produced a cutline score of 6,750, which meant than 841 offerors’ scores would exceed the cutline, a number that was 573 percent higher than the RFP’s 125 award maximum target. Id. at 5. The mode analysis produced a cutline score of 9,800, which meant than 255 offerors’ scores would exceed the cutline, a number that was 104 percent higher than the RFP’s 125 award maximum target. Id. The mean analysis produced a cutline score of 8,960, which meant than 661 offerors’ scores would exceed the cutline, a number that was 429 percent higher than the RFP’s 125 award maximum target. Id. at 5. Because the mode analysis result of 255 proposals was the closest to the 125 award maximum target, as compared to the other two analyses, the agency selected the mode cutline of 9,800 for the small business category. Id.

After establishing the 9,800 point cutline, the agency then made an additional adjustment to reflect what the agency stated were “substantial” increases in government-wide small business goals. Id. at 6. The agency concluded that the 255 proposals, which represented 104 percent of the RFP’s maximum target award, “would not be large enough to meet the increased demand for this category” as reflected by the increased goals. Id. The next-closest of the 3-filter analyses results was the mean. Id. This analysis, however, yielded too many proposals, at 429 percent above the maximum target award. Id. The agency therefore looked to the next “significant cluster” of self-scores below the 9,800 point cutline, which was 9,770 points. Id. This new point total added 26 additional offerors, which the agency concluded was an efficient number that also accommodated the increased small business goals. Id.

The agency’s application of the 3-filter analyses led to the selection of the mode for the following categories: small business, 8(a), WOSB, VOSB, SDVOSB, and HUBZone. Id. at 6, 9, 12, 15, 19, 22. The analysis led to the selection of the mean for the following categories: IEE, ISBEE, and ELB. Id. at 25, 27, 32. The agency used the differentiation method for the OTSB category. Id. at 27. The final cutlines, number of offerors advancing to phase 2, and percentages above the RFP’s targets were as follows:[13]

Socio-economic Category	RFP Award Estimate	Preferred Analysis Method	Self-Scoring Point Cutline	Proposals Above the Cutline	Cutline Above RFP Maximum Award Estimate
SB	100 - 125	Mode	9770	259	207%
8(a)	20 - 40	Mode	9740	123	308%
WOSB	20 - 40	Mode	9740	87	218%
VOSB	20 - 40	Mode	9700	62	155%
SDVOSB	20 - 40	Mode	9770	47	118%
HUBZone	20 - 40	Mode	9770	59	148%
IEE	20 - 40	Mean	8990	12	120%
ISBEE	75 - 125	Mean	9080	17	170%
OTSB	5 - 10	Differentiation	9000	127	102%
ELB	5 - 10	Mean	9470	76	190%

 

AR, Tab AA, SSD at 11.

Revised Explanation in Karsun and Neev Protests

While the record and agency’s responses to the protests stated that the self-scores for all proposals were validated, and that the cutlines and overall phase 1 selection decisions were based on validated self-scores, the agency’s response to supplemental protests filed by Karsun and Neev provided significantly revised explanations for the evaluations.[14] Supp. COS (B-419956.248 et al.), June 5, 2023[15], at 8. In particular, the agency clarified that the term “validated” as it was used to describe the evaluation of offerors’ self-scoring points, referred to two different processes, which the agency subsequently termed an “initial validation” and a “hard validation.”

The agency report provided by the agency to Karsun and Neev contained the same documents and explanations provided to all other protesters that challenged the validation of proposals and establishment of the phase 1 cutlines. Supplemental protests filed by Karsun’s and Neev’s noted that the cutline methodology spreadsheet (AR, Tab BB.1) contained column E titled “Self-Scored Total Points,” which appeared to be the basis upon which the agency ran its 3-filter mathematical analyses to establish the cutlines.[16] See AR, Tab BB.1, Cutline Methodology Spreadsheet. The protesters also noted that the spreadsheet, as provided by the agency, contained a number of hidden columns (F though L), including column I titled “Validated Score,” which listed scores for some, but not all offerors. See id. The protesters argued that this information from the spreadsheet showed that the agency relied on proposed unvalidated self-scores, rather than validated scores, to establish the cutlines.

In response to these arguments, the contracting officer explained on May 23 that the agency used “hidden columns I through L” to support what the agency called a “what if analysis.” Supp. COS (B-419956.248), May 23, 2023, at 10-11. The contracting officer explained, however, that the information in the hidden columns was “independent and did not affect our cutline determination.” Id. (emphasis added). The contracting officer further explained that “column E titled ‘Self-Scored Total Point’ [Tab BB.1] represents the validated self-score of each offeror” and was used “to determine the cutline based on the validated self-scores.” Id. at 11.

The protesters challenged these representations, pointing to information in the record indicating that the agency used the self-scores, not validated scores, to make the cutline determination. They noted that the data in column E (Self-Scored Total Points) of the cutline methodology spreadsheet (AR, Tab BB.1) corresponded to the data in column D (Self Score) of the All Offerors tab of the SSA master tracking spreadsheet (AR, Tab BB). See AR, Tab BB, SSA Master Tracking Spreadsheet, Tab All Offerors. The protesters further noted that in the SSA master tracking spreadsheet (AR, Tab BB), column D (Self Score) contained different scores than were in column E (Evaluated Score) of the same spreadsheet, which showed the “evaluated” self-scores that had been adjusted. See id. The protesters therefore argued that the agency’s representation that column E (Self-Scored Total Points) of the cutline methodology spreadsheet (AR, Tab BB.1) contained validated scores was not accurate--as the SSA master tracking spreadsheet indicated those scores had been adjusted at some point--and that the agency’s representation that data in this column supported the cutline calculations was therefore also not accurate.

In a June 5 response to a request from our Office to explain the apparent discrepancy, the contracting officer described the self-scoring data in all of the spreadsheet columns discussed above as “validated,” and disputed the protesters’ assertion that the agency had used unvalidated self-scores to establish the cutlines. Supp. COS (B-419956.248 et al.), June 5, 2023, at 8-9. In this supplemental statement, however, the contracting officer for the first time explained that the term “validation” referred to two different processes. Id. at 8. In this regard, the agency identified two steps:

Step 1: The agency sorted all proposals by total points within each category and performed a validation. As outlined in the cutline methodology document, all self-scores and the associated documents included in volume 1 of the proposal were reviewed. The government reviewed all points applied by the offeror in their Self-Scoring Sheet for accuracy. If at any point, there was a discrepancy noted in the self-score points assigned by an offeror, the evaluator documented and adjusted the offerors self-score points as needed. This validation by the government established a new score considered the “government validated score.” This step was completed for all offerors.

Step 2: The agency conducted a second more granular validation to determine if the methodology applied to establish the initial cutline was valid. This process included an additional review of any applicable offerors response to J.6 Self Scoring Sheet Templates signature requests and/or clarifications.[17]

Supp. COS (B-419956.248 et al.), June 5, 2023, at 8 (emphasis added). In subsequent declarations by the contracting officer and an agency statistician who worked on the CIO-SP4 procurement, the agency referred to the “Step 1” process as an “initial validation,” and the “Step 2 process as a “hard validation.” Supp. COS (B-419956.248 et al.), June 8, 2023, at 1-2; Decl. of Agency Statistician (B-419956.248 et al.), June 7, 2023, at 1-2.

Following an additional request from our Office to clarify these new and revised explanations in the contracting officer’s June 5 declaration, the agency filed a June 7 declaration from its statistician, which contradicted the prior statements of the contracting officer regarding the cutline methodology spreadsheet (AR, Tab BB.1) and SSA master tracking spreadsheet (AR, Tab BB). As noted above, the contracting officer stated that the cutlines established in the cutline methodology spreadsheet relied on column E (Self-Scored Total Points), and did not rely on hidden column I (Validated Score). Supp. COS (B-419956.248), May 23, 2023, at 10-11. The statistician, however, contradicted this explanation, stating: “The data in Column E [Self-Scored Total Points] of the Cutline Methodology Document (AR, Tab BB.1) was derived from the agency’s initial validation of all offerors’ self-scores. This data was not used to develop the cutline.” Decl. of Agency Statistician (B-419956.248 et al.), June 7, 2023, at 1 (emphasis added).

The statistician further clarified that the agency established the cutlines based on the following steps: (1) conducted an initial validation, which was reflected in the scores in column E (Self-Scored Total Points) of the cutline methodology document spreadsheet (AR, Tab BB.1); (2) conducted “preliminary” 3-filter analyses of the proposals based on their initial validation; (3) conducted a hard validation of offerors’ self-scores, reflected in the scores in column E (Evaluated Score) of the SSA master tracking spreadsheet (AR, Tab BB); (4) conducted “another” or “second” 3-filter analysis to establish the cutlines; (4) copied the hard validated self-scores into hidden column I (Validated Score) of the cutline methodology document to confirm that the “second” 3-filter analyses was correct. Id. at 2-3.

After these cutlines were established, the agency then made the manual adjustments to some cutlines to account for the increased government-wide small business participation goals. Id. at 3. These cutlines were then incorporated back into the SSA master tracking spreadsheet (AR, Tab BB) and used to determine which proposals advanced past phase 1 of the competition. Id. at 3-4.

The statistician stated that the “results of the second [3-filter] analysis confirmed that the initial analysis performed in the Cutline Methodology Document” were valid with regard to the cutlines that relied on the mode analysis, i.e., the small business, 8(a), WOSB, VOSB, SDVOSB, and HUBZone categories. Id. at 2. With regard to the business categories that did not rely on the mode, “the Mean and Differentiation calculation[s] vary slightly.” Id. at 3. The statistician did not address the significance of the variance of the mean and differentiation calculations.

The agency statistician also stated that the analysis discussed in his declaration “has been incorporated into the SSA Master Tracking Sheet (AR TAB BB) for [GAO’s] review.” Id. at 3; AR, Tab BB-Attachment 1, SSA Master Tracking Sheet--Hard Validation Data. The agency statistician did not explain whether the newly-produced version of the SSA master tracking spreadsheet reflected contemporaneous analyses, or whether it was a post-hoc analysis prepared in response to the protest.

On June 8, the day after filing the agency statistician’s declaration, the contracting officer provided another declaration acknowledging that the agency’s review of the protester’s allegation identified errors.[18] Supp. COS (B-419956.248 et al.), June 8, 2023, at 1. In this regard, the contracting officer stated: “During the agency’s review of the categories that relied on the Mean and Differentiation methodology there were slight variances noted in both the IEE and ELB category cutlines based on the validated scores of all offerors.” Id. As a result of these new calculations, the agency stated that it would lower the cutlines for the IEE and ELB categories, and that offerors affected by these changes “will be notified that they have successfully completed Phase 1.” Id.

The contracting officer’s June 8 filing provided another version of Tab BB for our Office’s review, representing that it was a “complete analysis” of the agency’s adjustments made in response to the protest. Id.; AR, Tab BB-Attachment 2-SSA Master Tracking Sheet DZ Analysis v4. The contracting officer stated that the “compete analysis” confirms that the agency’s cutlines, as revised, are reasonable and that the protests--aside from those the agency identified as being affected by the newly revised cutlines for the IEE and ELB categories, should be denied. The contracting officer’s statements regarding the second of the two revised versions of the SSA master tracking spreadsheet, particularly with regard to the post-hoc adjustments to certain cutlines, shows that both versions relied on post-hoc analyses that were not reflected in the contemporaneous record.

We discuss the significance of these shifting and incomplete explanations below. In particular, we address the remaining uncertainty that the agency conducted a validation of all proposals that complied with the RFP requirements, and whether the agency relied on unvalidated self-scores to establish the cutlines. We also address our concern that this information was provided to only two protesters, despite the fact that nearly all of the other protesters raised arguments for which the revised explanations would have been relevant.

1. The agency did not validate offerors’ self-scores

Certain protesters argue that NIH failed to validate any of the proposals, and instead accepted offerors’ proposed self-scores without any evaluation. These protesters contend that the agency therefore improperly used the unvalidated scores to establish the cutlines for each socio-economic category and determine which proposals advanced past phase 1 of the competition. These and certain other protesters also argue that the agency’s representations regarding its validation of proposals are not reasonable because the agency did not provide an adequate contemporaneous record showing that it validated all offerors’ proposed self-scores. Based on our review of the record, we find no merit to the contention that the agency failed to validate any of the proposals. We agree, however, that the agency did not adequately document its validation of proposals, and therefore sustain the protests on this basis.

The agency’s response to the protests did not provide specific documents showing the evaluation of individual proposals, for example detailing which proposals were found to have merited all of their self-scored points and which proposals received deductions based on the agency’s conclusions that certain self-scored points were not merited. Instead, the agency provided: (1) the SSA master tracking spreadsheet, showing each offeror’s self-score and any adjustments made, and (2) the final cutline methodology memorandum, which described the process by which the agency validated proposals and established the cutlines. The agency argued that these documents provided adequate documentation that the agency validated all offerors’ self-scores. E.g., MOL (B‑419956.268) at 28 (“As explained, supra, the Agency not only validated offerors’ self-scores, but that it also documented its validation process. See generally, AR Tab X, Cutline Methodology; see also AR Tab BB Master Tracking Sheet.”).

We agree with the agency that the SSA master tracking spreadsheet shows, for numerous proposals, that the agency deducted self-scoring points, and that these deductions show that the agency validated these proposals. In this regard, as discussed in part III of the decision below, several protesters challenge the agency’s validation of their proposals and the deduction of self-scoring points. On this record, we find no basis to conclude that the agency failed to validate any of the proposals.[19]

We agree with the protesters, however, that the record provided by the agency does not show that it validated all of the proposals. In general, we think the agency could have reasonably demonstrated that it had validated all proposals based on the SSA master tracking spreadsheet, which showed the proposed and evaluated scores for each proposal, and the narrative description of the validation process set forth in the final cutline methodology memorandum. As discussed above, however, the agency’s responses to Karsun’s and Neev’s supplemental protests show that the agency’s initial responses to the protesters and the descriptions of the validation process in the final cutline methodology memorandum were incomplete and ultimately misleading.

The agency’s subsequent explanations in the Karsun and Neev supplemental protests contain significant revisions to the narrative set forth in the final cutline methodology memorandum, as well as contradictory information as provided by the contracting officer and agency statistician. Accordingly, we cannot determine from the documentation provided, whether the agency in fact validated all of the vendors’ scores consistent with the solicitation requirements. As further discussed below in the 3rd argument in this part of the decision, the agency’s response to the protests shows a lack of hard validation for all proposals, reliance on partially-validated scores to establish cutlines, and misleading responses from the agency on this point.

For these reasons, in conjunction with our review of the agency’s process for determining the cutlines for the phase 1 evaluation, we find that the lack of documentation, combined with misleading and contradictory explanations regarding how the validation occurred, precludes us from finding that the phase 1 evaluation was reasonable. See Ohio KePRO, Inc., supra; Solers Inc., supra. We therefore sustain the protests on this basis.

2. The cutlines relied on unstated criteria and unreasonable methods

Certain protesters argue that the agency’s phase 1 cutlines relied on unreasonable considerations to guide the 3-filter analysis, and also contend that the 3-filter analyses used mathematical techniques that did not provide meaningful results. We find no merit to these arguments.

First, certain protesters argue that NIH’s evaluation relied on considerations that were not consistent with the terms of the solicitation and were otherwise improper. As discussed above, the agency stated that the 3-filter analyses that was applied to scores for each socio-economic category was guided by the estimated number awards identified in the RFP. See AR, Tab X, Final Cutline Methodology Memorandum at 3. The agency also stated that the establishment of the cutlines was guided by the overall consideration of efficiency. E.g., Supp. MOL (B-419956.201 et al.) at 3-5. The protesters contend that the agency could not reasonably rely on either consideration in establishing the cutlines.

The RFP did not explain how the agency would determine which proposals were amongst the highest rated for the phase 1 competition. With regard to the consideration of the estimated awards, offerors knew or should have known that the agency would be required to establish a cutline based on phase 1 self-scoring points, and also knew the RFP identified an estimated number of awards. Although the protesters contend that the estimated number of awards should not have influenced the agency’s determination of the highest rated proposals, we see nothing in the RFP which prohibited this consideration. Moreover, the agency did not strictly apply a hard cap on the number of proposals that advanced past phase 1 based on the RFP’s stated estimated numbers of awards for each socio-economic category. Rather, the agency used the upper number of each range as a guide in establishing cutlines that ultimately included between 102 and 308 percent of proposals above the higher numbers of the estimated awards. See AR, Tab AA, SSD at 11.

With regard to the overall consideration of efficiency, the protesters contend that the agency improperly limited the number of proposals advancing past phase 1 of the competition in order to limit the burden of evaluating proposals in phases 2 and 3 of the competition. Our Office has explained that agencies may not refuse to evaluate proposals based on the factors set forth in a solicitation simply because the volume of proposals received is large or that the effort to comply with the solicitation’s stated evaluation scheme is burdensome. See Kathpal Techs. Inc., Computer & Hi-Tech Mgmt, Inc., B-283137.3 et al., Dec. 30, 1999, 2000 CPD ¶ 11-12.

Here, however, the RFP explained that proposals would be evaluated based on the phase 1 self-scoring criteria, and only the highest rated proposals would advance to phases 2 and 3. RFP at 174. The RFP further stated that the phase 2 and phase 3 evaluation criteria were not part of the phase 1 evaluation. Id. at 174-76. We therefore do not agree, as the protesters contend, that the agency’s actions here are similar to those in Kathpal Techs. Inc. where we found that an agency refused to evaluate proposals based solely on the perceived burden of doing so. Rather, the RFP clearly anticipated that the phase 1 evaluation would be a means for the agency to limit the number of proposals that would be evaluated in phases 2 and 3 of the competition. On this record, we find no basis to conclude that the agency improperly considered either the estimated awards or the interests of efficiency when establishing the phase 1 cutlines.

Next, certain protesters argue that the agency’s 3-filter analyses relied on unreasonable mathematical techniques. As discussed above, the 3-filter analysis involved 3 different mathematical techniques: differentiation, mean, and mode. Differentiation identified the highest score where the next-lowest score was at least 100 points lower; the mean identified the average of all scores; and mode identified which score occurred the most frequently. The agency applied these three techniques to offerors’ self-scores in each of the socio-economic categories to determine which technique produced a score that resulted in a number of offerors moving to phase 2 that was closest to the maximum anticipated awards identified in the RFP. AR, Tab X, Final Cutline Methodology Memorandum at 3.

The protesters argue that the agency does not explain why any one of these techniques produced a meaningful result. The protesters also argue that the agency’s methodology was flawed because the agency did not use a technique that produced a meaningful result for all of the socio-economic categories. Rather, the agency applied three different techniques and selected the technique that produced a result that fit the answer the agency sought, i.e., the result that was closest to the maximum anticipated award for a particular socio-economic category. See Supp. MOL (B-419956.201 et al.) at 5-8.

We agree that the agency does not clearly explain why any of the methods used in the 3-filter technique produced meaningful results. In fact, the agency acknowledges that reliance on only one of the techniques would have produced undesirable results for certain of the socio-economic categories. See id. at 7.

Notwithstanding the agency’s acknowledgment that it chose amongst different methodologies to obtain the preferred result for each socio-economic category, as opposed to choosing a single methodology that yielded a consistent and meaningful result across all categories, we do not find that the agency’s approach was unreasonable in a manner that merits sustaining the protest. The agency appears to have used the 3-filter analyses to find a mathematically derived result that was nearest to the maximum anticipated awards for each socio-economic category as a proxy for the otherwise undefined standard of highest rated. Because we find that consideration of the maximum anticipated awards and efficiency are both reasonable considerations in connection with the RFP’s criterion of the highest-rated proposals, we do not think that the agency’s approach here was unreasonable.

3. The agency used unvalidated scores to establish the cutlines

Having concluded that the agency’s general approach to using the 3-filter analysis as an aid to setting the cutlines for phase 1 does not provide us a basis to sustain the protests, we next address certain protesters’ arguments that the 3‑filter analysis results were flawed because the agency used unvalidated, rather than validated self-scoring points. Here, we find that the record does not show that the agency’s cutlines were based on validated self-scoring points, as the agency claims. We also find that the agency’s responses to the protests rely on explanations that have shifted several times, as well as post-hoc rationales that are not supported by the contemporaneous record. For these reasons, we sustain the protests.

As discussed above, the agency’s response to all protests challenging the validation of proposals and use of the 3-filter analysis to establish the cutlines stated that for each socio-economic category, the agency validated all offerors’ self-scores, used the 3-filter analysis to establish the cutlines based on those validated scores, then adjusted the cutlines to address increased government-wide contracting goals. AR, Tab X, Final Cutline Methodology Memorandum at 3; e.g., Supp. MOL (B-419956.201 et al.) at 3-5. In response to two supplemental protests, the agency acknowledged that the record appeared to show that the cutlines were based on unvalidated, i.e., the offerors’ initial self-scores, rather than validated scores. The agency unambiguously stated, however, that this was not the case:

In all instances, the Agency validated self-scores prior to establishing the cutline for the different socioeconomic categories. In other words, the Agency did not base the cutline on offerors’ self-scores, but rather, based the cutline on offerors’ validated self-scores. The cutline methodology unintentionally uses language that could imply that the agency based the cutline on offerors’ self-scores, but this was not the case.

MOL (B-419956.261 et al.) at 4 n.5.

In contrast to these relatively straightforward explanations for the phase 1 evaluations, NIH’s responses to supplemental protests filed by Karsun and Neev clarified for the first time that the term “validated” meant different things at different times during the course of the procurement. As discussed below, an “initial validation,” appears to have involved a check that the vendor had submitted the required information to support its score without making any score adjustments. A “hard validation,” on the other hand, appears to have occurred when the agency actually analyzed whether the submitted information supported a vendor’s allocation of its self-scored points, and making corresponding adjustments to the self-scores. Additionally, as described above, the agency’s responses to Karsun’s and Neev’s supplemental protests included contradictory explanations between the contracting officer and agency statistician, as well as two revised versions of contemporaneous documents that the agency stated were for the purpose of explaining the agency’s revised explanations.

Notwithstanding the lack of clarity in the contemporaneous record and the agency’s responses to the protests, we think that hard validation, as described by the agency, was the review anticipated by the RFP to the extent it was the step that involved all required adjustments to the proposed self-scores. For example, decreasing an offeror’s self-score because the offeror did not provide sufficient documentation to support an experience example. Any initial validations that did not involve all of the required evaluations, were necessarily not the hard validation required by the RFP. Because the agency used offerors’ scores to determine the cutlines for each socio-economic category, it follows that the agency should have used only the hard validated scores to establish the cutlines, and not the initial validated scores.

In this regard, section L of the RFP stated that offerors were required to complete a self-scoring spreadsheet, which would show the total of the claimed points, as follows:

The total amount of points an offeror receives is shown in cell E19. This is the score the government will validate and then use to determine whether the offeror advances to phase II or is eliminated from the competition. Only the offerors who score the highest will advance to the next phase.

RFP at 157 (emphasis added). Similarly, section M of the RFP stated that “[d]uring phase 1, the government will validate the offeror’s completed self-scoring. Only the highest rated offerors will advance to phase 2 of the evaluation.” Id. at 174.

Karsun and Neev argue that neither the contemporaneous record, nor the agency’s revised responses to the protests shows that NIH conducted a hard validation of self-scores for all proposals, nor does the agency show that it used hard validated scores to establish the cutlines. We agree with the protesters. Despite multiple attempts to explain its actions and revising its initial responses to the protest, including the submission of revised explanations and spreadsheets that reflect post-hoc analyses, the agency has not demonstrated that its evaluation was reasonable and complied with the terms of the solicitation. We address two points that illustrate our concerns with the record and the agency’s responses.

First, as explained above, the agency produced a cutline methodology spreadsheet which showed the self-scores and calculations used by the agency to establish the cutlines for each socio-economic category. The data in column E (Self-Scored Total Points) of this spreadsheet (AR, Tab BB.1) appears to show the proposed self-scores, as none of these scores appear to contain any adjustments. Although we did not require the agency to produce the proposals of all 1,150 offerors that competed for awards, our review of the proposals of the protesters that challenged adjustments to their proposed self-scores shows that their proposed self-scores were reflected in column E (Self-Scored Total Points) of the cutline methodology spreadsheet (AR, Tab BB.1), without adjustment. The score adjustments challenged by these protesters are not reflected in this column and appear only in hidden column I (Validated Score) of the cutline methodology spreadsheet (AR, Tab BB.1) and column E (Evaluated Score) of the SSA master tracking spreadsheet (AR, Tab BB).[20] On this record, in the absence of any other information provided by the agency, we conclude that the initial validation the agency states is reflected in column E (Self-Scored Total Points) of the cutline methodology spreadsheet (AR, Tab BB.1) did not involve a detailed review of self-scores as none of those scores were adjusted.

Compounding this problem is the fact that the record indicates the agency used the scores from column E (Self-Scored Total Points) of the cutline methodology to establish an initial cutline for each of the socio-economic categories. E.g., AR, Tab BB.1, Cutline Methodology Spreadsheet, SB Tab. Although the agency has provided conflicting statements about whether these scores were used to establish cutlines, the cutline methodology spreadsheet along with the final cutline methodology phase 1 memorandum indicates that the agency did use these scores to establish at least the initial cutlines. The record indicates that the agency then identified those offerors that were above these initial cutlines and conducted a hard validation only on these offerors.

Second, the SSA master tracking spreadsheet shows that the agency did not conduct a hard validation of all proposals. As noted above, the agency initially stated that the SSA master tracking spreadsheet demonstrated that the agency validated all proposals--prior to explaining that the term “validated” referred to an initial validation and a hard validation. In response to Karsun’s and Neev’s supplemental protests, the agency states that column E (Evaluated Score) of the SSA master tracking spreadsheet (AR, Tab BB) represents the hard validated scores for all offerors’ proposals. Decl. of Agency Statistician (B‑419956.248 et al.), June 7, 2023, at 1. Our review of the SSA master tracking spreadsheet shows that of the 1,150 proposals received, only 199 of the proposals (approximately 17 percent) received an adjusted score indicative of a hard validation. Of the 433 proposals that advanced to phase 2 of the competition, 152 of those proposals (approximately 35 percent) were assigned an adjusted self‑score. See AR, Tab BB, SSA Master Tracking Spreadsheet, Successful P1 Offerors Tab.

In contrast, none of the remaining proposals with initial validated scores (listed in column D, Self-Score) below the initial cutlines for all of the socio-economic categories were assigned an adjusted self-score (listed in column E, Evaluated Score). As a group of protesters[21] argues, it “strains credulity” that the self-scores of approximately a third of the proposals that advanced to phase 2 of the competition required adjustment, while none of the proposals below the cutlines required adjustment.[22] Comments (B‑419956.207 et al.) at 5. On this record, in the absence of any other information provided by the agency, we conclude that the hard validation, upon which the agency states the cutlines were established, is reflected in column E (Evaluated Score) of the SSA master tracking spreadsheet (AR, Tab BB), and was conducted only for offerors above the initial cutlines for each socio-economic category. This shows, in turn, that the agency’s cutlines were based on only a partial and incomplete hard validation of offerors’ self-scores, to the extent they were based on hard validated scores at all.

In sum, despite NIH’s numerous attempts to clarify the record in response to Karsun’s and Neev’s protest, the agency does not clearly explain whether the agency conducted a hard validation of all offerors’ self-scores and used those hard validated scores to establish the cutlines. In fact, the record strongly indicates that the agency used the initial validated (and unadjusted) self-scores to establish the initial cutlines, performed a hard validation only on those offerors whose scores were above the initial cutlines, and then used the incomplete set of hard validated scores only as a way to check its analysis but did not revise the initial cutlines. To the extent any of the agency’s responses can be construed as claiming that the agency hard validated all proposed scores and used those hard validated scores to establish the cutlines, we find that the agency’s answers are not supported by the contemporaneous record.

Finally, the agency argues in response to the Karsun and Neev supplemental protests that it does not matter whether the agency used validated or unvalidated scores. In this regard, the contracting officer contends:

Regardless, of the use [of] unvalidated or validated scores to derive the cutline, our objective was to select the most qualified offerors based on the anticipated awards outlined in the RFP. Therefore, the methodology to derive the cutline would have yielded the same results whether the agency used validated or unvalidated self-score points.

Supp. COS (B-419956.248 et al.), June 5, 2023, at 8.

We disagree with the contracting officer’s assertion because the RFP clearly anticipated that the validated scores would be used to identify the highest rated proposals for phase 1, specifically what the agency now characterizes as hard validated scores. To the extent the agency’s validation and 3-filter analysis processes attempted to establish a mathematically objective cutline, the failure to use hard validated scores had an effect on these processes. In fact, the agency’s supplemental filing on June 8, 2023, which included the second of two newly-produced versions of the SSA master tracking spreadsheet--which uses what the agency contends is hard validated scores, without explaining the basis for the claim--acknowledged errors in the calculation of its cutlines that required partial corrective action. At any rate, we cannot speculate as to what the outcome would have been had the agency conducted a hard validation on the scores of all proposals as required by the solicitation. Based on the record provided by the agency to our Office for review, we are unable to conclude that the agency’s evaluation was in accordance with the terms of the solicitation. For these reasons, we sustain the protests.

4. The agency failed to consider all evaluation factors

Certain protesters argue that the agency improperly used self-scoring point cutlines to determine which proposals would advance past phase 1 of the competition, and did not consider the other evaluation factors identified in the solicitation. These arguments take several forms, none of which we find have merit.

First, certain protesters contend that the use of specific cutlines did not reflect a meaningful assessment of whether an offeror could successfully perform the solicitation requirements. In this regard, the protesters argue that the solicitation did not disclose that the agency would use self-scoring points as a cutline for phase 1, or failed to specify in advance what cutlines the agency would use for each socio-economic category.

As discussed above, the RFP clearly advised that the only criteria to be used to evaluate proposals in phase 1 of the competition were self-scoring points, and that only the “highest rated” proposals would advance to phases 2 and 3. RFP at 174. For these reasons, offerors knew or should have known that the agency would impose a cutline of some type based on self-scoring points to determine whether proposals advance past phase 1 of the competition. Further, the solicitation did not specify in advance what the cutlines would be, and protesters therefore knew or should have known that this would be a matter within the agency’s discretion.

Our Bid Protest Regulations require that protests based upon alleged improprieties in a solicitation that are apparent prior to the closing time for receipt of initial proposals must be filed before that time. 4 C.F.R. § 21.2(a)(1). Here, these challenges to the use of self-scoring points after the time for receipt of initial proposals are therefore untimely. See id.

Certain protesters also contend that the agency’s use of cutlines was improper because it resulted in the elimination of proposals from the competition without consideration of their price or technical approaches. In support of these arguments, the protesters contend that the agency’s evaluation here was similar to that found unreasonable by our Office in Kathpal Techs. Inc. In Kathpal Techs., Inc., we sustained the protest because the agency’s evaluation of proposals for the award of an IDIQ contract improperly failed to consider all required evaluation factors prior to excluding the protesters’ proposals from proceeding to the next phase of the competition. Kathpal Techs., Inc., supra, at 9‑12.

Unlike in Kathpal Techs., Inc., however, the solicitation here specifically provided that the phase 1 competition would be limited to specific criteria--validated self-scoring points--and that only the highest rated proposals would proceed to the next phases of the competition, where other factors such as technical approach and price would be considered. The RFP provided that price and technical approach would only be considered in phase 3 of the competition--and would only be considered for proposals that advanced past phases 1 and 2. For these reasons, we find no basis to conclude that the agency’s evaluation was inconsistent with our decision in Kathpal Techs., Inc.

In sum, we find that offerors knew or should have known that there would be a cutline established for each socio-economic category, that it would be based on validated self-scoring points, and that the solicitation did not specify in advance where the cutline would be. Under these circumstances, the agency had discretion to set the cutline for phase 1 at a point to advance only the “highest rated” proposals, provided the agency used a reasonable methodology to do so.[23]

5. Untimely Challenges to the Terms of the Solicitation

Certain protesters challenge NIH’s phase 1 validation of self-scores on grounds that given the high cutline scores for most socio-economic categories, certain solicitation criteria, though presented as optional phase 1 self-scoring criteria, were in effect unstated solicitation terms that imposed unduly restrictive mandatory “requirements” or “go/no-go criteria.” Allowing these criteria to determine the outcome was fundamentally unfair because only small businesses that formed an MPJV or CTA with large business team members could meet them, disadvantaging other small business vendors in the competition. E.g., Protest (B‑419956.268) at 8; Protest (B‑419956.193) at 6. We conclude that these arguments are untimely challenges to the terms of the solicitation.

Agencies must specify their needs in a manner designed to permit full and open competition, and may include restrictive requirements only to the extent they are necessary to satisfy the agencies’ legitimate needs or as otherwise authorized by law. 41 U.S.C. § 3306(a). Where a protester challenges a specification or requirement as unduly restrictive of competition, the procuring agency has the responsibility of establishing that the specification or requirement is reasonably necessary to meet the agency’s needs. Remote Diagnostic Techs., LLC, B‑413375.4, B‑413375.5, Feb. 28, 2017, 2017 CPD ¶ 80 at 3‑4. We examine the adequacy of the agency’s justification for a restrictive solicitation provision to ensure that it is rational and can withstand logical scrutiny. DynCorp Int’l, LLC, B‑418742.2, Sept. 25, 2020, 2020 CPD ¶ 318 at 6. A protester’s disagreement with the agency’s judgment concerning the agency’s needs and how to accommodate them, without more, does not show that the agency’s judgment is unreasonable. Emax Fin. & Real Estate Advisory Servs., LLC, B‑408260, July 25, 2013, 2013 CPD ¶ 180 at 4.

The RFP advised potential offerors as to how self-scoring points could be claimed in phase 1 of the procurement. As relevant here, the RFP conveyed that offerors could claim: (1) 100 points per example of experience performing IT projects that directly support Historically Black Colleges and Universities in accordance with EO 13779, for a maximum of 300 points; (2) 300 points for possessing an earned value management system; (3) 300 points for possessing an acceptable estimating system; and (4) 200 points for possessing an approved purchasing system. RFP at 165-168.

The RFP permitted offerors to propose as a MPJV that has been approved by the SBA, or contractor teaming arrangement (CTA). RFP at 147. The RFP stated that a small business could propose as a CTA only under the following conditions:

To be considered a small business, the other members of the CTA must all be small businesses, some other socioeconomic category of a small business, or an other than small business that has an SBA-approved mentor-protégé agreement with the eligible socio-economic business whose status the CTA is relying upon to compete for award.

RFP at 149. An MPJV or a qualifying small business CTA could claim “the experience, capabilities, business systems, and certifications” for all members of the CTA or MPJV. RFP at 157. Under these provisions, therefore, a small business offeror could claim the experience of a large business if: (1) the offeror was an MPJV and the large business was the mentor firm; or (2) the offeror was a qualifying small business CTA that included an MPJV where the large business was the mentor firm.

The protesters specifically challenge the four self-scoring criteria identified above. E.g., Protest (B‑419956.268) at 9-15. They contend that the challenged self-scoring criteria could be claimed only by small businesses in MPJVs or CTAs who partnered with large businesses.[24] Thus, by setting very high cutlines for the small business categories (SB, WOSB, VOSB, SDVOSB, HUBZone) the agency unfairly shut out small businesses that did not form an MPJV or CTA with large business team members.

Specifically, because NIH set cutlines of between 9,740 and 9,770 points for the small business socio-economic categories, the inability to claim points for nearly all of the four challenged criteria precluded a small business offeror from advancing past phase 1 of the competition. The protesters argue that the high cutlines effectively made the challenged criteria unstated mandatory requirements that were unduly restrictive of competition for small businesses. The protesters further contend that these criteria were inconsistent with the RFP’s stated goal of giving “qualified small businesses a greater opportunity to participate” in these requirements, and giving government agencies a mechanism to help meet their socio-economic contracting goals. E.g., Comments (B-419956.194) at 6 (citing RFP at 7); see, e.g., First Supp. Protest (B‑419956.185) at 3-4.

NIH primarily contends that these protest grounds are untimely challenges to the terms of the solicitation. In this regard, the agency argues that the solicitation clearly informed offerors as to how points would be awarded, and to the extent any offeror thought the RFP’s instructions and evaluation scheme were unfair or improper, that offeror was required to file a protest with our Office prior to the deadline for receipt of proposals, in accordance with our regulations at 4 C.F.R. § 21.2. E.g., MOL (B-419956.194) at 15.

Further, the agency maintains that it did not impose minimum mandatory requirements or apply unstated evaluation criteria in validating proposals, because the solicitation, including the attached self-scoring temple in the J.6 form, informed offerors that their proposals would be validated based on the criteria in question, as well as the point values that could be self-scored for each of those criteria. E.g., MOL (B-419956.193) at 7-8; MOL (B‑419956.197, B‑419956.198) at 16. Thus, according to the agency, although no single criterion included in the solicitation was required, all offerors were aware that failure to obtain points under a single, or multiple, criteria would decrease the offerors’ overall scores, and increase the risk of its elimination from the procurement following phase 1. E.g., MOL (B-419956.194) at 15.

We agree with the agency that the protesters’ challenges to these solicitation criteria amount to untimely challenges to the terms of the solicitation. All offerors were on notice from the terms of the solicitation that: (1) there was a limited number (10,000) of total points that could be self-scored in phase 1 of the competition; and (2) an offeror could form teaming arrangements, including arrangements that included large businesses, in order to claim points for many of the solicitation criteria. Therefore, offerors were aware, prior to the deadline for submission of proposals, that for every point they could not claim under phase 1 self-scoring, either alone or through teaming arrangements, there was an increased risk of other offerors achieving higher self-scores. Challenges to these solicitation criteria were thus required to be filed prior to the deadline for receipt of proposals in order to be timely. 4 C.F.R. § 21.2(a)(1). That the cutlines for some socio-economic categories were set at a high percentage of the available points did not convert the solicitation criteria challenged here (or any of the self-scoring criteria, for that matter), into go/no-go requirements; rather, these were merely the point totals claimed by the highest rated offerors.[25]

In sum, we do not find that the agency’s establishment of the cutlines for the socio-economic category converted any of the self-scoring criteria into mandatory requirements. Certain protesters contend that, had they known that the cutline would be so high, they would have formed a MPJV or a CTA with a MPJV member. E.g., Protest (B‑419956.184) at 26. Again, however, the RFP advised that only the highest rated proposals would advance past phase 1 of the competition, and that the only criteria for the phase 1 competition were self-scoring points. RFP at 174. Offerors therefore knew or should have known that offerors that formed MPJVs or CTAs with a MPJV member could potentially self-score more points and therefore be ranked ahead of offerors that were not able to self-score these points. We therefore dismiss these arguments as untimely challenges to the terms of the solicitation. See 4 C.F.R. § 21.2(a)(1).

6. Challenges to additional adjustments to cutlines

Certain protesters challenge NIH’s adjustments to the cutlines for some of the socio-economic categories, which were made after the application of the 3-filter analysis and intended to reflect increased government-wide small disadvantaged business (SDB) participation goals. We find no merit to these arguments.

These protesters argue that the adjustments to the cutlines made to address changes to the government-wide SDB goals were unreasonable because the increases from the 3‑filter analysis cutlines were not as large as the changes to the government-wide goals cited by the agency. For example, because the SDB contracting goal cited in the source selection document was set to increase from 10 percent to 15 percent by fiscal year 2025, which represents a 50 percent increase, the protesters contend that the agency should have similarly adjusted cutlines to allow an additional 50 percent of offerors to advance to phase 2 of the competition. See AR, Tab AA, SSD at 7. These protesters also argue that the adjustments were not sufficient to ensure that the increased goals would be met because the agency advanced fewer proposals past phase 1 of the competition for each socio-economic category as compared to the total number of awards made for those same categories under the predecessor CIO-SP3 contracts.

The Small Business Act requires each agency to set goals for participation of small business concerns, including SDBs. 15 U.S.C. § 644(g)(2). Our Office has explained that prior to setting aside a procurement, the FAR requires an agency to consider its progress in fulfilling its small business participation goals prior to selecting a particular socio-economic category under which to set aside the procurement. EDWOSB Transformer Services, LLC, B-416683, Oct. 15, 2018, 2018 CPD ¶ 357 at 4. We have also explained, however, that an agency’s failure to meet its set-aside goals does not require any particular procurement to be set aside, and that a protest based on such a failure does not state a valid basis of protest. AeroSage, LLC; SageCare, Inc., B‑416279, July 16, 2018, 2018 CPD ¶ 243 at 6 n.8. In this sense, small business contracting goals are aspirational, and the only “enforcement mechanism” in place is the requirement for agencies to produce annual reports to Congress and the President as to agencies’ level of success in meeting goals for small business contracting, including a remediation plan in the event goals are not reached. See id. at § 644(h); Dynamic Educ. Sys., Inc. v. United States, 109 Fed. Cl. 306, 324 (2013).

In sum, neither the Small Business Act nor EO 14901 set mandatory government-wide procurement minimums that agencies are required to meet overall, or in any particular procurement. Additionally, as relevant here, small business contracting goals are measured by the value of contracts awarded to small businesses as compared to the total value of all contracts awarded for a fiscal year, rather than the number of small business contracts awarded or the number of small business firms that are awarded contracts. See 15 U.S.C. § 644(g)(1). For these reasons, we find that the protesters do not demonstrate that the agency was required to make adjustments to the cutlines that mirrored the increased government-wide SDB goals.

As discussed above, we find that the record does not show that the agency reasonably established the cutlines for each socio-economic category. Although these cutlines were the starting point upon which the agency made additional adjustments to reflect increased government-wide contracting goals, we find no basis to conclude that the method used to make these additional adjustments was unreasonable. We therefore find that these arguments challenging the adjustments based on increases to the government-wide goals do not provide a basis to sustain the protests.

7. Small Business Offerors Had a Reasonable Basis to Expect a Lower Cutline

A number of protesters contend that the solicitation gave offerors a reasonable expectation that a small business offeror’s proposal would advance past phase 1 of the competition if it was validated as having at least 9,100 points. We find no merit to these arguments.

The RFP identified 10 task areas that contractors would perform. RFP at 23. Small business offerors were required to propose to perform at least eight task areas; large businesses (OTSBs and ELBs), in contrast were required to propose to perform all 10 task areas.[26] Id. at 159-60. The phase 1 evaluation criteria required offerors to identify experience examples for the functional areas they proposed to perform, and allowed offerors to self-certify points for up to three examples per functional area, with higher points available for higher contract values. RFP at 159‑64.

The protesters note that if a small business offeror (1) proposed to perform the minimum eight task areas and self-certified the maximum number of points for experience under those areas, and (2) self-certified all remaining available non-experience points, the maximum possible total self-score would be 9,100 points. Based on this calculation, the protesters contend that a small business offeror could have reasonably assumed that its proposal would be amongst the highest rated, and thereby advance past phase 1 of the competition, if it was validated to have at least 9,100 points.

We find that the protesters’ assumptions here were not reasonable. In this regard, it was clear from the terms of the solicitation that--regardless of the number of non-experience points claimed--a small business offeror that proposed to perform all 10 task areas could earn 900 more points than a small business offeror that propose to perform only the minimum number of task areas. See RFP at 159-64. The protesters do not explain why it was reasonable to believe that the phase 1 criteria provided that a proposal that merely met the minimum requirement would be automatically considered one of the highest-rated proposals. On this record, we find no basis to sustain the protests.

8. Unequal treatment of small business and other than small business offerors

Certain protesters that competed for awards in the small business, 8(a), WOSB, VOSB, SDVOSB, and HUBZone categories contend that the agency’s cutlines were unreasonable because they were higher than those established for large businesses (OTSB and ELB). In this regard, the protesters argue that while most small business categories needed to receive validated self-scores of over 97 percent of the available points, OTSB offerors needed to receive only 90 percent (9000 points) and ELBs 94 percent (9400 points) of the available points. See AR, Tab AA, SSD at 11. We find no merit to these arguments.

As discussed above, the RFP made clear that offerors competing for awards in each socio-economic category competed only against other offerors in the same category. RFP at 145-46. For this reason, the establishment of a higher or lower cutline for a particular socio-economic category had no effect on the interests of offerors in a different category. Additionally, the scoring criteria that applied to small business offerors for the points that could be claimed for experience were different than the criteria that applied to OTSB and ELB offerors. Id. at 159-64. For example, a small business, 8(a), WOSB, VOSB, SDVOSB, or HUBZone offeror seeking to claim the maximum number of points for a corporate experience example under a particular task needed to demonstrate an obligated/funded value of over $7,000,000, whereas an OTSB or ELB offeror would need to demonstrate an obligated/funded value of over $31,000,000. Id.

For these reasons, we find no merit to the protester’s contentions that the agency’s establishment of the cutlines reflected improper or unequal treatment of OTSB or ELB offerors as compared to other offerors. We therefore find no basis to sustain the protests.

9. The cutlines were improper competitive range determinations

Certain protesters argue that the establishment of the self-scoring point cutlines for the phase 1 evaluation was an improper competitive range determination that failed to satisfy the requirements of FAR part 15. Specifically, the protesters contend that the agency’s phase 1 cutlines were improper because they excluded offerors from the remainder of the competition without considering their price or technical proposals. We find no merit to these arguments.

In competitions under FAR part 15, an agency shall establish a competitive range “if discussions are to be conducted.” FAR 15.306(c). In establishing a competitive range, the agency must consider the offerors’ respective prices or costs in making its competitive range determination, and may not eliminate an otherwise technically acceptable proposal from the competition without considering price or cost. See Ryan P. Slaughter, B‑411168, June 4, 2015, 2016 CPD ¶ 344 at 6; Arc‑Tech, Inc., B‑400325.3, Feb. 19, 2009, 2009 CPD ¶ 53 at 3.

For each of the socio-economic category cutlines, the agency explained that it “did not further adjust the cutline and settled on this cutline to limit the number of offerors moving into Phase 2 for purposes of efficiency [in accordance with] FAR 52.215-1, while increasing the number of potential awards.” AR, Tab X, Final Cutline Methodology Memorandum at 6-7. As several protesters note, this FAR provision discusses efficiency in the context of establishment of a competitive range:

If the Contracting Officer determines that the number of proposals that would otherwise be in the competitive range exceeds the number at which an efficient competition can be conducted, the Contracting Officer may limit the number of proposals in the competitive range to the greatest number that will permit an efficient competition among the highest rated proposals.

FAR provision 52.215-1(f)(4).

The agency did not conduct discussions in this procurement. AR, Tab AA, SSD at 21. Despite the agency’s reference to efficiency in connection with FAR provision 52.215-1, we do not find that the establishment of the phase 1 cutlines constituted a competitive range, as that term is used in the provision or FAR section 15.306(c). Rather, the establishment of cutlines was part of a process set forth in the terms of the solicitation that provided for consideration solely of validated self-scoring points. See RFP at 174. In this regard, the RFP plainly stated that only the highest rated proposals would advance past phase 1 of the competition, and that the phase 2 and 3 evaluations--which were to consider pass/fail criteria, price, and technical approach--would take place only after the phase 1 evaluation. Id. at 174-76. The solicitation was therefore clear that cutlines would be established, and that the agency would do so without regard to the phase 3 criteria of price and technical merit. On this record, we find no basis to sustain the protests.

10. Use of the cutlines was a de facto nonresponsibility determination

Certain protesters argue that the establishment of the self-scoring point cutlines constituted improper de facto nonresponsibility determinations that required the agency to refer the proposals of unsuccessful small business offerors to the SBA for a certificate of competency (COC) determination. We find no merit to these arguments.

Under the Small Business Act, agencies may not find a small business nonresponsible without referring the matter to the SBA, which has the ultimate authority to determine the responsibility of small businesses under its COC procedures. 15 U.S.C. § 637(b)(7); FAR subpart 19.6; FitNet Purchasing Alliance, B‑410263, Nov. 26, 2013, 2014 CPD ¶ 344 at 6‑7. The SBA’s regulations require a contracting officer to refer a small business concern to the SBA for a COC determination when the contracting officer has refused to consider a small business concern for award of a contract or order “after evaluating the concern’s offer on a non-comparative basis (e.g., pass/fail, go/no go, or acceptable/unacceptable) under one or more responsibility-type evaluation factors (such as experience of the company or key personnel or past performance).” 13 C.F.R. § 125.5(a)(2)(ii); see AttainX, Inc.; FreeAlliance.com, LLC, B‑413104.5, B‑413104.6, Nov. 10, 2016, 2016 CPD ¶ 330 at 4; Coastal Envtl. Grp., Inc., B‑407563 et al., Jan. 14, 2013, 2013 CPD ¶ 30 at 4.

Here, the protesters contend that the application of the cutlines made a particular level of points a mandatory requirement for advancing past phase 1 of the competition. Such a mandatory requirement, in turn, represented a finding that small business offerors are not capable of performing the contract without possessing the experience or certifications necessary to claim those self-certification points. As explained above, however, we do not find that the establishment of the cutlines for the socio-economic categories made any particular criterion a mandatory requirement for advancement past phase 1 of the competition. Similarly, we do not conclude that the cutlines constituted a pass/fail evaluation. Instead, per the terms of the solicitation, proposals were evaluated on a comparative basis, and only the highest rated proposals advanced.

An agency’s comparative assessment of small business offerors’ proposals does not constitute a responsibility determination. See DA Defense Logistics HQ, B‑411153.3, Dec. 2, 2015, 2015 CPD ¶ 358 at 8. Where an agency excludes lower-rated, but technically acceptable proposals from award consideration, the agency is not required to refer the excluded proposals to the SBA for a COC determination. Zolon Tech, Inc., B‑299904.2, Sept. 18, 2007, 2007 CPD ¶ 183 at 8. On this record, we find no basis to sustain the protests.

In sum, we find that the agency cannot demonstrate that the phase 1 evaluation decisions were reasonable. The record shows that the agency attempted to use a 3‑filter analysis to identify with precision a specific number upon which to make each cutline. Although the agency made subsequent adjustments to the cutlines to reflect increased government-wide contracting goals, they were dependent on the figures produced by the 3-filter analysis.

As discussed above, the record provided by the agency and the agency’s responses to the protest do not demonstrate that the agency hard validated the self-scores for all proposals, and that such hard validated scores were the basis for its original or post-hoc calculations. Further, the agency’s explanations for the process by which it established the cutlines is unclear because they are based on contradictory explanations and post-hoc calculations and adjustments to the cutlines that are also not clearly explained.

We recognize the possibility that the data comprising the offerors’ self-scoring points and the 3-filter analysis technique selected by the agency might have yielded the same or substantially similar cutlines even if the agency performed a hard validation on all proposals. Again, however, neither the contemporaneous record nor the agency’s responses to the protests clearly explain how the agency validated proposals or established the cutlines for each socio-economic category. For these reasons we are unable to find, as the agency urges, that any errors in the calculations or lack of clarity about the data used for those calculations had no effect on the outcome of the phase 1 decisions.

In addition to these concerns, we find that the agency failed in its obligation to provide a complete and accurate response to the arguments made by protesters other than Karsun and Neev. Although other protesters clearly challenged the basis upon which the agency validated proposals and established the phase 1 cutlines, the agency responded with an inaccurate and misleading explanation that relied on the overly-simplified term “validated.” As the agency’s subsequent responses to Karsun and Neev explained, the agency’s explanation that it validated all self-scores was not accurate because the agency actually conducted two different processes, an “initial validation” and a “hard validation.” The agency’s responses to the other protesters deprived them of the opportunity to review and comment on an accurate version of the agency’s evaluation process.

For all these reasons, we sustain the protests challenging the phase 1 evaluation with regard to the validation of proposals and the establishment of the cutlines.

Part II Failure to Follow the Solicitation’s Award Criteria

Certain protesters argue that NIH’s establishment of the phase 1 cutlines was unreasonable because the agency used those cutlines as de facto award decisions, thereby departing from the RFP’s award criteria. In this regard, the protesters contend that the RFP anticipated a best‑value tradeoff award following the phase 3 evaluation, and that the agency abandoned this award basis by simply making award to all offerors whose proposals were above the phase 1 cutlines. The protesters argue that, had the agency followed the RFP’s award criteria, it would have set the phase 1 cutlines lower to allow for best‑value tradeoffs between a larger pool of proposals. For the reasons discussed below, we find that the protesters are not interested parties to pursue these arguments.

As discussed above, the RFP stated that proposals would be evaluated in three successive phases. RFP at 173. The phase 1 evaluation consisted of validating offerors’ self-scoring sheets and identifying the highest rated offerors. Id. at 174. The phase 2 evaluation assessed mandatory go/no-go criteria. Id. For phase 3, the RFP identified the following evaluation factors: (1) health IT capability; (2) management approach, which had three equally-weighted subfactors of program management, resources, and corporate commitment; (3) past performance, and (4) price. Id. at 176.

The RFP explained that award would be made on a “best value” basis, as follows:

The government will employ the source selection process described in FAR 15.101, which defines best value as using any one of a combination of source selection approaches. After the completion of phase 3, the government will use a selection methodology that awards contracts to offerors whose proposals represent the best value to the government at fair and reasonable prices.

Id. at 173.

The RFP identified the relative importance of the four phase 3 factors as follows: “Health IT capability is more important than management approach. Management approach is more important than price. Price is more important than past performance.” Id. at 178. The RFP stated that proposals would be assigned adjectival ratings under the health IT capability and management approach factors and subfactors of outstanding, good, acceptable, marginal, or unacceptable. Id. at 177-78. The past performance factor would be assigned one of the same ratings, with the additional possible rating of neutral. Id. at 178. After the factors and subfactors were rated, “[a]n overarching rating based on those subfactors will then be assigned to their parent factors[,]” and when “all factors have received their ratings, one single rating will then be assigned to the offeror’s proposal.” Id. at 177.

The protesters argue that the RFP required a tradeoff between the price and non-price factors because the RFP specified the relative weights assigned to those factors for the phase 3 evaluation. NIH contends that the RFP did not require a tradeoff, and instead gave the agency “flexibility” to employ any of the best value award techniques permitted in FAR section 15.101. Supp. MOL (B-419956.202 et al.) at 8. The agency states that the agency selected for award proposals that were the “highest rated” and that had “fair and reasonable” prices. Id. The source selection decision confirms that the agency made award to the offerors with the “highest rated proposals,” taking into consideration all three phases of the evaluation, that proposed “fair and reasonable” prices. AR, Tab AA, SSD at 21-22.

Generally speaking, the identification of evaluation factors with relative weights, along with the relative weight of price, implies that the agency will weigh the non-price factors against price in a tradeoff. See FAR 15.101-1(b) (“When using a tradeoff process . . . [a]ll evaluation factors and significant subfactors that will affect contract award and their relative importance shall be clearly stated in the solicitation.”). Thus, the solicitation’s provision of the relative weights of the phase 3 evaluation factors, including price, supports the protester’s interpretation of the RFP as providing for a tradeoff award decision.

Complicating this analysis, however, is the solicitation’s explanation that a proposal would be assigned a “single rating” for the non-price factors. RFP at 177. It is unclear how price would be weighed against a “single rating” for the technical proposal, because the past performance component of the single rating was less important than price. We therefore conclude that the solicitation’s provision of a single rating for the non-price factors supports the agency’s interpretation of the RFP as permitting award to proposals that offered the highest ratings at a fair and reasonable price.

For these reasons, we find the plain language of the solicitation, giving effect to all of its parts, does not clearly show how the agency was to select proposals for award. As noted above, the RFP’s award criteria referred to FAR section 15.101 and stated that this provision allowed for a range of possible best-value award techniques--but did not specify which technique would be used. RFP at 173. Although the phase 3 factors suggested two possible best-value award techniques--a price/technical tradeoff or highest-rated, fair and reasonable price--these approaches are inconsistent with each other.

An ambiguity exists when two or more reasonable interpretations of the terms or specifications of the solicitation are possible. See Ashe Facility Servs., Inc., B‑292218.3, B‑292218.4, Mar. 31, 2004, 2004 CPD ¶ 80 at 10. A patent ambiguity exists where the solicitation contains an obvious, gross, or glaring error, while a latent ambiguity is more subtle or nonobvious. Id. at 11. A solicitation is not ambiguous unless it is susceptible to two or more reasonable interpretations. LCLC Inc./CfMRF, B‑414357, May 22, 2017, 2017 CPD ¶ 153 at 5. A party’s interpretation need not be the most reasonable to support a finding of ambiguity; rather, a party need only show that its reading of the solicitation is reasonable and susceptible of the understanding that it reached. The HP Grp., LLC, B‑415285, Dec. 14, 2017, 2017 CPD ¶ 385 at 5. A solicitation provision that is patently ambiguous must be challenged prior to the time for receipt of initial proposals. 4 C.F.R. § 21.2(a)(1); J&J Worldwide Servs., B‑418148.3, June 30, 2020, 2020 CPD ¶ 312 at 5. A solicitation provision that is latently ambiguous must be challenged within 10 days of when the protester knew or should have known of the ambiguity. The Boeing Co., B‑311344 et al., June 18, 2008, 2008 CPD ¶ 114 at 37‑38 n.53.

Our Office has explained that where a solicitation provides for award on two different and conflicting bases, the solicitation is patently ambiguous.[27] KPMG LLP, B‑420949, B‑420949.2, Nov. 7, 2022, 2022 CPD ¶ 280 at 11. We need not, however, resolve whether the RFP is patently ambiguous or what the basis of award should have been because the protesters cannot demonstrate that they are interested parties to challenge this issue.

Our Office may only consider protests filed by interested parties. 31 U.S.C. § 3551(2); 4 C.F.R. § 21.1(a). An interested party is “an actual or prospective bidder or offeror whose direct economic interest would be affected by the award of a contract or by the failure to award a contract.” 4 C.F.R. § 21.0(a)(1). Determining whether a party is interested involves the consideration of a variety of factors, including the nature of the issues raised, the benefit or relief sought by the protester, and the party’s status in relation to the procurement. REEL COH Inc., B‑418095, B‑418095.2, Jan. 10, 2020, 2020 CPD ¶ 55 at 7. Generally, a party will not be deemed to have the necessary economic interest to maintain a protest if it would not be in line for award if its protest were sustained. Id.

The protesters argue that, had NIH followed the terms of the solicitation and made a best-value tradeoff award decision using the phase 3 criteria, it would have allowed more proposals to advance past phase 1 in order to have a broader range of choices for the best-value tradeoff decision. The RFP, however, did not specify where phase 1 cutlines would occur or the size of the pool of successful phase 1 offerors. More importantly, even if the protesters were correct that the RFP required the agency to conduct a best-value tradeoff, the RFP did not require the agency to establish a pool of offerors of any particular size. In this regard, the RFP did not prohibit the agency from setting a phase 1 cutline that created a pool of successful offerors, and then selecting all of those proposals for award on a best-value tradeoff basis. In other words, even if the protesters were correct as to the basis for award, we find no basis to conclude that the agency would have been required to set the phase 1 cutlines lower.

On this record, we find that the protesters are not interested parties to challenge the evaluation of proposals under the phase 2 or 3 evaluation criteria or the apparent award decisions based on those criteria. See 4 C.F.R. § 21.0(a)(1); 21.1(a). We therefore find no basis to sustain the protests.

Part III Challenges to Adjustments of Protester’s Self-Scores

Eight protesters challenge NIH’s adjustments to their proposed self-scores during the validation of proposals. For the reasons discussed below, we sustain the protest arguments by Sky Solutions, but find no basis to sustain any of the other protests. We discuss first our conclusions with regard to DecisionPoint to provide the appropriate background.

DecisionPoint

DecisionPoint Corporation argues that NIH improperly deducted self-scoring points from its claimed total because the agency used the wrong date to calculate the value of its experience references. The protester contends that the agency improperly deducted 390 points from its claimed total of 9,800 points, and had the agency not done so the protester would have advanced past phase 1 under the small business, VOSB, and SDVOSB categories. Protest (B‑419956.309) at 14-15. We find no merit to this argument.

The solicitation contained two different dates relevant to the evaluation of experience self-scoring points: (1) the date for consideration of an experience example, and (2) the date for calculating the value of the experience example. The RFP provided that experience examples “must be from the last three years prior to the date [] the solicitation was originally released (May 25, 2021).” RFP at 159. The protester’s challenge does not concern this date.

With regard to the date for calculating the value of experience examples, the initial RFP was issued on May 25, 2021, and stated that “[t]he dollar value of the corporate experience example is the total value of the contract including options.” AR, Tab A.2, Initial RFP at 164. On June 21, the agency issued RFP amendment 3, which incorporated questions and answers (Q&As), including the following concerning the calculation of the dollar value of leading edge technology experience:

Dollar values are calculated for Leading Edge Technology experiences, by calculating the obligated [value] up to the date of submission - obligated not contract ceiling, options, [not to exceed], etc. - dollar amounts for each experience. These experiences cannot have terminated more than 3‑years prior to the CIO-SP4 proposal close date.

AR, Tab D.8, RFP amend. 3, Response to Q&A No. 56, at 18. The due date for proposal submission was August 27, 2021.

Following the issuance of our Office’s decision in Computer World Services Corp.; CWS FMTI JV LLC, the agency issued RFP amendments 12 through 16 which revised certain parts of section L of the solicitation. As relevant here, RFP amendment 12 was issued on December 15, 2021, and permitted offerors to make changes to their experience examples, but only if those examples were “affected by the changes in section L.5.2.1 through L.5.2.4,” which concerned experience for firms competing as MPJVs. AR, Tab M.1, RFP amend. 12, Cover Letter at 1; Tab M.4, RFP amend. 12 at 146. These changes addressed the maximum number of experience examples that could be submitted by mentor firms and the minimum requirements for experience examples of protégé firms. RFP amend. 12 at 158-165. RFP amendment 12 set a deadline for proposal revisions of January 21, 2022. RFP amend. 12, Cover Letter at 1. This deadline was subsequently extended to January 28, then to February 11. AR, Tab P.1, RFP amend. 15, Cover Letter at 1.

The agency issued RFP amendment 16 on February 3, 2022, and affirmed that offerors whose proposals were affected by the revisions in RFP amendment 12 could submit revised proposals by February 11. AR, Tab Q.1, RFP amend. 16 Cover Letter at 1. For offerors not affected by the revisions, i.e., non-MPJV offerors, the agency stated the following: “Offerors who already submitted their proposals as of August 27, 2021 are only required to sign attachment J.9 Acknowledgement of Amendments and email a copy to [NIH].” Id.

On March 11, the agency issued a notice advising as follows: “In accordance with the solicitation, the agency will validate the scores to ensure the scores are only based on total obligated contract values within three years prior to May 25, 2021.” AR, Tab R, CIO-SP4 Notice, Mar. 11, 2022, at 1. This notice did not permit offerors to submit revised proposals.

On April 20, the agency issued a notice revising the guidance provided in the March notice, as follows:

Total obligated (funded) contract values for [all experience examples] . . . will be evaluated in accordance with the solicitation, i.e., total obligated (funded) contract values will be calculated by calculating the total obligated contract values up to the date of proposal submission. (see Amendment 0003, Question and Answer Number 56, page 18).

AR, Tab S, CIO-SP4 Notice, Apr. 20, 2022, at 1. The April notice effectively rescinded the change from the March notice. This notice also did not permit offerors to submit revised proposals.

DecisionPoint raises two primary arguments. First, the protester contends that the agency was required to calculate the value of experience examples as of February 11, 2022, the date for MPJV firms to submit revised proposals. Although the protester is not an MPJV offeror, it contends that it understood the RFP’s statement that values would be calculated “up to the date of proposal submission” to apply to the date for revised proposals. The protester therefore argues that the agency unreasonably used the initial proposal due date of August 27, 2021, to calculate the value of its experience examples.

Second, the protester contends that the agency must have treated MPJV offerors unequally from non-MPJV offerors because the protester’s interpretation of the appropriate date for calculating the value of experience examples should have led the agency to evaluate MPJV offerors’ revised proposals based on the February 11, 2022, due date. Assuming this to be the case, the protester argues that the agency treated it unequally as compared to MPJV offerors.

NIH contends that the solicitation provided that the value of experience examples must be calculated as of the date of initial proposal submission, i.e., August 27, 2021. MOL (B-419956.309) at 11. Although RFP amendments 12-16 provided for submission of revised proposals for MPJV offerors affected by changes in RFP amendment 12, the agency contends that the submission of a revised proposal did not affect the solicitation’s explanation that the value of the experience examples would be based on the August 27, 2021, date. Id.

We note that the Q&A in RFP amendment 3 referred to the self-scoring criteria for leading edge experience, and did not broadly refer to all experience examples, i.e., corporate experience, leading edge technology experience, federal multiple award experience, and EO 13779 work experience. Additionally, the language cited by the agency concerning the date of proposal submission was not present in RFP amendment 16, and was instead provided to offerors in the March and April 2022 notices.

Nonetheless, as of the April 2022 notice, offerors were on notice that the value of experience examples would be calculated on the following basis: “Total obligated (funded) contract values for [all experience examples] . . . will be evaluated in accordance with the solicitation, i.e., total obligated (funded) contract values will be calculated by calculating the total obligated contract values up to the date of proposal submission.” AR, Tab S, CIO-SP4 Notice, Apr. 20, 2022, at 1.

Prior to amendment 12, there was only one proposal submission date, August 27, 2021. RFP amendments 12-16 used the term “proposal submission” and did not state whether this meant initial or revised proposals. The April 2022 notice also did not specify whether “proposal submission” referred to the initial proposal due date of August 27, 2021, or the February 11, 2022, date for MPJV offerors to submit revised proposals. To the extent there was any lack of clarity about the dates, we find that there was, at best, a patent ambiguity that cannot now be timely challenged. See 4 C.F.R. § 21.2(a)(1); Ashe Facility Servs. Inc., supra; J&J Worldwide Servs., supra.

In any event, we do not think the protester reasonably interprets the term “proposal submission” to mean “revised proposals.” The agency explains that it did not apply the February 11, 2022, date to calculate the value of experience examples because to do so would have resulted in unfair and unequal circumstances. MOL (B-419956.309) at 11. In this regard, if all offerors experience examples were evaluated as of the February 11, 2022, date, only MPJV offerors would have been provided the opportunity to submit new examples. We agree with the agency that such circumstances would have been unfair, and that the agency therefore reasonably used the initial proposal submission date to calculate the value of proposals.

With regard to the protester’s argument that NIH evaluated MPJV offerors based on the February 11, 2022, date, and non-MPJV offerors based on the August 27, 2021, date, the agency expressly states that it did not do so; rather it used the August 27, 2021, date to calculate the value of all offerors’ experience examples. Id. at 12. To the extent the protester disputes this representation, the protester does not offer any specific evidence to support its arguments. On this record, we find no basis to sustain the protest.

Sky Solutions LLC

Sky Solutions LLC argues that the agency failed to reasonably evaluate its proposal because the agency’s validation of its self-score deducted 470 points from a total of 9,770 points, resulting in a validated self-score of 9,300 points. Consolidated Protest (B‑419956.269) at 17-21. We sustain this argument with respect to the agency’s deduction of 60 points from Sky Solution’s self-score for one particular task order.

The protester first argues that the agency improperly deducted points from its self-score for an order under a BPA issued by HHS for case and document management system (CDMS) requirements valued at $2.9 million. According to the protester, the order, which was completed in September 2017, was placed under a BPA with an ordering period that fell within the solicitation’s 3-year experience window. Id. at 18-19. The agency states that the BPA order was not considered because the order itself was not within the 3‑year window established in the solicitation, which ran from May 25, 2018, to May 25, 2021. See COS (B-419956.212, B-419956.269) at 8-9; MOL (B‑419956.205 et al.) at 25-26. The protester contends that the 3-year window consideration of experience examples did not apply to orders under BPAs, and that the entire ordering period of the BPA should be considered for purposes of determining whether the order under that BPA is within the 3-year window. We disagree.

The RFP provided that for IDIQ contracts or BPAs, an “experience example” can be “an order,” or a “collection of orders.” RFP at 158. Regardless of whether a BPA order is one order or a collection, the RFP specified that all “experience examples must be from the last three years prior to the date [] the solicitation was originally released (May 25, 2021).” Id. at 159, 161, 163, 164. On this record, we find that the protester’s disputed BPA order, which terminated in September 2017, was not within the RFP’s 3-year window of May 25, 2018, to May 25, 2021. We therefore conclude that the agency reasonably deducted the points from the protester’s validated self-score.

Sky Solutions next argues that the agency failed to properly calculate the value of its information technology support services (ITSS) task order with NIH. The protester contends that the agency improperly valued this order at $2.8 million, and deducted 60 points from the self-scored total of 150 points for this order. The protester further argues in its comments on the agency report that data from the Federal Procurement Database and included in its proposal shows that the obligated value of the order, as of the competed date of August 31, 2019, was $10.2 million. Comments (B‑419956.269) at 8. The protester asserts that the agency’s supplemental responses to the protester’s arguments did not address this argument regarding the ITSS task order.

We agree with the protester that the agency does not meaningfully explain the basis for the 60-point reduction to the protester’s validated self-score for the ITSS order. A 60‑point adjustment would increase the protester’s validated score to 9,360 points, leaving it well below the 9,770 point cutline for the small business category. Nonetheless, in light of our discussion in part I of the decision sustaining the protest based on the agency’s establishment of cutlines for phase 1 of the competition, we cannot conclude at this time that the protester is not prejudiced by this apparent evaluation error. We therefore sustain the protest.

Inserso Corporation and IS CIO JV

Inserso Corporation and IS CIO JV,[28] argue that for one of their experience examples, the Air Force Civil Engineer Center’s Information Technology Support Services (AFCEC ITSS) contract, submitted under the corporate experience and leading edge technology solicitation criteria, the agency improperly decreased their scores by over 1,000 points. Protest (B-419956.261) at 12‑16. In this regard, the protesters argue that because the AFCEC ITSS experience example involved performance under a requirements contract, it should have been scored according to the solicitation’s general rules for calculating self-scoring points, which would have permitted the protesters to claim points for “[t]he total obligated [funded] value of the AFCEC ITSS contract including exercised options[.]” Id. at 15 (citing RFP at 159) (emphasis added by protester).

As stated above, the RFP provided that for purposes of calculating self-score points, “[t]he dollar value utilized for experience in [the corporate experience and leading edge technology] sections [] is determined by the total dollars that were obligated (funded).” RFP at 158. The RFP further provided that for corporate experience examples, the dollar value is “the total obligated (funded) value of the contract including exercised options.” Id. at 159. With respect to IDIQ and BPA contracts, the RFP advised that “[e]xperience examples can be either a collection of orders or one single order placed under an IDIQ contract or BPA.” Id. at 158. For each of the RFP’s experience criteria at L.5.2.1-L.5.2.4, the RFP required “[a]ll [] examples must be from the last three years prior to the date [] the solicitation was originally released (May 25, 2021).” Id. at 159, 161, 163, 164.

IS CIO JV and Inserso Corporation submitted proposals under the OTSB and ELB socio-economic categories. AR, Tab DD.76.c, IS CIO JV Proposal Volume 1, Section 1 at 3; AR, Tab DD.79.b, Inserso Corporation Proposal Volume 1, Section 1 at 3. Under the L.5.2.1 corporate experience and L.5.2.2 leading edge technology criteria, the protesters provided as an experience example the AFCEC ITSS contract, a requirements contract performed over a time period spanning from April 2012 to August 2018. AR, Tab DD.76.e, IS CIO JV Proposal Self-Scoring Documentation at 438, 448. The protesters scored this example at 1,350 points for corporate experience and 600 points for leading edge technology, contributing to an overall self-score of 9,910 points for IS CIO JV, and 9,770 points for Inserso Corporation. Protest (B‑419956.261) at 9; Protest (B‑419956.264) at 9. NIH reduced IS CIO JV’s and Inserso Corporation’s overall self-scores to 8,710 points and 8,540 points, respectively, finding that though the protesters had submitted multiple “task orders” under the AFCEC ITSS requirements contract that together totaled in excess of $57 million, only “task orders with period of performance [] of 5/25/2018 through 5/25/2021 would be counted.” COS (B‑419956.261 et al.) at 9.

Inserso Corporation and IS CIO JV argue that a plain reading of the solicitation required NIH to validate all delivery orders placed under the requirements contract, regardless of the time they were performed. The protesters argue that a requirements contract should be evaluated under the solicitation’s general rule for calculating self‑scoring experience points, as all delivery orders were essentially “exercised options,” and at least one of the orders was performed during the allowable 3-year time period. See Supp. Protest (B-419956.261) at 5-6. The protesters’ further contend that because the solicitation contained a specific provision for orders placed under IDIQ contracts and BPAs, but not requirements contracts, requirements contracts must not fall under this provision.[29] See Protest (B-419956.261) at 15-16.

In the alternative, the protesters argue that, at best, the solicitation was latently ambiguous with respect to the treatment of requirements contracts. Comments (B‑419956.261) at 11. The protester contends that because its reading of the solicitation--that delivery orders under a requirements contract should be treated as exercised options--was reasonable, the agency should revalidate the protesters’ proposals utilizing their interpretation. Id.

NIH contends that requirements contracts--another type of indefinite-delivery contract--were “logically encompassed” by the provision covering IDIQ contracts and BPAs. MOL (B-419956.261 et al.) at 8. The agency argues that requirements contracts, like other delivery and task order contracts, do not contemplate the procurement of a firm quantity of supplies or services, and instead provide for the issuance of orders for delivery of supplies or services during the period of the contract. Id. Further, at the time of award of IDIQ contracts, BPAs, and requirements contracts, there is no obligated value funded to the contractor; instead, it is only when a delivery order is issued that the value of the contract is obligated. Id. at 8‑9. Thus, the agency argues that it interpreted the solicitation reasonably and in a manner consistent with the solicitation’s provisions advising that only orders performed within 3 years of May 2021 would be considered when it validated only one of the protesters’ delivery orders under the AFCEC ITSS requirements contract. Id. at 9.

Here, we find reasonable NIH’s interpretation of the solicitation and validation of the protesters’ proposals. The RFP provides both that experience examples must be from the 3 years prior to release of the solicitation, and that the dollar values used to calculate the number of self-score points based on experience examples would be determined by the total dollars that were obligated. RFP at 158. Despite not specifically addressing requirements contracts in the solicitation, we find no basis to question the agency’s decision to consider individual delivery orders under a requirements contract separately, in a similar manner to the orders in an IDIQ contract or BPA, because the orders are individually funded. The agency’s approach is also supported by the FAR, which states that a requirements contract is a type of indefinite-delivery contract, and is also known as a delivery-order or task-order contract, where the exact times or quantities of future deliveries are not known at the time of contract award. See FAR 16.501-2(a). As there was only one delivery order funded and performed within the requisite 3-year time frame, we find the agency acted reasonably and in a manner consistent with the solicitation when it considered only a single order, excluded other delivery orders from consideration, and based the number of points the protesters’ self-scores merited on the dollar amount obligated for the single order considered.

Further, we find unreasonable the protesters’ interpretation of the solicitation’s self‑scoring provisions, when the solicitation is read as a whole and in a manner giving effect to all of the solicitation’s provisions. Throughout the solicitation, the agency makes clear that there was a 3-year time period in which experience examples must have been performed; that is, performance must not have terminated prior to May 2018. When read as a whole with the rest of the solicitation, the RFP’s provision for IDIQ and BPA contracts provides that an offeror could claim self-scoring points for multiple orders performed under indefinite‑delivery type contacts, so long as those orders were performed in the requisite 3‑year time frame. In light of this language, we find the protesters’ interpretation of the solicitation to be unreasonable. In this regard, the protesters’ interpretation would require all delivery orders placed under a requirements contracts to be considered for purposes of calculating self‑scoring points, so long as one delivery order was performed in the requisite 3-year time frame. This interpretation is not supported by the solicitation. We therefore deny this protest ground.

Inserso Corporation next argues that the agency failed to properly calculate the value of its Defense Health Agency, Global Service Center (DHA GSC) contract by not considering multiple task orders performed under the single-award IDIQ contract. Protest (B‑419956.264) at 18-20. According to the protester, because the DHA GSC contract was a single-award IDIQ, the agency should have considered “the sum of all orders” placed under the IDIQ contract, regardless of whether they were performed within the 3-year time period established by the solicitation. Comments (B-419956.261 et al.) at 17-19.

The agency states that it reasonably adjusted the protester’s score based only on the task orders performed, and with funding obligated, during the requisite 3-year time period. MOL (B‑419956.261 et al.) at 14. We agree with the agency.

As discussed above, in order to be able to count points for an IDIQ or BPA order, the RFP required the orders to be performed between May 2018 and May 2021, and the monetary value of those orders would be based on funding obligated up to the August 27, 2021 deadline for receipt of initial proposals. Further, the relevant language of the RFP did not distinguish between multiple-award and single-award IDIQ contracts. See RFP at 158. On this record, we find that performance on the protester’s disputed orders terminated before May 2018, and therefore conclude that the agency reasonably deducted points from the protester’s validated self-score.[30]

AgilisTEK, OM Partners, and A-Tek

Three protesters, AgilisTEK, LLC, OM Partners JV 2, LLC, and A-Tek, Inc., argue that NIH improperly decreased their self-scores under the corporate experience criterion, specifically with respect to experience examples that were orders placed under IDIQ contracts or BPAs. In each protester’s case, the agency deducted points for IDIQ or BPA order experience examples after determining that the orders were performed or funds for the orders were obligated, outside of the timeframe established by the terms of the solicitation. The protesters all argue that the agency’s validation of self‑scoring in this regard was unreasonable and inconsistent with the terms of the solicitation, or in the alternative, that the solicitation was latently ambiguous with respect to how orders placed under IDIQ contracts or BPAs would be scored. Protest (B‑419956.311) at 11‑12. The protesters also argue that they were treated unequally to other offerors, because the agency used a “more restrictive” method for calculating the size of “umbrella contracts (i.e. IDIQs or BPAs) for purposes of the [c]orporate [e]xperience evaluation compared to traditional contracts.” Protest (B‑419956.310) at 14.

NIH argues that its validation of the protesters’ proposals was reasonable and consistent with the terms of the solicitation. In this regard, the agency contends that the RFP clearly provided that experience examples needed to be within the 3-year timeframe of May 25, 2018, through May 25, 2021, and that the total obligated dollar values of these examples, for purposes of claiming self-scored points, could be calculated up until the due date for receipt of proposals of August 27, 2021. MOL (B‑419956.310) at 2, 17. Further, the agency argues that with respect to IDIQ and BPA orders, the solicitation advised that though offerors could bundle orders for purposes of claiming experience points, the RFP still required that those orders complied with the established timeframe requirements. Id. at 15. The agency contends that there was no ambiguity in this regard, and that any other reading of the solicitation was unreasonable. Id. at 20. Finally, the agency argues that the protest ground alleging that it used a “more restrictive” method for calculating the size of IDIQ and BPAs amounts to an untimely challenge to the terms of the solicitation. Id. at 23.

As discussed above, in order to be able to count points for an IDIQ or BPA order, the RFP required the orders to be performed between May 2018 and May 2021, and the monetary value of those orders would be based on funding obligated through the August 27, 2021, deadline for initial proposals.

The agency decreased AgilisTEK’s score with respect to BPA orders that were both performed and had funding obligated in September 2021, past the respective deadlines.[31] AgilisTEK argues that the challenged orders should count towards its self‑score because the orders had funds obligated prior to the deadline for receipt of revised proposals in February 2022. The agency decreased OM Partners’s score with respect to orders issued under a multiple award contract because the orders were performed before the May 2018 beginning of the allowable period of performance. OM Partners contends that because between May 2018 and May 2021 at least some performance on the multiple award contract occurred, or some other orders had funds obligated, the challenged orders should also count towards its self-score. Finally, the agency decreased A-Tek’s score with respect to BPA orders that, though funded prior to August 27, 2021, were fully performed after the May 25, 2021, cutoff for performance established by the solicitation. A-Tek argues that the challenged orders should count towards its self-score because during the May 2018 through May 2021 timeframe it performed other orders issued under the underlying BPA.

Given our discussion above of the solicitation language and the fact that the RFP clearly established a window in which orders had to be performed, and explained the date up to which the value would be calculated, we find the agency’s point deductions reasonable and consistent with the solicitation language. The agency properly deducted points from each protester’s self-score where the order either was not performed, or had not been obligated funding, within the relevant time periods identified by the solicitation. We therefore deny these protest grounds.

Further, we find that the solicitation was unambiguous regarding NIH’s method for calculating points for experience examples submitted in the form of orders placed under IDIQ contracts or BPAs. The RFP consistently referred to the 3-year window in which experience examples had to be performed, including in the corporate experience solicitation criterion at issue here. When the solicitation is read as a whole and in a manner giving effect to all of its provisions, an interpretation of the solicitation that enables individual orders that were entirely performed and funded outside of the timeframes established is unreasonable. We therefore find the solicitation was not ambiguous in this regard, and deny these protest grounds.

Finally, we find the protesters’ argument that NIH’s method for calculating points with respect to IDIQ contracts and BPAs was improperly restrictive amounts to an untimely challenge to the terms of the solicitation. The solicitation clearly stated that it would treat orders issued under an IDIQ or a BPA differently than other contracts. Accordingly, the basis for any challenge to the reasonableness of this approach was apparent from the face of the solicitation and thus was required to be filed with our Office prior to the deadline for receipt of proposals in order to be considered timely.

MiamiTPSI LLC

MiamiTPSI LLC argues that the agency failed to credit its proposal with self-scored points for two contracts under task area 9, enterprise resource planning, resulting in an improper deduction of 300 points from its claimed total of 9,290 points. Protest (B‑419956.306) at 8. The protester argues that had the agency properly credited these points, its proposal would have advanced past the phase 1 cutline of 8,990 points for the ISBEE socio-economic category.[32] Id. We find no merit to this argument.

NIH explains that although the protester’s proposal claimed credit for 450 points in connection with task area 9, the proposal contained only one J.6 form that claimed self-scored points for that task area, with a maximum point value of 150 points. MOL (B‑419956.306) at 7. The agency therefore deducted 300 points from the protester’s self-score. Id.

MiamiTPSI LLC contends that its proposal claimed experience points for two contracts that were not credited by the agency in connection with task area 9: the Olympia Agile Release Train (ART) contract with the Department of Agriculture, and the Disaster Credit Management System (DCMS) contract with SBA. Supp. Comments (B‑419956.201 et al.) at 13-16. The protester’s J.6 forms for each of these contracts stated that the contracts “cover[] the following task areas,” identifying tasks 2, 3, 4, 5, 6, 7, 8, and 10. AR, Tab DD.112.c, MiamiTPSI Self-Scoring Sheet Documentation at 9, 42. Neither of the J.6 forms identified task area 9. Id.

The protester contends that, notwithstanding the fact that its J.6 forms for these two contracts did not state that the proposal sought credit for task area 9, the agency should have known that the protester intended to do so. The protester first notes that the RFP did not expressly instruct offerors to identify in their J.6 forms the task areas claimed for self-scored points. Supp. Comments (B-419956.201 et al.) at 15. While we agree that this instruction was not specifically provided, the protester does not explain how the agency was to know how an offeror sought to claim points for a contract referenced in a J.6 form without this information. Moreover, the protester did state in its J.6 forms the task areas for which it sought self-scoring credit; it simply failed to identify task area 9 on these J.6 forms. We do not find that the protester’s argument shows that the agency’s evaluation, which relied on the information provided in the protester’s J.6 forms, was unreasonable.

The protester next argues that the agency should have been aware of its intent to reference task area 9 for these two contracts because other narrative potions of the supporting documentation provided in its proposal included “sticky notes” that are used in Adobe PDF documents to add annotations. Id. at 15-16. The protester notes that certain of these sticky notes contained annotations which stated “Task 9: [enterprise resource planning].” See AR, Tab DD.112.c, MiamiTPSI Self-Scoring Sheet Documentation at 336, 489. As discussed above, the RFP required offerors to provide J.6 forms to support the self-scored points, and the protester submitted two J.6 forms that did not state that the referenced contracts were submitted for the purpose of claiming points under task area 9.

On this record, we do not find that the protester’s inclusion of sticky note annotations elsewhere in its proposal reasonably put the agency on notice that the protester intended its J.6 forms to claim points for task area 9.[33] In this regard, it is an offeror’s responsibility to submit a well-written proposal, with adequately detailed information which clearly demonstrates compliance with the solicitation requirements and allows a meaningful review by the procuring agency; an offeror that fails to do so runs the risk that the agency will, as here, unfavorably evaluate the offeror's proposal. Enterprise Servs., LLC, et al., B‑415368.2 et al., Jan. 4, 2018, 2018 CPD ¶ 44 at 7. We therefore find no basis to sustain the protest.

CONCLUSION AND RECOMMENDATION

We sustain the protests challenging NIH’s decision not to advance the protesters’ proposals past phase 1 of the competition because the record provided by the agency and the agency’s responses to the protests do not show that the agency (1) reasonably validated all offerors’ proposed self-scores, and (2) reasonably established the cutlines for the socio-economic categories. We also sustain part of Sky Solutions’ protest challenging the agency’s validation of its self-score.

We recommend that the agency reevaluate proposals consistent with the discussion above, in a manner that ensures that all offerors’ self-scores are validated, as required by the solicitation. Specifically, we recommend the agency:

• · Hard validate all self-scores, as the agency uses that term to denote a validation that complies with the terms of the solicitation and reflects all required evaluations and adjustments;
• · Reassess the arguments raised by Sky Solutions in part III of this decision, and make any required adjustments to its self-score in the agency’s validation of proposals;
• · Make new cutline analyses for each socio-economic category that are based on the hard validated self-scores for all offerors within those groups; and
• · Make a new determination of which proposals advance past phase 1 of the competition based on the results of these new evaluations and analyses.

We further recommend that the agency document the basis for its validation of proposed self-scores, establishment of the cutlines, adjustments to the cutlines, and the determination of which proposals advance past phase 1 of the competition. The agency should ensure that this documentation fully and clearly explains the basis for the agency’s evaluations and phase 1 decision.

We also recommend that the agency reimburse the protesters’ reasonable costs, including attorneys’ fees, associated with filing and pursuing their challenges to the validation of self-scores and establishment of the phase 1 cutlines, as addressed in the first and third arguments in part I of this decision. 4 C.F.R. § 21.8(d). This recommendation applies to all protesters who challenged the validation of proposals and the establishment of the cutlines.

We recognize that this recommendation applies to protesters other than Karsun and Neev, which were the protesters that identified the discrepancies in certain of the documents provided by the agency that prompted the agency’s revisions to its explanation for the evaluations. As discussed above, however, the agency’s responses to Karsun’s and Neev’s protests show that the agency’s initial responses to all of the protests that raised these arguments were incomplete and misleading with regard to the validation of proposals and establishment of the cutlines. Because the other protesters were not provided a record that allowed them to meaningfully pursue their grounds of protest, we conclude that recommendation of costs is appropriate here. Additionally, we recommend that the agency reimburse Sky Solutions LLC’s reasonable costs, including attorneys’ fees, associated with filing and pursuing its challenges to the validation of its proposed self-scores in part III of this decision.[34] 4 C.F.R. § 21.8(d).

The protesters’ certified claims for costs, detailing the time expended and costs incurred, must be submitted to the agency within 60 days after the receipt of this decision. 4 C.F.R. § 21.8(f).

The protests are sustained.

Edda Emmanuelli Perez
General Counsel

APPENDIX A – List of Protesters

The following firms challenge their exclusion from phase 1 of the competition conducted by NIH under RFP No. 75N98121R00001:

B-419956.184, B-419956.199, Systems Plus, Inc., of Rockville, Maryland

B-419956.185, B-419956.289, B-419956.320, CANN Softtech, Inc., of Herndon, Virginia

B-419956.186, B-419956.282, Dfuse Technologies, Inc., of Ashburn, Virginia

B-419956.187, B-419956.286, Red Oak Solutions, LLC, of Ashburn, Virginia

B-419956.188, B-419956.290, White Oak Solutions, LLC, of Paterson, New Jersey

B-419956.189, B-419956.292, ShorePoint, Inc., of Herndon, Virginia

B-419956.190, B-419956.293, JSSA, Inc., of Andover, Massachusetts

B-419956.191, B-419956.294, Knowledge Management, Inc., of Tyngsborough, Massachusetts

B-419956.192, 2050 Technology, LLC, of Westminster, Colorado

B-419956.193, JarWare, LLC, of Vienna, Virginia

B-419956.194, iDoxSolutions, Inc., of Bethesda, Maryland

B-419956.195, cFocus Software, Inc., of Largo, Maryland

B-419956.196, B-419956.319, SOFITC JV, LLC, of Piscataway, New Jersey

B-419956.197, B-419956.327, Spatial Front, Inc., of Bethesda, Maryland

B-419956.198, ImpactOne JV, LLC, of Bethesda, Maryland

B-419956.201, B-419956.334, Technology Solutions Provider, Inc., of Reston, Virginia

B-419956.202, B-419956.323, A1FedImpact, LLC, of Reston, Virginia

B-419956.203, B-419956.324, Saliense Consulting, LLC, of McLean, Virginia

B-419956.205, B-419956.330, Xfinion, Inc., of Bethesda, Maryland

B-419956.207, Hendall, Inc., of Rockville, Maryland

B-419956.208, Syneren Technologies, Corp., of Arlington, Virginia

B-419956.209, iVision, Inc., d/b/a iVision Consulting, Inc. of Washington, DC

B-419956.210, CWS FMTI JV, LLC, of Luray, Virginia

B-419956.213, B-419956.329, Astor & Sanders Corporation, of McLean, Virginia

B-419956.214, B-419956.328, Computer World Services Corporation, of Falls Church, Virginia

B-419956.215, B-419956.326, DevTech Systems, Inc., of Arlington, Virginia

B-419956.216, B-419956.325, Criterion Systems, LLC, of Vienna, Virginia

B-419956.219, B-419956.321, B-419956.342, Cyquent, Inc., of Rockville, Maryland

B-419956.220, B-419956.322, B-419956.340, Audacious Inquiry, of Catonsville, Maryland

B-419956.227, ICS-TSPi, LLC, of Reston, Virginia

B-419956.230, SRG-TSPi, LLC, of Stafford, Virginia

B-419956.231, Horizon Industries, Ltd., of Vienna, Virginia

B-419956.233, MASAI Technologies Corporation, of Frederick, Maryland

B-419956.234, CTIS, Inc., of Rockville, Maryland

B-419956.241, JCS Solutions, LLC, of Fairfax, Virginia

B-419956.242, TSC-ITG JV, LLC, of Reston, Virginia

B-419956.248, B-419956.336, B-419956.337, Karsun Solutions, LLC, of Herndon, Virginia

B-419956.252, Neev-KS Technologies, LLC, of Oakton, Virginia

B-419956.259, ASSYST, Inc., of Sterling, Virginia

B-419956.260, Platinum Business Services, LLC, of Clarksville, Maryland

B-419956.261, B-419956.314, B-419956.315, IS CIO JV, of Vienna, Virginia

B-419956.264, Inserso Corporation, of Vienna, Virginia

B-419956.268, Credence Dynamo Solutions, LLC, of Vienna, Virginia

B-419956.269, B-419956.317, Sky Solutions, LLC, of Chantilly, Virginia

B-419956.270, Blue Grove Solutions, LLC, of College Park, Maryland

B-419956.271, Ennoble First-Macro Solutions, LLC, of Chantilly, Virginia

B-419956.272, OCT Consulting, LLC, of McLean, Virginia

B-419956.273, Swain Online, Inc., d/b/a Swain Techs, of Horsham, Pennsylvania

B-419956.274, Katmai Management Services, LLC, of Anchorage, Alaska

B-419956.277, Capital Data Partners JV, LLC, of Silver Spring, Maryland

B-419956.278, Network Management Resources, Inc., d/b/a NMR Consulting, of Huntsville, Alabama

B-419956.279, mPower, Inc., of North Bethesda, Maryland

B-419956.280, ADG Tech Consulting, LLC, of Herndon, Virginia

B-419956.281, USmax Corporation, of Fairfax, Virginia

B-419956.283, Ripple Effect Communications, Inc., d/b/a Ripple Effect, of Rockville, Maryland

B-419956.284, B-419956.333, MicroTechnologies, LLC, of Vienna, Virginia

B-419956.295, B-419956.316, A Square Group, LLC, of Frederick, Maryland

B-419956.296, B-419956.341, eKuber Ventures, Inc., of Vienna, Virginia

B-419956.298, The Electric On-Ramp, Inc., of Houston, Texas

B-419956.306, MiamiTSPi, LLC, of Reston, Virginia

B-419956.309, DecisionPoint Corporation, of Gaithersburg, Maryland

B-419956.310, B-419956.345, AgilisTEK, LLC, of Bethesda, Maryland

B-419956.311, B-419956.346, OM Partners JV 2, LLC, of Reston, Virginia

B-419956.313, B-419956.347, A-Tek, Inc., of McLean, Virginia

 

---