ARiA

Decision

Matter of: ARiA

File: B-422365; B-422365.2; B-422365.3

Date: May 28, 2024

DIGEST

1. Request that protest be dismissed as outside the Government Accountability Office’s jurisdiction is denied where successful participation in the initial phases of a three-phase competition was not merely to result in cash prizes, but was also to serve as a prerequisite to compete for a small business innovation research direct to phase II contract award.

2. Protest challenging the aforementioned competition is dismissed where the various allegations are founded on an untimely challenge to the terms of the competition announcement and where the various additional allegations fail to establish a valid basis of protest.

ARiA, a small business of Madison, Virginia, protests the Department of the Army’s decision not to select its concept white paper pursuant to the Army’s “xTechScalable Artificial Intelligence (AI) Competition Announcement” for technology solutions. The competition announcement provided for three rounds of competition that ultimately may result in a small business innovation research (SBIR) direct to phase II contract award. The protester raises various challenges to the first round of competition.

We dismiss the protests.

BACKGROUND

The SBIR Program

By way of background, the SBIR program is conducted pursuant to the Small Business Innovation Development Act and section 9 of the Small Business Act, 15 U.S.C. § 638, which requires certain federal agencies to reserve a portion of their research and development funds for awards to small business concerns. See 15 U.S.C. §§ 638(e)(4), (f). The SBIR program has three phases. In phase I, firms submit proposals to conduct research, or research and development, on specified topics. Id. § 638(e)(4)(A). Phase I awardees may become eligible to compete for phase II awards which would involve prototyping, id. § 638(e)(4)(B), and phase III awards which would involve further pursuit of work derived from, extending, or completing efforts made under prior SBIR funding. Id. § 638(e)(4)(C).

The procurement at issue here concerns a particular type of phase II award--that is, direct to phase II awards, which are made under a congressionally approved pilot program. This program allows the Department of Defense the flexibility to award phase II SBIR contracts to small business concerns that complete the requirements of a phase I project, despite not having received a phase I award. 15 U.S.C. § 638(cc); see, e.g., PredictiveIQ LLC, B‑421436.2, July 7, 2023, 2023 CPD ¶ 186 at 2. In other words, as the Small Business Administration (SBA) explains, while receipt of a phase I award is usually “a pre-condition” to receive a traditional phase II award, the direct to phase II pilot program “provides agencies with an efficient and effective way to award non-dilutive capital to businesses with more mature technology than is appropriate for a phase I award.” SBA Comments at 4.[1]

The Procurement

On December 19, 2023, the Army posted a competition announcement[2] seeking eligible small businesses to submit concept white papers addressing “scalable techniques for adversarial AI,” to include “comprehensive models capable of defending against universal AI threat vectors.” Agency Report (AR), Tab 3, Competition Announcement at 2-3.[3] The competition announcement was posted by the Army xTech Program, which: “hosts Army prize competitions, connecting businesses with Army and [Department of Defense] experts to build solutions for current problems”; and “allows businesses to compete for cash prizes and potential follow-on contracts to accelerate and transition their transformative technology solutions into the Army.” COS/MOL at 1 n.2.

The competition announcement provided for three rounds of competition that ultimately may result in a direct to phase II contract award. Competition Announcement at 1. The authorities for the procurement, as cited in the competition announcement, included 15 U.S.C. § 638, as well as 10 U.S.C. § 4025 (Prizes for Advanced Technology Achievements) and 10 U.S.C. § 4022 (Prototype Projects). Id.

For the first round, the competition announcement provided instructions for small businesses to submit concept white papers using a tool called “Valid Eval”[4] and required all concept white papers to be submitted “using the template found on the Valid Eval registration page.” Id. at 3; AR, Tab 4, Concept White Paper Template. The competition announcement provided that the concept white papers would be evaluated and ranked considering the following scoring criteria: introduction, problem/solution, potential for impact, technology and concept viability, company viability, and proposal quality. Competition Announcement at 3-4. The competition announcement provided that the Army would select up to eight applicants to each receive a $10,000 cash prize and move on to the second round. Id. at 4.

The second round provided for the agency to select up to four applicants to each receive a cash prize and move on to the third round, in which those applicants could submit proposals for a direct to phase II award of up to $2 million each. Id. at 4-5. The competition announcement provided that the Army would issue “a separate announcement with detailed instructions” for those applicants’ “invitation to submit a direct to phase II Army SBIR proposal” at that time. Id. at 5.

On or before the January 18, 2024 initial closing date, the agency received 133 valid concept white papers, including one from ARiA. The agency evaluated the concept white papers, using Valid Eval consistent with the terms of the competition announcement, and selected the top eight applicants to proceed to the next round of competition. ARiA was ranked at 20th and was not selected. COS/MOL at 2, 7-8; see also AR, Tab 7, Scalable AI Analysis Report.

On February 15, the agency notified ARiA that it was “not selected to move on to part [two] of the competition,” and ARiA also received a detailed Valid Eval feedback report. AR, Tab 8, Notice of Non-Selection at 1. That same day, ARiA filed its first protest with our Office. The agency filed a request for dismissal based on lack of jurisdiction, which we declined, as discussed in detail below. After receiving the agency’s reports responding to its protest allegations, ARiA also filed two supplemental protests.

DISCUSSION

ARiA challenges the agency’s decision not to select its concept white paper in the first round of the competition. The protester argues the agency “took actions that were arbitrary and capricious by applying improper, inconsistent, and arbitrary criteria in the evaluation of ARiA’s proposal in a manner so as to uniquely disadvantage ARiA and bias and prejudice the contracting process against ARiA.” Protest at 1. In its various protest submissions, ARiA has raised arguments that are in addition to, or variations of, those specifically discussed below. While we do not specifically address all of ARiA’s arguments, we have considered all of them and find that they afford no basis on which to maintain the protest under our Bid Protest Regulations.

As discussed below, we conclude that ARiA’s protests are untimely and fail to state a valid basis of protest. Before we address those conclusions, we first address a procedural question of jurisdiction that was raised by the agency.

Jurisdiction

After ARiA filed its first protest, the agency filed a request for dismissal arguing that our Office lacks jurisdiction to review the protest. In the agency’s view, because the protester challenges the first round of this competition, “the contest at issue in ARiA’s protest involves the award of cash prizes rather than the award or proposed award of a contract.” Req. for Dismissal at 7, citing David Frankel, B‑408319, June 7, 2013, 2013 CPD ¶ 144 (dismissing protest challenging agency’s selection of winners of a prize contest because the transaction did not involve the award or proposed award of a contract).

Under the Competition in Contracting Act of 1984 and our Bid Protest Regulations, we review protests concerning alleged violations of procurement statutes or regulations by federal agencies in the award or proposed award of contracts for the procurement of goods and services, and solicitations leading to such award. See 31 U.S.C. §§ 3551(1), 3552; 4 C.F.R. § 21.1(a).

Here, the Army notes that the first round--as well as the second round--of the competition contemplated the award of cash prizes and explains that, as cited in the competition announcement, its authority to award cash prizes rests in 10 U.S.C. § 4025.[5] Req. for Dismissal at 6, citing Competition Announcement at 3‑4. The Army argues that the first and second rounds of competition were distinct from the third round of competition, which “will be conducted under a completely different statute (i.e., 15 U.S.C. § 638)” for its authority to award direct to phase II contracts. Req. for Dismissal at 9, citing Competition Announcement at 5.

By focusing its argument solely on the award of cash prizes, however, the agency ignores the other terms of the competition announcement. As noted above, the first round also contemplated selecting the top eight applicants to move forward in the competition. Moreover, through the successive rounds, the competition ultimately may result in a direct to phase II contract award. Competition Announcement at 3-5.

In other words, the Army’s position asks our Office to view the first round in a vacuum, separate from the entire procurement contemplated by the competition announcement. In contrast, SBA characterizes the first and second rounds of competition as appearing “necessary to identify the only entities eligible to submit a proposal for the Direct to Phase II contract award” and “to operate [in a manner] similar to a down-select or competitive range process that agencies may use to eliminate non-viable or low-scoring proposals from consideration for award.” SBA Comments at 5. We agree with SBA’s view.

In this context, the Army’s reliance on our decision in David Frankel, supra, is unavailing. Req. for Dismissal at 7. Under the unique circumstances of that decision, our Office concluded that we lacked jurisdiction to consider a protest challenging the results of a prize contest conducted by the Federal Trade Commission pursuant to the authority of the America COMPETES Reauthorization Act of 2010, 15 U.S.C. § 3719,[6] which did not involve the award or proposed award of a contract. See David Frankel, supra. In contrast, the competition announcement here contemplates a multi-phase procurement that ultimately may result in the proposed award of a contract.

Under these circumstances, we conclude that our Office has jurisdiction to hear the protest, and we declined to grant the agency’s request for dismissal and dismiss the protest at that time.[7]

Timeliness

While we declined to grant the agency’s request for dismissal based on a lack of jurisdiction, as discussed above, and after the protester filed two supplemental protests, we conclude that the various protest allegations are dismissible on other procedural grounds. As requested by our Office, the parties submitted supplemental briefings on some procedural questions that were raised through the development of the record.

First, the agency argues, and we agree, that ARiA’s protests are untimely. The crux of ARiA’s various allegations is its disagreement with the Valid Eval system used by the agency to evaluate submissions in this procurement. In its various filings, ARiA repeatedly contends that the agency used “an invalid test instrument” and questions various aspects of the agency’s “derelict . . . use of a software-assisted source-selection process,” including specific allegations about the processes, mechanisms, and algorithms used by Valid Eval, and the potential for an organizational conflict of interest posed by the role of Valid Eval as “a contractor acting as a nongovernmental advisor to the Agency.” See, e.g., Protest at 4; Supp. Protest at 21, 24, 28-29; 2nd Supp. Protest at 7.

As noted above, the competition announcement provided for the use of Valid Eval and required all concept white papers to be submitted “using the template found on the Valid Eval registration page.” Competition Announcement at 3; AR, Tab 4, Concept White Paper Template. As the agency explains, “[t]he evaluation criteria were published in the prize competition announcement and detailed information on these criteria . . . was provided in the concept white paper template and on the Valid Eval signup page, which was provided to all applicants and required to be used for a submission to be deemed valid for evaluation.” COS/MOL at 11. In a supplemental briefing to our Office, the agency explains, more specifically:

To gain access to the Valid Eval xTechScalable AI Competition registration page, applicants first had to agree to the “Valid Eval Usage Requirements” by accepting these requirements and the cookies associated with use of the webpage. The Valid Eval xTechScalable AI Competition webpage makes it obvious that the Valid Eval tool is being used as part of the competition process. Applicants using the registration page are clearly on notice that they are submitting their application using the Vali[d ]Eval tool.

Agency’s Supp. Briefing at 3‑4. The agency further notes that the Valid Eval registration page “provides a wealth of information about Valid Eval and what it does” in the evaluation process, such as “develop[ing] a robust rubric” and “partner[ing] with government agencies to set the standard for managing group evaluations at scale.”[8] Id. at 3-5 (emphasis omitted).

The protester does not dispute these facts and the agency’s characterization of the information in the competition announcement and readily available about Valid Eval. Instead, ARiA asserts that its protests are timely based on more specific information received through the development of the record. However, ARiA’s position is belied by its assertion that its protests are based on its “examination” of “the processes and procedures used by the Valid Eval software tool and the role of Valid Eval as a service provider based on information provided by the Agency.” Protester’s Supp. Briefing at 10.

Pursuant to our Bid Protest Regulations, a protest based on alleged solicitation improprieties that are apparent prior to the deadline for submitting proposals must be filed before that deadline. 4 C.F.R. § 21.2(a)(1); see, e.g., ASRC Fed. Data Sols., LLC, B‑417655 et al., Sept. 18, 2019, 2019 CPD ¶ 325 at 7 (“[i]t is well-settled that a party who has the opportunity to object to allegedly improper or patently ambiguous terms in a solicitation, but fails to do so prior to the time set for receipt of quotations, waives its ability to raise the same objection later”).

Our timeliness rules reflect the dual requirements of giving parties a fair opportunity to present their cases and resolving protests expeditiously without unduly disrupting or delaying the procurement process. A Squared Joint Venture, B‑413139, B‑413139.2, Aug. 23, 2016, 2016 CPD ¶ 243 at 8. More specifically, underlying our timeliness rules regarding solicitation improprieties is the principle that challenges which go to the heart of the fundamental ground rules by which a competition is conducted should be resolved as early as practicable during the solicitation process. Id.

Based on our review of the record, given that the competition announcement provided for the agency’s use of Valid Eval, ARiA’s attempts to “examin[e]” Valid Eval, at this point, constitute an untimely challenge to the terms of the competition announcement. ARiA’s challenge to the terms of the competition announcement is based on the use of Vaid Eval. As a representative example, ARiA questions the agency’s use of “scores computed by the Valid Eval software tool and the ranking of those scores provided by the Valid Eval software tool,” and argues for our Office to consider “statistical measures of interrater reliability [that] are used to assess whether a test instrument is valid.” Supp. Protest at 8-9; Comments at 8. In the protester’s view, it timely argued that “[t]he test instrument is not valid because the resulting ratings are random.” Protester’s Supp. Briefing at 12 (emphasis omitted). These specific arguments, again, are founded on the protester’s disagreement with the Valid Eval system used by the agency in this procurement.

As another example, ARiA argues it could only have developed the “empirical and methodological claims” in its protests based on “[t]he unfolding of new information in the course of the protest” provided by the agency. Protester’s Supp. Briefing at 10‑11. However, our decisions have repeatedly concluded that a protester need not await perfect knowledge before filing a protest. See, e.g., Peraton, Inc., B‑416916.11, Feb. 8, 2021, 2021 CPD ¶ 88 at 6, citing Valkyrie Enters., LLC, B‑414516, June 30, 2017, 2017 CPD ¶ 212 at 3‑4 n.2.

In this regard, the record shows that ARiA was, or should have been, aware prior to the January 18, 2024, closing of the competition announcement of the agency’s use of Valid Eval, i.e., the fundamental ground rules by which this competition was being conducted. Under these circumstances, we conclude that the protester, by waiting until February 15 and later to file its protests, raises an untimely challenge to the terms of the competition announcement. Accordingly, we dismiss the protests concerning the use of Valid Eval.

Failure to State a Valid Basis

Finally, to the extent ARiA raises other challenges to the agency’s first round evaluation and selection decision, we find that the protester has not established a valid basis for challenging the agency’s actions.[9]

Our Bid Protest Regulations require that a protest include a detailed statement of the legal and factual grounds for the protest, and that the grounds stated be legally sufficient. 4 C.F.R. §§ 21.1(c)(4), 21.1(f), and 21.5(f). These requirements contemplate that the protester will provide, at a minimum, either allegations or evidence sufficient, if uncontradicted, to establish the likelihood that the protester will prevail in its claim of improper agency action. Midwest Tube Fabricators, Inc., B‑407166, B‑407167, Nov. 20, 2012, 2012 CPD ¶ 324 at 3.

Under the circumstances presented here, the protester’s various filings do not include sufficient information to establish the likelihood that the agency violated applicable procurement law or regulations. While ARiA claims that it was “uniquely disadvantage[d]” in this procurement, Protest at 1, nowhere in its various filings does ARiA engage with the substance of its concept white paper and the evaluation and, critically, explain whether or how it merited higher ratings. For example, while ARiA draws quotes from the individual evaluator ratings of its concept white paper to question “[t]he wide variation on these scores across judges, which statistically cannot be distinguished from random assignments,” and argues that the “judges were not competent,” Protest at 5, 8, the protester does not explain what it proposed or how it should have been evaluated differently.

As another example, as noted above, for the first round of competition, the competition announcement provided that the concept white papers would be evaluated and ranked considering the following scoring criteria: introduction, problem/solution, potential for impact, technology and concept viability, company viability, and proposal quality. Competition Announcement at 3-4. Yet, ARiA does not explain whether and how its concept white paper addressed these criteria and what ratings it reasonably should have received.

Moreover, ARiA does not explain whether or how it merited high enough ratings to move forward in the competition given that, as noted above, the competition announcement provided for the first round selection of the top eight applicants to move forward in the competition, and the record shows that ARiA was ranked at 20th out of the 133 valid concept white papers received by the agency. Competition Announcement at 3‑5; COS/MOL at 7, citing AR, Tab 7, Scalable AI Analysis Report. In a supplemental briefing to our Office, ARiA generally contends that “a reasonable and rational evaluation of ARiA’s proposal and the resulting ratings and score has a reasonable possibility and substantial chance of ranking it among the top eight highest-scoring proposals.” Protester’s Supp. Briefing at 2. This general and conclusory statement, however, still does not remedy its failure to meaningfully challenge the agency’s actions.

Instead, ARiA repeatedly returns to its general claim that “the ratings are random and the result of chance, not a rational process.” See, e.g., 2nd Supp. Protest at 9. The protester’s assertions of improper agency action alone, without more, do not establish the likelihood that the agency violated applicable procurement law or regulations, and we dismiss the protests accordingly.

The protests are dismissed.

Edda Emmanuelli Perez
General Counsel

 

---