BNP Education Partners LLC d/b/a Marzano Research

DOCUMENT FOR PUBLIC RELEASE
The decision issued on the date below was subject to a GAO Protective Order. This redacted version has been approved for public release.

Decision

Matter of:  BNP Education Partners LLC d/b/a Marzano Research

File:  B-420247 

Date:  January 12, 2022   

DIGEST

Protest challenging evaluation of protester’s proposal and exclusion of proposal from the competitive range is denied where the agency’s evaluation was largely reasonable and there is no evidence that the protester was prejudiced by evaluation errors.  Further, the agency’s competitive range determination was reasonable and consistent with the solicitation.

BNP Education Partners LLC d/b/a Marzano Research (Marzano), a small business of Denver, Colorado, protests its exclusion from the competitive range under request for proposals (RFP) No. 91990020R0032 issued by the Department of Education for regional education laboratories.[1]  Marzano contends that the evaluators ignored or misread information in the firm’s proposal, applied unstated evaluation criteria, or otherwise evaluated in a manner inconsistent with the solicitation.

We deny the protest.

BACKGROUND

On February 2, 2021, using the procedures of Federal Acquisition Regulation (FAR) part 15, the agency issued the solicitation seeking “entities to serve as the Regional Educational Laboratory (REL) for nine of the 10 REL regions.”  Agency Report (AR), Tab A, Contracting Officer’s Statement (COS) at 3; Tab C, RFP at 527.[2]  The solicitation explained that the purpose of a REL “is to assist practitioners and policymakers in their work to improve outcomes for learners in [the REL’s] region from early childhood to adulthood by supporting stakeholders in the generation and use of research, evidence, and evidence-based practices.”  RFP at 527, 564.  To achieve this purpose,

RELs:  (1) conduct applied research and development; (2) design and implement training, coaching, and technical support activities that emphasize building capacity to use data and information to drive change; and (3) disseminate scientifically valid research, evidence-based practices, and supporting materials that allow stakeholders to apply this knowledge to their own practice.

Id. at 564.  The protester is an incumbent REL provider.  Protest at 1; see also AR, Tab E, Technical Evaluation Panel Consensus Memorandum (Tech. Eval.) at 11 (“Marzano has had success in collaborating with other education entities and partners through operating the REL Central [region] contract for two cycles.”).

The solicitation contemplated award of nine cost-plus-fixed-fee contracts, each with a 1‑year base period and four 1‑year option periods.  RFP at 527, 652.  At the time of the solicitation’s issuance, the agency estimated the combined value of the nine contracts to be approximately $23.4 million.  COS at 3.  The solicitation provided that the agency would make award using a best-value tradeoff methodology after establishing a competitive range and conducting discussions.  RFP at 652, 700.  The solicitation set forth six non-price evaluation factors, which in descending order of importance were:  (1) quality and credibility of the technical approach; (2) quality of the staff and leadership team; (3) quality of the management plan; (4) organizational capacities, experiences, and resources; (5) past performance; and (6) small business participation.  Id. at 700.  The non-price factors combined were significantly more important than price.  Id.

The solicitation provided that the evaluators would assign proposals adjectival ratings of excellent, satisfactory, marginal, or unsatisfactory for each non-price factor.  RFP at 708-710.  The agency received four proposals in response to the solicitation, including Marzano’s submission.  COS at 15.  The evaluators assessed Marzano’s proposal as follows:

	Marzano Rating	Strengths, Weaknesses, Significant Weaknesses, Deficiencies, and Risks
Technical Approach	Unsatisfactory	4 strengths, 6 weaknesses, 6 significant weaknesses, 3 deficiencies, 4 risks
Staff and Leadership Team	Satisfactory	7 strengths, 2 weaknesses, 2 significant weaknesses
Management Plan	Satisfactory	2 strengths, 2 weaknesses, 1 significant weakness
Organizational Capacities, Experience, and Resources	Excellent	3 strengths
Past Performance	Excellent	1 strength
Small Business Participation	Excellent	2 strengths

 

AR, Tab E, Tech. Eval. at 2-13.  Based on their assessment of Marzano’s proposal, the evaluators recommended eliminating the firm from the competitive range.  Id. at 2.  The contracting officer concurred in the recommendation, and excluded Marzano from the competitive range because its technical proposal had a rating of unsatisfactory for the technical approach factor.  COS at 16.  The contracting officer concluded that although Marzano’s proposal was assigned ratings of satisfactory and excellent for the other five factors, “these ratings were not enough to overcome” the proposal’s receipt of a “high‑risk” rating of unsatisfactory for the most important factor--technical approach.  AR, Tab H, Pre-Award Debriefing at 15.

After being notified of its elimination from the competitive range and receiving a pre‑award debriefing, Marzano filed this protest with our Office.  See AR, Tab F, Unsuccessful Offeror Notice; Tab H, Pre-Award Debriefing; Tab K, Contract Specialist Statement at 2; Tab L, Pre-Award Debriefing Transmittal Email. 

DISCUSSION

Marzano challenges the agency’s evaluation of the firm’s proposal under the three most important factors--technical approach; staff and leadership team; and management plan.  Protest at 10-35.  Specifically, Marzano challenges each of the weaknesses, significant weaknesses, deficiencies, and risks assessed in its proposal under these three factors.  Id.  Marzano argues that the evaluators ignored information in or misread the firm’s proposal, applied unstated evaluation criteria, or otherwise evaluated in a manner inconsistent with the solicitation.  Id.  Marzano acknowledges that, based on the record, “it is clear that the sole reason Marzano was removed from the competitive range was due to the Unsatisfactory rating under Factor 1 [technical approach] resulting from the unreasonably assigned significant weaknesses and deficiencies.”  Comments at 25-26; see also Protest at 2. 

As representative examples of the challenged evaluation assessments, we discuss the three deficiencies assessed in Marzano’s proposal under the most important factor--technical approach.  While we do not discuss Marzano’s challenges to the weaknesses and significant weaknesses assessed in its proposal, we have considered each of the protester’s arguments and find that none provides a basis to sustain the protest.

First Deficiency

As part of their proposed technical approaches, the solicitation required offerors to submit concept papers for partnership projects related to solicitation task 4--training, coaching, and technical support (TCTS) for evidence use--and subtask 5.1--conducting original and applied research and reporting research findings.  RFP at 662-663.  The solicitation provided that the agency would evaluate the “quality and credibility” of an offeror’s proposed technical approach based in part on whether a proposal demonstrated “a clear understanding of high-leverage state and district-level needs within the region and propose[d] reasonable and well-thought-out solutions about how to best serve those needs.”  Id. at 700, 702.  The agency also would assess whether an offeror’s proposed plan was “clearly written, coherent, well organized, and demonstrate[d] a commitment to detail and high-quality work.”  Id. at 702.  Further, the agency would examine if an offeror’s “[p]roposed approach for completing work in partnership” was “complete and feasible within the proposed timeline,” and provided “detailed, well‑thought-out discussions as to how the proposed work will have a demonstrable, credible, and direct relationship to partners’ short-, medium-, and long‑term outcomes, and how progress toward those outcomes will be measured.”  Id.

As relevant here, Marzano proposed a project with the South Dakota District Alliance focused on “[DELETED].”  AR, Tab D, Marzano Proposal at 31.  Marzano also proposed a project with the Wyoming District Partnership focused on “[DELETED].”  Id. at 57.  The record reflects that, in reviewing whether Marzano’s proposal reflected “a clear understanding of high‑leverage state and district-level needs within the region and propose[d] reasonable and well-thought-out solutions about how to best serve those needs,” the evaluators assessed a multi-part deficiency related to Marzano’s proposed South Dakota and Wyoming projects.  AR, Tab E, Tech. Eval. at 6-7.  Marzano challenges almost every aspect of the assessed deficiency.  See Protest at 23-26.  As discussed below, none of the challenges provides a basis to sustain the protest.[3]

In reviewing a protest challenging an agency’s evaluation of proposals, our Office will not reevaluate proposals nor substitute our judgment for that of the agency, as the evaluation of proposals generally is a matter within the agency’s discretion.  Environmental Restoration, LLC, B-417080, Feb. 5, 2019, 2019 CPD ¶ 155 at 4.  Rather, we will review the record to determine whether the agency’s evaluation was reasonable; consistent with the stated evaluation criteria, applicable procurement statutes and regulations; and adequately documented.  Id.; Cyberdata Techs., LLC, B‑417816, Nov. 5, 2019, 2019 CPD ¶ 379 at 4.  An offeror’s disagreement with an agency’s judgment, without more, is insufficient to establish that the agency acted unreasonably.  Environmental Restoration, LLC, supra at 4; Dictyon LLC, B-405504, Nov. 16, 2011, 2011 CPD ¶ 253 at 3.

  First Deficiency--South Dakota Project

The record reflects that the evaluators considered “the overall focus” of Marzano’s proposed South Dakota project to be “appropriately clear and narrow,” but found that it did not “appear to address a high-leverage need.”  AR, Tab E, Tech. Eval. at 6.  Specifically, the evaluators concluded that Marzano’s proposal did not “make a compelling case that there is any need related to [DELETED].”  Id.  Referencing a graph included in Marzano’s proposal, the evaluators noted that “BIPOC students already have fairly high participation rates:  Asian, Black, and Latino students participate at rates within 4 percentage points of white students, and students of 2+ races have higher rates than white students.  (American Indian/Alaskan Native students do have a large participation gap.).”  Id. citing AR, Tab D, Marzano Proposal at 33.  Further noting that “[e]ven with relatively similar CTE participation, BIPOC grad rates are still lower than white students,” the evaluators concluded that, based on the information in the proposal, it did “not appear that [DELETED] is a high-leverage driver for increasing graduation rates.”  AR, Tab E, Tech. Eval. at 6.

Marzano argues that the “evaluators seem to have imposed their own judgments about what constitutes high leverage instead of reviewing whether the requesting stakeholders consider an issue of high leverage.”  Protest at 23‑24.  Specifically, Marzano points to a letter of interest in the project from eight South Dakota stakeholders (i.e., state agencies, institutions, and school districts).  Id. at 24 citing AR, Tab D, Marzano Proposal at 435‑436.  Marzano also points to information in its proposal indicating that “stakeholders in South Dakota identified this need as high leverage, based on prior research conducted in the state,” which showed that “compared to their peers, CTE concentrators were more likely to graduate from high school on time, enroll in a postsecondary institution, and earn a postsecondary award within 5 years of graduation.”  Protest at 24 citing AR, Tab D, Marzano Proposal at 32.  Marzano maintains that “the 15-percentage point [CTE participation] gap for American Indian/Alaskan Native Students is compelling, even if the other groups do not have as wide of a gap.”  Protest at 24 citing AR, Tab D, Marzano Proposal at 33.  The agency cites to the contemporaneous evaluation record, and contends that the evaluators reasonably assessed Marzano’s proposal.  Memorandum of Law (MOL) at 14.

Contrary to the protester’s assertions, our review of the record does not reflect that the evaluators’ assessment was a substitution of their judgment for that of the proposed partners regarding what constitutes a high leverage need.  Rather, the evaluators noted that the proposal failed to provide an adequate explanation of a connection between, what the protester acknowledges are not wide gaps in CTE participation for BIPOC student groups other than American Indian/Alaskan Natives, and the stakeholders’ high leverage need of increasing graduation rates among BIPOC students.  AR, Tab E, Tech. Eval. at 6. 

It is an offeror’s responsibility to submit a well-written proposal, with adequately detailed information that clearly demonstrates compliance with the solicitation requirements, and an offeror risks having its proposal evaluated unfavorably where it fails to submit an adequately written proposal.  Environmental Restoration, LLC, supra at 9.  Further, the solicitation here expressly provided that “[a]ll information necessary to evaluate the technical soundness and management capabilities of the offeror will be contained in the technical proposal,” and cautioned that the evaluators would “not make assumptions or guess.”  RFP at 655-656.  On this record, we have no basis to conclude that the agency erred in its evaluation. 

  First Deficiency--Wyoming Project

With respect to Marzano’s proposed Wyoming project, in assigning the deficiency, the evaluators noted that “the overall focus is again appropriately narrow (implementing personalized competency-based education in selected districts), but the proposal’s description of the background context and need for this partnership is confusing and hard to follow; and the specific need related to personalized competency-based education (PCBE) in the specific districts is not clear.”  AR, Tab E, Tech. Eval. at 7.  Marzano argues that the agency ignored information in its proposal explaining that “to address challenges and inequities associated with the move to remote and blended learning as a result of the coronavirus pandemic,” Wyoming is developing a digital learning plan “with a renewed emphasis on PCBE, as the state has shifted to remote and blended learning in response to COVID‑19.”  Protest at 24-25 citing AR, Tab D, Marzano Proposal at 59.

Based on our review of the proposal’s description of the background context and need for the project, we have no basis to question the evaluators’ assessment that the proposal was not clear.  See AR, Tab D, Marzano Proposal at 57-61.  Nor is Marzano’s disagreement with the evaluators’ judgment, without more, sufficient to establish that the assessment was unreasonable.  Environmental Restoration, LLC, supra at 4.

  First Deficiency--Proposed Project Completion Exceeded Contract Term

The evaluators also noted that both Marzano’s South Dakota project and its Wyoming project “include[d] long-term outcomes that are labeled as ‘aspirational’ and are very unlikely to be achieved within the 5-year REL contract [period].”  AR, Tab E, Tech. Eval. at 7.  The evaluators found the inclusion of these outcomes to be contrary to the solicitation’s requirement that “long-term outcomes should be achievable within the 5‑year contract period.”  Id. citing RFP at 580 (“While acknowledging that partners’ long‑term goals of interest may not be achievable within the timeline of the REL cycle, RELs should identify targets for their projects’ long-term outcomes that are (1) consistent with the partner’s long-term goal but (2) attainable within the period of performance.”).

Marzano acknowledges that it included its partners’ “north star” goals in the long-term outcomes sections of its logic models for the South Dakota and Wyoming projects, but contends it did so “to show how they serve as a ‘north star’ for the REL work,” as required by the solicitation, and that they were “not to be measured during the five years of the contract.”  Protest at 25 citing RFP at 570;[4] see also AR, Tab D, Marzano Proposal at 38, 65.

As noted above, it is an offeror’s responsibility to submit a well-written proposal.  Marzano acknowledges that its proposal included aspirational “north star” goals in the long-term outcomes sections of the logic models for its South Dakota and Wyoming projects.  While Marzano expresses its belief that its proposal sufficiently conveyed that these aspirational “north star” goals were not to be measured in the same fashion as the other long-term outcomes included in these sections of the logic models, the evaluators concluded otherwise.  Based on the record, we have no basis to question the evaluators’ conclusion related to this sub-element of the first deficiency.  Nor does the protester’s disagreement with the conclusion provide such a basis.

Also related to this part of the first deficiency, Marzano takes issue with the evaluators’ assessment that the proposed logic models for its South Dakota and Wyoming projects failed to include “a baseline metric to identify the key need and measure their overall progress.”  Protest at 25 citing AR, Tab E, Tech. Eval. at 7.  Marzano points to information in the proposal’s narrative descriptions of the two projects that provides the baseline metrics, and contends that separate inclusion of the baseline metric in the logic model was not required by the solicitation.  Protest at 25. 

Our review of the record reflects that Marzano’s proposal did include baseline metrics for both its proposed South Dakota project and its proposed Wyoming project.  AR, Tab D, Marzano Proposal at 33, 59.  Additionally, the agency does not identify, nor could we find, a requirement in the solicitation for offerors to include baseline metrics specifically in the logic models--as opposed to the narratives--for their proposed projects.  See MOL at 14; RFP at 663 (requiring submission of “[a] logic model and narrative” for each proposed project that includes “a baseline metric,” but not delineating whether the metric should be in the logic model, narrative, or both).  Accordingly, we cannot conclude that the evaluators reasonably assessed this sub‑element of this part of the first deficiency.

Similarly, Marzano takes issue with the evaluators’ finding that the firm’s logic model for its proposed Wyoming project was “missing short-term outcomes for multiple projects/activities.”  Protest at 25 citing AR, Tab E, Tech. Eval. at 7.  In this regard, Marzano (1) represents that some of the proposed activities contribute only to medium or long-term outcomes; (2) argues that the solicitation provided offerors with flexibility in the type of outcomes they proposed; and (3) contends that the agency “held Marzano to an improper standard” by requiring short-term outcomes to be associated with each activity within a project.  Protest at 25 citing RFP at 579.  In its report responding to the protest, the agency does not identify a solicitation requirement for short-term outcomes to be tied to each activity for a proposed project, nor does it contend that Marzano’s reading of the solicitation’s provision related to offeror flexibility in defining outcomes is unreasonable or otherwise improper.  See MOL at 14.  As such, we cannot conclude that the evaluators reasonably assessed this sub‑element of the first deficiency.

  First Deficiency--Proposed Measurement Plan

The evaluators found issues with “[t]he measurement plans for both” Marzano’s South Dakota and Wyoming projects.  AR, Tab E, Tech. Eval. at 7.  For Marzano’s South Dakota project, the evaluators noted three separate issues with the proposed measurement plan.  The evaluators first concluded that the project will not result in the ability “to measure whether partners have skills to do CART [classification and regression tree] analysis on their own.”  Id.  The evaluators next found that the firm’s “measurement plan shows that they expect the medium term outcomes to be measurable in 2024 so they won’t likely achieve the long term outcomes one year later in 2025 (also not clear why they are measuring long-term outcomes in year 4 of the contract).”  Id.  Third, the evaluators expressed concern that “using the results of the research studies to identify CTE program modifications in 2024 isn’t appropriate as a short term outcome and the second study won’t be done until 2025 or 2026.”  Id.

Marzano maintains that its proposed activity “uses the results of research conducted at the district level (CART analyses) in 2022 and not the results of the study,” and points, not to its proposal, but to a solicitation provision that indicated “short-term outcomes can include partners ‘identifying a specific practice that they wish to adopt or adapt.’”  Protest at 26 citing RFP at 579.  Marzano does not explain, and we are unable to discern from the protest pleadings, how the cited solicitation provision relates to the evaluators’ findings or the protester’s challenge to them.  See Protest at 26; Comments at 17.  Based on the record here, we have no basis to question the reasonableness of the evaluators’ assessment of these sub-elements of this portion of the first deficiency. 

Similarly, for Marzano’s Wyoming project, the evaluators noted four separate issues with the proposed measurement plan.  AR, Tab E, Tech. Eval. at 7.  The evaluators found that the proposal was not clear regarding “why the outcome of understanding what is needed to conduct RCEs [rapid-cycle evaluations] will be measured in 2024 when the related project will end in the middle of 2023,” and that the proposal was unclear regarding “why the long-term outcomes are being measured in 2025.”  Id.  The evaluators also concluded that because the proposed research study will not “be done until the end of the contract” it is unclear “how it will lead to the [proposed] medium-term outcomes.”  Id.  Additionally, the evaluators expressed concern related to the firm’s proposed long-term outcomes for its Wyoming project that in order “to measure an increase in reports of student ownership and engagement over time you would need to administer that survey more than once.”  Id. 

Marzano maintains that “the medium-term outcome is that Wyoming develops a system of supports to implement PCBE--the study can help inform continued adjustments to the system of supports.”  Protest at 26; Comments at 17 citing AR, Tab D, Marzano Proposal at 64-65.  While Marzano’s proposal does suggest such a medium-term outcome, Marzano has not explained how the inclusion of this outcome in the proposal relates to or renders unreasonable the evaluators’ concerns regarding how this proposed outcome would be facilitated by a research study that will not be done until the end of the contract.  See id.  In our view, this protest argument amounts to nothing more than Marzano’s disagreement with the agency’s evaluation of the firm’s proposal, which is insufficient to establish that the agency acted unreasonably.  Environmental Restoration, LLC, supra at 4, 7.

Marzano also argues that “the Agency erroneously assumes that Marzano proposed to only administer the survey once.”  Protest at 26; Comments at 17 citing AR, Tab D, Marzano Proposal at 65, 77-78.  Marzano’s proposal provides that the firm will administer the survey “[DELETED],” and that the firm “[DELETED].” AR, Tab D, Marzano Proposal at 79.  While we cannot say it would be unreasonable to read the proposal’s reference to refining the surveys as indicating an intent to re‑administer the surveys after making any refinements, the proposal does not specify that this was Marzano’s plan.  The proposal also did not specify how many times Marzano proposed to administer the survey, or that the survey would be administered more than once.  See AR, Tab D, Marzano Proposal at 65, 77‑78. 

While Marzano expresses its belief that its proposal sufficiently conveyed the firm’s intent to administer its proposed survey more than once, the evaluators found otherwise.  Based on our review of the record, we cannot conclude that the evaluators’ reading of the proposal was unreasonable.  Consequently, we have no basis to question the reasonableness of the evaluators’ assessment of this sub-element of this part of the first deficiency. 

Second Deficiency

The second deficiency assessed by the evaluators in Marzano’s proposal again relates to the firm’s proposed South Dakota and Wyoming projects, both of which addressed Marzano’s approach to providing partners with TCTS support.  The solicitation provided that the agency would evaluate whether offerors’ “[p]roposed approaches to TCTS demonstrate an ability to develop and execute high-quality TCTS in partnership.”RFP at 702.  The record reflects that the evaluators concluded Marzano’s proposal did “not describe intensive enough support to achieve the intended capacity building outcomes for the participants,” in either the firm’s South Dakota or its Wyoming project.  AR, Tab E, Tech. Eval. at 7. 

  South Dakota Project

With respect to Marzano’s South Dakota project, the evaluators noted that the firm proposed “to teach participants [DELETED]--all in [DELETED] total contract hours over [DELETED] coaching sessions.”  AR, Tab E, Tech. Eval. at 7.  The evaluators did not consider this to be “enough time spent on each topic, or enough opportunity to practice the new skills, to reasonably expect the partners to learn them.”  Id.  Similarly, for Marzano’s Wyoming project, the evaluators noted that the firm:

aims to teach participants [DELETED]--all in [DELETED] total contract hours over [DELETED] sessions.

Id.  Again, the evaluators found that this “is far too much information, and far too technically complex information, to expect non-researchers to learn how to do in such a short project.”  Id.

Marzano maintains that “[t]he time allotted with additional work time between meetings is enough time to improve partners capacity by engaging them in the ‘design, execution, and evaluation of their activities[.]’”  Protest at 26.  Marzano further contends that it only proposed “participants develop skills to conduct CART analyses, not that they are able to do this on their own.”  Id.  Additionally, with respect to its South Dakota project, Marzano represents that its proposal “did not say that [the firm] is going to teach participants how to develop logic models or teach participants how to develop research plans,” rather, Marzano said it would “guide” participants through such development.  Id. citing AR, Tab D, Marzano Proposal at 41 (“[DELETED].”). 

The record reflects, however, that the logic model for Marzano’s South Dakota project included a short‑term outcome that “District partners develop skills to conduct CART analysis to inform program decisions.”  AR, Tab D, Marzano’s Proposal at 38-39.  Marzano does not explain, and we do not see, a distinction between a training resulting in participants having “skills to conduct” a CART analysis and a training resulting in participants being able to conduct a CART analysis “on their own.”  Based on the record here, we find no basis to question the evaluators’ conclusion that [DELETED] hours is an insufficient amount of instruction time to result in participants for the South Dakota project successfully developing the skills proposed as one of Marzano’s short-term outcomes.  

  Wyoming Project

With respect to its Wyoming project, the agency assessed this deficiency based on the evaluators’ conclusion that the protester proposed to teach participants the skills needed to conduct RCEs on their own.  Marzano disagrees with this assessment and argues that its proposal “did not say [the firm] is aiming to teach the participants how to develop theories of action; develop research questions; assess the availability and quality of data . . . etc.  Rather, Marzano stated that it will lead the participants through a process to do all of these activities.”  Protest at 27 citing AR, Tab D, Marzano Proposal at 77-78.  Marzano also points out that the logic model for its Wyoming project does not include an outcome providing “that the partners will learn to do all of this.”  Id. 

The record reflects that the short-term outcomes included in the logic model for Marzano’s Wyoming project are that the partners “develop an understanding of the research base of PCBE” and “of what is needed to conduct RCEs.”  AR, Tab D, Marzano Proposal at 65-67.  In its report responding to the protest, the agency does not explain why, for example, the evaluators read Marzano’s proposal to lead the participants through a process of developing research questions for a specific RCE as equating to Marzano proposing to teach the participants the skills necessary to conduct RCEs on their own.”  See MOL at 14-16.  In our view, leading participants through the development of research questions for an RCE is not the same as training participants to execute an RCE.  As a result, based on our review of the record, and without explanation by the agency to justify its determination that these two objectives are similar, we cannot conclude that the evaluators reasonably assessed this portion of the second deficiency related to Marzano’s Wyoming project.

Third Deficiency

The third deficiency assessed in Marzano’s proposal relates to its proposed “toolkit.”  The solicitation required offerors to propose a project for development of a “practitioner-friendly” toolkit designed “to help educators use evidence-based practices in classrooms.”  RFP at 606, 664.  As defined by the solicitation, a toolkit consists of three parts:  (1) initial diagnostic and on-going monitoring instruments; (2) professional development resources; and (3) steps for institutionalizing supports for evidence-based practice.  Id. at 607-608.  The solicitation explained that a successful awardee’s development of its toolkit would involve “two forms of evaluation”--one to “evaluate the usability of the tool with its intended audience,” and a second to “evaluate the efficacy and implementation of the finished toolkit’s Professional Development Resources.”  Id. at 609-610.  The solicitation established that “[b]oth the toolkit, including usability testing, and evaluation of the toolkit must be completed within the five-year contract period,” and “on a timeline that would allow for the draft evaluation report to be submitted to [the agency] no later than September 2025.”  Id. at 610-611.

The solicitation established that an offeror’s proposed toolkit project “shall reflect a realistic development plan,” and “should provide a detailed description of the product and how it will be developed.”  RFP at 664-665.  The solicitation provided a template offerors were required to use to submit information about their toolkit project.  Id. at 666, 685-687.  As relevant here, the template required offerors to “[d]escribe the partner(s) that will be involved in the development of the toolkit and their role throughout the process.”  Id. at 686.  Also relevant, the template included a section titled “Timeline, Deliverables, and References,” that required offerors to provide “dates for submitting the proposal, key milestones in toolkit development, submitting the usability memo, and key evaluation milestones.”  Id. at 687.  The formatting of this section in the template includes both a narrative paragraph bullet and a graphic table for offerors’ proposed timelines.  Id.  With respect to evaluation of offerors’ proposed toolkits, the solicitation set forth that the agency would assess, in part, whether a proposal “[c]learly and completely addresse[d] each component outlined in the Instructions to Offerors,” and whether it included “clear, feasible plans and timelines for the development of the toolkit (including usability testing) and related evaluation that are informed by a well‑thought-out ‘premortem’ of the proposed work.”  Id. at 703.

The record reflects that the evaluators considered Marzano’s description of how its toolkit would be developed to be “very vague” and to have left out “a lot of important information.”  AR, Tab E, Tech. Eval. at 8.  Specifically, the evaluators expressed concerns with Marzano’s description of its proposed toolkit development and its proposed toolkit implementation.  Id.

  Toolkit Development

Regarding development of the toolkit, the evaluators found that Marzano’s proposal did “not describe the development process or timeline.”  AR, Tab E, Tech. Eval. at 8.  Marzano does not dispute that it did not discuss the timeline for its toolkit in the narrative section of its proposal.  Protest at 27.  Rather, Marzano contends that “[t]he Solicitation did not require the timeline to be described within the narrative section,” and points to a “Timeline and Deliverables” graphic table in its proposal, which it maintains satisfied the solicitation requirements.  Id. citing AR, Tab D, Marzano Proposal at 153‑154.  The protester’s contention is not supported by the record.  As noted above, the solicitation required offerors to submit their toolkit projects using a designated template that included both a narrative and graphic description section for an offeror’s proposed timeline.  RFP at 687.  Accordingly, we find that the evaluators reasonably assessed this element of the third deficiency.

The evaluators also concluded that Marzano’s proposal did not describe “what the roles of the [DELETED] development team or [DELETED] advisors will be.”  AR, Tab E, Tech. Eval. at 8.  Additionally, the evaluators noted it was “unclear how involved the development team will be in the development or how the offeror will marshal the feedback of so many people.”  Id.  Marzano again argues that the solicitation did not require this information to be provided.  Protest at 27.  Marzano also contends that through the use of “revision lines” in its timeline table, its proposal showed that Marzano would have “continued interaction with” its partners and that it would be “interacting consistently over those periods of months as a partnership to accomplish the work.”  Id. citing AR, Tab D, Marzano Proposal at 153-154. 

This argument also is not supported by the record.  As noted above, the solicitation expressly required offerors to “[d]escribe the partner(s) that will be involved in the development of the toolkit and their role throughout the process.”  RFP at 686.  Neither the portion of its proposal cited by Marzano in support of its argument, nor any other portion we reviewed, includes the required description of the role of each partner throughout the toolkit development process.  See generally AR, Tab D, Marzano Proposal at 109-155, and specifically at 134-135.  Therefore, we find that the evaluators also reasonably assessed this element of the third deficiency.

The evaluators further noted as an issue of concern that Marzano’s proposal did not describe “how usability testing results will inform development or revisions, or why revisions after usability testing are expected to take 10 months to incorporate.”  AR, Tab E, Tech. Eval. at 8.  Marzano maintains that “usability testing was planned to align with resource development, so the 10-month period aligns with the projected rollout of specific resources.”  Protest at 27 citing AR, Tab D, Marzano Proposal, at 125-126.  The section of its proposal cited by Marzano in support of its argument describes the professional development resources that will be included in its toolkit, not the usability testing process or timeline.  AR, Tab D, Marzano Proposal at 125-127.  Based on the record here, we have no basis to question the evaluators’ assessment of this element of the third deficiency.

  Toolkit Implementation

Regarding implementation of the toolkit, the evaluators found that Marzano did not

explain what teachers will be ‘testing’ in the improvement cycles, what kinds of supports they may get from their school leaders, whether teachers are working independently on their own action plans and improvement cycles or whether that is collaborative, how often teachers meet to use the PD [professional development] resources, or what specific content is taught in the toolkit. 

AR, Tab E, Tech. Eval. at 8.  Marzano maintains that it “described and planned for developing the toolkit and process with its partners to increase usability and feasibility,” and that its approach “aligns with the Solicitation requirement to involve stakeholders.”  Protest at 28 citing RFP at 666.  The page of the solicitation cited by Marzano in support of its argument required offerors to include in their proposals “[a] key stakeholder letter from a leader or decisionmaker for a school, district, or postsecondary institution partner” demonstrating an “interest in partnering in tool development and usability,” and describing “how this toolkit will address a high leverage need” for the stakeholder.  RFP at 666.  Marzano does not explain, and we cannot discern from the protest pleadings, how noting the required involvement of stakeholders addresses the concern raised by the evaluators.  As such, we have no basis to question the evaluators’ assessment of this element of the third deficiency.

The evaluators also noted that Marzano’s proposal was “not clear” about

what actual process teachers and school leaders go through on a daily, weekly, or monthly basis while working through the toolkit over the course of the school year with the large number of resources included--what using the toolkit looks like in more detail than the high-level vision of teachers implementing improvement cycles. 

AR, Tab E, Tech. Eval. at 8.  Marzano contends that this information “was beyond the scope of the Solicitation requirements” and argues that the task is “to actually develop the toolkit.”  Protest at 28.  Marzano’s argument is not supported by the record.  Rather, the solicitation required offerors to submit “a description of plans for stakeholder involvement, including how the development team will ensure the usability of the toolkit by intended audience members through a formative evaluation.”  RFP at 665.  Further, the template offerors were required to use to describe their toolkit projects required a description of “how usability will be tested, and how those results will be used to improve the design of the toolkit.”  Id. at 686.  Accordingly, we find that the evaluators reasonably assessed this element of the third deficiency.

Additionally, regarding Marzano’s toolkit proposal, the evaluators were unclear about “what the continuous improvement cycles are meant to improve; whether the intent is for users to modify the toolkit itself, to improve implementation fidelity of the toolkit components, or to improve something else.”  AR, Tab E, Tech. Eval. at 8 citing AR, Tab D, Marzano Proposal at 119-120.  Referencing the same portion of its proposal noted by the evaluators, Marzano argues that it clearly described its intended improvement cycle process and further outlined it in the logic model for its proposed toolkit project.  Protest at 28 citing AR, Tab D, Marzano Proposal at 118-121.  Again, Marzano’s argument, in our view, amounts to nothing more than its disagreement with the evaluators’ assessment that the firm’s proposal lacked clarity.  Based on our review of the record, we have no basis to question the evaluators’ conclusion, nor does the protester’s disagreement with the conclusion provide such a basis.

Competitive Prejudice

Competitive prejudice is an essential element of every viable protest, and when an agency’s improper actions did not affect the protester’s chances of receiving award, there is no basis for sustaining the protest.  Environmental Restoration, LLC, supra at 9 n.9.  Here, the record reflects, and Marzano acknowledges that, the primary basis of the firm’s exclusion from the competitive range was the evaluators’ assessment of multiple deficiencies and significant weaknesses in the firm’s proposal, resulting in assignment of a rating of unsatisfactory under the most important evaluation factor--technical approach.  AR, Tab E, Tech. Eval. at 2; COS at 16; AR, Tab H, Pre-Award Debriefing at 15; Protest at 2; Comments at 25-26.

As discussed above, the evaluators assessed three deficiencies in Marzano’s proposal under the technical approach factor, each of which contained multiple parts.  Also as noted above, based on the record, we have no basis to question any of the evaluators’ assessments forming the basis of the third deficiency.  With respect to the first deficiency, while we agreed with the protester regarding a small minority (two) of the multiple assessments supporting that deficiency, in large measure the agency’s evaluation assessments were reasonable.  We are, however, unable to conclude that the evaluators reasonably assessed approximately half of the conclusions forming the basis of the second deficiency.  Consequently, we cannot rule out the possibility that correction of the evaluation errors related to the second deficiency could result in the assessed deficiency being revised to a significant weakness or lesser weakness by the evaluators.

Notwithstanding this possibility, however, Marzano’s proposal would continue to have been assessed two deficiencies and multiple significant weaknesses under the technical approach factor.[5]  Here, the solicitation defined a rating of unsatisfactory as representing a proposed “solution that fails to meet the minimum performance requirements,” “contains numerous inconsistencies, significant weaknesses, and/or deficiencies,” and “may contain strengths; however, any strengths are outweighed by the significant weaknesses and/or deficiencies.”  RFP at 709.  Additionally, the solicitation established that an unsatisfactory proposal would require “extensive revision” to rise to the level of a satisfactory or excellent rating, and presented “a high risk” to the agency.  Id.  Accordingly, even if the evaluation errors related to the assessment of the second deficiency were corrected, the rating of unsatisfactory, as defined by the solicitation, would continue to be reasonably applicable to Marzano’s proposal under the technical approach factor--the most important evaluation factor.  See e.g., Procentrix, Inc., B-414629, B-414629.2, Aug. 4, 2017, 2017 CPD ¶ 255 at 11 (“In light of the number of remaining significant weaknesses and deficiencies assigned to Procentrix’s quotation, we conclude that there is no reasonable possibility that the agency’s award decision would be affected by the elimination of this one deficiency.”).

Competitive Range Determination

Marzano does not raise any specific challenges to the agency’s competitive range determination, but argues that had the agency not committed the numerous alleged evaluation errors, “in particular, the errors under Factor 1 [technical approach],” the firm’s proposal would have received higher ratings and been included in the competitive range.  Protest at 2.  Contracting agencies are not required to include a proposal in the competitive range when the proposal is not among the most highly rated.  See FAR 15.306(c)(1); Cyberdata Techs., LLC, supra at 6.  The determination of whether a proposal is in the competitive range is principally a matter within the judgment of the procuring agency.  Cyberdata Techs., LLC, supra.  We will review that judgment only to ensure that it was reasonable and in accord with the solicitation and applicable statutes and regulations.  Id.  Here, the record provides no basis for us to question the agency’s decision to exclude Marzano’s proposal from the competitive range due to the evaluators’ assessment of multiple deficiencies and significant weaknesses in the proposal and resulting assignment of a rating of unsatisfactory under the most important evaluation factor.

The protest is denied.

Edda Emmanuelli Perez
General Counsel

---