Copilot Studio Guide: Agent Evaluation and Regression Checks

InYourLeague Research · 2026-08-27

Answer in brief

Build a representative Copilot Studio test set, document a baseline, repeat each evaluation, and review regressions before making evidence-based release decisions. Treat evaluation for every release candidate as a recommended internal control, not a universal Microsoft requirement, and keep responsible AI and content-safety reviews separate. Neither supplied official source specifies a selectable model ID.

Key facts at a glance

Failure modes and verification

FAQ

What should a foundational Copilot Studio test set contain?

It should cover key supported, prohibited, and out-of-scope scenarios. Each case should include a prompt, an optional expected response, observable acceptance criteria, and an appropriate evaluation method.

Should test chat or agent evaluation be used?

Use both. Test chat supports interactive exploration of a session, while agent evaluation provides structured and repeatable execution across multiple cases. Microsoft describes their complementary roles in About agent evaluation.

How should the baseline pass rate be calculated?

For each run, divide passed cases by evaluated cases. After multiple runs, calculate the average pass rate as the arithmetic mean of the per-run pass rates and retain the individual results so variability remains visible.

Is an 80–90% pass rate a required release threshold?

No. The evaluation checklist presents 80–90% as a realistic target to adjust to business needs. It is neither a measured product-specific baseline nor a universal release gate.

What release rule should an organization adopt?

A defensible internal rule is to require no unresolved must-pass-core regressions, attainment of a business-approved average pass-rate target, no unexplained material decline from the baseline, and completion of separate safety reviews. This is an editorial recommendation, not a Microsoft mandate.

Do the supplied sources identify a selectable model ID?

Neither supplied official source specifies a selectable model ID. Record only verifiable metadata, such as the agent version, baseline date, test-set version, language, user profile, and relevant configuration changes.

Sources and freshness

• Official source
• Official source
• Last verified: 2026-08-27

Extended guide

Recommended approach

Treat Copilot Studio agent evaluation as a repeatable quality-control process: design representative cases, define acceptance criteria and methods, establish a dated baseline, repeat the same set, and review regressions before deciding whether to release a change. Microsoft describes evaluation as iterative from design through deployment and regression detection. Requiring this evidence for every release candidate is an editorial recommendation for stronger internal governance, not a universal Microsoft requirement.

Neither supplied official source specifies a selectable model ID. Evaluation records should therefore include only verifiable metadata, such as the agent version, baseline date, test-set version, user profile, language, and relevant instruction, knowledge, tool, or routing changes.

1. Design the test set

Start with one case for each key supported scenario and one for each important prohibited or out-of-scope behavior. Each case should state the prompt, optional expected response, explicit acceptance criteria, and evaluation method. Define success in observable terms: correct content, completion of the main task, required tone and format, permission handling, citations, handoffs, and boundary behavior. Begin with a small foundational set, then expand coverage across the must-pass core, robustness to paraphrases and context, architecture behavior such as tool calls and routing, and edge cases.

2. Select evaluation methods

Match the method to the acceptance criteria. Copilot Studio can compare a response with an expected response or a quality standard, assign case-level scores, and apply multiple methods within a test set. Manual review is also valid when evaluators apply the documented criteria consistently. Use test chat for interactive exploration and agent evaluation for structured, repeatable execution. When diagnosing a result, inspect the response, transcript, activity map, and resources used. Evaluation measures correctness and performance; it does not replace responsible AI or content-safety reviews.

3. Establish the baseline

Run the foundational set, record every case as passed or failed, calculate the per-run pass rate, and save the agent version and baseline date. The supplied evidence contains no observed product-specific baseline, so none should be invented. Microsoft suggests an average pass-rate target of 80–90%, adjusted to business needs. That range is guidance, not a measured Copilot Studio result or a universal release gate.

4. Repeat runs consistently

Because agent responses can vary, run the same set multiple times. Record each run’s pass rate and calculate the average pass rate as the arithmetic mean across runs. For a defensible comparison, the editorial recommendation is to keep cases, criteria, methods, user profile, language, and relevant configuration constant unless the variable itself is under test. Copilot Studio also supports automated execution through Power Platform REST APIs, connectors, flows, and CI/CD workflows.

5. Review regressions

Compare the candidate with the documented baseline at both aggregate and case levels. Investigate human-judged failures that the evaluation marked as passes, confirmed failures, increased variability, and every regression in the must-pass core. Separate test-case defects—ambiguous prompts, incorrect assertions, or weak criteria—from agent-design defects involving instructions, knowledge, tools, routing, citations, or guardrails. Correct the responsible layer and rerun the affected set. Recommended full-suite triggers include model changes, major knowledge updates, new tools or connectors, and production incidents.

Organization-defined release rule

A practical internal rule is: release only when there are no unresolved must-pass-core regressions, the average pass rate meets the business-approved target, no material decline from the baseline remains unexplained, and separate safety reviews are complete. Record exceptions, owners, and rationale. This is a proposed governance rule, not a Microsoft-mandated threshold.

See About agent evaluation and the agent evaluation checklist.

Model availability note: The official source does not specify a selectable model ID.