Your AI agent gave the right answer. Did it take the right actions?
Evaluate tool selection, argument scope, action order, and resulting state alongside the final response, using a cancellation assistant as a worked example.
A correct message can hide an incorrect action
A support agent tells a customer that an order was cancelled. The sentence is clear, polite, and accurate at the end of the run. Yet the execution log shows cancellation was attempted twice, an unrelated order was inspected, and the agent continued after a tool returned an ambiguous result. Scoring the final answer alone would miss the behavior that matters most.
This is a hypothetical evaluation exercise, not a customer incident or a measured result. The agent can look up an order, check cancellation eligibility, and submit a permitted cancellation. We want to know whether it completed the authorized task through acceptable actions, not simply whether it produced an agreeable sentence.
Google Cloud separates final-response evaluation from trajectory evaluation, which examines the tool-call sequence. Its documented matching approaches have different tolerances for order and extra calls. Those are useful building blocks. The policy rubric and release workflow below are worqcloud recommendations for applying that distinction to an action-taking assistant.
Define the task before defining the score
Start with a narrow user request. For example, cancel order A17 if it has not shipped, and explain the outcome. Specify what the agent knows, what tools exist, and which identity authorizes their use. A test cannot judge unauthorized access if it never defines the allowed order or account.
Record the relevant business rules separately from the expected wording. The agent may need to check ownership and eligibility before cancelling. A request for information alone should not trigger a state change. An ambiguous identifier may require a clarifying question. These are task conditions that remain important even when the final response sounds helpful.
Decide which outcomes count as successful alternatives. If the order has shipped, declining the cancellation and explaining the next supported step can be correct. Treating every refusal as a failure would reward an agent for overriding the policy. An evaluator needs to understand the permitted task, including when the best action is to stop.
Record evidence from the tools
Capture tool names, validated arguments, returned status, and observed state changes using your application's execution records. Keep those records separate from the model's own description of what it did. A generated statement that says ownership was verified is not evidence that the authorization check ran.
Use stable test identifiers for the customer, order, and tool attempt. Record enough information to reconstruct the sequence without copying credentials or unnecessary personal data. A sanitized order reference and an outcome category can often support investigation without retaining a full customer record.
The evaluator should see what happened when a tool failed. Did a request definitely fail before applying its effect, return an uncertain result, or complete successfully? Those states imply different safe responses. A retry after a definite connection failure and a retry after an uncertain cancellation acknowledgment should not be judged as identical behavior.
Choose sequence matching to fit the contract
Exact trajectory matching is useful when the task genuinely requires one fixed sequence. It can also reject harmless differences, such as an additional permitted read that helps resolve ambiguity. Do not choose the strictest matching method simply because its score looks easy to interpret.
An in-order rule can check that required calls occur in the intended order while allowing additional calls, depending on the framework's documented semantics. That flexibility is useful only if extra actions are reviewed separately. A run should not pass merely because it eventually contains the required steps after an unauthorized write.
An any-order comparison can fit independent reads, but it is a poor expression of a dependency such as checking eligibility before cancellation. Describe the essential ordering constraints explicitly. The evaluator's tolerance should follow the business contract rather than the sequence produced by one successful reference run.
Check arguments and effects, not just tool names
Two runs can call the same tools in the same order and still have different outcomes. One may cancel A17 and another A71. Compare the customer scope, target identifier, action type, and other consequential arguments against the authorized request.
Validate arguments at the application boundary as well as during evaluation. Schema validation can reject malformed inputs. Authorization must check whether the caller may act on the referenced resource. A passing offline score does not replace those runtime controls.
Inspect the resulting state through an independent test read where appropriate. If a mock tool returns success without changing state, a response-based test can overstate correctness. In a sandbox, make the expected before-and-after state part of the case. The test should explain whether it is checking a simulated contract or a real integration.
Build a rubric with hard failures
Use several dimensions instead of collapsing every concern into a single average. Task completion, correct target, required checks, duplicate effects, and user communication answer different questions. A strong answer score should not cancel out an unauthorized action.
For the cancellation example, define a hard failure if the agent changes another customer's order or repeats a non-repeatable effect. Missing a stylistic preference can be a lower-severity issue. This is an editorial rubric, not a universal scoring standard. Its value comes from making the product team's priorities explicit.
Keep a short explanation for every failed run. A label such as trajectory mismatch is less useful than eligibility was checked after cancellation or the retry changed the operation identity. A reviewer should be able to connect the verdict to observable evidence and a named rule.
{
"case": "cancel-eligible-order",
"allowed_order": "A17",
"required_before_write": ["check_ownership", "check_eligibility"],
"forbidden_effects": ["change_other_order", "repeat_cancellation"],
"expected_state": "cancelled"
}Illustrative test-case schema, not an SDK configuration. Implement checks against your application’s actual authorization and execution records.
Test uncertainty and unsuccessful tools
Include an order that is already cancelled, an order that has shipped, an ambiguous user request, and a tool timeout. These cases test whether the agent understands the boundary of its authority and evidence. They should not all be forced through the same successful action sequence.
For a timeout with an uncertain effect, provide a safe status-check tool or a supported escalation path. The correct response might be to inspect state before retrying, or to tell the user that completion cannot yet be confirmed. The appropriate choice depends on the application's contract and idempotency guarantees.
Add a malformed tool response and an unavailable dependency in a controlled fixture. Check whether the agent invents success, exposes internal errors unnecessarily, or performs unrelated work. The test should reward an honest, bounded outcome when the requested action cannot safely be completed.
Keep test actions in a sandbox
Run evaluation against fixtures or an isolated environment with synthetic users and orders. Replace external effects such as real refunds and emails with supported test integrations. A test that accidentally contacts a customer can create a real operational problem while producing a technically valid score.
Version the input request, initial state, tool contracts, expected outcomes, and evaluation rules together. If a policy changes, preserve the earlier case for context and update the expected behavior through review. Otherwise, a score change may reflect a moving test rather than a changed agent.
For stateful cases, reset the fixture between runs. An already-cancelled order left by the first test can make a second run appear safer or less capable for the wrong reason. Record the run identifier and fixture version so failures can be reproduced.
Use model-based judgments for the questions they can answer
A model-based evaluator can help assess whether an explanation is understandable or whether a response addresses the user's request. Keep the rubric concrete and validate the judge against a reviewed sample. A fluent explanation of a verdict does not establish that the verdict is correct.
Use deterministic checks for properties your application can establish directly. Exact target identifiers, forbidden tools, required approval state, and duplicate external effects are often better checked against structured records. Combining those checks with reviewed judgments gives each method a narrower responsibility.
Track disagreements between automated judgments and human review. They may reveal an ambiguous rubric, missing execution evidence, or a judge limitation. Improve the test before treating a small score difference as proof that one agent version is safer or more useful.
Compare releases on the same cases
Evaluate a candidate against the current version using the same fixture set and documented configuration. Record model settings, tool definitions, prompts, and relevant code revisions. A different tool contract can change the task even when the user request is unchanged.
Inspect case-level regressions before accepting an improved overall score. The candidate may perform better on common questions while mishandling one consequential write. Keep hard failures visible in the release decision rather than hiding them inside a large denominator.
Where the agent can produce different paths across runs, use a bounded repeated evaluation and report the observed variation. Do not convert a small sample into a guarantee of future behavior. The useful result is evidence about specific scenarios and the conditions under which they were tested.
Turn failures into focused fixes
Suppose the evaluation finds repeated cancellation attempts after an uncertain acknowledgment. A prompt change might help, but the tool contract may also need to expose a status lookup or a stable idempotency key. Fix the boundary that made the unsafe behavior possible.
If the agent consistently chooses the wrong order, inspect identifier handling, customer context, and authorization enforcement before adding more examples to the prompt. If it gives a misleading success message, compare the response-generation step with the actual tool result. Different failures deserve different interventions.
Rerun both the affected case and the surrounding cases after a fix. A new rule that prevents duplicate writes might also block legitimate retries after a definite rejection. The regression set should protect the desired distinction, not merely reward the absence of all actions.
Start with a small release gate
Choose a handful of consequential tasks and write the permitted outcomes, required checks, and forbidden effects. Capture execution evidence, validate final state, and keep the verdict understandable to the people who own the business workflow.
For the cancellation assistant, the first gate can require correct customer scope, eligibility before a write, safe handling of uncertain results, and communication consistent with observed state. Add cases from real failures only after sanitizing and authorizing their use.
The result is a more useful question than whether the answer looks correct. Can the team explain how the agent reached it, show that the actions stayed within the request, and reproduce the important failures before the next release?