Testing the Unpredictable: Agent Trajectories
Agents do not follow a fixed execution path. When given the same prompt twice, they may call different tools in a different order. Standard unit tests looking for exact matches fail. Instead, we perform Trajectory Evaluation, analyzing the sequence of steps, tool selections, and final outputs.
LLM-as-a-Judge: We evaluate intermediate agent steps using a highly capable evaluator model that compares the agent's actual trajectory against a reference gold-standard set.
Trajectory Score Metric
Evaluators check for three key criteria: correctness of final output, efficiency (number of loops run), and accuracy of tool calls (avoiding redundant tool activations).