Early enterprise use of AI focused on report drafts, meeting summaries, and email copy. AI produced a first version. People reviewed it, added context, and used it in their work.

As models improved, organizations began building AI agents for specific workflows. These agents retrieve data, call tools, and move work across business systems. Customer inquiries, support tickets, and revenue data updates can all change the state of a live system.

When an AI agent says a task is complete, teams still need to know whether the intended outcome was recorded in the system and whether any unintended changes occurred. If people must verify those conditions by hand after every execution, review work accumulates across the operation.

That reality has sharpened the need for criteria that assess the outcomes AI leaves behind and the consistency of those outcomes over repeated runs. Microsoft and Hugging Face recently released ThinkingBox, a benchmark designed around that problem.

‍

ThinkingBox: Evaluating AI Work Outcomes

ThinkingBox models workflows in which actions change the state of a business system, including order processing, delivery exceptions, customer support, insurance, and internal IT support. An AI agent may retrieve order details, check shipment status, look up customer information, search policies, and create support tickets across multiple steps.

Take a delayed delivery. An agent retrieves the order and shipment details, checks the customer profile, searches the late-delivery compensation policy, and creates a support ticket. It may use the required tools correctly and apply the policy as expected, yet still mark the ticket as resolved while the delivery exception remains open.

The agent reports that the work is complete. The database still shows an unresolved delivery exception, and the customer's request has not been addressed. The execution trace may look valid, even though the final system state fails to meet the workflow objective.

Source: Zhuochun Li et al., “One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows”, Figure 1(A), CC BY 4.0.

‍

How Should Organizations Evaluate AI Work Outcomes?

A work run leaves an execution record: the data an agent retrieved, the policies it referenced, the tools it called, and the response it generated. Those records help teams diagnose failures and improve future runs. Evaluation also needs to inspect the system state after the work is done.

For a delivery-exception workflow, an evaluation can cover the following criteria.

When business data and policy definitions are organized, teams can define success conditions for each workflow. AI agents then retrieve data, recommend actions, or apply changes within those conditions. High-impact exceptions remain subject to human review and approval. Execution outcomes and change history become evidence for the next evaluation.

AI Evaluation compares answer quality, terminal state, unintended side effects, and repeatability against workflow-specific success criteria.

‍

One Successful Run Does Not Establish Repeatability

The ThinkingBox benchmark contains 507 business scenarios in which processing outcomes remain in systems such as orders, shipments, and tickets. Each scenario was run independently 20 times from a fresh initial state. This setup separates a single successful run from dependable performance across repeated runs.

The paper reports pass@1 as the single-run success rate and pass^20 as a measure of repeated-run reliability. Claude Opus 5 scored 66.50% on pass@1 and 47.53% on pass^20. Kimi-K3 scored 57.37% and 17.60%, respectively. A model may find a successful path once yet fail to reproduce it consistently.

A single successful run shows that the work can be completed. Results across repeated runs reveal whether it can be performed consistently.

Enterprise workflows often repeat daily or weekly. Teams need to see whether the same criteria hold across repeated requests, whether failures cluster under specific conditions, and whether failed runs leave incorrect states behind.

‍

Turning Evaluation Results Into Workflow Improvements

Evaluation creates a record. Its value comes from how teams use that record to identify the conditions that produce errors or exceptions and refine business rules and data structures.

For example, when a compensation policy is applied incorrectly for a specific customer status, the underlying issue may involve poorly structured policy conditions, missing data connections, or an exception that should have required human approval.

Execution outcomes and review history let organizations investigate failures through the structure of the workflow. Teams can refine data definitions, clarify policy criteria, reset conditions that require human approval, and verify the effect of those changes in the next evaluation.

Enhans AgentOS is designed to connect data and business context across real workflows, so execution outcomes and review history can inform the design of subsequent work. Data, institutional knowledge, and review criteria developed in one project can provide a starting point for the next.

People define success conditions, review high-impact exceptions, and approve the scope of automated processing. AI agents access data and execute work within those conditions. Evaluation provides a way to assess work quality across both roles.

In enterprise settings, reliability depends on whether an organization can sustain expected outcomes across repeated work and operate a process for reviewing and improving those results.

‍

What defines success for your AI workflows?

Enhans can help define expected outcomes, evaluation criteria, and human review conditions to clarify which workflows are ready for AI execution and what review conditions they require.

Contact