Most agent benchmarks are built on static environments: a frozen snapshot of a web interface, a sandboxed replica with predictable state. These benchmarks tell you how an agent performs in a controlled setting. They don't tell you whether the agent can handle a site that changes its checkout flow, enforces session expiry, or presents a CAPTCHA mid-task.
CartBench is a structured evaluation suite for commerce agents built from Machine Commerce behavioral traces. Test cases are designed to run against live retail environments with live state, so every score is earned against the real thing.
Status: In design. The task taxonomy, scoring dimensions, and failure categories are drafted, and the live test harness is next. Planned for early 2027.
What CartBench evaluates
CartBench is planned to cover all four task categories in the Commerce Agent Traces taxonomy, tested across a representative sample of major US retail environments:
- Discovery: Finding a specific product from a natural language description and site navigation starting from the homepage.
- Comparison: Identifying and comparing multiple candidate products against a set of criteria, including price, specifications, and availability.
- Cart management: Adding, modifying, and removing items in a cart across sessions, including variant selection, quantity changes, and coupon application.
- Checkout: Completing a purchase end-to-end through authentication, shipping, payment, and order confirmation under real friction conditions.
Scoring
CartBench will score agents across three dimensions:
- Task completion: Binary success or failure. The task either reached its defined end state or it didn't. No partial credit for getting close.
- Step efficiency: The number of actions the agent took divided by the minimum viable action count for that task on that environment. An agent that completes a checkout in 40 steps when 18 are sufficient is less capable than one that completes it in 20. Efficiency scores surface agents that complete tasks but do so with unnecessary or repeated actions.
- Failure mode robustness: How the agent handles the friction points that real retail environments introduce: authentication walls, CAPTCHA challenges, session expiry, form-fill errors, and navigation dead ends. Robustness is scored by failure mode category, making it possible to identify exactly where a given agent breaks down.
CartBench build stages
Example test case format
| Field | Example |
|---|---|
| Task | Find a specific product from a natural language description, starting at the homepage |
| Constraints | Target product, price ceiling, variant requirements, shipping deadline |
| Score | Task completion (pass/fail), step efficiency (steps vs. minimum viable path), failure mode robustness (5 categories) |
Why live environments are necessary for a useful score
Static benchmarks decay. When the environment doesn't change, a score that holds stable over time only tells you that the agent hasn't regressed relative to a fixed target. It doesn't tell you whether the agent has kept pace with the environments it actually runs in.
Live environments introduce a signal that static benchmarks cannot: score drift over time. An agent whose CartBench score on a given retail environment declines over six months, without any model changes, has become brittle relative to that environment. The benchmark caught a real capability regression before it surfaced in production.
This is why CartBench versions will be timestamped and why score drift over time will be treated as signal rather than noise. A benchmark that can only tell you how your agent compares to a snapshot is measuring the wrong thing.
Benchmark versioning
Each CartBench release will be timestamped and versioned. Test cases will be drawn from live retail environments at the time of release. Scores from different versions are not directly comparable - the environments change, and that is the point. Teams tracking agent quality over time should run evaluations against the same version, then compare version-to-version scores to understand how agent capability has changed relative to environment change.
binary - no partial credit
vs. minimum viable path
categories scored
Access
CartBench is planned for early 2027. We are offering early access to labs and product teams actively building or evaluating commerce agents. Early access participants will have input into the task taxonomy and scoring rubrics before the benchmark is finalized. To join the waitlist, contact us at founders@generalmachines.ai.