Case note
Before and after: measuring AI quality, cost, and time
A practical buyer question: after you change an AI workflow, how do you know the change helped, looking at quality, cost, and time together?
A single metric will mislead you. Quality without cost can hide a more expensive path to the same accepted output. Cost without quality can hide cheaper calls that reviewers throw away. Time without the other two can hide a faster draft that creates a longer review. Compare a before window and an after window, same workflow and done definition, with all three read from records.
This is a how-to for measuring a change. It is not an argument that activity dashboards are a substitute for ROI. That "why" belongs with Why AI ROI beats dashboards. What follows is the measurement sheet itself.
Use the same workflow, cost center, and unit of work on both sides. Write down what you changed. Then read quality from work records (accepted, rewritten, rejected, reopened), cost from spend exports for that scope, and time from a cycle-time field you already trust.
If a field is missing on one side of the window, you cannot compare it. Say so. Do not fill the hole with a modeled figure.
Synthetic example. A team changes how a support draft is assembled. They write a sheet first: review outcome, billed usage, and time to accepted reply from the same tickets, for matching windows before and after. This illustration includes no totals and is not a customer result.
For a change in model routing, such as a reasoning model versus a cheaper or open-weight route, apply this sheet rather than judging from unit price alone: The true cost of reasoning models vs cheap models. See also proof.
What to inspect
- Work records with review outcomes and cycle time, filtered to the workflow under change.
- Spend and usage exports filtered to the same project, model route, or cost center.
- A written note of the change and the timestamps that bound each window.
- Join keys (ticket id, project id) that let a quality row, a spend row, and a time row refer to the same work.
One next action
Before you change anything, write a one-page measurement sheet that names the workflow, the three metrics (quality, cost, time), the source system for each, the before window, and the after window. Then make the change. After the after window closes, fill the sheet from those sources only.
If you want help reading the filled sheet against a pack of exports, Overdrive helps you maximize your AI spend from inside your AI assistant. It is a remote MCP server with OAuth sign-in, and it works today in Claude, ChatGPT, and Grok Bot (setup steps). Start with a labeled sample pack and expand the lineage on each figure. Using Overdrive with your own company data is still in security review. The plan is for you to connect the tools you already use instead of uploading files, and those connectors are not available yet.
Claims and caveats
- This post claims that a workflow change should be judged with quality, cost, and time together, compared in matching before and after windows, and tied to source rows.
- It does not claim that any particular change improves those metrics. It reports no before and after results, and it does not treat an activity dashboard as the measure of the change.
- The short illustration is a synthetic example. It is not customer data and it is not a result.