Superintelligence and Productivity: Metrics for the First 90 Days

Superintelligence and Productivity: Metrics for the First 90 Days

Which metrics to track in the first 90 days of an AI agent to prove real productivity gains against a baseline before you expand its use in your company.

•

Want to apply this in your company?

Book a call with Juan →

To measure the real gain from an AI agent in its first 90 days, track a few metrics: time to task completion, handoff rate to a person, error rate in a reviewed sample, rework by your team, and cost per resolved task. Compare all of them against a baseline collected before the agent goes live. Without that comparison, the feeling of higher productivity is just an opinion.

Key takeaways

  • Start with a baseline of the real process before any agent goes live.
  • Choose a small set of metrics, split across speed, quality and control.
  • Measure separately what the agent resolves on its own and what gets handed off to a person.
  • Log errors and decisions, because the most expensive mistake is often the one nobody saw.
  • Use the metrics to decide whether to expand, adjust or stop, not to evaluate people.

Why measure before you expand

A demo usually impresses, but operations carry the real cost. An agent tends to look very productive on easy questions, which are also the most frequent ones, and to fail on edge cases, which are the ones that generate complaints and rework. For that reason, the goal of the first 90 days is not to prove that the technology works in theory. It is to find out whether it improves a specific flow in your company, with your customers and your rules.

Our method organizes this work in four steps: map the current process, limit data and actions, pilot a real flow with logs and a success metric agreed in advance, and then transfer the operation and decide whether it will be expanded, adjusted or stopped. The metrics in this article serve mainly the pilot step, because that is where the decision gets made.

Baseline: what to collect before the agent

The baseline is a snapshot of the process as it works today. Without it, any improvement or decline has no reference point. The work is simple, but it has to be done with discipline:

  • Choose a flow with a clear start and end, such as answering order status questions, booking an appointment or triaging a quote request.
  • Collect data over a representative period, including busy weeks and quiet weeks.
  • Record the volume of cases, the average time to completion and how often a person had to correct or redo something.
  • Document the current rules and the exceptions the team already knows by heart.
  • Define in writing what counts as success and what counts as failure, before the pilot starts.

This last step is the one most often missing from projects. If the team only decides what success means after seeing the results, the metric ends up justifying a conclusion that was already chosen.

Speed and coverage metrics

The first group answers the most obvious question: did the process get faster, and did it handle more cases? Three metrics are usually enough.

The first is time to resolution, measured from the first contact to the closing of the case, not just to the agent's first reply. A quick reply that resolves nothing is not a gain. It only moves the work to later.

The second is the rate of resolution without human intervention, meaning the share of cases the agent closes within the defined rules. Read it together with quality, because an agent can resolve a lot simply by accepting poor answers.

The third is the handoff rate to a person and the time a case waits in the human queue after the transfer. If the agent transfers everything, the gain disappears. If it transfers too little and makes mistakes, the problem shows up in quality. This metric shows the balance between the two sides.

Quality, rework and risk metrics

The second group is what separates a useful pilot from an operation that creates hidden work. The main metric is the accuracy rate in a sample reviewed by a person. The sample should be fixed, made of real cases anonymized or simulated, and rerun whenever instructions or tools change. Judging the model by how confident it sounds is not enough.

Rework is the most revealing measure. Count how many times the team corrected a reply, redid a quote, reopened an appointment or explained to a customer something the agent got wrong. If correction time grows along with volume, the speed gain has simply been transferred to review.

Finally, track risk events: answers outside the approved policy, actions attempted beyond the permission granted, personal data exposed in a reply, and complaints traceable to the agent. These should be counted explicitly, with the date and the matching case in the log, so the team can investigate the cause and not just record that a problem happened.

The published agent evaluation guidance from Anthropic and OpenAI's evals guide describe the logic of building sets of test cases and running them repeatedly. It is worth adapting that logic to your process, without copying metrics that make no sense for your business.

Cost per resolved task: what to count honestly

Cost is the most neglected metric, because people usually count only the tool subscription. A more honest calculation includes the cost of the tool and its infrastructure, the time spent by whoever sets up and maintains the agent, the time for human review and the time spent fixing failures. Dividing that total by the number of tasks actually resolved gives a unit cost that can be compared with the previous process.

The unit economics framework from the FinOps Foundation is a good reference for thinking through this calculation. It does not provide figures for your company, but it helps separate fixed cost, variable cost and hidden cost. Do not compare cost without comparing quality: a cheap task that requires careful review can end up costing more than it appears to.

Illustrative scenario: a scheduling agent over 90 days

Illustrative scenario, no real data: a small clinic wants to use a WhatsApp agent to reschedule appointments and answer questions about preparation. In the baseline phase, the team finds that most of its time goes to rescheduling and confirmations, and that the notes field is often incomplete. Before switching the agent on, the team defines that success requires shorter resolution time, no rescheduling outside the real calendar, and stable rework.

In the pilot, the agent only books free slots and transfers to the front desk any request involving an exception to the insurance rules. In the first review cycle, the logs show that most transfers come from patients asking about a specific procedure the rules did not cover. The team adjusts the instructions and adds that case to the fixed test set. At the end of the period, the decision is to expand only to rescheduling, keeping exceptions with the front desk.

The point of the scenario is not the result, which here is hypothetical, but the sequence: baseline, written criteria, fixed sample, transfer log and an explicit decision at the end.

A simple guide for deciding at the end of the 90 days:

  • Expand, when the success metric was met on the sample, rework stayed stable or fell, and risk events are under control.
  • Adjust, when speed improved but one type of case concentrates errors or transfers, and the rule can be corrected without rewriting the flow.
  • Stop, when quality cannot support the operation or the review cost cancels out the time saved.

Before testing changes in production, it is worth validating the flow with simulated data. This practice is described in our article on how to test AI agents with mocks and a safe sandbox.

Measuring the agent without watching your team

Metrics should evaluate the system, not the people. When each attendant's time becomes an AI-linked ranking, the team starts hiding errors and avoiding difficult cases, and the measurement loses its value. The practical rule is to measure the flow: volume, time, transfers, errors and rework across the process as a whole.

There is also a data protection dimension. Logs of customer conversations contain personal data and need a defined purpose, restricted access and a retention period. Recording only what is needed to evaluate the agent is usually enough and reduces risk. The article on measuring AI productivity without turning it into surveillance goes deeper into this balance.

How to build the 90-day dashboard

You do not need a sophisticated system to start. A spreadsheet with the baseline, a case register with status and transfer reason, and a weekly review of a sample will handle most of the work. As the agent grows, instrumentation should include traces and technical metrics, and the OpenTelemetry semantic conventions for generative AI help standardize that data.

What matters is that the dashboard is reviewed against one fixed question: are we closer to expanding, adjusting or stopping? If the answer does not change over the weeks, the chosen metrics are probably not measuring what matters. To understand what an agent does today before promising any gain, read the article on superintelligence in practice and what an agent does today.

Frequently asked questions

How many metrics should I track in the first 90 days? Track a small set, split across speed, quality and control. More than that usually produces an attractive dashboard and few decisions. Each metric needs a written definition and a named person responsible for reading it.

Can I use the model's own self-assessment as a quality metric? Not as the only measure. The model's self-assessment can help with triage, but accuracy must be checked by a person on a fixed sample of real or simulated cases. That sample should be rerun after every change to instructions or tools.

When should I consider the pilot finished? The pilot ends when the success criteria written at the start have been evaluated with enough data, not when the calendar runs out. If one area had little volume during the period, say so in the decision. The conclusion can be to expand, adjust or stop, and all three are valid results.

Do I need a specific tool to measure the agent? Not necessarily. Well-structured logs, a case spreadsheet and periodic review are enough for most initial flows. Observability tools make sense when volume grows and you need to trace each call, but defining the metrics comes before choosing a tool.

Sources