When finance teams first look at AI costs, they usually see a price per million tokens. It looks tiny. Then someone compares two models, sees that one is several times cheaper per token, and the decision seems obvious.

It usually isn't.

Your business doesn't pay for tokens. It pays for finished work. One invoice posted correctly. One supplier email answered. One claim checked. That is the number I care about, and it is the one we put in front of clients: cost per task.

Eddie Siegel, the CTO of Ode, said the same thing in a recent interview. You care about the cost per task, not the cost per token. It is one of those ideas that sounds obvious once said, and is still ignored in most AI budgets I see.

Why the token price misleads

A cheap model that gets the task wrong more often is not cheap.

Every mistake has a cost. Someone has to spot it. Someone has to fix it. Sometimes the mistake reaches a supplier or a customer before anyone notices. That cost is real, and it is usually far bigger than the AI bill.

A cheaper model may also need more steps to get to the same answer. More back and forth, longer instructions, more checks. The price per token is lower, but it uses more of them.

And the token price ignores the people around the system. If one model needs a person to check every output and another only needs a person to check the odd ones, the difference in staff time can matter more than anything on the AI invoice.

What goes into cost per task

When we measure cost per task in a pilot, we look at the whole picture for one unit of work.

  • The AI cost for that task, on that model, at your real volume.
  • The checking cost. How many tasks a person needs to review, and how long that takes.
  • The fixing cost. How often it gets something wrong, and what it takes to put right.
  • The running cost. Hosting, monitoring and keeping it accurate over time.

Then we compare it with what the task costs today when your team does it by hand. That is the money number the pilot is judged on.

Why we test more than one model

This is where being independent matters. We don't sell a model, so we have no reason to favour one.

In a pilot we build tests from your own real cases, usually 100 to 200 of them. Then we run Claude, OpenAI and open models through the same tests. You get a scorecard: accuracy on your cases, and cost per task, side by side.

The results often surprise people. The most expensive model isn't always the best for a given task. The cheapest isn't always the cheapest once checking and fixing are counted. And the answer can be different for two tasks in the same company. One model may be better at reading scanned invoices. Another may be better at drafting replies.

Without tests on your own work, you are choosing on brand and price lists. With them, you are choosing on facts.

A simple way to think about it

When someone asks me which model is cheapest, I ask three questions back.

What is the task? Not "AI for finance". One specific task, like matching invoices to purchase orders.

How do you know if it was done right? If we can't say what right looks like, we can't measure anything.

What does a mistake cost you? A wrong tag on an internal email is cheap. A wrong payment to a supplier is not.

With those three answers, the cost per task becomes something you can measure, not guess.

Models change. Your tests don't have to.

New models ship every few months. Prices drop. Quality moves. If your system is tied to one model and you never measured anything, every new release is either noise or a big risky change.

If you have tests built from your own work, it is simple. When a better model ships, we run it through your tests. If it wins on accuracy and cost per task, we switch. If it doesn't, we stay. We build it so you can switch without starting again.

That is a big part of what our ongoing care is for.

What this means for your budget

If you are approving AI spend, ask for cost per task, not a token price. Ask what it was tested on, and whether those were your real cases. Ask how mistakes are caught and what they cost. Ask what happens when a better model comes out.

If the answers are vague, the budget is a guess.

In our 10-day pilot, a model scorecard with cost per task is one of the things you get, along with the money number, the tests and a working first piece of the system. See how the 10-day pilot works.