The problem
Why do AI pilots stall before production?
A pilot is built to impress a room. Production is built to survive an ordinary Tuesday. The two need different things, and most teams only plan for the first.
The demo runs on a handful of clean examples. Someone on the project team watches every answer. Nobody counts the cost per task, because there are only twenty tasks. Nobody asks who fixes it at 2am, because nothing runs at 2am.
Then the pilot is called a success, and the hard questions arrive at once. Can it reach the live system? Who approves the output? What does it cost at full volume? What does legal think? Who owns it now? Each question is fair. Together, they stop the project.
In my experience the model is rarely the reason. The work between a good demo and a dependable system is unglamorous, and nobody budgeted for it. So the pilot sits in a shared drive, and the task goes back to being done by hand.
This guide is the list of that work. Eight gates. Check each one before you call a pilot done.
Read this first
Who is this guide for, and how do you use it?
It is for the person who has a pilot, or is about to fund one. A CTO, a product lead, an operations head, a finance lead with a stack of invoices. You do not need to write code to use it.
Start with the scorecard. Then work through the gates in order. Each chapter covers why the gate matters, what good looks like, how to score it, one test to run this week and the red flags to watch for.
Score with the people who do the work, not only the people who built the pilot. The builders know what the system can do. The team knows what the task really needs.
Already read the playbook? The playbook is the order of steps to run a pilot. This guide is the check before you call it ready. Use them together.
The framework
What is the production-readiness scorecard?
Eight gates sit between a working demo and a system your team relies on. Each one is a question you can answer with evidence, not a feeling.
Score each gate 0, 1 or 2. Zero means missing. One means partly there, or true only in the demo. Two means true on real work, with proof you could show someone outside the project.
Eight gates, sixteen points. The score is not a prediction and it is not a certification. It is a checklist with numbers, so you can see where the work is and agree on what comes next.
Be strict. If you are torn between two scores, take the lower one. A gate you round up now is a gate you meet again after launch, in front of more people.
Gate 1 · Data access
Can the system reach real data, live?
Most pilots run on an export. Someone pulled a spreadsheet, cleaned it a little and handed it over. Fine for a demo. In production, the system reads from the live source, with the same permissions and the same mess your team sees every day.
What good looks like
The system reads from the real source of truth: the ERP, the inbox, the document store, the CRM. Access goes through a service account with only the permissions it needs. Missing fields, odd formats and late records are handled, or sent to a person.
How to score it
0
Runs on a one-off export or hand-picked samples.
1
Connected to a copy or a test environment, not the live source.
2
Reads live data through a proper service account, with messy records handled.
Test this week
Pick twenty items that arrived yesterday. Not the clean ones. Run them through the pilot without touching them first. Count how many it could not even read.
Red flags
- Someone still cleans the data by hand before each run.
- Access runs through a person's own login.
- Nobody can say which system is the source of truth.
Gate 2 · A real test set
Do you have a test set built from real cases?
A demo proves the system can be right. A test set shows how often it is right, and where it goes wrong. Without one, every change to a prompt or a model is a guess, and every debate about quality is an opinion.
What good looks like
A fixed set of real cases, with the right answer written down by someone who does the work. Messy cases included, in the mix they actually arrive. You run it after every change, and the result is a number. The set stays with you, so you can test any model or vendor later.
How to score it
0
Quality judged by eyeballing a few outputs.
1
A test set exists, but it is small, clean or written by the builders.
2
Real cases, answers from the people who do the work, run on every change.
Test this week
Ask the person who does the task today to write the right answer for twenty-five recent cases. Run the pilot on them and compare. You now have the start of a test set, and an honest first number.
Red flags
- The test cases were chosen because they worked.
- Nobody re-runs the tests when the prompt or model changes.
- Accuracy is quoted, but nobody can show the cases behind it.
Gate 3 · Human review
Who checks the output, and when?
Every AI system gets some answers wrong. The question is who catches them before they cost money or trust. In the pilot, the project team checks everything. That does not scale, and it does not survive the team moving on.
What good looks like
A written rule for what a person must approve. Usually anything that leaves the company, moves money or affects a person. Cases the system is unsure about, or was never tested on, go to a person by default. Reviewers see why the system gave its answer, and their corrections flow back into the test set.
How to score it
0
No review rule. Or the builders check everything by hand.
1
Review happens, but the rule is informal or rests on one person.
2
A written rule, routed through the tools people already use, with corrections captured.
Test this week
Write one sentence: "A person must approve any output that..." If your team cannot agree on the ending in one meeting, score this gate zero.
Red flags
- Review is a button people click without reading.
- The system never says when it is unsure.
- Corrections are made, then thrown away.
Gate 4 · Cost per task
What does one finished task cost?
Pilot costs hide. Twenty tasks a day cost almost nothing in model fees. At full volume, with retries, long documents and human review, the picture changes. If nobody works it out before launch, finance will work it out after.
What good looks like
One number: the full cost of one finished task. Model fees at real volume, the time people spend checking, the cost of fixing mistakes, and hosting and monitoring. Set next to what the task costs by hand today. If the AI version is not clearly better, you find out before you scale, not after.
How to score it
0
Nobody has worked it out.
1
Model fees are known. Review time and the cost of mistakes are not.
2
Full cost per task at real volume, next to a measured baseline.
Test this week
Take last month's volume for the task. Multiply by the model cost per item from your pilot logs. Add the review minutes per item at your team's hourly cost. Put it next to today's cost. One line on a spreadsheet.
Red flags
- Cost is quoted per token, never per task.
- Nobody measured the starting point, so there is nothing to compare.
- The business case assumes no one reviews anything.
Gate 5 · Speed
Is it fast enough for the way people work?
Speed decides whether people use it. A reply that makes a customer wait gets abandoned. A batch that lands after the morning cut-off is late, however accurate it is.
What good looks like
You know the time the task allows: seconds for a person waiting, minutes for an inbox, hours for a nightly batch. The system meets it at real volume, including the slow cases: long documents, retries, busy hours. When it runs slow, the person waiting can see progress, or the work falls back to today's process.
How to score it
0
Speed never measured outside the demo.
1
Measured on a few cases, not at real volume or busy hours.
2
Meets the time the task allows at real volume, with a fallback when it does not.
Test this week
Run your test set at the volume of a busy hour. Time the slowest tenth, not the average. Ask the people who do the work whether that wait is acceptable.
Red flags
- Only the average is reported.
- Every request goes to the largest model, even the simple ones.
- No timeout. A stuck request just hangs.
Gate 6 · Security and personal data
Where does the data go, and who signed off?
Security and legal reviews tend to arrive at the end, when changes cost the most. They belong at the start. A pilot that sends data somewhere your policy does not allow is not close to production, however well it performs.
What good looks like
You know what data the system touches and how sensitive it is. You know where each model processes requests, not only where data is stored, and you have it in writing. Business AI plans that do not train on your data. Personal data handled under the rules that apply to you, with your data protection lead or counsel involved. Your security team has signed off on this exact setup.
How to score it
0
Not reviewed. Or the pilot used consumer tools or personal accounts.
1
Reviewed in principle, not for the exact models, regions and data in use.
2
Signed off for the exact setup, with processing locations in writing.
Test this week
For every model in the pilot, write down where requests are processed and whether the provider may train on them. If anyone has to guess, you have found the work.
Red flags
- Test data was pasted into a consumer chat app.
- The storage region is known. The processing region is not.
- Legal will "take a look before launch".
Gate 7 · Ownership
Who owns it after launch?
A pilot has a project team. A production system needs an owner: one named person who answers for whether it works, every week, long after the project team has moved on.
What good looks like
A named owner on the business side who knows the task. A named technical owner who can fix it. A short runbook: how it works, how to check it, what to do when something looks wrong, and how to switch it off. Time for both owners is part of their actual job, not stacked on top of it.
How to score it
0
No owner beyond the pilot team.
1
A sponsor, but nobody with the time or a runbook.
2
Named business and technical owners, a runbook, and time set aside.
Test this week
Ask two questions. If accuracy drops next month, who notices? Who fixes it? If either answer is a team name or a shrug, score this gate zero.
Red flags
- The owner is "the AI team".
- Only the vendor that built it understands how it works.
- Nobody has been shown how to switch it off.
Gate 8 · Rollout and rollback
How will it go live, and how do you roll it back?
A launch to everyone at once turns small problems into big ones. A staged rollout finds the cases your test set missed while the damage is still small and easy to undo.
What good looks like
A plan in stages: a small group first, then more, measured the same way at each step. A success line written before launch. Monitoring that shows accuracy, cost and speed in production. A rollback that returns the work to today's process in minutes. A plan to re-run the test set when a model changes or a better one ships.
How to score it
0
The plan is to switch it on for everyone.
1
A staged plan, but no rollback and no monitoring in production.
2
Staged, monitored, with a tested rollback and a re-test plan for new models.
Test this week
Run a fire drill. Switch the system off in a test environment and send the work back to the manual process. Time it. If nobody knows how, that is your score.
Red flags
- Success gets defined after launch.
- Model updates go live without re-running the test set.
- The old process was retired on day one.
The score sheet
How do you score your pilot?
One row per gate. Circle 0, 1 or 2. Add them up. Sixteen is the maximum.
The AI Pilot-to-Production Field Guide: score sheet
| Gate | 0 | 1 | 2 | Score |
|---|---|---|---|---|
| 1. Data access | Runs on a one-off export or hand-picked samples. | Connected to a copy or a test environment, not the live source. | Reads live data through a proper service account, with messy records handled. | ___ |
| 2. A real test set | Quality judged by eyeballing a few outputs. | A test set exists, but it is small, clean or written by the builders. | Real cases, answers from the people who do the work, run on every change. | ___ |
| 3. Human review | No review rule. Or the builders check everything by hand. | Review happens, but the rule is informal or rests on one person. | A written rule, routed through the tools people already use, with corrections captured. | ___ |
| 4. Cost per task | Nobody has worked it out. | Model fees are known. Review time and the cost of mistakes are not. | Full cost per task at real volume, next to a measured baseline. | ___ |
| 5. Speed | Speed never measured outside the demo. | Measured on a few cases, not at real volume or busy hours. | Meets the time the task allows at real volume, with a fallback when it does not. | ___ |
| 6. Security and personal data | Not reviewed. Or the pilot used consumer tools or personal accounts. | Reviewed in principle, not for the exact models, regions and data in use. | Signed off for the exact setup, with processing locations in writing. | ___ |
| 7. Ownership | No owner beyond the pilot team. | A sponsor, but nobody with the time or a runbook. | Named business and technical owners, a runbook, and time set aside. | ___ |
| 8. Rollout and rollback | The plan is to switch it on for everyone. | A staged plan, but no rollback and no monitoring in production. | Staged, monitored, with a tested rollback and a re-test plan for new models. | ___ |
| Total, out of 16 | ___ | |||
A zero on security or ownership blocks launch, whatever the total. skcript.com/ai-consulting/guide
Reading the score
What should you do with your score?
One rule first. A zero on security or ownership blocks launch, whatever the total.
0 to 7
Not ready yet.
Stop adding features. Go back to the gates that scored zero. Data access, the test set and ownership are the usual suspects. If the gaps run deep, restarting on one narrower task is often faster than patching.
8 to 12
Close, with known gaps.
Most of the work is done. Write a short plan for every gate below 2, each with an owner and a date. Do not open it to the whole team until security and rollback both score 2.
13 to 16
Ready to roll out in stages.
Go live with a small group first. Keep measuring the same way. Score it again after the first month in production. Scores slip when volume, data or models change.
Next step
Should you close the gaps yourself, or bring in help?
Most of these gates are work your own team can do. Data access, ownership and rollout depend more on your organisation than on outside expertise.
Help pays off when the gaps are technical and the people are busy: building a test set, testing several models fairly, working out cost per task, wiring the system into the tools your team already uses.
If you hire, bring this scorecard to the first call. Ask how they would score your pilot, and what they would do about each gate below 2. A good partner answers gate by gate. A weak one talks about the model.
Our 10-day pilot builds one task on your real data with a test set, cost per task and a person checking, from day one. For a pilot that already stalled, the 5-day pilot check-up comes first. Both are fixed prices, published on our pricing page.