An AI pilot reaches production when five things are true: you measured the task before you started, you tested on your own messy cases, what you built runs inside your real tools, someone on your side owns it, and the team uses it every day with numbers you can read. Miss any one of those and the pilot stalls, no matter how good the model is.

That is the whole playbook in a paragraph. The rest of this post turns it into steps you can follow, in order, whether you run the work yourself or hire someone like us at Skcript Applied Labs.

Why AI projects fail

Almost every company I talk to in Dubai, Abu Dhabi or Riyadh has a version of the same story. They bought the licences, ran a pilot, liked the demo, and then the work quietly went back to being done by hand.

In my experience that rarely happens because the model is weak. It happens in the gap between a demo and a system a team relies on. I wrote about the five gaps in detail in Why AI pilots stall. Here is the short version, because the playbook is built from them.

  1. Nobody measured a starting number. "It felt faster" is not a result.
  2. The tests used clean demo data. Real work has crooked scans, missing fields and emails that switch between Arabic and English.
  3. The demo was thrown away. A chat window that never touches your ERP or inbox is not a system.
  4. Nobody owns taking it live. A sponsor in the meeting is not an owner for the next step.
  5. Success meant "looks good in a meeting". Applause is not adoption.

Each step below closes at least one of these.

Step 1: Pick one task, not a department

"AI for finance" is not a project. "Supplier invoice intake" is. Pick one piece of work that eats your team's week, happens often, and is heavy on text or documents. Invoices and purchase orders, supplier and customer emails, contracts, claims, onboarding files and internal requests are good places to look.

Check the volume early. If a task happens a few times a month, the cost of building and looking after an AI system can be more than it saves. That is one of the cases in When not to use AI.

Closes: scope creep, which feeds every other gap.

Step 2: Measure the baseline and the money number

Sit with the people who do the task and watch them do it. Write down how long one item takes, how many come in each week, how often something goes wrong and what that costs. From those, work out the money number: what the task costs today and what fixing it would be worth.

If you can't find a number worth chasing, stop here. That is a good outcome, and it cost you a day instead of a quarter.

Closes: gap 1.

Step 3: Write the success line before you build

Write one sentence everyone signs: the team uses this for X tasks a day, with Y% needing a human check, and the money number moved by Z. That sentence is what the pilot is judged on.

If you can't fill it in, you are measuring theatre. I have seen pilots declared a success because the board liked the video, and then nobody opened the tool. That is a definition of success problem, not a model problem.

Closes: gap 5.

Step 4: Name an owner and settle security on day one

Going live needs a named person on your side who knows how the task really works, can approve access and can say what "good" looks like. In our pilots that person gives about two hours a week.

Bring IT and security in now, not after the demo. Decide where the data for this task is allowed to go. This is where "our data can't leave the country" needs to turn into specific answers: where it is stored, where the model processes it and who can see it. I cover that in Keeping your data in the UAE. For the legal side, see PDPL and DIFC AI compliance in the UAE, or EU AI Act readiness for SMEs if you are in Europe.

Closes: gap 4, and removes the most common late blocker.

Step 5: Build a test set from real cases

Collect 100 to 200 real cases of the task with the right answers attached. The right answer is whatever your best person would have done. Often it already exists, because your team did the work by hand: the approved invoice, the reply that went out, the claim decision.

Keep the messy ones in on purpose. If the system only works on clean cases, you want to find that out now, not after go-live. And you don't need a data clean-up first. A set of real examples is enough to start, as I explain in Do we need to fix our data first?.

The test set is the most useful thing you will build. It checks every change for the rest of the project, and it stays with you to check any AI tool you try later.

Closes: gap 2.

Step 6: Test more than one model, and score cost per task

Run a small shortlist through the same test set with the same instructions. We usually include Claude, OpenAI and at least one open model you could run on servers you control. Don't tune one and leave the others on defaults.

Score more than accuracy:

  • Accuracy on your cases, and which ones each model got wrong.
  • Type of mistake. A wrong date is not the same as a wrong amount.
  • Cost per task. The model cost at your real volume, plus the time people spend checking, plus the cost of fixing mistakes, plus hosting and monitoring.
  • Speed, where it matters for the work.
  • Where the data goes for that exact model and setup.

Cost per task is the column that changes decisions. A cheaper model that gets more wrong is not cheap, because every mistake needs someone to spot it and fix it. I wrote about this in Cost per task, not cost per token, and about the scorecard itself in How we test AI models on your work.

Test each step of the task, not just the whole thing. The best model is often different for different steps, and sometimes a plain rule beats all of them for one step. Keep the rule.

This is a natural stop point. If the results are poor, stop, and keep the numbers and the test set.

Step 7: Build the first working piece inside your tools

This is where most pilots fall short, so give it the most time. Build the first working piece of the real system: reading from the inbox, the ERP or the shared drive, and writing back in a way a person can check.

Two rules keep it safe:

  • A person approves anything important before it goes anywhere.
  • The AI only handles the kinds of cases it was tested on. Everything else goes to a person, as it does today.

Build it so a model can be swapped without rebuilding everything around it. You will want that later.

Closes: gap 3. Nothing you keep from here gets thrown away.

Step 8: Run it on real work and measure again

Run it on the current week's work, next to your team. Then measure the same way you did in step 2, so the before and after sit side by side on time, cost and accuracy. Check the success line from step 3.

Watch whether people actually use it. If the team routes around the tool, find out why before you scale it.

Step 9: Decide, with a fixed plan

The output is a decision: go live, change the approach, or stop. If it is go, you need a fixed price and a plan for the full rollout, with the result it must hit agreed in writing.

In our work, the first eight steps fit into a 10-day AI pilot with a fixed price of USD 7,500, about AED 27,500. Here is what it actually delivers. The rollout to the whole team, connected to every system that matters, with the rare cases covered, is a separate 4 to 6 week plan.

Step 10: Roll out, then keep it honest

The rollout widens what the system handles, one tested case type at a time. The test set keeps checking every change. The owner from step 4 becomes the person who runs it, with a runbook that says what to do when something looks wrong.

After launch, models keep changing. When a better one ships, run it through your tests. Switch only if it wins on accuracy and cost per task. With tests built from your own work, that check takes a day instead of a project.

A one-page AI implementation roadmap

If you want the checklist version:

Step Output Gap it closes
Pick one task Task with volume worth the effort Scope
Baseline Starting numbers and money number 1
Success line One signed sentence 5
Owner and security Named owner, data rules settled 4
Test set 100 to 200 real cases with answers 2
Model tests Scorecard with cost per task 2
First working piece Running inside your tools 3
Real-work run Before and after numbers 5
Decision Fixed rollout plan or a stop All
Rollout and care Live system, retested on new models All

If your pilot already stalled

Start there. Test what you have on your real work and check it against the five gaps. Usually one or two of them explain most of why it stopped. We run a 5-day pilot check-up for exactly this.

The lesson I keep relearning is that the gap between a demo and a system is careful engineering, honest measurement and patience. It is not glamorous. It is also where all the value is.

If you have a stalled pilot or a task that eats your team's week, start a chat with us on the site, or book a call through the Applied Labs page. If you are still choosing a partner, my guide to AI consulting in the UAE covers what to ask, and our AI consulting page shows how we work in the UAE and Europe.