AI pilots that actually ship

Most AI pilots die at week four. Not because the model was wrong, not because the prompt was poorly written, not because the team was weak. They die because nobody decided, on day one, what a successful week-eight looked like.

I have shipped six AI pilots in the last twelve months. Four landed. Two did not. The two that stalled are still technically interesting — a fitment-data extractor and a support triage agent — but neither one lives in production, and neither one will, because the business that asked for them could not tell me what "working" meant.

The single biggest failure mode

Scope creep is not the killer. Evaluation debt is. A pilot without an evaluation harness is a pilot without a finish line — and pilots without finish lines are re-scoped every fortnight until the budget runs out and everyone quietly moves on.

“The question is never "does the model work?" The question is "how do you know when it is wrong?"”— A line I steal from myself on every scoping call

A useful evaluation harness does not need to be clever. It needs to be written down before the first prompt is tuned. A hundred labelled examples, a target accuracy, and a weekly regression run. That is it. Teams that do this ship in six weeks. Teams that promise to do this "once the model feels closer" ship in never.

What I do on day one

• Write the ROI model before writing any code — hours saved, deflection rate, cost per task
• Build the evaluation set before the first prompt — one hundred inputs, one hundred expected outputs
• Pick a kill metric — the number that, if missed at week six, means I stop
• Wire telemetry from the first deploy — cost per call, latency, token usage, error rate
• Ship a working but unimpressive version to real users by week two

Every one of those is boring. None of them involve tuning a prompt. All of them are the difference between a pilot that lands and a pilot that turns into a Slack thread nobody opens.

When to kill a pilot

The hardest conversation in AI work is the one where you tell a client their pilot should be killed. It is also the one they will thank you for six months later, because the alternative is a tool that costs £8,000 a quarter and deflects exactly nothing. If the kill metric is missed at week six, I write a post-mortem and I move on. That clause in the scope protects everyone.

Pilots ship when someone in the room is allowed to say no. If that person does not exist on your side or mine, the pilot will run until the money is gone. Name them on day one.