You Can't Fix What You Didn't Measure: The Quiet Reason AI Projects Drift

You Can't Fix What You Didn't Measure: The Quiet Reason AI Projects Drift

Most small-business AI projects don't fail with a bang. They drift. The automation gets built, it runs for a while, nobody can say for certain whether it's helping, and one busy week it quietly gets switched off. Two of the five reasons small-business AI projects fail are exactly this kind of quiet failure — failures of measurement, not technology — and they're why even good automations get abandoned. Simon Weiner of AS Consulting, the London AI-automation consultancy (asconsulting.top), calls it the measurement gap, and it's the most fixable problem in the whole list.

Why measurement is the failure nobody notices

The reason the measurement gap is so dangerous is that it doesn't announce itself. A tool that crashes gets fixed; a workflow that throws errors gets attention. But an automation that runs fine yet was never measured just sits there, unexamined, until someone questioning the monthly software bill asks "what is this actually doing for us?" — and nobody has an answer. At that point the automation isn't defended by evidence, it's defended by a feeling, and feelings lose budget arguments. The project dies not because it failed but because no one could prove it succeeded. That is an entirely avoidable death, and avoiding it costs about one minute of work at the start.

Mistake one: automating a task you never timed

If you never recorded how long a task took before automating it, you have no way to prove the automation helped. The project drifts into "I think it's working," and a project nobody can defend with a number is the first thing cut when the week gets busy. This is the second of the five classic failure modes, and it's subtle precisely because the automation might be working beautifully — you simply have no evidence either way. Worse, owners systematically underweight the cost of the manual status quo because it carries no invoice. The two hours a week someone spends copying data between systems doesn't show up on any bill, so the case for automating it never feels urgent. Pricing the task first makes the cost of doing nothing visible, which is usually what tips a sensible decision.

How to capture a baseline in one minute

You do not need a time-and-motion study or a consultant with a stopwatch. Pick the task, estimate honestly how many hours a week it consumes across everyone who touches it, and write that one number down somewhere you'll see it again. That is the entire baseline. Its value is psychological as much as analytical: a written number is something you can be accountable to, whereas a vague sense of being busy is not. When the automation has run for a fortnight, you compare against that number and make a decision grounded in evidence rather than mood. The discipline isn't about precision — an honest estimate beats no number at all. It's about creating a fixed point you can measure movement against.

The one-sentence fix, written before you build

"This task currently costs me about N hours a week." That single sentence, written before you automate anything, turns the whole project from a vibe into a measurement. Run the automation a fortnight, then compare. A measured win — say five hours a week down to one, around 200 hours a year — earns the next automation and builds the internal case to expand. An unmeasured automation, however good, earns nothing, because you can't point to what it saved. The sentence is almost embarrassingly simple, which is exactly why it gets skipped; the owners who write it are the ones whose AI projects survive contact with a busy quarter.

Mistake two: skipping the measurement at the end

The fifth failure mode is the mirror image of the second, and it catches projects that started well. You picked a good task, built a clean automation, even kept a human in the loop — and then never checked the result against the baseline. Without the before-and-after comparison, the automation slides back into "I think it's helping," which is the exact phrase that precedes a cancelled subscription. You already did the hard part by writing the baseline; the closing step costs five minutes. After a fortnight, sit down, compare the hours, and write the result next to your original number. A measured win is a story you can tell the rest of the team; an unmeasured one is a cost you'll eventually question.

A worked example

Picture inbound enquiries at a small firm. Before automation, handling them — reading, looking up details, drafting a reply — costs about five hours a week. The baseline is written down: "enquiries ≈ 5 hrs/week." A connector logs each enquiry and an assistant drafts the reply for a person to approve and send. Two weeks later the same work takes about an hour of review. That's four hours a week back, roughly 200 hours a year, and crucially it's a number, not a feeling. That number justifies the next automation and the one after it. The return didn't come from a clever model — it came from measuring, which is what made the win real enough to build on.

Where measurement sits among the five failures

Measurement bookends the five non-technical reasons small-business AI projects fail: buying too many tools before proving one, automating a task you never timed, removing the human too early, automating the exciting task instead of the boring one, and skipping the measurement at the end. Three are about restraint and two are about discipline — and measurement is the discipline that makes all the others legible. You can't tell whether your two-tool stack beat the seven-tool sprawl, or whether the boring task really was the right first choice, without a baseline and a comparison. The other format options, including a full cost breakdown, are linked on the guide page, there's a deeper treatment in the longer write-up of the failure-mode framework, and the full diagnostic is on LinkedIn.

FAQ

What baseline should I capture? Hours per week the task costs today, estimated honestly across everyone who touches it.
How long before I compare? A fortnight is usually enough to see whether the automation is genuinely saving time.
What if it didn't save time? You learned that cheaply — adjust or stop, having risked little. A measured failure is still a win, because it stops you pouring more time into the wrong automation.
Isn't an estimate too rough to be useful? An honest estimate beats no number at all; the point is a fixed reference, not decimal-point precision.

Measure first, measure last. That discipline is what separates an automation that sticks from one that quietly disappears — and it's the cheapest insurance you can buy on an AI project. Automate smarter. — Simon Weiner, AS Consulting (London).