We were asked to cover two thousand templates

The brief was to put an AI assistant across the whole template library. Two thousand of them. The number was in the deck, it was in the objective, and it had the pleasing quality of sounding like ambition. We shipped against a fraction of it, and the feature worked better than it would have if we’d done what was asked.

This is the most common conversation I have in AI product work, and it almost never starts as a disagreement about scope. It starts as a disagreement about what the number means.

Where the number comes from

Nobody picks “all of them” out of malice. It comes from a reasonable place: if the AI works, it should work everywhere, and any gap is a place where a user hits a wall. Partial coverage feels like an unfinished product.

It is also legible in a way that quality isn’t. “Two thousand templates covered” is a sentence you can put in a steering deck. “The forty templates people actually use, done properly” requires you to explain yourself, and explaining yourself in a steering committee is a cost people avoid.

What the data usually says

The first thing I did was look at how the library was actually used — the work behind the AI change assistant case study. This took an afternoon, and it is the step that most often gets skipped, because everyone already believes they know the answer.

The distribution was the one you would expect if you thought about it for a minute and nobody ever does: heavily concentrated. A small proportion of templates carried most of the real volume. A long tail of them had been used a handful of times, ever, some by a single person, some years ago, several by someone testing something.

That changes the shape of the problem entirely. Building for the whole library means spending most of your effort on cases that will produce almost no value, while the cases that carry the load get the same generic treatment as everything else. You end up with something that is technically comprehensive and noticeably mediocre at the exact moments it matters most.

The argument that works

Turning that into a scope change is not a data problem. The data is easy. It’s a framing problem, and the framing that works is not “we should do less.”

Nobody wants to hear that. It sounds like a team asking for an easier target, and a sponsor who has already promised the number to somebody else has a strong reason not to accept it.

What works is: the same effort, concentrated where it will be felt. The scope is not smaller. It is aimed. You are not proposing to deliver less value; you are proposing to deliver more of it, and you have the usage distribution to show why.

Then make the tail explicit rather than pretending it doesn’t exist. It is not abandoned — it falls back to the existing path, which is exactly what those users have today, so nobody loses anything. That single sentence removes most of the objection, because the fear underneath “cover everything” is usually that someone will hit a hole and complain.

Why this matters more for AI than for ordinary features

Scope discipline is not new. But AI raises the stakes on it for three reasons.

Quality is not uniform across cases. A conventional feature either works or it doesn’t. An AI feature works well on cases that resemble its examples and degrades on the ones that don’t. Spreading across everything means spreading into the region where it is worst, and users do not average their experience — they remember the bad one.

Evaluation gets harder in proportion to scope. You can build a genuinely good evaluation set for forty cases, and assess each of them properly at intake. For two thousand, you will end up asserting quality rather than measuring it, which means you find out about problems from users.

Trust is asymmetric and it is shared. One confidently wrong answer costs more than several right ones earn, and the damage isn’t contained to that case. People don’t conclude “this feature is unreliable on obscure templates.” They conclude “this feature is unreliable,” and they stop using it for the cases where it was excellent.

What I’d take from it

• Look at usage before you agree to a coverage number. An afternoon with the data is usually enough to change the shape of the whole project.
• Reframe from less to aimed. “Same effort, concentrated where it lands” is a scope argument a sponsor can carry upwards. “We’ll do a subset” is not.
• Give the tail a defined fallback. Most resistance to narrow scope is fear of holes. Name what happens outside the scope and the fear goes away.
• Let evaluation constrain ambition. If you cannot build a test set for the scope you’ve agreed, the scope is too wide, whatever the deck says.

The brief said two thousand. The evidence said something much narrower, and the feature that shipped was better for it — faster to use, easier to trust, and measurable in a way the comprehensive version would never have been.

Briefs are hypotheses, in scope decisions as much as in oversight design. Somebody wrote that number before anyone had looked at how the thing was used, which is exactly when numbers like that get written. The job is not to deliver it. The job is to go and find out whether it was right, and to bring back something better than an opinion when it wasn’t.