Where AI genuinely improves a product

Two years into the great AI retrofit, most products have gained an AI feature, and most of those features share a fate: enthusiastic launch, a screenshot on social media, then a usage curve that looks like a ski slope. Meanwhile a smaller set of AI capabilities has quietly become load-bearing — things users would riot to lose.

The difference between the two groups is rarely model quality. It’s product judgment about where intelligence belongs. After building, evaluating, and sometimes deliberately rejecting AI features across our own Labs work, here’s the pattern we trust.

AI pays where these four conditions meet

1. The input already exists. Winning AI features feed on material the user has anyway — documents, voice notes, photos, tickets, history. Features that require users to compose a prompt are asking for new labor and get abandoned first. The best interfaces don’t have a prompt box at all; they have a context and a button.

2. The output is checkable. Users trust what they can verify cheaply. A draft reply they can skim, an extraction with the source highlighted, a suggested route with the reasoning visible. When verification costs more than doing the task manually, trust never compounds — and unverifiable confident output is how products lose users permanently.

3. “Pretty good” is genuinely useful. Language models are probabilistic. Where a 90% answer saves real time (a first draft, a triage suggestion, a summary), AI shines. Where 90% is a lawsuit (dosage, payments, legal conclusions), AI belongs at most in the approach road — never at the decision. Map your product’s tolerance honestly before choosing the feature.

4. Volume makes the difference compound. Ten seconds saved once is nothing; ten seconds saved two hundred times a day per employee is a business case. Point AI at the repeated motion, not at the showpiece.

Run any proposed AI feature through those four gates and the roadmap sorts itself with unsentimental speed.

The features that keep passing the gates

Across our Labs studies, the same shapes keep surviving scrutiny: speech-to-structure (voice notes becoming typed records — field reports, inspections, meeting decisions); retrieval with citations (finding and grounding answers in a private corpus, where the citation is the feature); first-pass triage (classification and routing with humans on the judgment calls); document intelligence (extraction and cross-referencing at volumes humans can’t sustain); and drafting inside a schema (never freeform, always into fields a human owns).

Notice what’s absent: the open-ended chatbot bolted onto a product. Conversation is an interface of last resort — it puts the burden of knowing what’s possible on the user.

Where we’ve said no

Judgment also means subtraction, and the “no” list is where credibility is earned. A subscription-analytics experiment of ours concluded that founders needed correct, explainable arithmetic, not an AI insights layer — a model summarizing revenue data added doubt, not value. A family-coordination concept rejected AI scheduling because in that domain predictable beats clever; a mis-guessed school pickup costs more than every optimization gains. A medication reminder needs reliability engineering, not intelligence.

The tell in each case was the same: AI would have been the pitch, not the product.

The uncomfortable operational truth

An AI feature is not shipped when it works in the demo. It’s shipped when you can answer: What’s the failure rate, on what evaluation set? What does the user see when the model is wrong or slow? What does a call cost at your volume, and what’s the latency budget? Who reviews the consequential outputs, and where’s the audit trail? What happens when the provider deprecates the model?

Teams that can’t answer these haven’t built a feature; they’ve scheduled an incident. This is why we treat evaluation harnesses and observability as part of the feature’s definition of done — the unglamorous half that decides whether the glamorous half survives contact with production.

The one-sentence test

If you can’t finish the sentence “this feature saves this person this many minutes on this task they already do, and they can check it was right in seconds” — you’re not looking at a product improvement. You’re looking at a press release.