AI Product Judgment
When the Roadmap Meets an LLM: How I Reorder Priorities
Most roadmap frameworks assume you can estimate effort and impact with reasonable confidence. AI features break that assumption, since the same feature can take a week or a quarter depending on how the model behaves on your actual data, and you often cannot know which until you have already started.
So instead of estimating effort upfront, I score three things before a feature enters the roadmap: how reversible the decision is if the model underperforms, how much of the value survives if the model gets 70% of real cases right instead of 95%, and whether a human fallback is cheap enough to ship alongside it. Features that fail on all three get pushed, no matter how exciting the demo looked.
The 70% and 95% are planning bars, set before there is any data. For each feature I write down what a pass looks like, as a yes or no call, so the numbers mean something once real cases come in.
I use it to force the uncomfortable conversation early, instead of three sprints into a feature that only works in the demo.
Where the three questions came from
In early 2023 I was the founding PM at Tanbii, a gamified app that nudges people to live greener. The founder wanted an AI feature, but nobody could say yet what it should do. There was no earlier product to learn from, and no one on our team of five had built with AI before.
The obvious move was to plug in one of the big models. I looked into it and decided against it. At the time, the main LLM APIs only took text, and what our users gave us was photos. For an app with no revenue yet, paying per call for a large model would have been expensive, and the extra wait on every call would have made the app feel slow.
So we went light. We had no data of our own, so I worked from what we could get: competitor data, a review of what similar apps were shipping, user feedback and a user survey. The goal was more users who came back every day, and that pointed to one feature: an AI image recognition daily streak, built on a third-party service. Using a vendor kept the cost down and made the MVP easier to test, and we tuned the feature with A/B tests after launch.
Here is how the three questions play out on features I've built since.
The three questions on real features
Can I undo it cheaply if the model underperforms?
In Standfirst, the classifier that decides whether a request becomes a blog post or a social post is plain rules, with no LLM call. So is the recommendation engine. Rules cost nothing, answer instantly, and I can see why they decided what they did. If they turn out too rough, adding a model later is a small change. Pulling a model out of a flow that users already depend on is much harder. Rules are also easy to test: a rule gives the same answer every time, so a short list of real requests with the expected label works as a regression test. Standfirst saves the rule's guess next to the type the user finally picks, and the rows where the two differ tell me when the rules have become too rough.
HatchFellow went through a reversal like this. The first version rotated a small, hard-coded question bank, and repeats became the main complaint, so I switched to generating questions daily, with a no-repeat history check on every new question.
How much of the value survives at 70%?
Advisor Agent answers questions about immigration status and veteran education benefits, where a 70% answer is a wrong answer. So the feature is split in two. The model reads the question and writes the reply. Then a code-based check, with no second model involved, confirms that every cited source exists and that every number in the reply appears in that source word for word. It runs on every reply before the user sees it, so it works as a guardrail. If an answer can't pass, the conversation goes to a person. The check catches invented sources and invented numbers. It can't tell whether a real source was read correctly, which is one reason a person stays in the loop.
Is a human fallback cheap enough to ship with it?
For the SEO Content Engine it was. The automation only writes drafts into a Google Sheet, and a person approves every one before anything is published. The review queue cost nothing to build.
In a chat product, a fallback only helps if you can see when it kicks in. During QA on Advisor Agent, the replies looked fine and testing showed a 0% hallucination rate. The test was grading the wrong system. The Claude call had failed on every request, and a simpler rules engine had answered instead. A rules engine can't make things up, so the eval passed for a reason that had nothing to do with the model. I caught it only because every reply was tagged with the engine that produced it. That tag is what eval teams call a trace: a record of which path produced each answer. A fallback that fires silently on every request looks exactly like a model that works, so a perfect score is a reason to check what produced it.
What this does to the order
When I run a backlog through these questions, the same order keeps coming out:
- Rules, wherever a rule is good enough.
- The model, on the parts where 70% still helps the user.
- A human check on anything where a wrong answer costs someone money, legal status or trust, shipped in the same release.
- Features that need 95% from day one and have no cheap fallback wait until a smaller, rule-based version shows the need is real.