Case study
advisor agent.
Answers prospective international students' Day 1 CPT questions before they reach sales: it recommends schools, cites a verified source for every answer, and hands the conversation to an advisor when it can't.
- Role
- Product owner, designer & builder (solo)
- Type
- Full-stack AI product (RAG + recommendation engine)
- Year
- 2025–2026
- Status
- Live 2025–2026 · offline since I left the company

The spark
I built this at my previous job, a marketing agency for universities, whose site day1cptuniversities.com serves international students on Day 1 CPT. The team answered the same handful of questions over and over by phone, text and email, for students whose situations were all slightly different. We already had those answers in our internal database of calls, texts, emails and web pages, so we wanted a chatbot that answers from them.
The problems to solve
- P1
The same questions, asked by people in different situations
A generic FAQ gives everyone the same answer. Students needed answers drawn from what the company already knew and matched to their own case.
- P2
A wrong answer misstates someone's immigration status
Wrong advice here means a wrong answer about someone's immigration status. That raised the bar from "the chatbot sounds right" to "every claim is provably grounded". A prompt alone can't promise that, so it had to be checked in code, with a person taking over when the check fails.
- P3
Partner schools come first, but only ones that fit
Partner schools had to lead every recommendation, but a partner that doesn't fit the student is a bad answer. A model told to recommend partners may follow that in one conversation and bend it in the next, so the rule can't depend on the model.
- P4
A silent fallback can hide that the AI never ran
During QA, the Claude understanding call failed on every request with a JSON schema validation error, and each time the system fell back to a simpler rules-based engine without logging it. The replies looked fine, so without a record of which engine wrote each reply, nobody could tell the AI hadn't run in that test round.
How it works
- 01
Core agent loop (RAG)
A question runs through Voyage AI (voyage-3) embeddings against a pgvector index of the company's internal database (call records, texts, emails and web pages), and Claude (claude-opus-5-5) reads the retrieved chunks before drafting a structured-output reply. Understanding the question and writing the answer are two separate calls.
SolvesP1P2
- 02
Grounding checked in code
Every reply must cite a retrieved source chunk, and every dollar figure or percentage in it has to appear word for word in that chunk. If either check fails, the answer is thrown out and the conversation goes to a human.
SolvesP2
- 03
Recommendation engine
Each reply recommends five schools. Slot 1 is always a partner school, and slots 2 and 3 go to partners as well; a non-partner school appears there only when no partner fits the student. Anything the data can't confirm is flagged.
SolvesP1P3
- 04
Widget, handoff & dashboard
The chatbot widget sits on the site. A handoff webhook sends the conversation to a human advisor whenever grounding fails or the student asks for one, and a metrics dashboard logs every conversation, including which engine (Claude or the rules-based fallback) wrote each reply.
SolvesP1P2P4
Product decisions
Source checks run in code
"Always cite your source" in a prompt is a request, and the model can forget it under pressure. So the code checks that the cited chunk exists and that every number in the reply appears in it word for word. The model can't argue its way past a check like that.
SolvesP2
Partner rules are in the ranking code
An LLM given "please recommend our partner schools" might follow it in one conversation and bend or drop it in the next. Writing the rule as ranking logic (a partner in slot 1, partners first in slots 2 and 3, non-partners only when no partner fits) means it holds every time.
SolvesP3
Logging which engine answered
During QA, the Claude understanding call failed on every request with a JSON schema validation error (a nullable-enum field written in a form Anthropic's structured-output validator rejects). Each time it fell back to a simpler rules-based engine, and nothing in the logs said so. The replies looked fine, but in that test round the AI hadn't run once. I only caught it because the QA harness tagged every reply with the engine that produced it.
SolvesP4
Built with
Demo
No live link: the product went offline after I left the company. A real screen below.

Outcome
After the schema fix, a full re-run showed the real Claude engine handling 55 of 55 test conversations, with one minor grounding slip: a correct fact, stated without that turn's source behind it. That's a citation-discipline gap; nothing was made up.
The core agent loop, recommendation engine, chatbot widget, handoff webhook and metrics dashboard went live in 2025 and ran through 2026. It went offline after I left the company. The email channel was never built.
What's next
- Had I stayed, the email channel was next: the same grounded answers, sent as replies to inbound enquiries.