Applies toLegacy bots (Bot Framework/LUIS, Dialogflow, Lex, QnA Maker successors, vendor chat widgets) moving to RAG on Azure OpenAI + Azure AI Search, Bedrock Knowledge Bases or similar
What changes, and what shouldn't
| Old bot | RAG assistant |
|---|---|
| Fixed intents, hand-written answers | Answers generated from retrieved documents |
| Fails on unseen phrasing | Handles phrasing well; can be wrong in new ways |
| Answers go stale when policies change | Updates when documents are re-indexed |
| Deterministic flows (reset password, book a slot) | Keep these deterministic. Expose them as tools, not free text |
The goal is not "replace everything with an LLM". It's "free-text questions get grounded answers; transactions keep their tested flows".
1. Mine the old bot
- Export intents, training phrases and answers. They're a ready-made test set.
- Export 3–6 months of transcripts: top questions, fallback ("I didn't understand") rate, handoff reasons.
- List the transactional flows and the systems they call.
2. Build the knowledge base
- Sources: help centre articles, policy documents, product docs. Assign an owner per source.
- Ingestion with chunking by headings, metadata (
source,url,updated,audience), and permission fields if content isn't public. - Hybrid search (keyword + vector) plus a reranker, which handles product codes and exact terms better than vectors alone.
Old bot answers that aren't in any document: either move them into the help centre (preferred) or index them as a "curated answers" source.
3. Assistant design
User question
├─ Is it a known transaction? (classifier or tool choice) ──► existing flow / API tool
└─ Otherwise ──► retrieve top passages ──► answer with citations
└─ no good passage ──► "I don't know" + handoff
System prompt essentials: answer only from provided passages, cite them, say when you don't know, never invent policy, stay in scope.
4. Evaluate before go-live
Build an evaluation set from:
- the old bot's intents (one or two phrasings each, with the expected answer/source),
- the top 200 real transcript questions,
- tricky ones: out-of-scope, ambiguous, adversarial.
Measure:
| Metric | Target example |
|---|---|
| Retrieval hit rate (right source in top 5) | ≥ 90% |
| Answer correctness (human or LLM-judge, spot-checked) | ≥ 85% on in-scope |
| Correct "don't know" on out-of-scope | ≥ 90% |
| Groundedness (claims supported by citations) | ≥ 95% |
5. Roll out side by side
- Internal users and support agents first ("agent assist" mode).
- A percentage of customer traffic, with thumbs up/down and an easy handoff.
- Compare against the old bot: containment rate, handoff rate, CSAT, complaint themes.
- Retire old intents as the new assistant proves itself; keep transactional flows.
6. Operate
- Content owners update documents; re-indexing is scheduled and monitored.
- Weekly review of negative feedback and "don't know" questions, which show gaps in the documentation.
- Safe mode and kill switches ready (see the kill switch series).
Warning
The old bot never invented a refund policy. An LLM can. Groundedness checks, citations and "don't know" behaviour are not optional for customer-facing assistants.
Checklist
- Intents and transcripts exported as test data
- Knowledge sources owned and indexed with metadata
- Transactions kept as deterministic tools
- Evaluation targets met
- Phased rollout with comparison metrics