An AI customer support system isn't a chatbot on your homepage. It's an internal tool that helps your support team respond faster and more consistently. Here's how I'd architect one.
Start with the support team's workflow
Before choosing technology, map the current workflow:
- Ticket arrives
- Agent reads ticket, searches docs/past tickets
- Agent drafts response
- Agent sends response (maybe after review)
- Ticket closed or escalated
AI should compress steps 2-3, not replace the agent entirely.
Architecture overview
Help Center + Past Tickets + Internal Wiki
↓
Document Ingestion Pipeline
(chunk → embed → store in pgvector)
↓
Support Agent Dashboard
↓
User asks question / ticket arrives
↓
RAG retrieval → Draft generation → Agent review → Send
Component breakdown
Document ingestion
- Sync from help center CMS, internal wiki, and resolved ticket archive
- Chunk documents by article/section boundaries
- Store metadata: product area, document type, last updated date
- Re-index on document changes (webhook or scheduled sync)
Retrieval pipeline
- Embed the ticket/question
- Vector search with metadata filters (product area, plan tier)
- Rerank top results for relevance
- Include 3-5 most relevant chunks as context
Draft generation
- System prompt: "You are a support assistant. Answer based only on the provided documentation. Cite sources. If unsure, say so."
- Include ticket context, customer plan tier, and retrieved docs
- Output: draft response + source citations + confidence score
Agent dashboard
- Show draft alongside source documents
- One-click send, edit before send, or reject
- Feedback buttons (helpful/not helpful) for continuous improvement
What I'd build in v1 vs. v2
V1 (4-6 weeks):
- Document ingestion from help center
- RAG-powered draft generation
- Agent review dashboard
- Basic analytics (usage, edit rate, send rate)
V2 (after validating v1):
- Auto-categorization and routing
- Multi-language support
- Customer-facing self-service (with lower confidence threshold)
- Automated quality evaluation pipeline
Pitfalls to avoid
- Skipping human review — agents need to verify before sending
- Indexing outdated docs — stale documentation produces wrong answers
- No confidence scoring — agents can't tell when to trust the draft
- Ignoring edit patterns — if agents always rewrite drafts, your prompts need work
- Measuring vanity metrics — "AI generated 1000 drafts" means nothing if agents rejected 90%
Success metrics
- Time to first response (should decrease)
- Consistency of answers across agents (should improve)
- Agent satisfaction with draft quality (track edit rate)
- Customer satisfaction scores (should maintain or improve)
Build for your support team first. Customer-facing AI comes after you've validated quality internally.