The demo worked perfectly in the meeting. Then you tried to ship it. Here's what changes between prototype and production — and what you need to build.
What the prototype had
- A prompt that works for 5 example queries
- Direct API calls to OpenAI with no error handling
- No authentication or rate limiting
- Manual testing ("it looks good!")
- Hardcoded context documents
What production needs
1. Evaluation framework
Before writing more code, define how you'll measure quality:
- Test set: 50-100 representative queries with expected answers
- Metrics: Retrieval accuracy, answer relevance, hallucination rate
- Baseline: Score your prototype against the test set
- Target: Define minimum acceptable scores before launch
Without this, you're guessing whether changes improve or degrade quality.
2. Retrieval reliability
Prototype RAG often uses a handful of documents that happen to work. Production RAG needs:
- Robust document ingestion with error handling
- Incremental indexing when source documents change
- Chunking strategy validated against your test set
- Reranking for improved retrieval precision
- Metadata filtering for scoped searches
3. Guardrails and fallbacks
Production AI must handle failure gracefully:
- Confidence thresholds: Don't show low-confidence answers
- Citation requirements: Every claim linked to a source document
- "I don't know" responses: When retrieval finds nothing relevant
- Fallback to existing workflow: When AI can't help, route to the manual process
- Input validation: Prevent prompt injection and abuse
4. Observability
You need to see what's happening in production:
- Log every query, retrieved chunks, and generated response
- Track latency, token usage, and cost per request
- Monitor error rates and failure patterns
- Alert on quality degradation (rising "I don't know" rate, increasing edit/rejection rate)
5. Cost and performance management
- Caching: Cache embeddings and frequent query results
- Model selection: Use smaller/cheaper models where quality is sufficient
- Rate limiting: Prevent abuse and control costs
- Batch processing: For non-real-time AI tasks, batch to reduce API calls
6. User experience polish
- Streaming responses (don't make users wait 5 seconds staring at a spinner)
- Loading states and progress indicators
- Clear indication when AI is generating vs. when it's done
- Easy feedback mechanism (thumbs up/down, report issue)
- Graceful degradation when AI service is unavailable
The timeline reality
| Phase | Prototype | Production | |-------|-----------|------------| | Prompt engineering | 2 days | 1-2 weeks (with eval) | | RAG pipeline | 3 days | 2-4 weeks | | UI integration | 2 days | 1-2 weeks | | Error handling | 0 days | 1 week | | Evaluation & testing | 0 days | 1-2 weeks | | Monitoring & ops | 0 days | 1 week |
A prototype that took a week to build typically needs 4-8 more weeks to reach production quality. Plan accordingly.
The most important shift
Stop thinking of it as "an AI feature" and start thinking of it as "a product feature that uses AI." It needs the same reliability, testing, monitoring, and user experience standards as any other feature you ship.
The demo proved the concept. Production proves the value.