RAG (Retrieval-Augmented Generation) is the most common pattern for building AI features that need to answer questions about your specific data. If you're a software engineer evaluating or implementing RAG, here's what you need to know.
The problem RAG solves
LLMs have knowledge cutoffs and don't know about your company's documents, codebase, or product data. Fine-tuning is expensive and slow to update. RAG gives the model relevant context at query time without retraining.
How it works
User question
↓
Embed question → Vector search → Retrieve relevant chunks
↓
Combine chunks + question → Send to LLM → Generate answer
Step 1: Ingestion. Split your documents into chunks. Generate vector embeddings for each chunk. Store in a vector database (pgvector, Pinecone, etc.).
Step 2: Retrieval. When a user asks a question, embed the question, find the most similar chunks via vector search, optionally rerank results.
Step 3: Generation. Send the retrieved chunks + the user's question to an LLM with instructions to answer based only on the provided context.
Key engineering decisions
Chunking strategy
How you split documents dramatically affects retrieval quality:
- Fixed-size chunks (500-1000 tokens) — simple but may split mid-sentence
- Semantic chunking — split on paragraph/section boundaries
- AST-aware chunking — for code, split on function/class boundaries
Bad chunking is the #1 cause of poor RAG quality.
Embedding model
Choose based on your content type. General-purpose models (OpenAI text-embedding-3-small) work for most text. Code-specific embeddings exist for codebase RAG.
Retrieval parameters
- Top-k: How many chunks to retrieve (typically 3-10)
- Similarity threshold: Minimum score to include a chunk
- Metadata filtering: Restrict search to relevant document types, dates, or categories
Reranking
Initial vector search is fast but imprecise. A reranker model can reorder results for better relevance before sending to the LLM.
When RAG works well
- Question-answering over documentation
- Support assistants with knowledge bases
- Internal search across wikis and wikis
- Codebase understanding assistants
When RAG struggles
- Questions requiring synthesis across many disparate sources
- Real-time data that changes frequently
- Highly structured data better served by SQL queries
- Tasks requiring reasoning beyond the retrieved context
Production considerations
- Citation: Always show users which sources were used
- Hallucination guardrails: Instruct the LLM to say "I don't know" when context is insufficient
- Incremental indexing: Update embeddings when source documents change
- Evaluation: Maintain a test set of questions with expected answers; measure retrieval accuracy and generation quality separately
RAG is not magic — it's a search system with an LLM synthesizer on top. Get the retrieval right, and generation quality follows.