Out of the box, an AI model knows a lot about the world but nothing about your company: your policies, products, contracts or support history. Retrieval-Augmented Generation (RAG) is the most common way to fix that without training a model. Here's how it works in plain terms.
The idea in one sentence
When someone asks a question, the system first finds the most relevant passages from your documents, then gives them to the LLM with the question and asks it to answer using only those passages, citing its sources.
The pipeline
1. Ingest
Collect documents from where they live (PDFs, Word files, Confluence, SharePoint, Google Drive, help-desk articles, databases) and extract clean text. Keep metadata such as title, source URL, date and access permissions.
2. Chunk
Split documents into passages of a few hundred words, ideally along natural boundaries like headings and sections. Good chunking is one of the biggest factors in answer quality.
3. Embed and index
Convert each chunk into an embedding, a list of numbers that captures its meaning, and store it in a vector index. Options range from pgvector inside your existing PostgreSQL to dedicated vector databases. For most business projects, pgvector is plenty and keeps the stack simple.
4. Retrieve
At question time, embed the question and find the closest chunks. In practice, the best results come from hybrid search, which combines semantic (vector) search with classic keyword search, followed by a re-ranking step that puts the most relevant passages first.
5. Generate
prompt = f"""Answer the question using only the sources below.
If the answer is not in the sources, say you don't know.
Cite sources as [1], [2].
Sources:
{numbered_chunks}
Question: {question}"""
The model writes an answer grounded in your content, with citations users can click to check.
What separates a demo from a production system
- Permissions. Users must only retrieve documents they're allowed to see. Filter by access rights at retrieval time, not after.
- Freshness. Re-index automatically when documents change, and remove deleted ones.
- "I don't know". The assistant should say so when the answer isn't in the sources, rather than guessing.
- Evaluation. Build a test set of real questions with known answers and measure accuracy whenever you change chunking, models or prompts.
- Feedback. Thumbs up/down on answers shows you where the knowledge base has gaps.
Is RAG still needed with huge context windows?
Modern models can read very long inputs, and for a handful of documents you can simply include them all. But for thousands of documents, frequent updates, per-user permissions and cost control, retrieval is still the practical approach. Many systems now combine the two: retrieve generously, then let a long-context model reason over the results.
Where RAG delivers value fastest
- Internal knowledge assistants for HR policies, IT help and SOPs,
- customer support answers drafted from product docs and past tickets,
- sales enablement, such as finding the right case study or pricing rule,
- compliance and legal search across contracts and regulations.
Want an AI assistant that actually knows your business? Talk to our Python and AI team about a proof of concept on your own documents.