Retrieval Augmented Generation Explained for Business

Retrieval augmented generation (RAG) is a technique where a language model first searches your own documents for relevant passages and then writes its answer based on those passages. It lets you ask questions about your contracts, manuals, product data or policies and get answers with a link to the source. The model is not retrained on your data: your content stays in your own storage and is retrieved per question.

The short version:

RAG connects a language model to your own knowledge base without training the model on your data.

Answer quality depends mostly on your documents and the retrieval step, not on which model you pick.

RAG reduces hallucinations but does not eliminate them, so citations and an explicit "I don't know" path are mandatory.

For most business questions, RAG is cheaper and easier to audit than fine-tuning.

What is retrieval augmented generation?

The term comes from a 2020 paper by Lewis and colleagues at Facebook AI Research: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. The idea was straightforward. A language model only knows what was in its training data. If you want it to answer questions about information it has never seen, you hand it that information at the moment of the question.

A general model like ChatGPT or Claude does not know your price list, your service terms or the troubleshooting guide for your product. Ask about them and you get a generic or invented answer. With RAG, the system first looks up the relevant passages in your own sources and puts them in the prompt. The model then answers based on what those passages say.

Is ChatGPT a RAG model? Not by itself. ChatGPT is a language model that uses RAG-like techniques when it searches the web or reads files you upload. The difference with your own RAG system is control: you decide which sources are included, who can see what, and where the data lives.

How RAG works, step by step

A RAG system has two parts: preparing your knowledge (once, and again whenever content changes) and answering a question (every time).

Preparing the knowledge base:

Collect sources. PDFs, Word files, SharePoint, Google Drive, your help desk, product data from your ERP.
Chunk. Split documents into pieces of a few paragraphs, with metadata such as file name, date and department.
Embed. Convert each chunk into a vector, a list of numbers that captures its meaning. Chunks with similar meaning end up close together.
Store. Save the vectors in a vector database or a regular database with vector search.

Answering a question:

The user's question is converted into a vector as well.
The system retrieves the closest chunks, usually combined with classic keyword search (hybrid search).
The top five to twenty chunks are sent to the model together with the question and instructions.
The model writes an answer and cites the sources it used.

A practical example: a field service company has hundreds of product sheets and repair guides. A technician asks an internal assistant, "What does error E4 on indoor unit type X mean and what should I check?" The system finds the right page in the manual, the model summarizes the steps and links to the page. No more scrolling through a folder of PDFs on a phone.

Why RAG reduces hallucinations but does not solve them

Language models predict text. When they lack information, they sometimes produce a plausible answer anyway. RAG helps because the model receives the facts, but three failure points remain:

Retrieval finds the wrong chunks. The model answers fluently, based on the wrong source. In our experience, most errors in RAG systems start here.
The source is wrong. An old version of a manual, two conflicting price lists, a half-updated policy. RAG exposes messy documentation very quickly.
The model fills gaps. Instead of saying the source is silent, it blends in general knowledge.

What works in production:

Always show the sources used, with a link to the original document.
Instruct the model to say "this is not in the available documents" when the answer is missing, and test that behavior.
Build an evaluation set of 50 to 100 real questions with known correct answers, and run it after every change.
Remove outdated documents from the index instead of leaving them next to the new version.

These are the same issues that sink many AI initiatives. We cover the pattern in why AI projects fail: rarely the model, usually data and ownership.

RAG vs fine-tuning: which one do you need?

A common question is whether you should train a model on your own data instead. That is fine-tuning, and it solves a different problem.

	RAG	Fine-tuning
What it does	Retrieves current information per question	Changes the model's behavior or style
Good for	Facts, documents, prices, procedures	Fixed tone, strict output format, classification
Freshness	Immediate: new document, new answer	Retrain on every change
Citations	Yes, per answer	No, knowledge is baked in
Access control	Filter per user at retrieval time	Impossible to separate after training
Cost profile	Storage plus tokens per question	Training runs plus hosting a custom model

For almost every question of the form "what do our documents say about X", RAG is the right tool. Fine-tuning makes sense when you want the model to learn a behavior, such as classifying invoices into your own chart of accounts, and you have thousands of good examples. Combining both is possible but rarely necessary: a well-built RAG system with clear instructions covers most business needs.

Is RAG obsolete now that models handle a million tokens?

Claude Opus 5.5 and Sonnet 5 have a 1 million token context window, which is hundreds of pages. For a small document set you can sometimes skip retrieval and send everything. For a knowledge base of thousands of documents you cannot, and even when it fits, you pay for every token on every question. You also still need per-user permissions. So RAG is not obsolete, but it is changing: systems retrieve larger chunks and let the model decide what else to look up. That is essentially what an AI agent does when search is one of its tools.

What does a RAG system cost?

There are three cost buckets: build, run and maintain.

Running cost per question. This is low. Assume a question with retrieved context uses 8,000 input tokens and 500 output tokens. With Claude Haiku 4.5 ($1 per million input, $5 per million output) that is roughly $0.01 per question. With GPT-6 Luna ($0.10 / $0.50) it is a fraction of that. With a heavier model like Claude Opus 5.5 ($4 / $20) it is around $0.04. Prices as of September 2026. At 5,000 questions a month, model costs land in the tens to low hundreds of dollars. Embeddings and storage add to that but are usually smaller.

Build cost. This depends on the number of sources, integrations and permission requirements. An internal assistant on a single document source is a different project from a customer-facing assistant that also pulls order status from your ERP. Most of the effort goes into connecting and cleaning sources, permissions and the evaluation set, not into the model.

Maintenance. Documents change, sources move, models get new versions. Plan for recurring work on the index and tests.

Privacy and data protection: where does your data go?

RAG sends pieces of your documents to a language model. If those contain personal data, privacy law such as the GDPR applies. Key points:

Business APIs do not train on your data by default. OpenAI states that its API platform and ChatGPT Business and Enterprise are not used for training by default. Anthropic says the same for its business Claude plans.
Data location differs by provider. OpenAI offers European data residency for Enterprise, Edu and the API. Anthropic's first-party API offers "global" or "us" as inference location, with no EU option. If you need to stay within the EU, factor that into your model and hosting choice.
Uncontrolled use is a data breach risk. The Dutch Data Protection Authority has warned that employees pasting personal data into chatbots against company policy can trigger a reportable data breach. A governed RAG system is one way to prevent that.
Permissions belong in retrieval. Filter by user rights when fetching chunks. Never let the model decide whether someone may read an HR file.

For more on this, read ChatGPT and GDPR for business use.

Where RAG works well, and where it does not

Good fits:

Internal knowledge assistant for procedures, manuals and policies.
Customer service drafts: the system drafts a reply from the knowledge base, an agent reviews and sends. See AI chatbots for customer service.
Proposals and tenders: reuse past proposals and product specs for a new bid.
Technical documentation for field technicians and support staff.

Poor fits:

Counting and calculating across data. "What was our Q2 revenue with customer X?" is a database question. Let an agent run a query against your ERP or accounting system instead.
Knowledge that is not written down. RAG cannot extract what only lives in your best employee's head.
High-stakes decisions without review. RAG produces a sourced draft, not a decision.

Frequently asked questions

What is RAG in AI?

RAG stands for retrieval augmented generation. A language model first retrieves relevant information from a defined set of documents and uses it to write its answer. This lets it answer questions about information outside its training data, such as your company documents, with citations.

What is the difference between an LLM and RAG?

An LLM (large language model) is the model itself, such as GPT-6 or Claude. RAG is an architecture around that model: a retrieval step that finds and passes the right context. The LLM writes the answer, RAG makes sure it has the right information.

Is RAG becoming obsolete with larger context windows?

No. Larger context windows make RAG more flexible because you can pass bigger chunks. For thousands of documents, per-user permissions and predictable cost per question, retrieving the right pieces is still necessary.

Will my data be used to train the model?

With the business APIs of OpenAI and Anthropic, not by default according to their own terms. Check data location and put a data processing agreement in place. Consumer accounts have different terms and are not suitable for company data.

How long does it take to build a RAG system?

A first working version on a single source can be ready within weeks. Most of the time goes into cleaning documents, connecting sources, permissions and testing with real questions. A production system with multiple sources and integrations fits in an MVP timeline of 4 to 8 weeks.

Next steps

Start with one well-defined question: what information do employees or customers look up most often, and where does it live? Collect 50 real questions and the documents that contain the answers. That tells you quickly whether RAG works for your case and what needs to happen to your documentation first.

Want to find out which knowledge sources in your business are good candidates? Take the free AI scan or see how we build AI agents and knowledge systems, with the code and data owned by you.

Get started today