Key points
- When an agent gives a wrong answer from your documents, the fault is usually in retrieval, not in the model.
- Split documents along their own structure, and keep each chunk labelled with where it came from.
- Vector search finds meaning and keyword search finds exact terms. Most real systems need both.
- Make the model cite its sources, and give it a clear instruction for when nothing relevant is found.
- A small set of real questions with known answers is enough to tell whether a change helped or hurt.
1. What retrieval-augmented generation is
A language model knows what was in its training data. It does not know your school's fee structure, your shop's return policy or the price list you updated last week. Ask it anyway, and it will often produce an answer that sounds reasonable and is invented.
Retrieval-augmented generation, usually shortened to RAG, is the standard remedy. Before the model answers, the system searches your own documents for passages related to the question and places them in front of the model. The model is then asked to answer from those passages.
The idea is simple, and that is the trap. The model can only be as right as the passages it is shown. If the search brings back the wrong passage, half a passage or an outdated one, the model will write a fluent answer on top of it. Most of this note is about the search, because that is where most of the errors begin.
2. Splitting documents into chunks
Documents are too long to search and pass to the model whole, so they are cut into pieces called chunks. Each chunk is indexed separately, and the search returns the few chunks that best match the question.
The easiest method is to cut every fixed number of characters. It is also the method that causes the most harm. A fixed cut does not care about meaning. It will separate a heading from the paragraph below it, split a table in two, or end a chunk in the middle of the sentence that holds the answer.
A better method follows the document's own structure. Cut at headings, then at paragraphs, and only cut inside a paragraph if it is very long. Keep tables and numbered lists whole. Attach the document title and section heading to every chunk, so a chunk that says "The fee is payable in two instalments" also carries the label that says which programme it belongs to.
Read twenty of your own chunks before building anything else. If a chunk does not make sense to you without its neighbours, it will not make sense to the model.
3. Why chunk size and overlap matter
Chunk size is a trade-off with a cost on both sides. Small chunks are precise: each is about one thing, so a match is likely to be on topic. But a small chunk can lose its context. "This does not apply to scholarship students" is useless if the rule it refers to sits in the chunk before.
Large chunks keep context together, but they mix several topics. The search finds them less reliably, because the one relevant sentence is diluted by everything around it. They also fill the model's input with text it does not need, which costs more and gives the model more to be distracted by.
Overlap softens the edges. Each chunk repeats a little of the end of the previous one, so a sentence that falls on a boundary appears complete in at least one chunk. There is no universal right size. A policy document with long, linked clauses needs larger chunks than a list of short questions and answers. Choose a starting point, then let the evaluation in section 10 decide.
4. Embeddings and vector search in plain words
An embedding model turns a piece of text into a long list of numbers. The useful property is that texts with similar meaning get similar lists. "What time do you open" and "your shop timings" share almost no words, but their number lists end up close together.
A vector store is a database built to hold these lists and find the nearest ones quickly. At indexing time, every chunk is converted and stored. At question time, the question is converted the same way, and the store returns the chunks whose numbers are closest.
This is search by meaning, and it is what lets a customer use their own words. It has weaknesses. It is poor at exact items such as a product code, a roll number, a person's name or an abbreviation, because those carry little general meaning. And it always returns something. The nearest chunk to an off-topic question is still the nearest, even when it is not relevant at all.
If your users write in Urdu, Roman Urdu or a mix, check that the embedding model you pick handles those well. Test it with real questions in each form before relying on it.
5. Hybrid search: keywords plus vectors
Keyword search is the older method. It looks for chunks that contain the words in the question and ranks them by how often and how distinctively those words appear. It has the opposite profile to vector search: excellent at exact terms, blind to paraphrase.
Hybrid search runs both and combines the results. A question about "the refund rule for order LHR-2291" needs meaning to find the refund policy and exact matching to find the order code. Neither method alone handles both halves.
Many vector stores and frameworks, including LangChain, offer a way to combine the two. If yours does not, running two searches and merging the lists in code is a reasonable start. For any knowledge base that contains codes, names, model numbers or section references, hybrid search is worth the extra step.
6. Re-ranking
The first search is built for speed. It compares the question against every chunk using numbers calculated in advance, so it is fast but approximate. The chunk ranked first is often good, and the truly best chunk is often sitting at position six.
A re-ranker is a second, slower pass. It takes the question and each candidate chunk together and judges how well that chunk answers that question. Because it reads the pair together, it is more accurate than the first search. Because it is slower, it is only applied to a shortlist.
The usual pattern is to fetch a generous shortlist with hybrid search, re-rank it, and pass only the top few chunks to the model. This matters because the model pays most attention to what it is given. Five strong chunks produce a better answer than twenty mixed ones.
7. Giving the model sources and making it cite them
How the chunks are presented matters as much as which ones are chosen. Give each chunk a label, such as its document title, section and date, and number them. Then instruct the model to answer only from these passages and to state which passage each claim came from.
Citations do two jobs. For the reader, they turn "trust me" into "see section 3 of the fee policy". For the builder, they make every answer checkable. When an answer is wrong, you can see at once whether the model misread a good passage or was handed a bad one. Those are different faults with different fixes.
Citations are also a quiet discipline on the model. An instruction to point at a source for each statement makes unsupported statements stand out, both to the model and to anyone reviewing the logs. It does not make invention impossible, so spot-check that cited passages do say what the answer claims.
An answer without a traceable source should be treated as a guess, however well it reads.
8. What to do when nothing relevant is found
This is the step most often left out, and it is the one that makes agents lie. Vector search always returns its nearest chunks. If a parent asks a school's agent about hostel facilities and the school has no hostel document, the search will still return something, perhaps a paragraph about the canteen. A model told to "use the context" will try.
The system needs an explicit path for "not found". There are three practical layers.
- A relevance threshold: if the best chunk's score from the search or re-ranker is below a set level, treat the result as empty.
- A clear instruction: if the passages do not contain the answer, say so and do not answer from general knowledge.
- A useful fallback: offer to pass the question to a person, and log it as a gap in the knowledge base.
9. Keeping the knowledge base current
A retrieval system is a copy of your documents taken at one moment. The documents keep changing and the copy does not, unless someone makes it. The classic failure is a new price list added beside the old one, so the search returns both and the model picks either.
Treat the index as something that is maintained, not installed. Each chunk should record its source file and the date it was indexed. When a document changes, its old chunks are deleted and the new version is indexed in their place. When a document is withdrawn, its chunks go too.
Give this a named owner and a routine. For a small business, that can be as plain as one shared folder that is the single source of truth, and a workflow in a tool such as n8n that re-indexes a file whenever it is replaced.
10. Evaluating with a small set of real questions
Without measurement, every change to chunk size or search settings is a guess. The measurement does not need to be elaborate. It needs to be real.
Collect thirty to fifty questions that actual users ask, in their actual wording. For each, write down the correct answer and which document section contains it. Include a few questions that your documents do not answer, where the correct response is "I do not have that information".
Then check two things separately for each question.
- Retrieval: did the correct section appear among the chunks passed to the model?
- Answer: given those chunks, was the final answer correct, complete and properly cited?
- Refusal: for the unanswerable questions, did the system decline instead of inventing?
Run the same set after every change. If retrieval fails, fix chunking or search. If retrieval succeeds and the answer is still wrong, fix the instructions. Vector stores and retrieval of this kind are part of what the TIDES AI bootcamp teaches, and this test set is the habit that matters most.
