Columns › 9. RAG and tools › Part 38
RAG Basics: Look It Up Before Answering, and Why That Matters
RAG 入門:讓模型先查資料再回答,核心價值在哪裡
Start from the three problems a model has when it answers on its own: it doesn’t know your data, its knowledge has a cutoff, and you can’t tell when it’s wrong. Then the two paths of RAG, its four core values, how it compares with long context, fine-tuning and plain search, and a minimal Go example that calls no API at all.
In this article
- What this article solves
- Step 1: Three problems with a model answering on its own
- Step 2: What RAG is: an open-book exam
- Step 3: The core value: why it’s worth doing
- Step 4: How to choose between the options
- Step 5: A minimal “look it up, then answer” in Go
- Step 6: When not to use RAG
- Step 7: How to read on
- Common pitfalls
- Key takeaways
- Next
The previous article, “Writing an MCP Server in Go: Connecting AI to OpenProject”, let the agent read tickets through MCP. A ticket tells the agent what to do, but how to do it is usually written somewhere else: on-call runbooks, API docs, incident write-ups. Part 9 is about exactly that: before the model answers, it looks things up in the team’s own material. This approach is called RAG (Retrieval-Augmented Generation).
The column has four articles, in order of difficulty. This one is the beginner level: only the ideas and the core value, no AI account needed. The next is intermediate: where RAG quality comes from. The one after is advanced: how to evaluate it, how it fails and how to keep hallucinations out. The last builds the whole pipeline in Go.
What this article solves
In the middle of the night the shop’s payment service starts returning PAYMENT_TIMEOUT. The person on call asks a chatbot, “Can I just retry a payment timeout?”, and it answers with confidence: “Yes, retry three times with exponential backoff.” That sounds reasonable, but the team’s runbook actually says: a charge must not be retried until its idempotency key is confirmed, or the customer may be charged twice (part 4, “Idempotency and Distributed Locks”, explained why).
The model isn’t lying. It simply doesn’t know your rules, so it answers with what payment systems in general probably do. This article covers:
- The three fundamental problems of a model answering on its own.
- RAG in one sentence, and what each of its two paths does.
- The four core values of RAG, and what it cannot fix.
- How it compares with long context, fine-tuning and plain search, and when to pick which.
- A minimal “look it up, then answer” in Go, to see what the data looks like at every step.
Step 1: Three problems with a model answering on its own
A large language model (LLM) is trained on a vast amount of public text. It’s very good at understanding questions, writing clearly and reasoning, but when the question is about your team it has three fundamental problems:
- It doesn’t know your data. Your runbooks, API docs, incident notes and order states never appeared in its training data. It can only guess.
- Its knowledge has a cutoff. It knows nothing that happened after training. The refund rule changed last week, the service launched yesterday: it has seen neither.
- You can’t tell when it’s wrong. Its answers are always fluent; a guess sounds exactly like a fact. This is what people call hallucination: not deliberate deception, but a model trained to produce the most answer-like text, not to say only what can be looked up.
The first two are “no data”; the third is “no evidence”. RAG deals with both at once: it hands the data to the model, and it makes every sentence of the answer point to its source.
Step 2: What RAG is: an open-book exam
The easiest way to picture it is an open-book exam. In a closed-book exam you rely on memory, and you don’t know when you remember wrong. In an open-book exam you first look up the right page, answer from what the book says, and can even note “see page 42”.
RAG lets the model sit an open-book exam, in two paths:
- Building the index (offline, in advance): split the documents into chunks, build an index that can find them quickly (usually a vector index; the next article goes into it), and store each chunk with where it came from, which file and which lines. Redo it only when documents change.
- Answering a question (online, every time): first use the question to retrieve the most relevant few chunks from the index; put them, numbered, into the prompt, and require the model to answer only from these chunks, marking which chunk each sentence uses.
Figure: the top row is the indexing done in advance; the bottom row is what happens for every question. The model appears only in the last step.
Notice that the model is responsible only for the last step: reading the chunks and writing the answer. Retrieval finds the material, not the model. This matters: most of RAG’s quality depends on how well retrieval finds things, not on how clever the model is.
Step 3: The core value: why it’s worth doing
RAG isn’t magic that makes AI smarter. Its value is concrete, in four points:
One: answers have sources you can check. Every sentence maps to a passage in some document. A reader can open the original to confirm; when the answer is wrong, you can tell whether the document is wrong or the model misread it. Where mistakes are costly, in payments, permissions or compliance, being checkable matters far more than sounding good.
Two: updating the data needs no retraining. When a rule changes, edit the document and rebuild the index for those chunks; the next question uses the new content. The model isn’t touched and nobody waits for training. You control how fresh the data is.
Three: permissions can be enforced. Retrieval can filter by identity first: support staff find only support documents, engineers also find internal runbooks. The model sees only the retrieved chunks, so what it never sees, it can’t say. That is far more reliable than telling the model “please don’t reveal confidential information”.
Four: the model can be swapped. The data and the index are yours; the model is only the last step that reads the chunks. Switch to a cheaper or stronger model tomorrow and the index stays, and so does the question set you evaluate with.
Equally, be clear about what RAG cannot fix:
- If the documents are wrong, contradict each other or are long out of date, RAG faithfully cites the wrong content. Garbage in, garbage out.
- If the question needs reasoning or calculation rather than lookup (say, “design a sharding scheme for us”), retrieval doesn’t help much.
- If retrieval misses the right chunk, the model can only say “I don’t know”, or worse, answer from unrelated chunks anyway. The next two articles deal with exactly these.
Step 4: How to choose between the options
To make a model “know your data”, there are four common approaches:
| Approach | How | Good for | Not good for |
|---|---|---|---|
| Put it in the context | Paste whole documents into every prompt | Very little data (a few documents), one-off analysis | Too many documents won’t fit; you pay for all of it every time; hard to enforce permissions |
| RAG | Retrieve relevant chunks, include only those | Many documents, frequent updates, citations, per-user permissions | Questions that need the whole corpus read (“how many of all our contracts…”) |
| Fine-tuning | Train the model further on your data | Changing the model’s tone, format, domain vocabulary | Making it “remember facts”: imprecise, can’t cite, redo on every change |
| Plain search | Keyword search, list documents for a person | Users know which words to search; they want the document itself | Users want one direct answer |
A common misconception is “fine-tune the model on our data and it will know our rules”. Fine-tuning is more like training a habit of speech than handing over a book to look things up in: the facts it remembers are imprecise, it can’t say where they came from, and every data change means training again. For facts and sources, use RAG; for style and format, consider fine-tuning. The two can also be combined.
Another common question: “Context windows are huge now. Do we still need RAG?” With only a few documents, putting them straight into the context really is simplest. But with hundreds of documents, edited daily, visible to different people differently, pasting everything every time is slow and expensive, and you can’t filter by identity. In practice the two are often combined: retrieval picks the relevant documents, and those few go into a longer context whole.
Step 5: A minimal “look it up, then answer” in Go
To see what the data looks like at each step, first write a version that calls no AI service at all: retrieval is the simplest possible “how many words the question and the chunk share”, and at the end it only prints the prompt that would go to the model. Real embeddings, vector indexes and calls to Claude are left for the implementation article.
// cmd/rag-toy/main.go: the smallest RAG, just to see how the data flows
package main
import (
"fmt"
"sort"
"strings"
)
// Chunk is a piece of a document that remembers where it came from
type Chunk struct {
ID string // the id the model cites
Source string // file and lines
Text string
}
var chunks = []Chunk{
{"c1", "runbooks/payment-timeout.md:12-18", "Do not retry a charge until its idempotency key is confirmed, or the customer may be charged twice."},
{"c2", "runbooks/payment-timeout.md:20-26", "First check the payment latency dashboard: is the bank slow, or our own database?"},
{"c3", "api/order.md:40-44", "An order is pending, paid, shipped or cancelled."},
}
// score: how many words of the question appear in the chunk (a real system uses embeddings plus keyword search)
func score(question, text string) int {
n := 0
for _, w := range strings.Fields(question) {
if strings.Contains(text, w) {
n++
}
}
return n
}
// retrieve returns the k chunks with the highest score; a score of 0 is left out
func retrieve(question string, k int) []Chunk {
type hit struct {
c Chunk
s int
}
var hits []hit
for _, c := range chunks {
if s := score(question, c.Text); s > 0 {
hits = append(hits, hit{c, s})
}
}
sort.SliceStable(hits, func(i, j int) bool { return hits[i].s > hits[j].s }) // ties keep their order
var out []Chunk
for i := 0; i < len(hits) && i < k; i++ {
out = append(out, hits[i].c)
}
return out
}
// buildPrompt puts the numbered chunks into the prompt and says to answer only from them
func buildPrompt(question string, found []Chunk) string {
var b strings.Builder
b.WriteString("Answer only from the chunks below; end every sentence with the [id] it relies on; if the chunks don't say, answer \"The documents don't say.\"\n\n")
for _, c := range found {
fmt.Fprintf(&b, "[%s] (%s) %s\n", c.ID, c.Source, c.Text)
}
fmt.Fprintf(&b, "\nQuestion: %s\n", question)
return b.String()
}
func main() {
q := "retry payment timeout"
found := retrieve(q, 2)
if len(found) == 0 {
fmt.Println("The documents don't say.") // nothing found: don't ask the model, just say so
return
}
fmt.Print(buildPrompt(q, found))
}Run go run ./cmd/rag-toy, and what it prints is everything that would be sent to the model:
Answer only from the chunks below; end every sentence with the [id] it relies on; if the chunks don't say, answer "The documents don't say."
[c1] (runbooks/payment-timeout.md:12-18) Do not retry a charge until its idempotency key is confirmed, or the customer may be charged twice.
[c2] (runbooks/payment-timeout.md:20-26) First check the payment latency dashboard: is the bank slow, or our own database?
Question: retry payment timeoutThis small example shows a few basic rules of RAG that the next three articles keep using:
- Every chunk carries its source (file and lines) and a short id for the model to cite.
- The prompt says to answer only from the chunks, and what to say when they don’t contain the answer.
- When retrieval finds nothing, don’t call the model at all. That is the cheapest and most reliable defence against hallucination.
- The model sees only the retrieved chunks:
c3has nothing to do with this question, so it never reaches the prompt.
Step 6: When not to use RAG
RAG has a cost: curating documents, building the index, maintaining and evaluating it. In these cases, hold off:
- Very little data. With three to five documents, put them straight into the prompt.
- The question needs reasoning, not lookup. “Why is this SQL slow?” needs the model to read the code, not to search the docs.
- The data isn’t curated yet. Contradictory, unmaintained, half-outdated documents: tidy them first. RAG makes mistakes more convincing, because it attaches a “source”.
- The answer lives in a live system. “What state is order 1234 in?” needs the order service, not a document. Use tool calling to let the model query the API; the implementation article puts retrieval and tools in one flow.
Step 7: How to read on
The rest of this column goes up in difficulty, one article after another:
- Intermediate: where quality comes from. Chunking, embeddings, vector databases, hybrid vector and keyword search, reranking, prompt assembly and metadata filters: how each stage shapes the answer.
- Advanced: evaluation, hallucinations and common failures. Building a question set, measuring retrieval and answers, blocking hallucinations layer by layer, where failures usually come from, and monitoring once it’s live.
- Implementation: a RAG pipeline in Go. Everything above, built: chunking, pgvector, hybrid search, Claude’s tool use, citation checks and tests.
Common pitfalls
Thinking RAG makes the model “learn” the data. The model learns nothing; every question retrieves and reads the chunks again. Break the index and you break the answers.
Judging only how polished the answer sounds. A polished answer can have no basis at all. First check that the cited chunk really supports the sentence.
Throwing every document in. Outdated drafts and documents with personal data get retrieved too, and the model cites them. Filter before anything enters the corpus.
Asking the model anyway when nothing is found. If retrieval returns nothing, say “the documents don’t say” instead of letting the model improvise.
Treating fine-tuning as a way to memorise data. For facts and sources, retrieve; fine-tuning is for style and format.
Key takeaways
- A model answering on its own has three problems: it doesn’t know your data, its knowledge has a cutoff, and you can’t tell when it’s wrong.
- RAG is an open-book exam: offline, chunk and index the documents; online, retrieve the relevant chunks first, then have the model answer only from them and cite its sources.
- Retrieval finds the material, not the model; most of RAG’s quality depends on retrieval.
- Four core values: checkable, data updates without retraining, permissions by identity, a swappable model.
- RAG can’t correct wrong documents, and doesn’t help with questions that need reasoning rather than lookup.
- For facts and sources use RAG, for style and format consider fine-tuning; with few documents just use the context.
- Chunks carry sources and ids, answer only from the chunks, don’t call the model when nothing is found: these three rules hold from the toy to a production system.
Next
Now that we know why RAG is useful, the next question is: why do some RAG systems answer precisely while others miss the point? The next article, “Where RAG Quality Comes From: Chunking, Embeddings, Vector Databases and Hybrid Search”, breaks retrieval into five stages and looks at how each one decides whether the answer is any good.