This resume is also available in English.

9. RAG and tools2 / 5 · All columns

Columns › 9. RAG and tools › Part 39

Where RAG Quality Comes From: Chunking, Embeddings, Vector Databases and Hybrid Search

RAG 的品質從哪裡來:切塊、embedding、向量資料庫與混合檢索

Split RAG into five stages and see how each one decides whether the answer is any good. Write heading-based chunking, cosine similarity and RRF fusion in Go, then cover how a vector database fits in and what to watch, metadata filters and the context budget.

  • RAG
  • AI
  • Go
  • Embedding
In this article
  1. What this article solves
  2. Step 1: Five stages, each one loses points
  3. Step 2: Chunking: size, overlap and splitting by structure
  4. Step 3: Embeddings: turning text into vectors you can compare
  5. Step 4: Hybrid vector and keyword search, merged with RRF
  6. Step 5: Vector databases: how to choose, how they fit, what to watch
  7. Step 6: Reranking: fetch many, then sort precisely
  8. Step 7: Prompt assembly: number of chunks, order and the context budget
  9. Step 8: Metadata filters: permissions, versions, language and time
  10. Common pitfalls
  11. Key takeaways
  12. Next

The previous article, “RAG Basics: Look It Up Before Answering, and Why That Matters”, said that in RAG, retrieval finds the material, not the model. This one is the intermediate level: with the same documents and the same model, why does one RAG system answer precisely and another miss the point? The answer is almost always on the retrieval side.

What this article solves

Someone on call asks: “Can I retry a charge on PAYMENT_TIMEOUT?” The system replies “The documents don’t say.” But the runbook does say. There are many possible reasons:

  • The sentence was cut at the boundary between two chunks, leaving half on each side.
  • Vector search treated “payment timeout” and “order timeout” as very similar, and the top results are all order documents.
  • A string like the error code PAYMENT_TIMEOUT is something vector search is bad at anyway.
  • It was found, but ranked ninth, and only the top five were kept.
  • It was found and put into the prompt, but buried under so many unrelated chunks that the model never used it.

This article breaks RAG into five stages, explains the trade-offs in each, and writes the key pieces in Go:

  1. The five stages, and how each one loses points.
  2. Chunking: size, overlap, splitting by structure, and the metadata each chunk carries.
  3. Embeddings: vectors, similarity, and using one model consistently.
  4. What vector search and keyword search are each good at, and merging them with RRF.
  5. Vector databases: how to choose, how they fit RAG, and what to watch.
  6. Reranking: fetch many, then sort them precisely.
  7. Prompt assembly: number of chunks, order, the context budget and the refusal rule.
  8. Metadata filters: permissions, versions, language and time.

Step 1: Five stages, each one loses points

Chunking size and overlap split by heading keep heading path Embedding one model multilingual new model: rebuild Retrieval vector + keyword RRF fusion permission filter Rerank fetch 30–50 keep the top few weigh the cost Prompt chunks carry ids context budget refuse if missing The five stages a question goes through (and what each one tunes) What an earlier stage drops, a later one cannot bring back: tune from left to right.

Figure: a question passes through five stages; under each is what it most often tunes. What an earlier stage drops, a later one cannot bring back.

Think of the five stages as one pipeline in series: if chunking is bad, a great embedding model won’t help; if retrieval misses, reranking has nothing to rank; if the chunk is retrieved but the prompt is badly assembled, the model may still not use it. So the order for tuning RAG is usually front to back: first make sure the right passage exists and can be retrieved, then worry about ranking and the prompt.

Step 2: Chunking: size, overlap and splitting by structure

Chunking decides the unit of retrieval. Too large, and one chunk mixes several topics, its vector gets blurry and it wastes space in the prompt; too small, and a sentence loses its context (“it must not be retried”: what is “it”?).

There are three common approaches:

  • Fixed length: one chunk every N characters. Simple, but it often cuts a sentence, a table or a code block in half.
  • By structure: split at Markdown headings, paragraphs and code blocks. A section stays together; the best fit for technical documents.
  • By meaning: use embeddings to detect where the topic changes. Sometimes better, but expensive and hard to debug; rarely the first step.

For technical documents, start with “split by heading, then cap the size”: first split at ## and ###; if a section is too long, split it by paragraph, keeping a little overlap (say the last sentence or two of the previous chunk) so a key sentence doesn’t land on the boundary.

Besides its text, each chunk carries metadata: source file, line, heading path (for example “Payment timeout › Can I retry?”), document type, last update, roles allowed to read it. The heading path is especially useful: put it in front of the text when embedding, and the “Can I retry?” chunk knows it’s about payment timeouts.

// internal/rag/chunk.go: chunk Markdown by heading; split an over-long section by paragraph
package rag

import "strings"

type Chunk struct {
	Source string // file path
	Line   int    // first line (from 1)
	Path   string // heading path, e.g. "Payment timeout › Can I retry?"
	Text   string
}

// Split cuts at ## and ###; each piece is at most max runes, split again at blank lines if longer
func Split(source, doc string, max int) []Chunk {
	var out []Chunk
	var heads []string // the current heading path
	var buf []string
	start := 1
	flush := func(next int) {
		text := strings.TrimSpace(strings.Join(buf, "\n"))
		if text != "" {
			for _, part := range bySize(text, max) {
				out = append(out, Chunk{source, start, strings.Join(heads, " › "), part})
			}
		}
		buf, start = nil, next
	}
	for i, line := range strings.Split(doc, "\n") {
		level := 0
		if strings.HasPrefix(line, "## ") {
			level = 1
		} else if strings.HasPrefix(line, "### ") {
			level = 2
		}
		if level == 0 {
			buf = append(buf, line)
			continue
		}
		flush(i + 1)
		if len(heads) >= level {
			heads = heads[:level-1]
		}
		heads = append(heads, strings.TrimSpace(strings.TrimLeft(line, "# ")))
	}
	flush(0)
	return out
}

// bySize cuts at blank lines (paragraphs) so each piece stays within max runes; a single
// paragraph that is itself too long is kept whole
func bySize(text string, max int) []string {
	var out []string
	cur := ""
	for _, p := range strings.Split(text, "\n\n") {
		if cur != "" && len([]rune(cur))+len([]rune(p)) > max {
			out = append(out, cur)
			cur = ""
		}
		if cur != "" {
			cur += "\n\n"
		}
		cur += p
	}
	if cur != "" {
		out = append(out, cur)
	}
	return out
}

(To keep the point clear, this leaves out the overlap and the rule that a ## inside a code block isn’t a heading; the implementation article adds them. When a section is split, every piece records the section’s first line; the implementation article gives each piece its own.)

How big? There’s no universal answer; measure with the evaluation method in the next article. A reasonable start for technical documents is a few hundred to about a thousand characters per chunk, with a sentence or two of overlap. Splitting by structure and keeping the heading path usually help more than hunting for the “best size”.

Step 3: Embeddings: turning text into vectors you can compare

An embedding model turns a piece of text into a list of numbers (a vector); texts with similar meaning point in similar directions. At query time the question is turned into a vector too, and the closest chunks are found. “How close” is most often measured with cosine similarity: the cosine of the angle between two vectors, where closer to 1 means more alike.

// internal/rag/cosine.go
package rag

import "math"

// Cosine returns the cosine similarity of two vectors; 0 if their lengths differ or either is zero
func Cosine(a, b []float32) float64 {
	if len(a) != len(b) || len(a) == 0 {
		return 0
	}
	var dot, na, nb float64
	for i := range a {
		dot += float64(a[i]) * float64(b[i])
		na += float64(a[i]) * float64(a[i])
		nb += float64(b[i]) * float64(b[i])
	}
	if na == 0 || nb == 0 {
		return 0
	}
	return dot / (math.Sqrt(na) * math.Sqrt(nb))
}

A few rules people trip over:

  • Index and query must use the same embedding model (the same version). Vectors from different models can’t be compared; changing models means rebuilding the whole index. Record in the index which model was used.
  • Languages must match up. With Chinese questions over English documents, choose a multilingual embedding model and test it with real questions in both languages.
  • Vectors are good at “means the same”, not at “exactly the same characters”. Error codes, order numbers, function names and version numbers are often missed by vectors. That’s why the next step adds keyword search.

Step 4: Hybrid vector and keyword search, merged with RRF

The two kinds of search complement each other:

Vector searchKeyword search (e.g. BM25, full-text)
Good atParaphrases, synonyms (“refund” finds “reverse the charge”)Exact strings (PAYMENT_TIMEOUT, order-1234, WithCancel)
Bad atError codes, ids, very short queriesDifferent words with the same meaning

So in practice you run both and merge the two rankings. Their scores are in different units (a cosine similarity and a BM25 score can’t simply be added), and the simplest stable way to merge is RRF (Reciprocal Rank Fusion): look only at ranks, never at scores. A chunk gets 1 / (k + rank) from each ranking, the sums are sorted again. k is commonly 60; it keeps first and second place from being too far apart.

Vector search ranking 1. c7 1/(60+1) 2. c2 1/(60+2) 3. c9 1/(60+3) Keyword search ranking 1. c2 1/(60+1) 2. c4 1/(60+2) 3. c7 1/(60+3) After RRF 1. c2 0.0325 2. c7 0.0323 3. c4 0.0161 4. c9 0.0159 Each chunk scores 1/(k+rank) per ranking (k=60), summed: only ranks matter, not scores. c2 is near the top of both, so it comes first; only vector search found c9, in third place.

Figure: vector search and keyword search rank separately; RRF turns ranks into scores and adds them, so chunks near the top of both rise to the top.

// internal/rag/rrf.go
package rag

import "sort"

// RRF merges several rankings (each a list of ids, best first) into one, best first
func RRF(k float64, rankings ...[]string) []string {
	score := map[string]float64{}
	for _, ranking := range rankings {
		for i, id := range ranking {
			score[id] += 1 / (k + float64(i+1)) // ranks start at 1
		}
	}
	ids := make([]string, 0, len(score))
	for id := range score {
		ids = append(ids, id)
	}
	sort.Slice(ids, func(i, j int) bool {
		if score[ids[i]] != score[ids[j]] {
			return score[ids[i]] > score[ids[j]]
		}
		return ids[i] < ids[j] // a fixed order for ties, so results are reproducible
	})
	return ids
}

For example, vector search ranks [c7, c2, c9] and keyword search ranks [c2, c4, c7]; RRF(60, …) gives [c2, c7, c4, c9]: c2 is near the top of both, so it comes first.

Step 5: Vector databases: how to choose, how they fit, what to watch

A vector database stores vectors and finds the nearest few quickly. With a few thousand chunks, keeping the vectors in memory and computing cosine similarity for each is fast enough; a dedicated index pays off at hundreds of thousands or millions of chunks. The choices fall into two groups:

  • Vector support in a database you already run: for example PostgreSQL’s pgvector extension, or vector fields in Elasticsearch/OpenSearch. Data, permissions, backups and transactions all stay in the setup you have, and keyword search lives in the same place.
  • A dedicated vector database: for example Qdrant, Milvus or Weaviate. Stronger when the data is very large and vector queries are the main load, but one more system to run, back up and monitor.

For most teams, start with the database you already use (the implementation article uses PostgreSQL + pgvector), and move only when the volume really outgrows it. When pairing it with RAG, watch for these:

  • It finds approximate nearest neighbours. Indexes such as HNSW and IVF trade a little accuracy for speed and may miss the truly closest few. Index parameters (pgvector’s HNSW ef_search, IVFFlat’s probes) tune exactly that trade-off; measure the change with the recall metric from the next article, not just the speed.
  • The distance function must match the embedding model. If the model says cosine, use cosine (in pgvector <=> is cosine distance, <#> negative inner product, <-> Euclidean distance), and the operator class the index was built with must match, or queries won’t use the index.
  • The dimension is fixed on the column. A vector(1024) column can’t hold a 1536-dimension vector. Changing embedding models means a new column or table, rebuilt in full, then switched over.
  • Filtering and approximate search interfere. “Find the nearest 10, then drop the ones the user can’t see” may leave 2, or none. Either let the database filter during the search (iterative scans since pgvector 0.8, filtered search in dedicated databases), or fetch more and filter afterwards, and test the case where the filter is very strict.
  • Don’t store vectors alone. Each row also holds the original text, source, line, heading path, content hash, embedding model name and metadata. Answers cite the text, filters need the metadata, rebuilds need the hash.
  • Follow document updates and deletions. Re-embed the chunks of a changed document and delete the rows of a removed one, or the model will cite content that no longer exists. Update incrementally by content hash, and regularly look for chunks in the index whose file is gone.
  • Hybrid search needs both indexes. Besides the vector index you need a full-text or keyword index; with both in one database, the RRF of step 4 is easy.

Step 6: Reranking: fetch many, then sort precisely

Vector and keyword search both have to be fast, so they’re a coarse filter: the question and each chunk were turned into vectors or index entries separately, never actually read together. Reranking puts the question and each candidate chunk together in front of a dedicated model (commonly a cross-encoder reranker, or an LLM asked to judge) and scores them one by one. Much more accurate, and much slower and more expensive.

So the usual setup has two stages:

  1. Hybrid search fetches a generous set of candidates, say 30 to 50 chunks.
  2. The reranker scores only those, and the top 5 to 8 go into the prompt.

Whether to add reranking depends on the evaluation: if the right chunk is often “retrieved but ranked low”, reranking helps a lot; if the right chunk isn’t retrieved at all, go back to chunking or retrieval, because reranking can’t rescue it.

Step 7: Prompt assembly: number of chunks, order and the context budget

With the chunks retrieved, the last step is putting them into the prompt. It’s often overlooked, yet it decides directly whether the model uses the right chunk:

  • More isn’t better. With 30 chunks, the model has to find the answer in a pile of unrelated content and is more likely to use the wrong one or miss the key one. Start around 5 and adjust with the evaluation.
  • Set a context budget. For example “at most N tokens of chunks”, dropping from the bottom of the ranking when over. Cost and latency stay under control too.
  • Every chunk carries an id and its source. The model cites the id; the program maps it back to file and line for the reader.
  • Merge adjacent chunks from the same document. Consecutive chunks 3 and 4 read better as one.
  • Write the rules out. Answer only from the chunks; cite an id in every sentence; if the chunks don’t say, say you don’t know; if chunks contradict each other, point it out rather than picking one.
  • Chunks are data, not instructions. If a document says “ignore the rules above”, that’s just document content. Put the chunks between clear markers and say so in the system prompt; the stronger defences come in the advanced and implementation articles.

Step 8: Metadata filters: permissions, versions, language and time

The metadata attached during chunking becomes filter conditions at retrieval time, and they must apply before or during retrieval, not afterwards by asking the model:

  • Permissions: retrieve only documents the signed-in user may see. Only a chunk the model never sees is one it truly can’t reveal.
  • Versions: when API v1 and v2 docs both exist, filter by the version the user asks about, or default to the current one.
  • Language: Chinese questions prefer Chinese documents, falling back to English.
  • Time and status: retired runbooks and drafts stay out of retrieval; incident notes can be sorted or weighted by date.

Filtering at the retrieval layer has another benefit: which chunks each answer used, and why those, can be traced from the logs.

Common pitfalls

Switching to a bigger model first. Wrong answers usually come from retrieval; first check whether the right chunk was retrieved.

Using vector search only. Error codes, ids and function names go missing; add keyword search and merge with RRF.

Changing embedding models without rebuilding the index. Old and new vectors can’t be compared, and retrieval quietly gets worse. Record the model name and version in the index.

More chunks is better. Too many dilute the key content and cost more time and money. Use a context budget.

Filtering after the approximate search. Take the top 10, then filter by permission, and often one or two remain, or none; filter during the search.

Leaving permissions to the model. “Please don’t reveal confidential documents” is not access control; filter at retrieval.

Dropping headings when chunking. Without “Payment timeout”, a chunk titled “Can I retry?” doesn’t tell its vector what it’s about.

Key takeaways

  • RAG is a pipeline of five stages, chunking, embedding, retrieval, reranking, prompt assembly; what an earlier stage drops a later one can’t recover, so tune front to back.
  • Chunk technical documents by heading, cap the size, keep a little overlap; each chunk carries its source, line, heading path and readable roles.
  • Index and query with the same embedding model; changing models means rebuilding; test multilingual setups.
  • Vectors are good at similar meaning, keywords at exact strings; run both and merge by rank only with RRF (1/(k+rank), k commonly 60).
  • Start with the vector database you already have (e.g. pgvector) and move when volume demands; it finds approximate neighbours, the distance function and dimension must match the embedding model, filter during the search, store text, source and hash in every row, and keep updates and deletions in sync.
  • Reranking fetches many and sorts precisely; it helps most when the right chunk is “retrieved but ranked low”.
  • In the prompt, control the number of chunks and the context budget, give every chunk an id, answer only from the chunks and say when they don’t know; chunks are data, not instructions.
  • Filter by permission, version, language and time at retrieval, never by asking the model.

Next

Every stage has something to tune, so how do you know whether a change made things better or worse? Trying a few questions by feel isn’t enough. The next article, “Evaluating RAG, Hallucinations and Common Failures: How to Know It Answers Right”, builds a question set, measures retrieval and answers separately, lays out how to block hallucinations, and lists the most common failures and their fixes.