Chapter 12: RAG, Retrieval & Agent Memory

When the Answer Is Not in the Window

Context windows are large in 2026, but they are not free and they are not infinite - and relevance still degrades with noise. Retrieval is how you put the right context in front of the model at the right time instead of pouring everything in and hoping. It is also how an agent reaches knowledge that lives outside the codebase entirely: documentation, tickets, past incidents, and institutional memory.

The RAG Pipeline

Retrieval-Augmented Generation (RAG) has a simple shape: split your sources into chunks, turn each chunk into an embedding (a vector), store the vectors, and at query time retrieve the chunks most similar to the question and add them to the prompt.

Two-row pipeline: index documents into chunks, embeddings and a vector store, then embed a question, retrieve nearest chunks, build a prompt and produce an answer. — RAG in one picture: index once, then retrieve the nearest chunks at query time and hand them to the model.

// Index: chunk -> embed -> store
const chunks = splitIntoChunks(doc, { size: 800, overlap: 100 });
for (const chunk of chunks) {
  const embedding = await embed(chunk.text);      // e.g. a text-embedding model
  await db.query(
    'INSERT INTO docs (content, embedding) VALUES ($1, $2)',
    [chunk.text, toVector(embedding)]
  );
}

// Query: embed question -> nearest neighbors -> augment prompt
const q = await embed(userQuestion);
const { rows } = await db.query(
  'SELECT content FROM docs ORDER BY embedding <=> $1 LIMIT 5', [toVector(q)]
);
const answer = await model.generate({
  system: 'Answer using only the provided context.',
  context: rows.map(r => r.content).join('\n---\n'),
  question: userQuestion
});

Embeddings, Vector Stores, and Hybrid Search

Vectors live in a store built for similarity search - pgvector inside PostgreSQL for teams that want one fewer system, or a dedicated vector database at scale. Pure semantic search misses exact terms (error codes, function names, SKUs), so production systems use hybrid search: combine keyword (BM25) and semantic results, then re-rank the merged set with a cross-encoder for precision. Chunking strategy matters more than the embedding model: too large and you bury the relevant sentence; too small and you lose context.

RAG vs Long-Context vs Agentic Search

There are three ways to get knowledge into a model, and they are not interchangeable:

* Long-context - just put the documents in the window. Best for a bounded, known set of sources that fit comfortably and are all relevant.
* RAG - retrieve the relevant slice from a large corpus. Best for large, slow-changing knowledge bases where most content is irrelevant to any one question.
* Agentic search - let the agent grep, glob, and call tools to find what it needs on demand. For codebases this often beats classic RAG: the agent follows imports and symbols the way a developer would, with no index to keep fresh.

The strongest systems blend them: agentic search over code, RAG over docs and tickets, and long-context for the handful of files the task is actually about.

Agent Memory

Retrieval brings in knowledge; memory brings in continuity. Agents use several kinds:

* Short-term (working) memory - the current session's conversation and scratchpad, bounded by the context window.
* Long-term memory - persisted facts and preferences (often plain markdown files, sometimes a vector store) that survive across sessions: "this team uses Vitest," "the staging deploy needs approval."
* Episodic memory - traces of past runs, used to learn from what worked and what failed.

Treat memory like any other context: version it, review it, and keep it current. Stale or poisoned memory is worse than none - an agent will confidently act on a fact that stopped being true three releases ago. The same hygiene that protects context files (Chapter 2) protects memory.