Retrieval‑Augmented Generation (RAG)

---

1. What is RAG?

Retrieval‑Augmented Generation is a two‑stage pipeline that augments a language model with external knowledge.

1. Retrieve – find the most relevant passages (or chunks) from a large document collection.
2. Generate – feed the retrieved context to a causal language model together with the user query; the model then writes a grounded answer.

RAG overcomes the “hallucination” problem of vanilla LLMs by grounding their output in real, up‑to‑date text.

---

2. Core Concepts & Techniques

Concept	What it means	Why it matters in the code
Document ingestion	Reading all *.txt files from a folder	Provides the raw knowledge base
Chunking	Splitting a long document into smaller, manageable pieces	Allows retrieval to focus on a fine‑grained semantic unit
Chunk size (50 words)	Number of words per chunk	50‑word chunks are small enough for a meaningful similarity score but large enough to capture local context
Chunk overlap (10 words)	Number of words shared between consecutive chunks	Keeps boundary words from being lost, preserves semantic continuity
Tokenization (words)	Splitting on whitespace	Simple, fast, and language‑agnostic
Embedding	Turning text into dense vectors	Enables efficient similarity search in a continuous vector space
Sentence‑Transformer (all-MiniLM-L6-v2)	Pre‑trained encoder that outputs 384‑dim embeddings	Lightweight, good for quick demos, still captures sentence‑level semantics
FAISS index	Library for fast nearest‑neighbour search	Allows sub‑millisecond retrieval even for thousands of chunks
Index type (IndexFlatL2)	Exact L2 (Euclidean) distance search	Simple, no training needed; suitable for demonstration
Retrieval (k=3)	Return the top‑k most similar chunks	Controls how many snippets feed into the prompt
Prompt construction	Assemble context + question + “Answer:” prefix	The language model expects a clear format to know where to start generating
Deterministic generation	set_seed(42) & do_sample=False	Guarantees reproducibility – essential for a teaching lab
Pad token fallback	Use EOS token if PAD is undefined	Avoids warning messages from the generation pipeline
Device selection	device=0 if torch.cuda.is_available() else -1	Runs on GPU if available; otherwise CPU
Data type (torch.float16)	Reduces GPU memory usage	Important when training larger models on limited hardware
Truncation	truncation=True in the pipeline	Keeps the input within the model’s maximum token limit
max_new_tokens	Limits how many tokens are produced	Controls answer length

---

3. Step‑by‑Step Walkthrough

3.1 Chunking

• Goal: Convert long documents into a collection of semantically coherent, overlap‑aware segments.
• Algorithm:
  1. Split raw text into words (text.split()).
  2. From start = 0 to len(words):
     • Create a slice of chunk_size words.
     • Append the joined string to chunks.
     • Move start forward by chunk_size - overlap.
  3. Return the list of chunks.
• Why overlap?
  Without overlap, the first words of the next chunk would be cut off, causing a semantic gap between adjacent chunks.
  Example: if “hiking” is the last word of chunk 1 and the first word of chunk 2, the two chunks might refer to the same concept but the model would treat them as separate. Overlap keeps those boundary words in both chunks.

3.2 Embedding

• What are embeddings?
  A fixed‑size dense vector (e.g., 384‑dim for MiniLM) that captures semantic meaning of the text.
• Why use Sentence‑Transformers?
  These models are specifically fine‑tuned for sentence‑level similarity.
  MiniLM is compact but still expressive enough for quick demos.
• model.encode(docs, convert_to_numpy=True)
  • Processes all chunks in a batch.
  • Returns a np.ndarray of shape (num_chunks, dim).

3.3 FAISS Index

• FAISS (Facebook AI Similarity Search) is a high‑performance library for vector similarity search.
• IndexFlatL2
  • Stores vectors in memory.
  • Uses exact Euclidean distance (L2).
  • No training phase, so it’s trivial to set up.
• Adding vectors

  index.add(embeddings)

• Searching
  index.search(query_embedding, k) returns the top‑k nearest vectors along with their distances.

3.4 Retrieval

• Query embedding
  The user’s question is embedded with the same model used for documents, ensuring homogeneous vector space.
• Nearest‑Neighbour search
  faiss.IndexFlatL2 returns indices and distances.
  We pick the first row (indices[0]) because the query is a single vector.
• Choosing k
  • Larger k gives more context but may dilute relevance.
  • Common values: 3–5 for demos; production systems might use 10–20.

3.5 Prompt Construction

• Structure

  Context:
  <chunk1>
  
  <chunk2>
  
  <chunk3>
  
  Question: <user_query>
  Answer:
  

• Why this format?
  The language model is conditioned on the entire string.
  The “Answer:” token marks the spot where generation should begin, and later we strip everything before the last “Answer:”.

3.6 Generation

• Pipeline
  transformers.pipeline("text-generation", ...) handles tokenization, model inference, and post‑processing.
• Determinism
  • set_seed(42) ensures that all random choices (e.g., token sampling) are the same every run.
  • do_sample=False forces the model to use greedy decoding.
• Pad token fallback
  Some GPT‑2 variants lack a PAD token; we set it to the EOS token to silence warnings.
• max_new_tokens
  Controls how many tokens beyond the prompt are generated.
  A larger value yields longer, potentially more detailed answers.
• Precision
  torch_dtype=torch.float16 (if a GPU is available) reduces memory usage and speeds up inference.

---

4. Practical Take‑aways & Extensions

Topic	Take‑away	Next steps
Chunk Size	Balances granularity and computational cost.	Try 100‑word chunks or sentence‑based chunking (using NLTK or spaCy).
Chunk Overlap	Helps preserve boundary context.	Experiment with 0, 5, 20 overlap.
Embedding Model	More powerful models yield better similarity but cost more memory.	Switch to all-roberta-large-v1 or paraphrase-MiniLM-L12-v2.
FAISS Index	Exact search is simple; for huge corpora use HNSW, IVF, or PQ.	IndexIVFFlat with training step for >10M vectors.
Retrieval Strategy	Use k > 1 to fuse multiple snippets.	Try weighted aggregation or re‑ranking with BM25.
Prompt Engineering	Format, context length, and phrasing influence answer quality.	Add “Use the following information:” or bullet points.
Determinism vs Creativity	Fixed seed + no sampling = reproducible but less diverse.	For creative writing, enable do_sample=True and a temperature.
GPU vs CPU	GPU accelerates both embedding and generation.	Monitor memory; use torch.cuda.empty_cache() if needed.
Scaling Up	For millions of documents, you’d need a distributed retriever (e.g., Elasticsearch, Milvus).	Use vector similarity services (Pinecone, Qdrant).
Evaluation	Compare generated answer to ground‑truth or use human annotators.	Compute precision@k, Rouge, BLEU, etc.

---

5. Glossary of Key Terms

• Chunk – A segment of text (here, ~50 words).
• Chunk Size – Number of words per chunk.
• Chunk Overlap – Words shared between adjacent chunks.
• Embedding – Dense vector representation of a text fragment.
• FAISS – Facebook AI Similarity Search, a fast library for nearest‑neighbour queries.
• IndexFlatL2 – An exact L2 distance index with no pre‑processing.
• Nearest‑Neighbour Search – Retrieval of items whose vectors are closest to a query vector.
• Prompt – The formatted input given to a language model.
• Deterministic Generation – Producing the same output every time (fixed seed, no sampling).
• Pipeline – High‑level abstraction that bundles tokenizer + model + device logic.

---

6. Visual Flow Diagram

Document Folder
       |
       v
   Chunking (50w, 10w)
       |
       v
Embedding (Sentence-Transformer)
       |
       v
  Embedding Matrix
       |
       v
    FAISS Index
       |
       +----------------+
       |                |
User Query          Query Encoding
       |                |
       +------> Retrieval (top-3)
                      |
                      v
             Context Chunks
                      |
                      v
            Prompt Construction
                      |
                      v
           Generation (GPT-2, temp=0)
                      |
                      v
               Final Answer

---

Closing Thoughts

The example above bundles together several foundational concepts in modern NLP:

1. Data preprocessing (chunking, overlap).
2. Dense representation learning (Sentence‑Transformers).
3. Efficient similarity search (FAISS).
4. Prompt engineering for grounding.
5. Deterministic generation for reproducibility.

By tweaking any of these components you can observe how the whole system reacts—an excellent way for learners to experiment with RAG and understand the delicate interplay between retrieval and generation. Happy hacking!