Chapter 6: Advanced RAG Techniques

Advanced RAG Techniques: Beyond Basic Retrieval

Basic RAG (retrieve top-k documents, pass to LLM) works well for simple queries, but real-world applications often need more sophisticated techniques to handle complex queries, improve accuracy, and optimize performance.

1. Query Expansion: Improving Retrieval Coverage

What it is: Query expansion rewrites or augments the user's query to include synonyms, related terms, or alternative phrasings before retrieval. This helps find documents that use different terminology than the original query.

Why it's needed: Users often write short, ambiguous queries. A query like "Python ML" could mean many things, and documents might use different terminology ("machine learning," "deep learning," "scikit-learn," etc.).

How it works:

1. Synonym expansion: Add synonyms for key terms (e.g., "ML" → "machine learning," "deep learning")
2. Related term expansion: Add related concepts (e.g., "Python ML" → "scikit-learn," "TensorFlow," "PyTorch")
3. LLM-based expansion: Use an LLM to generate query variations or rewrite the query more clearly
4. Retrieve with expanded query: Use the expanded query for retrieval (or retrieve for each variation and combine)

When to use: When queries are short, ambiguous, or use abbreviations. Particularly useful for technical domains with lots of synonyms and jargon.

2. Multi-Query Retrieval: Generating Multiple Query Variations

What it is: Instead of retrieving with a single query, generate multiple query variations, retrieve documents for each variation, then combine and deduplicate the results.

Why it works: Different phrasings of the same question might retrieve different relevant documents. By generating multiple variations, you increase the chance of finding all relevant information.

How it works:

1. Generate query variations: Use an LLM to generate 3-5 different phrasings of the original query
2. Retrieve for each: Run retrieval for each query variation
3. Combine results: Merge all retrieved documents, removing duplicates
4. Rerank: Rerank the combined results to get the most relevant documents

When to use: For complex queries where a single phrasing might miss relevant documents. Particularly effective when combined with query expansion.

3. Parent-Child Chunking: Balancing Precision and Context

What it is: A hierarchical chunking strategy where you store small, precise chunks for retrieval (children), but also store larger parent chunks that contain surrounding context.

The problem it solves: Small chunks are better for precise retrieval (more likely to be fully relevant), but they lack context. Large chunks have more context but are less precise (may contain irrelevant information).

How it works:

1. Create parent chunks: Split documents into larger chunks (e.g., 1000 tokens) that preserve context
2. Create child chunks: Split each parent into smaller chunks (e.g., 200 tokens) for precise retrieval
3. Store both: Embed and store both parent and child chunks, with metadata linking children to parents
4. Retrieve children: Use small child chunks for retrieval (precise matching)
5. Include parent context: When a child is retrieved, also include its parent chunk in the context sent to the LLM

When to use: When you need both precise retrieval (small chunks) and rich context (large chunks). Common in production RAG systems.

4. Metadata Filtering: Reducing Search Space

What it is: Apply filters based on document metadata (date, author, category, source) before performing vector similarity search. This reduces the number of documents to search.

Why it's important: Searching 1 million documents is slow. If you can filter to 50,000 relevant documents first (e.g., "only documents from 2023"), search becomes much faster and more accurate.

How it works:

1. Extract metadata: When indexing, extract and store metadata (date, category, author, etc.)
2. Apply filters: Before vector search, filter documents by metadata criteria
3. Search filtered set: Perform vector similarity search only on filtered documents
4. Return results: Return top-k documents from the filtered search

When to use: When documents have meaningful metadata and queries can be scoped (e.g., "recent articles," "technical documentation," "from specific author").

Context Compression: Fitting Retrieved Information into LLM Windows

The Problem

After retrieving top-k documents, you might have 10,000+ tokens of context. But LLMs have context window limits (e.g., GPT-4: 8K-128K tokens, Claude: 100K-200K tokens). If your query + context + answer exceeds the limit, you need to compress the context.

Context Compression Techniques

1. Summarization

What it is: Use an LLM to summarize each retrieved chunk, reducing token count while preserving key information.

How it works:

• For each retrieved chunk, create a summary prompt: "Summarize this text in 2-3 sentences: [chunk text]"
• Generate summaries (much shorter than originals)
• Use summaries as context instead of full chunks

Trade-offs: ✅ Reduces tokens significantly. ❌ May lose important details. ⚠️ Adds latency (need to summarize each chunk).

2. Sentence Extraction

What it is: Extract only the most relevant sentences from retrieved chunks, discarding the rest.

How it works:

• Score each sentence in retrieved chunks for relevance to the query (using embeddings or LLM)
• Select top-N most relevant sentences
• Use only these sentences as context

Trade-offs: ✅ Very fast. ✅ Preserves exact information (no summarization loss). ❌ May lose context between sentences.

3. LLM-Based Compression

What it is: Use an LLM to compress the entire retrieved context into a shorter version that preserves information relevant to the query.

How it works:

• Prompt: "Compress this context to [target tokens] while preserving all information relevant to: [query]"
• LLM generates compressed version
• Use compressed context for final answer generation

Trade-offs: ✅ Most intelligent compression. ✅ Preserves query-relevant information. ❌ Expensive and slow. ❌ May introduce hallucinations.

4. Relevance-Based Prioritization

What it is: Instead of compressing, prioritize chunks by relevance score and include only the most relevant ones until you hit the token limit.

How it works:

• Sort retrieved chunks by similarity score (most relevant first)
• Add chunks to context one by one until you reach token limit
• Discard remaining chunks

Trade-offs: ✅ Fast and simple. ✅ No information loss (just truncation). ❌ May miss important information in lower-ranked chunks.

Best Practices

• Combine techniques: Use summarization for less relevant chunks, full text for most relevant
• Reserve tokens for answer: Leave 20-30% of context window for the LLM's answer
• Monitor compression quality: Track answer quality with and without compression

Iterative Retrieval: Multi-Hop and Self-RAG

1. Multi-Hop Retrieval

What it is: If the initial answer is incomplete or the model needs more information, generate follow-up queries, retrieve additional context, and repeat until the answer is complete.

Why it's needed: Complex questions often require information from multiple documents. A single retrieval might not find all necessary information.

How it works:

1. Initial retrieval: Retrieve top-k documents for the original query
2. Generate answer: Attempt to generate answer from retrieved context
3. Check completeness: Determine if answer is complete (using LLM or heuristics)
4. Generate follow-up query: If incomplete, generate a new query to find missing information
5. Retrieve again: Retrieve documents for the follow-up query
6. Combine context: Add new context to existing context
7. Generate final answer: Generate answer from combined context
8. Repeat if needed: Continue until answer is complete or max iterations reached

When to use: For complex, multi-part questions that require information from multiple sources. Common in question-answering systems.

2. Self-RAG: Autonomous Retrieval and Generation

What it is: An advanced technique where the model itself decides when to retrieve, what to retrieve, when to generate, and when to stop—making the RAG system more autonomous.

How it works:

1. Retrieve decision: Model decides if retrieval is needed (some queries can be answered from its training data)
2. Query generation: If retrieval needed, model generates the retrieval query
3. Retrieval: Retrieve documents using generated query
4. Relevance check: Model evaluates if retrieved documents are relevant
5. Generate or retrieve again: If relevant, generate answer; if not, retrieve again with different query
6. Self-critique: Model evaluates its own answer for completeness and accuracy
7. Iterate or stop: If answer is good, stop; if not, continue retrieving and generating

Advantages:

• ✅ More autonomous: Doesn't always retrieve (saves cost when not needed)
• ✅ Adaptive: Adjusts retrieval strategy based on query complexity
• ✅ Self-correcting: Can identify when answers are incomplete and retrieve more

Challenges:

• ❌ Complex to implement: Requires fine-tuning or prompt engineering
• ❌ Higher latency: Multiple decision points add time
• ❌ Cost: More LLM calls (retrieve decisions, query generation, self-critique)

When to use: For high-value applications where accuracy is critical and you can afford the complexity and cost. Still experimental but promising.

Advanced RAG Mathematical Models

Advanced RAG techniques involve mathematical models for generation, context management, and quality evaluation. Understanding these formulas helps you optimize context usage, implement iterative retrieval, and measure answer quality.

from sentence_transformers import SentenceTransformer
from typing import List
import re

class QueryExpander:
    """
    Query expansion system that adds synonyms and related terms.
    
    Improves retrieval by finding documents that use different
    terminology than the original query.
    """
    
    def __init__(self):
        self.embedder = SentenceTransformer('all-MiniLM-L6-v2')
        # In production, use a proper synonym dictionary or LLM-based expansion
        self.synonym_dict = {
            "machine learning": ["ML", "artificial intelligence", "AI", "deep learning"],
            "neural network": ["neural net", "NN", "deep learning model"],
            "python": ["Python programming", "Python language"],
            "tutorial": ["guide", "how-to", "instructions", "walkthrough"]
        }
    
    def expand_with_synonyms(self, query: str) -> str:
        """
        Expand query by replacing terms with synonyms.
        
        Args:
            query: Original query
            
        Returns:
            Expanded query with synonyms
        """
        expanded_terms = []
        words = query.lower().split()
        
        for word in words:
            expanded_terms.append(word)
            # Check if word is part of a key phrase
            for key, synonyms in self.synonym_dict.items():
                if key in query.lower():
                    expanded_terms.extend(synonyms)
        
        # Remove duplicates and join
        expanded_query = " ".join(list(set(expanded_terms)))
        return expanded_query
    
    def expand_with_llm(self, query: str, num_expansions: int = 3) -> List[str]:
        """
        Use LLM to generate query variations.
        
        Args:
            query: Original query
            num_expansions: Number of variations to generate
            
        Returns:
            List of expanded query variations
        """
        # In production, use OpenAI API or similar
        # This is a simplified example
        prompt = f"""Generate {num_expansions} different ways to ask this question:
"{query}"

Return only the variations, one per line:"""
        
        # For demonstration, return manual variations
        # In production, call LLM API here
        variations = [
            query,  # Original
            f"Explain {query}",
            f"What is {query}?",
            f"Tell me about {query}"
        ]
        return variations[:num_expansions]

# Example usage
expander = QueryExpander()

query = "Python machine learning tutorial"
expanded = expander.expand_with_synonyms(query)
print(f"Original: {query}")
print(f"Expanded: {expanded}")

# Use expanded query for retrieval
# retriever.retrieve(expanded)

from typing import List, Set
from sentence_transformers import SentenceTransformer

class MultiQueryRetriever:
    """
    Multi-query retrieval that generates multiple query variations
    and combines results for better coverage.
    """
    
    def __init__(self, base_retriever):
        """
        Initialize multi-query retriever.
        
        Args:
            base_retriever: Underlying retriever (HybridRetriever, etc.)
        """
        self.base_retriever = base_retriever
        self.embedder = SentenceTransformer('all-MiniLM-L6-v2')
    
    def generate_query_variations(self, query: str, num_variations: int = 3) -> List[str]:
        """
        Generate query variations using embedding similarity.
        
        Args:
            query: Original query
            num_variations: Number of variations to generate
            
        Returns:
            List of query variations
        """
        # In production, use LLM to generate variations
        # For demonstration, create simple variations
        variations = [query]  # Include original
        
        # Add variations based on query structure
        if "?" in query:
            variations.append(query.replace("?", ""))
            variations.append(f"Explain {query.lower()}")
        else:
            variations.append(f"{query} explanation")
            variations.append(f"What is {query}?")
        
        return variations[:num_variations]
    
    def retrieve_multi_query(self, query: str, top_k_per_query: int = 10, final_k: int = 5) -> List[dict]:
        """
        Retrieve using multiple query variations and combine results.
        
        Args:
            query: Original query
            top_k_per_query: Number of results per query variation
            final_k: Final number of results after combining
            
        Returns:
            Combined and deduplicated results
        """
        # Step 1: Generate query variations
        query_variations = self.generate_query_variations(query, num_variations=3)
        print(f"Generated {len(query_variations)} query variations")
        
        # Step 2: Retrieve for each variation
        all_results = []
        seen_documents = set()
        
        for variation in query_variations:
            results = self.base_retriever.retrieve(variation, top_k=top_k_per_query)
            
            # Deduplicate and add to combined results
            for result in results:
                doc_text = result['document']
                if doc_text not in seen_documents:
                    all_results.append(result)
                    seen_documents.add(doc_text)
        
        # Step 3: Rerank combined results (optional but recommended)
        # For simplicity, sort by hybrid_score
        all_results.sort(key=lambda x: x.get('hybrid_score', 0), reverse=True)
        
        # Step 4: Return top-k
        return all_results[:final_k]

# Example usage
# multi_retriever = MultiQueryRetriever(hybrid_retriever)
# results = multi_retriever.retrieve_multi_query("machine learning", final_k=5)

from typing import List
import re

class ContextCompressor:
    """
    Compresses retrieved context to fit within context window limits
    while preserving the most relevant information.
    """
    
    def __init__(self, max_tokens: int = 2000):
        """
        Initialize context compressor.
        
        Args:
            max_tokens: Maximum tokens allowed in compressed context
        """
        self.max_tokens = max_tokens
    
    def compress_by_relevance(self, query: str, documents: List[str], 
                              relevance_scores: List[float]) -> str:
        """
        Compress context by keeping only most relevant parts.
        
        Args:
            query: User query
            documents: List of retrieved documents
            relevance_scores: Relevance scores for each document
            
        Returns:
            Compressed context string
        """
        # Sort documents by relevance
        doc_scores = list(zip(documents, relevance_scores))
        doc_scores.sort(key=lambda x: x[1], reverse=True)
        
        # Add documents until we hit token limit
        compressed_parts = []
        current_tokens = 0
        
        for doc, score in doc_scores:
            doc_tokens = len(doc.split())  # Approximate token count
            if current_tokens + doc_tokens <= self.max_tokens:
                compressed_parts.append(doc)
                current_tokens += doc_tokens
            else:
                # Truncate last document if needed
                remaining_tokens = self.max_tokens - current_tokens
                if remaining_tokens > 50:  # Only if meaningful space left
                    words = doc.split()[:remaining_tokens]
                    compressed_parts.append(" ".join(words) + "...")
                break
        
        return "\n\n".join(compressed_parts)
    
    def compress_by_summarization(self, documents: List[str]) -> str:
        """
        Compress by summarizing documents (simplified example).
        
        In production, use LLM-based summarization.
        """
        # Simplified: extract first sentence of each document
        summaries = []
        for doc in documents:
            sentences = doc.split('.')
            if sentences:
                summaries.append(sentences[0] + '.')
        
        return " ".join(summaries)

# Example usage
compressor = ContextCompressor(max_tokens=1000)

documents = [
    "Machine learning is a subset of artificial intelligence...",
    "Deep learning uses neural networks...",
    "Python is a programming language..."
]

scores = [0.95, 0.85, 0.60]

compressed = compressor.compress_by_relevance(
    "What is machine learning?",
    documents,
    scores
)
print(f"Compressed context ({len(compressed.split())} tokens):")
print(compressed)

class AdvancedRAG:
    """
    Complete advanced RAG system with query expansion, multi-query,
    context compression, and generation.
    """
    
    def __init__(self, retriever, llm):
        self.retriever = retriever
        self.llm = llm
        self.query_expander = QueryExpander()
        self.multi_query = MultiQueryRetriever(retriever)
        self.compressor = ContextCompressor(max_tokens=2000)
    
    def query(self, user_query: str, use_expansion: bool = True,
              use_multi_query: bool = True, use_compression: bool = True):
        """
        Complete advanced RAG query pipeline.
        
        Args:
            user_query: User's question
            use_expansion: Whether to expand query
            use_multi_query: Whether to use multi-query retrieval
            use_compression: Whether to compress context
        """
        # Step 1: Query expansion (optional)
        if use_expansion:
            expanded_query = self.query_expander.expand_with_synonyms(user_query)
        else:
            expanded_query = user_query
        
        # Step 2: Multi-query retrieval (optional)
        if use_multi_query:
            retrieved = self.multi_query.retrieve_multi_query(
                expanded_query,
                top_k_per_query=10,
                final_k=10
            )
            documents = [r['document'] for r in retrieved]
            scores = [r.get('hybrid_score', 0.5) for r in retrieved]
        else:
            retrieved = self.retriever.retrieve(expanded_query, top_k=10)
            documents = [r['document'] for r in retrieved]
            scores = [r.get('hybrid_score', 0.5) for r in retrieved]
        
        # Step 3: Context compression (optional)
        if use_compression:
            context = self.compressor.compress_by_relevance(
                user_query,
                documents,
                scores
            )
        else:
            context = "\n\n".join(documents)
        
        # Step 4: Generate answer
        prompt = f"""Context:
{context}

Question: {user_query}

Answer based on the context above:"""
        
        # In production, use actual LLM API
        answer = f"Generated answer for: {user_query}"  # Placeholder
        return answer

# Example usage
# advanced_rag = AdvancedRAG(hybrid_retriever, llm)
# answer = advanced_rag.query("What is machine learning?")

pip install sentence-transformers langchain openai