Context Compression
Context compression is a technique in RAG systems that reduces the amount of retrieved context passed to the LLM while preserving the most relevant information. The goal is to fit the most useful content within the LLM's context window, reduce processing costs, and improve response quality by eliminating irrelevant or redundant information. Without compression, RAG systems typically retrieve more chunks than can fit in the context window, or they include chunks that contain irrelevant information alongside the relevant parts. Context compression addresses this through several approaches: extractive compression which selects the most relevant sentences or passages from the retrieved chunks, abstractive compression which uses an LLM to summarize the retrieved content, filtering which removes chunks below a relevance threshold, and reranking which reorders chunks by relevance and selects only the top-k. The most common approach is the reranker, a specialized model that scores each retrieved chunk for relevance to the query and selects the most relevant subset. Rerankers are more accurate than the initial vector similarity because they use cross-encoder architectures that jointly encode the query and each chunk, capturing fine-grained relevance signals that bi-encoder embeddings may miss. More advanced compression techniques include LLM-based compression where the LLM generates a compressed summary of the retrieved context, instruction-based compression where the system prompts the LLM to focus on specific aspects of the retrieved content, and structured compression that extracts and organizes information into a predefined format like tables or bullet points. Context compression is particularly important for complex queries that require synthesizing information from multiple documents, where the total retrieved content may far exceed the available context window. The compression ratio is a key design parameter — too aggressive compression may lose important information, while too little compression may not provide enough benefit. Modern RAG systems typically use a combination of techniques, starting with filtering and reranking, followed by extractive or abstractive compression if needed.
In Financial Services
Real-World Example
A large investment bank implements context compression in its RAG system for earnings analysis. For a query about 'Apple's Q3 2026 revenue breakdown by segment,' the system retrieves 50 chunks from 15 different documents including the earnings release, 10-Q filing, analyst reports, and news articles. The reranker scores each chunk and selects the top 15 most relevant chunks, compressing the context from 35,000 tokens to 8,000 tokens. The LLM then generates a response based on the compressed context, including revenue figures for iPhone, Services, Mac, iPad, and Wearables segments. The system reports that context compression improved response accuracy by 15% on a benchmark of 500 financial queries, while reducing per-query costs by 60%. The bank also finds that the reranker's relevance scores provide a useful confidence signal — when the top-ranked chunks have low relevance scores, the system can flag the response for human review. The compressed context also makes the system's reasoning more transparent for compliance purposes, as the relevant source passages are clearly identified.
Why It Matters for Finance
Context compression is a critical optimization for financial RAG systems because it directly addresses the tension between retrieval quantity and generation quality. Retrieving more documents improves recall but can overwhelm the LLM's context window and introduce noise. Context compression resolves this tension by allowing the system to retrieve broadly while presenting only the most relevant information to the LLM. For financial institutions, where document volumes are large and queries are complex, compression is essential for building RAG systems that are both accurate and cost-effective. The technique also provides a natural confidence signal through reranker scores, which can be used for quality monitoring and escalation workflows. As context windows continue to grow, compression becomes even more important, as the ability to process more tokens does not eliminate the need to prioritize the most relevant information.
Related Terms
Explore in Finatune
Frequently Asked Questions
What is context compression in RAG?
Context compression is a technique that reduces the amount of retrieved context passed to the LLM while preserving the most relevant information. It uses reranking, filtering, and summarization to fit the most useful content within the LLM's context window.
How does context compression help with large financial documents?
Context compression helps by filtering and prioritizing the most relevant passages from large financial documents like 10-K filings. A reranker scores each chunk for relevance to the query, selecting only the top passages, which reduces token usage and improves response quality.
When should I use context compression for financial document RAG?
Use context compression when queries require synthesizing information from multiple financial documents, when retrieved chunks exceed the LLM's context window, or when you want to reduce inference costs while maintaining or improving response accuracy.