Chunking and Retrieval-Augmented Generation (RAG): Complete Guide
What Is Chunking and How Does It Work in RAG Systems
Chunking is a methodology used to make large amounts of information more manageable by dividing extensive or complex documents into shorter text portions, known as chunks, each characterized by its own semantic coherence. This process is particularly relevant in the context of LLMs (Large Language Models), which operate within a context window of limited size and, consequently, cannot fully analyze very large files or entire archives in a single processing operation.
Within a Retrieval-Augmented Generation (RAG) architecture, each chunk is represented through a mathematical vector, known as an embedding, and subsequently stored in a vector database. When a request is submitted, the system identifies and retrieves the most relevant content segments in relation to the query. This allows the model to receive only the useful context required to formulate the answer, improving its accuracy and helping to reduce the risk of hallucinations or AI hallucinations.
How to Split Documents into Chunks
The creation of chunks generally begins with a preliminary pre-processing phase, which is required to clean the content and make it suitable for subsequent processing. During this phase, elements that do not provide informative value can be removed, such as page numbers, repetitive headings and HTML tags.
Once the text has been prepared, a segmentation algorithm is applied to divide it according to predefined criteria or based on syntactic and semantic characteristics. In more advanced processes, the generated chunks can also be enriched with metadata, such as titles, content summaries or source information. The resulting segments are then converted into embeddings and inserted into the vector storage infrastructure.
Main Chunking Techniques for RAG Systems
The way in which a document is divided depends on its structure, the nature of the information it contains and the type of search the system needs to perform. The main strategies include:
- Fixed-Size Chunking: divides the content into segments of the same length, using characters, words or tokens as a reference. It is computationally fast, but not very flexible and may interrupt sentences or concepts at the wrong point.
- Recursive Chunking: works through a hierarchy of natural separators, such as paragraphs, sentences and words, seeking to preserve the syntactic structure and coherence of the content during segmentation.
- Semantic Chunking: identifies changes in topic by analyzing the semantic similarity between consecutive sentences through embeddings. In this way, segment boundaries are mainly created when the meaning or topic being discussed changes.
- Agentic and Dynamic AI Chunking: relies on AI agents or LLMs to analyze the content and the user’s intent, allowing the system to dynamically identify the most appropriate splitting points and associate specific metadata with the chunks.
- Hierarchical and Sliding Window Chunking: preserves the relationships between parent and child elements within a document or uses partially overlapping segments to ensure greater continuity between consecutive blocks.
How to Choose the Optimal Chunk Size
Defining the correct chunk size requires finding a balance between the amount of information contained in each segment and the need to maintain a sufficiently coherent context. Excessively small chunks can separate information that should be interpreted together, resulting in incomplete or fragmented answers.
Conversely, overly large segments may contain a significant amount of irrelevant information, introducing greater noise into the retrieval process. This can reduce the accuracy of vector search and lead to an increase in API-related costs.
For search and question answering (Q&A) activities, blocks ranging from 200 to 500 tokens are generally used, while for summarizing particularly complex content, the size can reach approximately 2,000 tokens. However, there is no universal size: the most effective configuration should be identified through practical testing and performance evaluations based on the specific use case.
Chunk Overlap: What It Is and What It Is Used For
Con Chunk Overlap si identifica la parziale sovrapposizione di contenuto tra due chunk consecutivi. L’overlap viene generalmente impostato tra il 10% e il 20%, mentre negli approcci basati su una finestra scorrevole può arrivare fino al 50%.
La funzione dell’overlap è principalmente quella di mantenere la continuità del contesto tra segmenti adiacenti. In questo modo si riduce il rischio che informazioni importanti, collocate proprio in prossimità del punto di separazione, vengano suddivise tra due blocchi o vadano perse nelle fasi successive di vettorializzazione.
È comunque importante evitare valori di overlap eccessivamente elevati. Una sovrapposizione troppo ampia comporta infatti una maggiore duplicazione delle informazioni, con conseguente aumento dello spazio necessario per lo storage nei database vettoriali e dei tempi richiesti per l’elaborazione.
Margot AI Agent and RAG
Margot AI is a platform of customizable AI agents designed to support the automation of Customer Service and business support. The solution includes several automation-focused modules, including an automatic ticket classifier, a virtual assistant, a conversational chatbot and a tool for managing the company Knowledge Base.
Margot can integrate with corporate Helpdesk, CRM and Ticketing environments, enabling workflow automation, the management and summarization of complex conversations, and the routing of requests to the appropriate recipients.
Margot’s cognitive system, based on a RAG architecture, enables accurate responses by retrieving information directly from company documents. Data is processed in compliance with privacy requirements and GDPR compliance, without using user data to train the models.
Contact our sales team to activate Margot and its customizable AI agents.