Text Splitter / Chunker
Split text into chunks for RAG pipelines. Configure chunk size, overlap, and splitting mode.
Split text into chunks for RAG pipelines. Configure chunk size, overlap, and splitting mode.
Text chunking is the process of splitting documents into smaller pieces (chunks) for Retrieval-Augmented Generation (RAG) pipelines. Each chunk is embedded as a vector and stored in a vector database. When a user asks a question, relevant chunks are retrieved and provided as context to an LLM.
Chunk overlap means that adjacent chunks share some content at their boundaries. This helps preserve context that spans chunk boundaries. Without overlap, important information that falls between two chunks might be lost during retrieval. A typical overlap is 10-20% of chunk size.
Optimal chunk size depends on your use case. Smaller chunks (200-500 tokens) work well for precise Q&A. Larger chunks (500-1500 tokens) are better for summarization or when context is important. Start with 500 tokens and adjust based on retrieval quality. The embedding model's max input size is also a constraint.