skillZs
★ LIVE SKILL TAGS ★
>>> LIVE SKILLS INDEX <<<
* OPEN SOURCE *
NO LOGIN, NO TRACKING
※ REAL INSTALL DATA ※
← back to all skills
latestaiagents/agent-skills68 installs

chunking-strategies

Optimize document chunking for RAG performance and retrieval quality. Use this skill when splitting documents, choosing chunk sizes, implementing semantic chunking, or improving RAG retrieval accuracy. Activate when: chunking, split documents, chunk size, text splitting, document processing, RAG performance, semantic chunking, overlap.

How do I install this agent skill?

npx skills add https://github.com/latestaiagents/agent-skills --skill chunking-strategies
view source ↗

Is this agent skill safe to install?

  • Gen Agent Trust Hubpass

    The skill provides comprehensive documentation and Python code examples for implementing various document chunking strategies in RAG pipelines. It uses standard industry libraries and follows best practices for text processing without any identified security risks.

  • Socketpass

    No alerts

  • Snykwarn

    Risk: MEDIUM · 1 issue

  • Runlayerwarn

    1/1 file flagged

What does this agent skill do?

Chunking Strategies for RAG

Chunk boundaries decide what the retriever can find. Pick a splitter that respects the document's structure, then check it against real queries.

When to Use

  • Designing a new RAG pipeline
  • Retrieval quality is poor because answers are split across chunk boundaries
  • Documents mix content types (code, tables, prose, HTML)
  • Balancing context-window limits against retrieval precision

Why Chunking Matters

Poor chunking causes:

  • Context fragmentation (answers split across chunks)
  • Irrelevant retrieval (too much noise in chunks)
  • Lost relationships (parent-child content separated)
  • Wasted tokens (chunks too large or too small)

Chunking Methods Comparison

MethodBest ForTrade-offImplementation
Fixed-size / recursiveUniform content, prototyping, general textIgnores meaning; recursive at least prefers paragraph and sentence breaksEasy
Document-aware (Markdown, HTML, code)Structured docsNeeds a splitter per formatEasy
SemanticProse where topics shift without headingsSlower; needs embedding calls at ingestMedium
Parent-childHierarchical docs, answers that need surrounding contextTwo stores to keep in syncMedium
Late chunkingChunks that depend on document-wide contextNeeds a long-context embedding model that exposes token embeddingsComplex

Which method retrieves best depends on your documents and queries; compare candidates with the evaluation at the end of this skill.

Pattern 1: Fixed-Size with Overlap

The baseline approach. RecursiveCharacterTextSplitter tries larger separators first (paragraphs, then lines, then sentences) and falls back to smaller ones, so it keeps paragraphs intact when it can. Start here and change only when evaluation shows a problem.

from langchain_text_splitters import RecursiveCharacterTextSplitter

def create_fixed_chunks(
    text: str,
    chunk_size: int = 512,
    chunk_overlap: int = 50
) -> list[str]:
    """
    Split text into fixed-size chunks with overlap.

    Guidelines:
    - chunk_size: 256-1024 tokens is a common starting range
    - overlap: 10-20% of chunk_size so sentences at a boundary appear in both chunks
    - length_function=len measures characters, not tokens. To size in tokens, use
      RecursiveCharacterTextSplitter.from_tiktoken_encoder(...) or pass a tokenizer's
      length function.
    """
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=chunk_overlap,
        length_function=len,
        separators=["\n\n", "\n", ". ", " ", ""]
    )
    return splitter.split_text(text)

Avoid when: documents have strong structure (headings, code, tables); use a document-aware splitter (Pattern 5) instead.

Pattern 2: Semantic Chunking

Group by meaning, not arbitrary boundaries:

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

def create_semantic_chunks(text: str) -> list[str]:
    """
    Split text based on semantic similarity between sentences.
    Keeps related content together.
    """
    embeddings = OpenAIEmbeddings()

    splitter = SemanticChunker(
        embeddings=embeddings,
        breakpoint_threshold_type="percentile",
        breakpoint_threshold_amount=95  # Higher = fewer, larger chunks
    )

    return splitter.split_text(text)

Trade-off: every sentence is embedded at ingest time, so indexing is slower and costs more than recursive splitting.

Custom Semantic Chunking

import numpy as np
from sentence_transformers import SentenceTransformer

def semantic_chunk(
    sentences: list[str],
    model_name: str = "all-MiniLM-L6-v2",
    threshold: float = 0.5
) -> list[list[str]]:
    """
    Group sentences by semantic similarity.
    """
    model = SentenceTransformer(model_name)
    embeddings = model.encode(sentences)

    chunks = []
    current_chunk = [sentences[0]]

    for i in range(1, len(sentences)):
        # Cosine similarity between consecutive sentences
        sim = np.dot(embeddings[i-1], embeddings[i]) / (
            np.linalg.norm(embeddings[i-1]) * np.linalg.norm(embeddings[i])
        )

        if sim >= threshold:
            current_chunk.append(sentences[i])
        else:
            chunks.append(current_chunk)
            current_chunk = [sentences[i]]

    chunks.append(current_chunk)
    return chunks

Pattern 3: Parent-Child Chunking

Retrieve small, return with context:

from llama_index.core.node_parser import (
    HierarchicalNodeParser,
    SentenceSplitter,
    get_leaf_nodes
)
from llama_index.core import StorageContext, VectorStoreIndex
from llama_index.core.retrievers import AutoMergingRetriever

def create_hierarchical_index(documents):
    """
    Create parent-child chunk hierarchy.
    Small chunks for retrieval, auto-merge to parents for context.
    """
    # Define chunk sizes for each level
    node_parser = HierarchicalNodeParser.from_defaults(
        chunk_sizes=[2048, 512, 128]  # Parent → Child → Leaf
    )

    nodes = node_parser.get_nodes_from_documents(documents)
    leaf_nodes = get_leaf_nodes(nodes)

    # Store all nodes
    storage_context = StorageContext.from_defaults()
    storage_context.docstore.add_documents(nodes)

    # Index only leaf nodes
    index = VectorStoreIndex(
        leaf_nodes,
        storage_context=storage_context
    )

    # Retriever auto-merges to parents when siblings retrieved
    retriever = AutoMergingRetriever(
        index.as_retriever(similarity_top_k=12),
        storage_context=storage_context,
        simple_ratio_thresh=0.3  # Merge if 30%+ siblings retrieved
    )

    return retriever

LangChain Equivalent: ParentDocumentRetriever

In LangChain 1.x the retriever lives in langchain-classic:

from langchain_classic.retrievers import ParentDocumentRetriever  # pip install langchain-classic
from langchain_core.stores import InMemoryStore
from langchain_text_splitters import RecursiveCharacterTextSplitter

retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,                 # indexes the small child chunks
    docstore=InMemoryStore(),                # holds the parents; use a persistent store in production
    child_splitter=RecursiveCharacterTextSplitter(chunk_size=400),
    parent_splitter=RecursiveCharacterTextSplitter(chunk_size=2000),
)
retriever.add_documents(docs)
parents = retriever.invoke("query")          # matches children, returns their parents

Small chunks give precise matches; returning the parent gives the model enough context to answer.

Pattern 4: Late Chunking

Embed full document first, then chunk - preserves global context:

def late_chunking(
    document: str,
    model,
    chunk_size: int = 512
) -> list[dict]:
    """
    PSEUDOCODE: `model.tokenize`, `model.encode_tokens`, and `model.decode` stand in
    for your embedding model's API. You need a long-context model that returns
    token-level embeddings; see the late chunking paper for a reference implementation.

    Late chunking: embed the whole document, then pool token embeddings per chunk,
    so each chunk vector reflects document-level context.
    """
    # 1. Get token-level embeddings for full document
    tokens = model.tokenize(document)
    token_embeddings = model.encode_tokens(tokens)

    # 2. Split into chunks
    chunks = []
    for i in range(0, len(tokens), chunk_size):
        chunk_tokens = tokens[i:i + chunk_size]
        chunk_embeddings = token_embeddings[i:i + chunk_size]

        # 3. Pool chunk embeddings (mean pooling)
        chunk_vector = np.mean(chunk_embeddings, axis=0)

        chunks.append({
            "text": model.decode(chunk_tokens),
            "embedding": chunk_vector
        })

    return chunks

Pattern 5: Markdown/Code-Aware Chunking

from langchain_text_splitters import (
    MarkdownHeaderTextSplitter,
    Language,
    RecursiveCharacterTextSplitter
)

def chunk_markdown(text: str) -> list[dict]:
    """Split markdown by headers, preserving structure."""
    headers_to_split_on = [
        ("#", "h1"),
        ("##", "h2"),
        ("###", "h3"),
    ]

    splitter = MarkdownHeaderTextSplitter(
        headers_to_split_on=headers_to_split_on
    )

    return splitter.split_text(text)


def chunk_code(code: str, language: str = "python") -> list[str]:
    """Split code respecting language syntax."""
    lang_map = {
        "python": Language.PYTHON,
        "javascript": Language.JS,
        "typescript": Language.TS,
    }

    splitter = RecursiveCharacterTextSplitter.from_language(
        language=lang_map.get(language, Language.PYTHON),
        chunk_size=1000,
        chunk_overlap=100
    )

    return splitter.split_text(code)


def chunk_html(html: str) -> list:
    """Split HTML by header tags; header text is added to each chunk's metadata."""
    from langchain_text_splitters import HTMLHeaderTextSplitter

    splitter = HTMLHeaderTextSplitter(
        headers_to_split_on=[("h1", "Header 1"), ("h2", "Header 2"), ("h3", "Header 3")]
    )
    return splitter.split_text(html)

Header-based splitters can produce sections longer than your target size; run a size-based splitter over oversized sections afterwards.

Metadata Enrichment

Attach metadata to every chunk at ingest. It enables filtering, citations, access control, and targeted re-indexing:

import hashlib
from datetime import datetime, timezone

def enrich(chunks: list, source_doc) -> list:
    for i, chunk in enumerate(chunks):
        chunk.metadata.update({
            "source": source_doc.metadata["source"],      # URL or file path, for citations
            "section": chunk.metadata.get("h2") or chunk.metadata.get("Header 2"),
            "chunk_index": i,
            "total_chunks": len(chunks),
            "content_hash": hashlib.sha256(chunk.page_content.encode()).hexdigest(),
            "has_code": "```" in chunk.page_content,
            "indexed_at": datetime.now(timezone.utc).isoformat(),
        })
    return chunks

The content hash lets you skip re-embedding unchanged chunks when a document is updated.

Chunk Size Guidelines

Starting points to tune, not measured optima (sizes in tokens):

Content TypeStarting SizeOverlap
Q&A / FAQ256-51225-50
Technical docs512-102450-100
Legal documents1024-2048100-200
Code500-100050-100
Conversations256-51250-100

Evaluation: How to Know If Chunking Is Good

def evaluate_chunking(chunks: list[str], test_queries: list[dict]):
    """
    Evaluate chunk quality with test queries.

    test_queries format:
    [{"query": "What is X?", "expected_chunk_contains": "X is..."}]
    """
    results = {
        "avg_chunk_size": np.mean([len(c) for c in chunks]),
        "chunk_size_std": np.std([len(c) for c in chunks]),
        "total_chunks": len(chunks),
        "retrieval_hits": 0
    }

    for tq in test_queries:
        # Check if expected content is in a single chunk
        for chunk in chunks:
            if tq["expected_chunk_contains"] in chunk:
                results["retrieval_hits"] += 1
                break

    results["hit_rate"] = results["retrieval_hits"] / len(test_queries)
    return results

This checks whether an answer survives chunking intact. To measure whether the retriever then finds it, use the retrieval metrics in rag-evaluation.

Structural Checklist

Sample chunks from each document type and check:

  • Chunks don't break mid-sentence
  • Code blocks stay intact
  • Tables aren't split across chunks (or each piece repeats the header row)
  • Headings stay with their content
  • Overlap preserves continuity at boundaries
  • Metadata is present and supports the filters you need

Best Practices

  1. Match chunk size to query length - Chunks should be similar size to expected queries
  2. Preserve meaning boundaries - Never split mid-sentence or mid-paragraph
  3. Include metadata - Add source, page, section info to each chunk
  4. Test with real queries - Evaluate on your actual use cases
  5. Consider the embedding model - Stay within its maximum input length; text beyond it is truncated
  6. Start with recursive splitting - Move to semantic or parent-child chunking when evaluation shows a gap

Quick Decision Tree

What type of content?
├─ Structured (headers, sections)
│   └─ Use: Markdown/recursive splitter + hierarchy
├─ Unstructured (prose, articles)
│   └─ Use: Semantic chunking
├─ Code
│   └─ Use: Language-aware splitter
└─ Mixed
    └─ Use: Parent-child with semantic leaves

References

Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.

<a href="https://skillzs.dev/skills/latestaiagents/agent-skills/chunking-strategies">View chunking-strategies on skillZs</a>