HeaderChunker and SectionChunker silently ignore IngestionChunkerOptions.OverlapTokens #7825

Description

IngestionChunkerOptions.OverlapTokens is public and documented, but only DocumentTokenChunker honors it. HeaderChunker and SectionChunker delegate to the internal ElementsChunker, whose constructor reads only MaxTokensPerChunk — OverlapTokens is never consulted. Setting it has no effect, with no warning and no exception.

The property is documented as "Gets or sets the number of overlapping tokens between consecutive chunks." Nothing indicates that only one of the four built-in chunkers supports it.

Root cause, verified against main (2026-10-11):

• HeaderChunker.ProcessAsync groups elements by header level and hands them to _elementsChunker.Process(document, chunkHeader, elements).
• SectionChunker does the same: _elementsChunker = new(options) in the constructor, then _elementsChunker.Process(document, context, elements).
• ElementsChunker (internal sealed) never reads the option — its constructor assigns only _tokenizer = options.Tokenizer; and _maxTokensPerChunk = options.MaxTokensPerChunk;.
• Only DocumentTokenChunker reads it: _chunkOverlap = options.OverlapTokens;.
• ElementsChunker.cs is 273 lines and contains 0 occurrences of OverlapTokens.

Suggested fix, either of these removes the inconsistency:

1. Honor the option in ElementsChunker. It already tracks the running token count and commits through a local Commit(), so carrying the committed chunk's tail into the next one is feasible — the same approach DocumentTokenChunker uses (GetIndexByTokenCountFromEnd over the committed content, with the resulting token count written back into the running counter so the overlap is charged to the next chunk's budget). That makes HeaderChunker and SectionChunker behave like DocumentTokenChunker.
2. If overlap is intentionally unsupported for structure-based chunking, make it explicit — document the limitation on IngestionChunkerOptions.OverlapTokens, or throw from the constructor. Silently dropping the option is the least helpful of the three behaviors.

I have a working local implementation of option 1 (roughly 15 lines inside ElementsChunker.Commit()) and can open a PR if that is useful.

Reproduction Steps

1. Create a .NET 10 console app referencing Microsoft.Extensions.DataIngestion 10.10.0-preview.1.26459.2 and Microsoft.ML.Tokenizers 2.0.0.

2. Run this. Note that HeaderChunker is in Microsoft.Extensions.DataIngestion while SectionChunker and DocumentTokenChunker are in Microsoft.Extensions.DataIngestion.Chunkers, so both namespaces must be imported.

using Microsoft.Extensions.DataIngestion;
using Microsoft.Extensions.DataIngestion.Chunkers;
using Microsoft.ML.Tokenizers;

Tokenizer tokenizer = TiktokenTokenizer.CreateForEncoding("cl100k_base", new Dictionary<string, int>());

IngestionChunkerOptions options = new(tokenizer)
{
    MaxTokensPerChunk = 256,
    OverlapTokens = 30,
};

foreach ((string name, IngestionChunker<string> chunker) in new (string, IngestionChunker<string>)[]
{
    ("HeaderChunker", new HeaderChunker(options)),
    ("SectionChunker", new SectionChunker(options)),
    ("DocumentTokenChunker", new DocumentTokenChunker(options)),
})
{
    List<string> chunks = [];
    await foreach (IngestionChunk<string> chunk in chunker.ProcessAsync(BuildDocument()))
    {
        chunks.Add(chunk.Content);
    }

    Console.WriteLine($"{name}: {chunks.Count} chunk(s)");
    for (int i = 1; i < chunks.Count; i++)
    {
        Console.WriteLine($"    chunk {i} -> chunk {i + 1}: {SharedBoundaryLength(chunks[i - 1], chunks[i])} chars shared at the boundary");
    }
}

static int SharedBoundaryLength(string previous, string next)
{
    int max = Math.Min(previous.Length, next.Length);
    for (int length = max; length > 0; length--)
    {
        if (previous.AsSpan(previous.Length - length).SequenceEqual(next.AsSpan(0, length)))
        {
            return length;
        }
    }

    return 0;
}

static IngestionDocument BuildDocument()
{
    IngestionDocument document = new("repro");
    IngestionDocumentSection section = new();
    section.Elements.Add(new IngestionDocumentHeader("# Operations handbook") { Level = 1 });

    for (int i = 1; i <= 4; i++)
    {
        section.Elements.Add(new IngestionDocumentParagraph(
            $"Section {i}. Redis memory alert runbook. Start by logging in and running INFO memory to check used_memory_human. "
            + "If used memory is above eighty percent of maxmemory, look for large keys: run redis-cli --bigkeys to scan the "
            + "whole keyspace, then confirm one by one whether each of the largest keys really needs to be kept."));
    }

    document.Sections.Add(section);
    return document;
}

Expected behavior

OverlapTokens = 30 should make HeaderChunker and SectionChunker produce consecutive chunks that share the last 30 tokens — the same behavior DocumentTokenChunker provides, and the same behavior the property documents.

If that is not supported for structure-based chunking, the API should throw or document the limitation instead of silently ignoring the option.

Actual behavior

All three chunkers share the same options instance and the same document. The code above prints:

MaxTokensPerChunk=256, OverlapTokens=30

HeaderChunker: 2 chunk(s)
    chunk 1 -> chunk 2: 0 chars shared at the boundary

SectionChunker: 2 chunk(s)
    chunk 1 -> chunk 2: 0 chars shared at the boundary

DocumentTokenChunker: 2 chunk(s)
    chunk 1 -> chunk 2: 143 chars shared at the boundary

DocumentTokenChunker shares 143 characters (about 30 tokens) at the boundary. HeaderChunker and SectionChunker share none.

No exception, no warning, no diagnostic entry — the option is silently dropped.

Regression?

Unknown. I have only tested 10.10.0-preview.1.26459.2, which behaves this way.

Known Workarounds

Wrap the chunker in a decorator that prepends the previous chunk's tail:

internal sealed class OverlapChunker(
    IngestionChunker<string> inner,
    Tokenizer tokenizer,
    int overlapTokens) : IngestionChunker<string>
{
    // Prepends the last `overlapTokens` tokens of the previous chunk to each chunk.
}

Note the subtlety this exposes: ElementsChunker guarantees only MaxTokensPerChunk for its own output, so the inner chunker must be constructed with MaxTokensPerChunk - OverlapTokens, otherwise the added overlap pushes chunks over the configured limit. That is exactly the semantics DocumentTokenChunker already implements internally; it is just not reachable from HeaderChunker.

Configuration

• Microsoft.Extensions.DataIngestion 10.10.0-preview.1.26459.2
• Microsoft.Extensions.DataIngestion.Abstractions 10.10.0-preview.1.26459.2
• .NET 10
• Windows

Other information

HeaderChunker is also the only one of the four built-in chunkers in Microsoft.Extensions.DataIngestion — SectionChunker, DocumentTokenChunker, ElementsChunker and SemanticSimilarityChunker are all in Microsoft.Extensions.DataIngestion.Chunkers. Not a bug in itself, but it is why the reproduction above needs two using directives.