Why one chunk spanned three others in our chunk visualiser

Our chunk visualiser draws every chunk of a document as a box over the original PDF page, and on one of our test documents a single box sprawled across the boxes of three other chunks. I read that as a chunking bug, duplicate content going into the index, and went looking for the line that produced it.

The chunker was emitting that chunk on purpose, as a retrieval aid for tables, so there was no line to fix. The change in v1.12.6 (tagged 11 June 2026, commit 74d55306) removed nothing. It changed which chunking strategy a new knowledgebase gets by default and left every existing knowledgebase as it was. It also added a banner to the visualiser that explains the old behaviour to anyone who still sees it.

What the visualiser showed

The chunk visualiser is a drawer in the knowledgebase admin. For a processed document it asks the RAG service for that document's chunks, matches each chunk's text back to the page and bounding box the OCR layout gave us, and draws the boxes in chunk order. It also marks a chunk as overlapping when it sits next to another chunk (by order) on the same page and their boxes intersect. The response even says how it worked that out: "overlap_method": "bbox_adjacent".

The document was 05_Apex_Logistics_Services_Agreement_2025, one of the small synthetic agreements in our test fixtures (8.2KB as a PDF, with tables, headings and sections). Under the token_based strategy it came out as five chunks, and one of them verbatim contained its neighbours. In the notes I wrote at the time it's recorded as:

chunk4 == chunk1 + chunk2 + chunk3

Counting every pair of chunks where one fully contains the other gave five duplicate-containment pairs across five chunks. On the overlay that looks like a chunker that has lost track of where it is, and a chunk that repeats three others word for word looks like the same text going into the index twice.

What token_based does with each table

token_based is the strategy our ingestion inherited from LightRAG, extended with table handling. When a knowledgebase is set to it, the chunking function is chunking_by_token_size_table_aware in a configuration file, bound to the knowledgebase's max_table_token_size. Its docstring says what it does:

For each table detected, creates TWO chunks:

1. Isolated table chunk: heading + table + footer (for analytics)
2. Context-enriched chunk: previous_chunk + table + next_chunk (for retrieval)

The mechanics go roughly like this. The function strips markdown separator rows (|-----|) to save tokens, then scans for pipe tables. For each one it looks back up to six lines for something that reads like a heading or caption, allowing for up to two blank lines in between, so the table's title travels with it. Prose between tables goes through the ordinary token-size chunker; each table becomes its own chunk tagged chunk_type='table'.

Then comes a post-processing pass, and this is where my "duplicate" came from. For every chunk tagged table, it finds the nearest non-table chunk before it and the nearest non-table chunk after it, joins the three with blank lines, and appends the result as a new chunk tagged table_context. It only does this if there is at least one neighbour; a table alone in a document gets no context chunk. The docstring's own example is a sleepover allowance table sitting between a paragraph about sleepover policies and a paragraph about exceptions, which produces:

Chunk 3: Chunk 0 + Chunk 1 + Chunk 2 (chunk_type='table_context')

That is precisely the shape the visualiser was drawing for 05_Apex. The reasoning behind it holds up. A bare table chunk is a grid of cells and numbers, and a question phrased in prose has little in common with it textually. The paragraph above a table usually names what it is and the paragraph below it often qualifies it, so a chunk that carries the table and both paragraphs gives retrieval something to match on, while the isolated table chunk stays clean for anything that wants just the grid.

Oversized tables get a variant of the same idea. When a table exceeds max_table_token_size (or twice the chunk size when that isn't set), it's split by rows into several table chunks, with the headers repeated, for questions about specific rows, plus one table_full chunk holding the whole table if it fits under the embedding model's 8,191-token limit. The context pass skips table_full chunks and, for a run of split table chunks, walks outward past all of them to find the prose on either side.

Two more details made this harder to see than it should have been.

The first is that chunk_type never reaches storage. It exists in the list the chunker returns and is gone by the time chunks are upserted, so there is no way to ask the store "which of these chunks is a table_context chunk?" after the fact. The visualiser couldn't label it as such even if we'd wanted it to.

The second is the enable_table_processing flag, which looks as if it should control all this. Setting it to False leaves the table-aware chunking on. That flag only decides whether the multimodal TableModalProcessor is registered, which is the path that sends tables to the LLM for entity processing. Under token_based the chunker itself is always table-aware, whatever that flag says. If you go looking for the switch that removes the context chunk, that's the one you'll find, and flipping it leaves the chunks exactly as they were.

The same document under both strategies

We had a second strategy sitting next to it. structure_aware chunking goes through chunk_markdown_file in a configuration file and works from the document's shape instead of a token budget. It splits on markdown headings (H1 to H4, keeping the heading text in the chunk) using LangChain's MarkdownHeaderTextSplitter, detects a heading immediately followed by a table, list, image or code block and keeps the pair together, and only then runs a secondary split on anything still too large. At v1.12.6 its configuration defaults were:

max_token_size: 1200
overlap_token_size: 100
max_table_token_size: 2000
max_list_token_size: 1800
keep_heading_with_list: True
include_table_caption: True
include_table_footnote: True

It emits no extra context chunk. Each chunk maps to one region of the document, and the heading that introduces a table already rides along with it, which covers part of what the context chunk was for.

Here is 05_Apex through each strategy, with the counts from my notes on the investigation.

The structure-aware split is much finer on a short, heavily sectioned agreement like this one (seventeen heading-aligned chunks against five), and no chunk in it contains another.

I did not measure retrieval quality between the two strategies on this document as part of this change, so I can't tell you the context chunk was worth or not worth its duplication in answer accuracy. This was a decision about what a new knowledgebase should get by default and what a person looking at their chunks should see, and I'll come back to what I'd measure next.

The lightrag initialisation code still describes structure_aware as "opt-in" in its log line and comments. That was true when those were written. Structure-aware chunking is handled a level up, in the document processor, which calls chunk_markdown_file when the knowledgebase's strategy says so, and the old token-based function stays bound underneath for backward compatibility.

Flipping the default without touching existing knowledgebases

Once I stopped treating it as a bug, the change was a default, and a default in this codebase lives in more than one place. A knowledgebase's chunking_strategy is written into its stored metadata at creation, and four separate code paths could supply the value when the caller didn't:

1. MetadataManager._build_config in a configuration file, which assembles the stored config: _val("chunking_strategy", "token_based") became _val("chunking_strategy", "structure_aware").
2. The KnowledgebaseCreateRequest schema in a configuration file, whose field default went from "token_based" to "structure_aware".
3. The per-user _defaults template in a configuration file, which seeds a new user's knowledgebase settings.
4. The equivalent _defaults template in the multi-user module, which seeds the same settings for users created there.

Miss any one of them and new knowledgebases created through that path would silently keep getting token_based. The admin configuration form had its own display fallback (configData.chunking_strategy || "token_based", in two places) and that followed suit, so the form doesn't show the old strategy for a knowledgebase whose config it hasn't loaded a value for.

What I deliberately did not change were the runtime fallbacks. All over the ingestion and query paths (the Celery tasks that rebuild a knowledgebase, the orchestration manager, startup, the query router) there is code shaped like this:

chunking_strategy=metadata.get("chunking_strategy", "token_based")

Those run against knowledgebases that already exist. An older knowledgebase might have no chunking_strategy field in its metadata at all, and when it was built, the absent field meant token_based. If I'd flipped those fallbacks too, the next rebuild or re-ingest of such a knowledgebase would quietly switch strategy underneath its existing chunks, entity graph and vectors. Leaving them alone means the new default only applies at creation time, which is where I wanted it.

One practical note from verifying it. Creating a knowledgebase through the agent API with no chunking_strategy in the request stored structure_aware, which is what we checked on staging. But the agent API's GET /knowledgebase/{id} response doesn't include the chunking strategy, so you can't confirm the default from there. The way to check is to load the stored metadata directly, through the metadata manager's load_knowledgebase_metadata.

Explaining the old behaviour where people see it

Changing the default does nothing for the knowledgebases already on token_based, and those are exactly the ones where the visualiser draws the spanning box. Their owners would still open the drawer, see one chunk sprawled over three, and draw the same conclusion I did.

Because chunk_type isn't stored, I couldn't mark the individual context chunk. What I could do was tell the frontend which strategy the whole knowledgebase uses. The /rag-chunks endpoint in a configuration file now reads the knowledgebase metadata and returns a top-level chunking_strategy alongside the chunks. It resolves an absent field to token_based, since that's the runtime fallback everywhere and a field-less knowledgebase really did run that strategy. It returns null only when the metadata can't be read at all, in which case the frontend shows nothing rather than guessing.

ChunkVisualiserDrawer.tsx then shows an amber info banner when, and only when, chunking_strategy === "token_based". It reads, in part:

Token-size chunking. This knowledgebase chunks by token count. For each table it adds a context-enriched chunk (the table plus its neighbouring text) to improve retrieval

and goes on to say that this chunk intentionally overlaps the others, that its box spans them below, and that new knowledgebases default to structure-aware chunking, "which follows the document layout and maps chunks 1:1 to regions." A structure-aware knowledgebase never shows the banner, because there is nothing to explain.

Of everything in this change, the banner is what I'd reach for first next time. The chunker was doing what its docstring said, and the visualiser, where we look at chunks, gave no hint of it.

What I'd check if you run a LightRAG-derived pipeline

If your ingestion inherited LightRAG's token chunking and someone has since added table handling, it's worth an afternoon with a document that has a few tables in it:

• Count the chunks per table. If you see two (or more, for a big table), look for a post-processing step that builds a context chunk out of the neighbours before you file it as a bug.
• Check whether any chunk type or role survives into your store. If it doesn't, any tooling that shows chunks can only explain them per knowledgebase, not per chunk.
• Check which flags control the chunker and which only control the multimodal processors. The names won't necessarily tell you.
• When you change a default, change it where values are created and leave the read-time fallbacks alone, or existing indexes will change strategy on their next rebuild.

The measurement I haven't taken is the one that would settle whether the context chunk earns its keep: the same question set against the same documents under both strategies, scored per question, with the table-answer questions broken out. Tables have cost us answers before (in one case they never reached the chunk stream at all, written up in Missing tables, not the extraction model, explain the accuracy gap), so that's the subset I'd want to see before saying anything firmer than "structure-aware is the clearer default".