The completeness illusion: four ways our document status was wrong

Every document uploaded to Certant ends up with one status: processed, pending, incomplete or failed. That value is the only thing a user has to go on, and in the first half of 2026 we found four separate ways it could be wrong.

Three of them made finished documents look broken and one made a degraded document look finished. They lived in different stages of the pipeline and in different code, but each was a check that confirmed a step had run and never confirmed the step had produced what a correct run would produce.

The requirement itself went into our fix notes in capitals back in January: "The status pill is ABSOLUTE - it MUST show the correct status of the document." By June I was asking the same question again about a different part of the system: "does the completion validation show docs fully ingested? users only have this value to go on."

Validation ran before the data had landed

The earliest of the four (fix notes dated 3 January 2026) was a large technical document that had finished ingesting and showed as FAILED or PENDING. Everything it needed was there: the vector store held 257 chunks with their embeddings, 903 entities, and relationships, and the GraphML file had 6,603 entries for it.

There were two bugs, and either would have been enough on its own.

The first was timing. Validation fired as soon as processing reported complete, but the last persistence step, _insert_done(), writes the vector store files and was still running. For large documents that step took 30 to 60+ seconds. Validation read half-written data and reported exactly what it saw. There was a retry, but it was three attempts at 1, 2 and 4 seconds, 7 seconds in total, for a write that could take a minute.

The second was counting. This is what the relationship check did:

relationships_count = 0
for chunk_id in chunks_list:
    if _count_grep_matches(vdb_relationships_file, chunk_id) > 0:
        relationships_count += 1  # Counts chunks, not relationships!
relationships_verified = relationships_count >= len(chunks_list)  # Wrong check!

It counted chunks that had at least one relationship, then demanded that number equal the total number of chunks. If 200 of 257 chunks had relationships, the count was 200 and verification failed, regardless of how many relationships existed in total. A chunk can legitimately extract zero entities and zero relationships, so a rule that every chunk must have one fails any document with even one such chunk, however long you wait.

The replacement counts unique relationship lines across all of the document's chunks and asks for a plausible density instead:

chunk_ids_str = '\n'.join(chunks_list)
grep_result = subprocess.run(['grep', '-F', '-f', '/dev/stdin', str(vdb_relationships_file)], ...)
unique_lines = set(grep_result.stdout.strip().split('\n'))
unique_relationships_count = len([line for line in unique_lines if line.strip()])

# Verify: expect at least 0.5 relationships per chunk (more lenient)
expected_min_relationships = max(1, int(len(chunks_list) * 0.5))
relationships_verified = unique_relationships_count >= expected_min_relationships

The retry went from three attempts to five, with exponential backoff of 1, 2, 4, 8 and 16 seconds (31 seconds in total). The wait doubles when checkpoints or a Celery task show the document is still being processed, or when the document has more than 200 chunks, which takes the ceiling to 62 seconds. The status log line changed too, from entities=200, relationships=200 (chunk coverage dressed up as a count) to unique counts with coverage shown alongside.

The longer retry is still only a better guess. A document that takes longer than a minute to persist would still beat it, and 0.5 relationships per chunk is a heuristic. By June the graph side of the validator had moved on from inferring completion out of file contents: it keys off explicit graph_extraction_status == 'completed' and graph_merge_status == 'completed' values written by the stages that did the work. That is the better answer, because the stage that wrote the data is the one that knows when it has finished.

A validator reading a storage backend we had moved off

The second bug turned up on 19 June 2026. Our knowledgebases started life on file-based vector storage, JSON files such as vdb_chunks.json, and PostgreSQL later became the default backend. The completeness validator was written against the files.

It had been partly updated. validators.py already had a PostgreSQL path through a StorageAccess object (an environment variable and friends). But the background validation task called the lightweight validator without passing a storage object, and storage=None meant "use the files". For a PostgreSQL-backed knowledgebase those files don't exist, so the validator found 0 chunks and marked fully ingested documents failed or incomplete.

The fix is small: pass StorageAccess.get_instance(kb_id) through, so a PostgreSQL knowledgebase is validated by counting what's in PostgreSQL. With that in place, a complete document that had been reading as zero counted 66 chunks. File-backed knowledgebases still fall back to the file path.

Before the fix went in, we cleared the validation cache and re-triggered validation to refresh the statuses, assuming a fresh run would correct them. The fresh run was the old file-based check, and it flipped 54 complete documents to failed. We reverted that. Re-running a check we didn't trust gave us the same wrong answer on more documents.

The same validator got the graph verification right. That part read the explicit completion statuses, so it didn't care which backend held the rows. The chunk count was the part inferring completeness from storage, and it inferred from the wrong storage.

The same investigation found a second, smaller problem in how results were written back. The FAILED branch collapsed everything to a bare failed. A document whose chunks were verified (searchable, what we call RAG-ready) but whose graph stage had failed got the same label as a document that never ingested at all. Those now land as pending_graph_extraction or pending_graph_merge, and a stale failed on a RAG-ready document can be upgraded.

Merge completion never set the overall status

The third bug was in the same week and is the simplest of the four.

Our pipeline is fast-first. A document is chunked and embedded so it's searchable early, then graph extraction and graph merge run in the background. After embedding, a document rests at pending_graph_extraction with naive_rag_ready set. Merge is the final stage, so when merge completes, the document is done.

The merge-completion code set graph_extraction_status and graph_merge_status to completed and never touched the overall status. The user-facing completion check reads that overall field, looking for processed, completed or complete. So a fully merged document kept its stale non-terminal status and was reported as incomplete.

Most documents looked fine, which made it hard to spot. 463 of them already showed processed, set by a separate reconciler. That reconciler never fired for the tail of documents whose merges had been stuck and later recovered, so the documents showing the wrong status were exactly the ones that had already had a rough time. Anyone spot-checking would mostly see green.

The fix sets status: "processed" in the merge-completion update itself, next to the two sub-statuses, so it no longer depends on the reconciler. It's idempotent: a document already processed is unchanged.

The chunker that fell back and reported success

The fourth bug (fixed 28 June 2026) runs the other way, and it's the one that worries me most.

Structure-aware chunking works from the markdown the parser produces, so it can keep a heading with its list and keep tables together (its config has settings such as max_table_token_size, keep_heading_with_list and include_table_caption). To do that it has to find the markdown file. Our remote OCR path writes it to {working_dir}/ocr_output/{docname}/vlm/{docname}.md. The discovery code searched config.parser_output_dir, which defaulted to somewhere else.

When discovery comes up empty, the processor logs a warning and carries on:

Markdown file not found for structure-aware chunking, falling back to token-based chunking with text_content

Token-based chunking cuts the plain text by size and knows nothing about where tables and headings start and end. The document then embedded, extracted and reported success like any other. Validation passed it, because there were chunks.

On the Aruma enterprise agreement, the fallback produced 40 chunks. With discovery fixed, the same document produced 405. A short test document went from about 3 chunks to 17. Nothing errored, and the only trace was one warning line in a worker log.

Our first patch searched recursively under output_dir, which was still the wrong base directory. The authoritative location turned out to be the one the parser itself logs when it extracts its results, and the fix that stuck adds a "Strategy 0" ahead of the others: search working_dir/ocr_output recursively and prefer an exact match on the document's name.

A false FAILED gets looked at, because someone sees a red pill and asks about it. A silent fallback leaves nothing on screen to ask about. The document is searchable and answers come back; they are built from chunks that ignore the document's structure, and there's no signal connecting a worse answer to a chunking decision made at ingest. The same shape turns up in the OCR bake-off, where under-extracted scans still landed as processed, and in the JSON extraction regression, where a thin graph sat behind a chunks > 0 success check.

What the four have in common

The causes are a race, a wrong storage backend, a missing field in an update and a wrong directory, in different code, fixed between January and June. The checks had a narrower question in common. The validator checked whether it could read chunks and the merge checked that its own step was complete, while the chunker only needed to have produced some chunks. None of them compared the result with what a correct run of that stage would have produced.

There was no single fix, and I don't think there is one. Each needed a check specific to its stage, and the ones I'd now apply to any multi-stage pipeline are these.

A fallback is a result, and it should be recorded where people look, next to the document, as "requested structure-aware, used token-based". A warning in a worker log only helps someone who is already looking for a problem.

A validator should read data through the same storage layer the writer uses. If it has its own idea of where data lives, it goes out of date the next time storage changes, which is what happened to ours.

The stage that finishes the work owns the terminal status. If completion is a separate reconciliation step, some tail of documents will fall outside whatever triggers it.

Validation should run after the thing it checks has signalled that it's finished, and should prefer explicit completion markers written by that stage over inferring completion from what's on disk. Our longer retry schedule is a patch.

Count the thing you mean. "Chunks with a relationship" and "relationships" are different numbers, and a check that confuses them fails any document where some chunks have no relationships.

Know which way each check fails, too. Three of ours reported failures that weren't real, which is annoying but visible. The fourth passed a document we use as a demo with a tenth of the chunks it should have had, and nothing on screen said so.