Getting Strata to corpus scale: a 149,000-token prompt and a DuckDB lock storm

On 12 June our tabular pipeline pulled 257 tables out of six enterprise agreements, matched 131 column groups across them, then failed to build a data model. The error:

Requested input length 149729 exceeds maximum input length 131071

Synthesis runs at temperature 0, so every retry failed identically, and the prompt only grows as documents land. Past that ceiling the knowledgebase gets no new ontology version while extract, profile and match keep cheerfully reporting success.

The same run had a second problem. The table profiler held a read-only DuckDB connection for forty minutes at a time, and extract tasks ran out of retries waiting for it.

What the 257-table run cost

The corpus was six Disability Services enterprise agreements, 1,222 pages, including a 414-page fully scanned Victorian public mental health agreement with about 56 pages of rate tables. The run took roughly 8.3 hours from upload to all-done, about 6 of them in a tabular phase that behaved like six serial per-document runs plus failures, for LLM work that would parallelise to about an hour.

The profiler held a read-only DuckDB connection per table across the LLM and embedding calls, because forecast-readiness detection afterwards probes the columns the LLM marks as temporal. A DuckDB writer needs full cross-process exclusivity, which even a read-only handle blocks, so one pinned reader meant no writer for the whole LLM latency. Extract tasks burned three retries (about 90 seconds) against 40-minute holds and went to failed; four documents needed manual rescue.

The final matcher consolidation round over all 257 tables ran past 75 minutes, beyond kombu's default 3,600 second visibility timeout, so Celery redelivered it mid-run and three match_and_model runs ended up racing on one knowledgebase through an unlocked version read-increment at two mint sites. I covered what that race did to the ontology in Building a data model from PDFs instead of designing one. The default applied because one of the Celery apps on the shared Redis broker set no broker_transport_options, so the 28,800 second value configured elsewhere never took effect.

Synthesis had a second bug as well. backing_tables came purely from the LLM echoing table references. Validation dropped unknown names but never completed the list, so at 257 tables attention dilution lost some. One financial ObjectType stuck at 11 of its 14 backing tables across six or more temperature-0 runs, and the missing tables dropped out of governed views without an error.

Schema families: sizing the prompt by table shape

Both synthesis bugs come from enumerating tables when the LLM's job is modelling shapes. One call did two jobs:

1. "Which tables have the same shape?" This grows with corpus size and is mechanical; no LLM needed.
2. "What do this shape's rows mean, and how does it relate to other shapes?" That is modelling judgement, growing with the number of distinct shapes, sublinear in corpus size. The same wage table across six employers and four years is one shape with 24 instances.

A deterministic grouping layer now sits between them. Tables whose column-name fingerprints match form a schema family, and synthesis gets one line per family: the representative column render, member and document counts, domain tags. The LLM proposes backing_families, and code expands each family into backing_tables and the per-table column mappings. With no table inventory to echo, nothing can drop silently.

Family assignment is sticky: a family stays on its ObjectType from the previous active version unless the new proposal explicitly moves or retires it, so every family is newly assigned, retained or visibly raw-only.

Measured on 6 July:

That is a 24x smaller prompt on a corpus 28% larger. The 40-document, 320-table run swept to 23 families, against a token budget of 100,000.

The current-ontology section grows with the stored model, so later runs on the same corpus measured 16,340 and 12,954 tokens, still far below the ceiling. For a heterogeneous cold start, a token estimator checks against STRATA_SYNTH_TOKEN_BUDGET and partitions novel families into batches, one proposal each, followed by a single merge pass over the proposed ObjectType summaries. That merge input is bounded by model size, so it cannot overflow; in the final scale verification it produced exactly one version.

Tables landing in already-modelled families attach deterministically and mint a new version with no LLM call. Document 500 that looks like documents 1 to 499 costs zero tokens and cannot fail on context length.

Snapshot, then think

The profiler fix is embarrassingly simple. Read everything in one short-lived connection, close it, then do the slow work with no handle open:

snapshot = read_snapshot(kb_id, table)      # ONE connection, sub-second
profile  = await think(snapshot, llm, embed)  # minutes, ZERO connections
payloads = CommentPayload(table, profile)     # buffered, not written

Since the LLM decides which columns are temporal, the snapshot pessimistically pre-materialises the distinct value series for every temporal candidate column: anything castable to DATE or TIMESTAMP, plus every VARCHAR. Snapshotting only the LLM-independent statistics would not have been enough.

Writes are funnelled into one reconcile step per document that applies every COMMENT ON statement and the dictionary merge in one lock window, then checkpoints; expected hold is tens of milliseconds. Read-side connects now spin-wait up to 30 seconds on the transient file lock instead of failing into a 60 second Celery retry, added after three documents dispatched at once with no wave separation made read connects fail instantly.

The acceptance test dispatches 3 documents of 20 tables each simultaneously:

The 40-document, 320-table ingest completed in about 18 minutes with zero rescues, against roughly 6 hours and 4 manual rescues for June's 257 tables.

The design document projected the 82-table Victorian document, roughly 68 minutes serial, at 9 to 12 minutes with per-table fan-out at 8. We never ran it that way, so it is only calibration.

Per-table fan-out is off by default because the first live run at 21 documents put a polling orchestrator in every slot of the 8-slot prefork pool while 19 subtasks queued with nowhere to run, and it thrashed until max retries. STRATA_PROFILE_FANOUT now defaults to 0, so parallelism comes from per-document orchestrators across slots, and an orchestrator work-steals any table whose subtask has not staged its result by the poll deadline. Matcher Phase B confirmations run in-task under asyncio.gather for the same reason.

One matcher per knowledgebase, and no lost work

The Celery fix set broker_transport_options on every remaining client of the shared broker, including a byte-identical second copy of the worker's task module, which could otherwise have silently restored the 3,600 second default. The sweep test found a fifth site the manual audit had missed, so a repo-level regression test now asserts the setting on every Celery( constructor pointing at the shared broker:

broker_transport_options={"visibility_timeout": 28800},

Celery settings cannot be the guarantee, because acks_late redelivery can always happen. Modelling runs under a per-knowledgebase Redis lock, strata_match_running:{kb_id}, taken with SET NX EX at a 7,200 second TTL with a renewal touch between phases. A round arriving during the lock returns skipped_already_running and sets strata_match_rerun:{kb_id}. On release, the holder deletes a set flag and enqueues exactly one debounced round.

The rerun flag also closes a gap in the older debounce, whose key is SET NX with a 30 second TTL equal to the countdown: a profile finishing at t+30.1s finds no key, sets a fresh one and mints a second concurrent round, which the run lock now skips and the handoff reruns. Live logs show skipped, rerun flagged on the concurrent arrivals and exactly one trailing round after release.

Both mint sites now allocate versions under a per-knowledgebase mutex with a compare-and-swap on the version pointer. The first cut aborted when the CAS failed, which was safe but useless, so it now re-stamps and retries at all three mint paths.

Where the bottleneck moved

At corpus scale the matcher profiled 1,888 columns, produced 375 candidate groups from its mutual top-k pass, and confirmed them one LLM call at a time at 15 to 25 seconds each: hours per round, re-confirming everything.

Match ids are deterministic hashes of sorted member sets, so the SchemaMatch store was already a confirmation cache, and the biggest win needed no new code paths: a round now only confirms groups whose membership changed. Confirmations then ran in parallel in-task, and later in batches per call.

That is about a 60x cut in round time for the same group count, with steady-state rounds close to free: one later run re-confirmed 21 groups out of 382. Keyed on exact membership, the cache first missed on every incremental ingest, since a new document changes membership; subset-extension and shrink inheritance forms fixed that.

What I would tell anyone building this

Identity must come only from deterministic inputs: our first fingerprint included profiler-derived flags, which wobble between runs, so identical shapes split into different families and the drift gate parked them for review; names-only fixed it.

Unit tests did not catch the design defects here. Live runs against real LLM traffic in an isolated VM did: the fan-out livelock at 21 documents, the drift rule that no-oped on 18 new matches, the coverage check that never converged when the LLM named a property differently from the matcher's canonical name, and a gate condition that took four iterations because the real 320-table corpus kept falsifying it. The unit suites stayed green throughout, and the module suite finished at 548 tests.

The settings:

STRATA_PROFILE_FANOUT=0              # per-table fan-out off; parallelism is per-document orchestrators
STRATA_MATCH_CONFIRM_CONCURRENCY=16  # Phase B confirmations in flight, in-task asyncio.gather
STRATA_MATCH_CONFIRM_BATCH=10        # column groups per confirmation call
STRATA_MATCH_MUTUAL_TOP_K=6          # 3 missed renames against large same-named cohorts
STRATA_SYNTH_TOKEN_BUDGET=100000     # under the 131,071 model max; over budget goes hierarchical