14 to 17 seconds down to 0.3: caches that expire when the data changes

The overview page for a knowledgebase in Certant fires six requests the moment it opens: the document list, the resolved configuration, the graph metrics, the metrics history, the recent activity and the knowledgebase record itself. On a 1,132-document knowledgebase on staging, four of those six took between 14 and 17 seconds, and they did it on every reload, including reloads where nothing in the knowledgebase had changed since the last one.

Every request did the same expensive read before its own work, six did it at once, and the caches we had expired on timers short enough that an unchanged knowledgebase got recomputed over and over. The fix drops a cache entry when something in its knowledgebase changes, whatever the entry's age. With that in place the TTLs can be 24 h, and the page reloads in 0.32 s.

What the page was doing

I timed the six overview requests from a script against staging, three passes in a row. This is the third pass before the fix, with the knowledgebase id replaced by {kb} and the query strings trimmed:

== before-v3.1.4 pass 3 wall 17.08s
  200  16.86s /api/agent/knowledgebase/{kb}/config/re...
  200  16.78s /api/metrics/history?scope=kb&metrics=documents,ready,processing,...
  200  16.79s /api/proxy/knowledgebase/{kb}/graph/met...
  200  16.98s /api/knowledgebase/{kb}/documents?limit...
  200   0.14s /api/knowledgebase/{kb}
  200  16.78s /api/kb/{kb}/activity?limit=5

All three passes, per request, in seconds:

The slow requests finish within about half a second of each other on every pass, whatever they are asking for. Which ones escape varies: the metrics history was fast only on pass 1, the graph metrics only on pass 2. Those two had short-lived caches (60 s and 300 s) and were fast when their entry was still warm. On pass 3, five of the six were slow.

The shared cost was the knowledgebase lookup. Every one of those handlers started by calling the same get_knowledgebase function to check the knowledgebase existed, and that function had no cache at all. Each call made an HTTP round trip into the RAG service, which read the knowledgebase metadata under a file lock and, for a loaded knowledgebase, built live instance stats. Building those stats constructed a brand-new storage access object on every call: a new document-status store, a new asyncpg pool, and the catalogue queries that run when a pool initialises. Back in the API, it then ran a COUNT(*) on the document status table to get the document count. That count was synchronous, and it ran directly on the event loop.

Our API is one uvicorn process, so a blocking database call on the event loop stalls every other request in that process until it returns. Six handlers each doing a full knowledgebase read, with blocking parts that queue behind each other, fits the clustering in the table.

The short TTLs made it worse. Graph metrics were cached for 300 s, which meant the four-query Apache AGE read that produces the entity and relation counts ran on the request path of the first page load after every idle five minutes. The documents payload lived for 15 s, activity for 10 s, metrics history for 60 s. None of those windows had anything to do with how often the data changed. On a knowledgebase that had finished ingesting, nothing had changed, but each expiry still forced a full recompute on the next page load.

Generation counters and change timestamps

Caching the knowledgebase record for 24 hours on a timer would mean a renamed knowledgebase keeps its old title on the page for a day. A short TTL avoids that, and a short TTL is what we already had. We added two small markers per knowledgebase in Redis instead, and every cached entry is checked against them when it's read.

The first is a generation counter, kb:gen:{kb}. Every write to a knowledgebase's metadata increments it, and so does every broader invalidation (a document processed, a graph extracted or merged, an upload, a delete, a knowledgebase update). The cached knowledgebase record and the cached resolved config each store the generation they were built under, and a read fetches both the entry and the current generation in one MGET:

raw, gen = client.mget([key, render_kb_gen_key(kb_id)])
current_gen = str(gen or "0")
...
if str(entry.get("gen", "")) != current_gen:
    return None, current_gen
return entry["payload"], current_gen

A rebuilt entry is written with the generation read before the rebuild started. If a config save lands while a rebuild is in flight, the rebuilt entry goes into Redis tagged with the old generation and is a miss on the very next read. Tag it with the generation read afterwards and you can pin the pre-change value for the full 24 hours.

The second marker is a timestamp, kb:changed:{kb}, holding the last time anything happened to the knowledgebase's documents. Every worker phase event writes it, as does every broader invalidation and every queue-history change (an upload, a retry, a reprocess, a connector ingest). The document list and the activity feed are checked against it: a cached payload is only served if it was built after the last change. Those payloads record when their build started, not when it finished, so a change that lands mid-build invalidates the payload that build produced.

Both markers live for 30 days, longer than any entry validated against them, so an expired marker can turn a cached entry into a miss but can't make a stale entry look current.

The rest is making sure every change touches a marker. The RAG service's metadata manager is where every metadata write lands (knowledgebase updates, every config save, worker-side status and count updates), so its write methods carry a decorator that drops the cached record and config and bumps the generation after a successful write. The knowledgebase update endpoint in the API drops them as well, belt and braces, because a stale title after a rename is the first thing anyone would notice. A handful of mutations never pass through a worker hook at all (clearing a knowledgebase's data, deleting the graph, marking a document complete by hand, a synchronous rebuild), so each of those now invalidates the knowledgebase's caches directly. And a global configuration change only takes effect when the RAG service restarts, so on startup it sweeps every cached record and resolved config.

A not-found result is never cached, and neither is a resolved config where any part failed to resolve. A knowledgebase with no fast LLM legitimately returns a null there, and treating every null as a failure made those miss on every read, so the RAG service now returns an explicit unresolved list and only a non-empty one blocks caching.

The document count needed one more change. The service already had a per-process memo holding counts for five minutes. On a change-invalidated miss, that memo could answer with the count from before the change, and the new cache would then hold the old number until the next change. The miss path now clears the memo entry before counting.

Single-flight for the first load

Caching the record fixes warm loads. It does nothing for the first load after an invalidation, when all six handlers find an empty cache at the same moment and all six go off to do the full read, which is the original problem again.

With single-flight, the first handler to miss registers a future for that knowledgebase and does the read. Anyone who arrives while it's in flight awaits that future instead of starting their own:

pending = _DETAIL_INFLIGHT.get(flight_key) if use_cache else None
if pending is not None:
    try:
        shared = await asyncio.shield(pending)
    except Exception:
        shared = None          # the leader failed: read ourselves
    else:
        return copy.deepcopy(shared)

Three details took a little care. Each waiter gets its own deep copy, because several handlers modify the record they get back (the detail endpoint merges config fields into it). If the leader fails or is cancelled, the future is resolved with an exception and the waiters fall back to their own read rather than hanging. And the in-flight table is keyed by the event loop's id as well as the knowledgebase id, because an asyncio future belongs to the loop that created it and awaiting it from another loop fails.

The graph metrics needed a different shape of the same idea. The worker's per-document hook fires several times per document during ingestion, and it used to delete the cached metrics each time. That turned every reader after every merge into a hard miss, with the four-query AGE read inline. Now the hook marks the entry stale by rewriting its computed_at to zero and keeping the remaining Redis expiry. The next reader gets the last value immediately and schedules exactly one background recompute, guarded by a recompute lock. Deletes on the API side (a document or knowledgebase deleted, data cleared, the graph deleted or rebuilt) still remove the entry outright, because after one of those the old counts are wrong.

There was a race left in that. A recompute that was already running when a change landed would finish and write the pre-change counts back as fresh. Entries are stamped with their build start, so a fresh entry older than 60 s is also compared against kb:changed:{kb}, and if it was built before the last change it is treated as stale. The 60 s floor is there because a knowledgebase mid-ingestion emits a phase event every few seconds, and without it every poll would kick off the AGE read again.

TTLs as a memory bound

With invalidation driven by change, the TTL stops being a correctness setting and becomes a limit on how long Redis holds something nobody has asked for. These are the windows before and after, from the diff:

We kept the short in-flight windows because while documents are processing the data changes every few seconds anyway. "Settled" means every document is in a terminal status with no graph extraction or merge queued or running.

The activity feed has one part that does move with the clock: its count of documents completed in the last 24 hours. A settled feed is cached only until the first of those documents drops out of the window, so the number never goes stale.

The metrics history tiles on the overview are daily snapshot levels (documents, ready, processing, failed, entities, relations). They only change when the daily snapshot job writes a new row, so the snapshot day is now part of the cache key and the hour is, again, just a memory bound.

Getting blocking work off the event loop

The commit also moves synchronous work in async handlers into asyncio.to_thread throughout the knowledgebase API: every config-service read and write, the document status reads, the JSON validation cache read, the activity feed build (which sorts every row and bulk-reads phase events from Redis). Every function in the new cache module does synchronous Redis I/O and is called the same way.

On the RAG service side, the metadata read moved off the loop too, and the internal knowledgebase read stopped building a new storage access object per call. Its document counts now come from the same resolver the knowledgebase list already used: one short-lived asyncpg connection and one workspace-scoped aggregate for a PostgreSQL-backed knowledgebase. The service already keeps a cached storage access instance, and I didn't reuse it here. It's process-global and unlocked, its asyncpg pool is bound to the loop of whichever thread first used it, and asyncio.to_thread hands successive calls to different threads.

After

Same knowledgebase, same script, three passes after deploying the fix:

Pass 1 was the first load after the deploy, with every cache cold. The graph metrics paid for the AGE read once (3.71 s) and nothing else went over 1.2 s. Pass 2 and 3 were warm, and the whole page came in at 0.42 s and 0.32 s against 14.48 to 17.62 s before.

The same staging run also timed the page in a browser. The title rendered at 845 ms on first load and 336 ms on a reload, with the page fully settled at 1,597 ms, 1,368 ms and 1,388 ms across the three loads.

The change shipped with 22 dedicated tests for the new caches, 416 passing across the affected API suites (plus 4 failures that predated it) and 66 passing in the RAG service. The cache key templates are written by both services, so a shared fixture file asserts they render byte-identical key names on both sides. If one side renamed a key, the other side's invalidations would silently stop matching anything.

If you want to do the same

1. Give each cached aggregate a generation counter, increment it wherever that thing is written, and store the generation an entry was built under with the entry. Read the generation before you rebuild.
2. For data that changes through background work, keep a last-changed timestamp and serve an entry only if its build started after it. Make both markers outlive the entries they validate.
3. Put single-flight in front of the expensive read, give every waiter its own copy, and let waiters fall back to their own read if the leader fails.
4. If you're on asyncio, move every synchronous call in an async handler to a thread.