Start free

Notes from the workshop.

Writing on AI, data engineering and the infrastructure underneath it, from the founder building Certant. Notes from the work, not the marketing.

Cover image for Why one chunk spanned three others in our chunk visualiser
Latest·Technical

Why one chunk spanned three others in our chunk visualiser

Our chunk visualiser showed one chunk containing its neighbours. The chunker did it on purpose for tables, so we changed the default strategy for new knowledgebases.

Daniel Voyce··10 min read
Read the post
37 posts

Technical

Engineering notes on data, models, infrastructure and the systems underneath them.

Certant Strata Data Model tab for the Workforce Telemetry knowledgebase, Object Types view, showing six object types synthesised from a stream: Department, Employee, EmploymentType, Location, PayFact and PayGrade, with semantic column types such as identifier_code and currency_aud
Daniel Voyce··10 min read

Streaming records into Strata with no continuous DuckDB writer

Producers POST batched NDJSON, we land immutable Parquet files, and a DuckDB view created once reads the folder at query time. No loader, no broker.

Read the post
Cover image for The completeness illusion: four ways our document status was wrong
Daniel Voyce··9 min read

The completeness illusion: four ways our document status was wrong

Four pipeline checks confirmed a step had run but not what it produced. Three reported failures that weren't real and one passed a degraded document.

Read the post
Cover image for 14 to 17 seconds down to 0.3: caches that expire when the data changes
Daniel Voyce··11 min read

14 to 17 seconds down to 0.3: caches that expire when the data changes

A knowledgebase page made six duplicate reads per load and took 14-17 s on every reload. Generation numbers, single-flight and 24 h TTLs got it to 0.3 s.

Read the post
Certant Data Model page, Tables tab, with three same-shape water quality tables for FY2021, FY2022 and FY2023
Daniel Voyce··9 min read

Getting Strata to corpus scale: a 149,000-token prompt and a DuckDB lock storm

Synthesis died at 257 tables. Schema families cut the prompt 24x and snapshot-then-think profiling made table profiling 6x faster.

Read the post
Cover image for A pending PostgreSQL lock blocks every reader, and it took us down twice
Daniel Voyce··10 min read

A pending PostgreSQL lock blocks every reader, and it took us down twice

A backup and a stuck transaction each left an ALTER TABLE waiting, and every SELECT queued behind it. How we found both and the rules we run by now.

Read the post
Cover image for Two locks that locked themselves: an fcntl.flock and a threading.Lock
Daniel Voyce··11 min read

Two locks that locked themselves: an fcntl.flock and a threading.Lock

Twice in seven weeks one thread asked for a lock it already held and froze our RAG service. Why, how we found it, and the reentrant fixes.

Read the post
Certant Strata "Fix the model" dialog showing typed correction proposals, each with the column evidence quoted underneath and an amber warn-and-allow callout
Daniel Voyce··9 min read

Teach it once: the corrections ledger

A Year column that meant years of experience broke our auto-built data model. Now one English sentence fixes it, and the fix survives every re-ingest.

Read the post
Cover image for Three instant retries turned one rate limit into an overnight 429 storm
Daniel Voyce··9 min read

Three instant retries turned one rate limit into an overnight 429 storm

An overnight ingest logged 307.2K rate-limit errors while the provider's concurrency sat flat. The gateway's instant retries were the cause.

Read the post
Cover image for One bad LLM response in 597 chunks: how FIRST_EXCEPTION failed our biggest documents
Daniel Voyce··11 min read

One bad LLM response in 597 chunks: how FIRST_EXCEPTION failed our biggest documents

A malformed 200 from the model cancelled a whole batch of good extraction results, and bigger documents were far more likely to hit one. The fix and the pattern.

Read the post
The Monitors panel with the edit-monitor form and the demo monitors below
Daniel Voyce··10 min read

From a threshold breach to an approved action

How a Strata monitor detects a breach, counts the blast radius from the ontology, and parks the fix in an approvals inbox until a person signs it off.

Read the post
Cover image for Autograd was on: 3 GB per highlighted chunk and eight OOM kills in three days
Daniel Voyce··9 min read

Autograd was on: 3 GB per highlighted chunk and eight OOM kills in three days

Citation highlighting ran a Hugging Face model with autograd enabled. Each chunk kept 3 GB it never gave back. One line of torch.inference_mode() fixed it.

Read the post
Cover image for Graph retrieval was 70x slower than it needed to be
Daniel Voyce··12 min read

Graph retrieval was 70x slower than it needed to be

Every index was in place and a one-hop graph lookup still took 28 seconds. Replacing Apache AGE Cypher with direct SQL made it sub-second.

Read the post
Certant Strata Data Model tab, Ask your data, showing the three-row turbidity result with five source-PDF provenance chips, the views used, and the expanded governed SQL
Daniel Voyce··8 min read

Ask your data in English, get governed SQL or an honest no

How Certant Strata turns a question into governed SQL with one temperature-0 call and a guard, and what the user sees when it refuses.

Read the post
Cover image for A $169-a-month VM beat serverless and Kubernetes for our stack
Daniel Voyce··11 min read

A $169-a-month VM beat serverless and Kubernetes for our stack

We priced Fargate, Cloud Run, Container Apps and managed Kubernetes against one $169/month VM, then kept the VM and built blue/green deploys on it.

Read the post
Cover image for Four data-loss bugs that came from one dict merge
Daniel Voyce··10 min read

Four data-loss bugs that came from one dict merge

Four vector-store bugs lost up to 71% of chunks and entities without raising an error. All four trace back to one unguarded write-behind merge.

Read the post
The Strata blueprint gallery in the AI Agent Builder, showing template cards for BP-01 to BP-13, each naming its Palantir Foundry / AIP analogue
Daniel Voyce··8 min read

Certant Strata and Palantir Foundry / AIP, feature by feature

I graded our ontology product against the nine things a Foundry / AIP engagement delivers. It scored three partials and six reds, and it leads at the document end.

Read the post
Three castles, each sealed inside its own floating glass dome with a Docker whale, illustrating one VM per worktree
Daniel Voyce··10 min read

Running the whole RAG stack on a MacBook, one Colima VM per worktree

What really runs locally on Apple Silicon: a VM per git worktree, a pgvector SIGILL that never showed on OrbStack, and a PaddleOCR segfault we never fixed.

Read the post
Cover image for Missing tables, not the extraction model, explain the accuracy gap
Daniel Voyce··9 min read

Missing tables, not the extraction model, explain the accuracy gap

Two knowledge bases, same corpus, different extraction models, the same 41.8% and zero disagreements. The failing questions all had their answers in tables.

Read the post
Cover image for A 0.6B model was starving on the CPU: moving citation highlighting to the GPU
Daniel Voyce··6 min read

A 0.6B model was starving on the CPU: moving citation highlighting to the GPU

Citation highlighting ate 50 s of a 71 s query on our DGX Spark. The encoder was tiny; the CPU was full. Moving it to the idle GPU: 3.6 s to 0.08 s per chunk.

Read the post
Diagram contrasting the four-model stack with the consolidated Gemma-4 stack, showing OCR at 0.42 s per page against 18 s per page
Daniel Voyce··11 min read

The Gemma-4 experiment: one model for chat, OCR and reranking

One Gemma-4 model runs chat, extraction and reranking on a DGX Spark. OCR stays a dedicated model: consolidated OCR runs 40x slower per page.

Read the post
Grafana dashboard for the DGX Spark stack during the OCR phase of a 50-document benchmark, showing generation at 363 tok/s, prompt prefill peaking at 3.69K tok/s, and 99.4 GiB of 121 GiB unified memory used
Daniel Voyce··10 min read

155 pages of OCR took 37 minutes. The bottleneck was one threading.Lock.

A lock around a single GPU model made OCR serial. Moving GLM-OCR to SGLang took a 155-page PDF to under five minutes. The RunPod and client-side details.

Read the post
PDF visualiser showing the Schedule J salary and allowance rate table with DDSO 1, 1Q, 2 and 2A classification groups, four effective-date column groups, and chunk overlay boxes numbered 5 and 6 on the right panel
Daniel Voyce··11 min read

Five OCR engines and one enterprise agreement: what actually read the rate tables

MinerU, Docling, PaddleOCR, GLM-OCR and Baidu Unlimited-OCR on the same wage schedules. Rowspans, a silent under-extraction bug, and what we run now.

Read the post
Stacked bar chart of the DGX Spark's 121 GiB unified memory pool filled three ways, showing gpt-oss-120B leaving 1-7 GiB free, Qwen3.6 leaving 28-30 GiB and Gemma-4 leaving about 55 GiB
Daniel Voyce··10 min read

Why Qwen3.6 with thinking off became our DGX Spark default

Swapping gpt-oss-120B for Qwen3.6-35B gave back 25 GiB on the GB10, and one per-call reasoning flag took a merge summary from 189 s to 4 s.

Read the post
Grafana panels during document ingest on the gpt-oss stack
Daniel Voyce··11 min read

The standard stack: gpt-oss-120B and a full RAG pipeline on one box

Four models in 121 GiB of unified memory, 1 to 7 GiB free, and the boot loop that took a 64 GB swapfile to fix. The DGX Spark reference build, warts included.

Read the post
Line chart of aggregate tokens per second against concurrent requests for Qwen3.6 with thinking off and Gemma-4, with saturation knees marked at 32 and 64 concurrent requests.
Daniel Voyce··9 min read

Most people tuning local LLMs optimise the wrong half of the problem

A full 50-document ingest pinned the GPU at 96% for 38 minutes, on prefill, not decode, while the celebrated decode levers barely moved throughput.

Read the post
Bar chart of unified memory used by three model stacks against a 121 GiB ceiling: gpt-oss-120B leaves 1 to 7 GiB free, Qwen3.6 leaves 28 to 30 GiB, Gemma-4 leaves about 55 GiB idle
Daniel Voyce··6 min read

128GB of VRAM is the most misread spec on the DGX Spark

The DGX Spark's 128GB is one pool shared by CPU and GPU, not VRAM: the memory budget, the OOM spike, and what to set.

Read the post
Timeline infographic of seven optimisation steps on a DGX Spark, each showing a before and after value, running from "won't boot" to a batched ceiling of 1,000 to 2,800 tokens per second.
Daniel Voyce··8 min read

Seven levers that took a DGX Spark from won't boot to 2,800 tokens a second

Seven config changes took my DGX Spark from OOM-killing itself on boot to 2,800 tokens a second, measured before and after.

Read the post
Verdict cards for the three models: Qwen3.6-35B at 2,805 tok/s batched with 28-30 GiB free, Gemma-4 26B consolidating chat, OCR and rerank, and gpt-oss-120B at 2,621 tok/s but only 1-7 GiB free
Daniel Voyce··9 min read

Which LLM should you run on a DGX Spark? gpt-oss-120B vs Qwen3.6-35B vs Gemma-4 26B

Three model stacks, one DGX Spark, a month of benchmarks: why Qwen3.6-35B with thinking off beats Gemma-4 26B and gpt-oss-120B on this box.

Read the post
Cover image for Asking for JSON halved our extraction recall. JSON Lines fixed it.
Daniel Voyce··11 min read

Asking for JSON halved our extraction recall. JSON Lines fixed it.

A graph-extraction regression traced to output shape: a closed JSON object tells the model to stop early. Removing the enclosing array recovered 2.7x.

Read the post
Bar chart comparing single-stream and 128-request batched throughput for gpt-oss-120B, Qwen3.6-35B and Gemma-4 26B on a DGX Spark, with 42.7x, 23.4x and 22x gain badges
Daniel Voyce··8 min read

Stop chat-testing your DGX Spark

A single chat window is the one benchmark the DGX Spark is built to lose. The two hardware numbers behind that, and the 22-43x you get back from batching.

Read the post
My FrankenNAS setup
Daniel Voyce··3 min read

Getting vLLM v0.9.2 working on CUDA 12.4 on QNAP (Driver 550.76)

Getting vLLM v0.9.2 running on CUDA 12.4 with an older QNAP NAS and Nvidia driver 550.76 using a custom Docker build.

Read the post
Photo by Fabrizio Chiagano
Daniel Voyce··7 min read

Moving from Ollama to vLLM: Finding Stability for High-Throughput LLM Serving

An engineer's journey from using Ollama to vLLM for running large language models locally, exploring the trade-offs between convenience and production-grade stability for high-throughput inference.

Read the post
Abstract visualization of data graph connections
Daniel Voyce··7 min read

GraphRAG Is the Logical Step From Rag — So Why the Sudden Hype?

GraphRAG builds on RAG by using graph structures to organize data relationships, providing LLMs with richer context while avoiding hallucinations.

Read the post
Autogluon architecture showing multi-layer stacking for forecasting
Daniel Voyce··11 min read

Revolutionising Machine Learning: Achieving Top 4% in Kaggle with AutoGluon in Just 7 Lines of Code

Discover how AutoML and AutoGluon can achieve competitive Kaggle rankings with just seven lines of Python code, automating feature engineering and model selection.

Read the post
GPU and data processing technology
Daniel Voyce··9 min read

Data Engineering: Fast Spatial Joins Across ~2 Billion Rows on a Single Old GPU

Explore GPU-accelerated spatial joins processing 2.3 billion rows on a consumer GTX 1060, comparing Parquet vs ORC performance for large-scale geospatial data.

Read the post
Colorful abstract shapes representing GPU computing and data processing
Daniel Voyce··8 min read

GPU Data Engineering, CSV to Parquet & ORC using Dask & RAPIDS.ai

Converting large CSV files to Parquet and ORC formats using Dask and RAPIDS.ai on consumer GPU hardware, with performance benchmarks and optimization strategies.

Read the post
7 posts

Product

What we are building with Certant, and how it works in practice.

Knowledgebase wizard with agreement and payroll export indexed, FDE drawer open
Daniel Voyce··9 min read

Four employees below their legal minimum: the Aruma scenario end to end

A screenshot-led walk through the Aruma pay-compliance run: bind a knowledgebase, ask in plain English, approve one correction, get four names back.

Read the post
Certant admin UI in dark mode with the AI FDE drawer open on the right, bound to the Aruma Disability Services EA knowledgebase, showing the greeting and a branch chip reading main
Daniel Voyce··10 min read

An agent with write access to your ontology

Our AI FDE can edit a live data model. The gate around it: a short-lived key scoped to you, three server-enforced tiers, and a signed exposure register.

Read the post
Certant PDF viewer on page 146 of 147, Schedule J minimum hourly rates, DDSO 2 at $21.34
Daniel Voyce··10 min read

Page 146: the number nobody was going to find

Two people paid below a legal minimum buried in a 147-page agreement, found and proved from the page, then fixed with a human signature.

Read the post
Certant dashboard in Simple mode showing workspace overview cards and the small AI FDE pill in the bottom-right corner
Daniel Voyce··8 min read

An engineer in a drawer: 274 seconds to a working ontology

Palantir sends a forward deployed engineer. We ship one in a drawer: 21 July 2026, it built a working ontology for 1.5 credits, failed three of five tasks.

Read the post
Certant KB settings, Advanced section, showing the Strata data model toggle card
Daniel Voyce··12 min read

Building a data model from PDFs instead of designing one

Certant Strata turns the tables inside a PDF corpus into a versioned data model. How the pipeline works, and the four ways it broke at corpus scale.

Read the post
Slide reading "Not a chatbot. Their company's intelligence." with four cards: grounded in your ontology, every answer cited, 100% on the box (on-device and air-gapped capable, no data egress), working on day one
Daniel Voyce··9 min read

What air-gapped actually means: every phone-home in the stack, audited

Ten outbound-call vectors in our own stack, the two that needed code, and the tcpdump proof that a locked-down box still answers.

Read the post
MindLattice Logo
Daniel Voyce··3 min read

Overdue update: The launch of MindLattice

An overview of MindLattice, a platform addressing data engineering challenges in AI transformation and serving underserved industries.

Read the post
28 posts

Informational

Practical background and business guidance for teams exploring AI.

AI Risk Management: A 2026 Guide
Daniel Voyce··15 min read

AI Risk Management: A 2026 Guide

Master AI risk management in 2026. Learn frameworks, assessment methods, tools, and best practices for regulated industries. Start free with Certant.

Read the post
How to Unify Fragmented Enterprise Data
Daniel Voyce··10 min read

How to Unify Fragmented Enterprise Data

Unify fragmented enterprise data across systems into a single knowledge base.

Read the post
How to Build Chatbots Easily: A 2026 Guide
Daniel Voyce··9 min read

How to Build Chatbots Easily: A 2026 Guide

Learn how to build chatbots easily in 2026. From no-code platforms to training on your own data, this guide covers tools, steps and real business examples.

Read the post
Unified Knowledge Base for HR Policies: 2026 Guide
Daniel Voyce··10 min read

Unified Knowledge Base for HR Policies: 2026 Guide

Build a unified knowledge base for HR policies that gives staff instant, verifiable answers.

Read the post
Benefits of Traceable AI in Finance: A 2026 Guide
Daniel Voyce··10 min read

Benefits of Traceable AI in Finance: A 2026 Guide

Traceable AI gives finance teams verifiable answers, faster audits and safer credit decisions. Here is what it changes in practice.

Read the post
Progress vs Seekr for Data Integration: 2026 Comparison
Daniel Voyce··9 min read

Progress vs Seekr for Data Integration: 2026 Comparison

Progress vs Seekr for data integration compared: pipelines, AI-ready data, governance and knowledge graphs for regulated firms. See which fits your stack.

Read the post
Enterprise Knowledge Management Platform Deployment Guide
Daniel Voyce··18 min read

Enterprise Knowledge Management Platform Deployment Guide

A practical deployment guide for enterprise knowledge management. Plan, migrate, govern and measure your platform rollout with confidence.

Read the post
Traceable AI in Regulated Sectors: Why It Matters
Daniel Voyce··12 min read

Traceable AI in Regulated Sectors: Why It Matters

Traceable AI regulated sectors: Traceable AI in regulated sectors reduces compliance risk and builds auditor confidence.

Read the post
Reducing Manual Data Processing in Healthcare
Daniel Voyce··10 min read

Reducing Manual Data Processing in Healthcare

Reducing manual data processing in healthcare cuts errors, frees clinical time and improves data quality.

Read the post
Glean vs Stardog for Knowledge Management in 2026
Daniel Voyce··11 min read

Glean vs Stardog for Knowledge Management in 2026

Glean vs Stardog for knowledge management: compare search, graphs and governance, plus a regulated-industry alternative with verifiable answers.

Read the post
How to Audit AI Generated Answers: A 2026 Guide
Daniel Voyce··11 min read

How to Audit AI Generated Answers: A 2026 Guide

Learn how to audit AI generated answers with a repeatable process for source verification, accuracy checks, and audit trails that satisfy regulators.

Read the post
How to Audit AI Knowledge Graphs: A Step-by-Step Guide
Daniel Voyce··12 min read

How to Audit AI Knowledge Graphs: A Step-by-Step Guide

Learn how to audit AI knowledge graphs effectively. Step-by-step process to verify accuracy, identify gaps, and ensure reliable answers in your system.

Read the post
Best Glean Alternatives for Enterprise Knowledge in 2026
Daniel Voyce··13 min read

Best Glean Alternatives for Enterprise Knowledge in 2026

Compare the best Glean alternatives for enterprise knowledge in 2026, from open-source search to verifiable knowledge graphs for regulated industries.

Read the post
Implementing Sovereign AI for Government Agencies
Daniel Voyce··11 min read

Implementing Sovereign AI for Government Agencies

A step-by-step guide to implementing sovereign AI for government agencies, covering data residency, governance, transparency and air-gapped deployment.

Read the post
A practical evaluation method for answers that need to stand up to checking.
Daniel Voyce··7 min read

How to Test Document AI Before Trusting Answers

A practical method for evaluating document AI with traceable citations, refusal tests, human review and inspectable calculations.

Read the post
Get Pricing for Automated Workflow Tools: 2026 Review
Daniel Voyce··11 min read

Get Pricing for Automated Workflow Tools: 2026 Review

Compare automated workflow tool pricing, uncover hidden costs, and calculate ROI before you buy. Get pricing for automated workflow tools with confidence.

Read the post
How to Verify AI Answers in Compliance: A 2026 Guide
Daniel Voyce··10 min read

How to Verify AI Answers in Compliance: A 2026 Guide

Learn how to verify AI answers in compliance with source citations, audit trails and human review. A practical 2026 guide for regulated teams. Start now.

Read the post
Glean vs Stardog: Knowledge Graph Comparison 2026
Daniel Voyce··9 min read

Glean vs Stardog: Knowledge Graph Comparison 2026

Glean vs Stardog knowledge graph comparison for 2026: see how each platform handles semantic modelling, governance and regulated-sector needs before you.

Read the post
Book a Demo for Knowledge Graph Software: 2026 Review
Daniel Voyce··14 min read

Book a Demo for Knowledge Graph Software: 2026 Review

Book a demo for knowledge graph software with confidence. Compare platforms, see what to ask, and learn where knowledge graph RAG fits. Start your free.

Read the post
Preventing AI Hallucinations in Regulated Industries
Daniel Voyce··12 min read

Preventing AI Hallucinations in Regulated Industries

Preventing AI hallucinations: Discover how regulated industries prevent AI hallucinations. Learn grounding techniques, risk frameworks, and verification.

Read the post
Enterprise AI Implementation Cost and Timeline Guide
Daniel Voyce··12 min read

Enterprise AI Implementation Cost and Timeline Guide

Enterprise AI implementation cost and timeline explained: what drives budgets, how long rollouts take, and what regulated teams should plan for. Start.

Read the post
Automating Contract Review for Financial Services in 2026
Daniel Voyce··12 min read

Automating Contract Review for Financial Services in 2026

How automating contract review for financial services cuts manual hours, flags risk and keeps auditors happy. Explore tools, workflows and best practice.

Read the post
Best Glean Alternatives for Regulated Firms in 2026
Daniel Voyce··14 min read

Best Glean Alternatives for Regulated Firms in 2026

Compare the best Glean alternatives for regulated firms in 2026, from data governance in AI search tools to compliance-ready knowledge base software.

Read the post
Scaling Knowledge Graphs for Enterprise Teams in 2026
Daniel Voyce··17 min read

Scaling Knowledge Graphs for Enterprise Teams in 2026

Scaling knowledge graphs for enterprise teams: governance, team structure, knowledge graph RAG and implementation best practices. Explore the 2026.

Read the post
Deploying Sovereign AI for Government Policy: 2026 Guide
Daniel Voyce··15 min read

Deploying Sovereign AI for Government Policy: 2026 Guide

Learn how deploying sovereign AI for government policy keeps data resident, meets frameworks, and supports air-gapped deployment. Start your sovereign AI.

Read the post
Buy Enterprise AI Compliance Solution: 2026 Buyer's Guide
Daniel Voyce··11 min read

Buy Enterprise AI Compliance Solution: 2026 Buyer's Guide

Ready to buy enterprise AI compliance solution? Compare features, frameworks, pricing models and deployment options before you commit. Start your free.

Read the post
Automated Policy Compliance for Large Enterprises
Daniel Voyce··12 min read

Automated Policy Compliance for Large Enterprises

Learn how automated policy compliance helps large enterprises reduce risk and save time. Explore best practices, examples, and a clear implementation.

Read the post
Scaling Internal Knowledge Bases for Mid-Sized Firms
Editorial Team··10 min read

Scaling Internal Knowledge Bases for Mid-Sized Firms

Scaling internal knowledge bases for mid-sized firms requires structure, not just software. Learn best practices, productivity gains, and software.

Read the post

Get new posts by email.

Occasional writing on AI, data engineering and what we're building. No marketing fluff.