155 pages of OCR took 37 minutes. The bottleneck was one threading.Lock.

A customer document, 155 pages of enterprise agreement, took about 37 minutes to OCR and usually didn't finish: the RunPod job blew through its 570-second polling window, the client gave up and posted a new job, and copies of the same document piled up IN_PROGRESS on the same endpoint.

The model was fine, the GPU was mostly asleep. Our OCR server wrapped a single transformers model in a threading.Lock, so every page waited its turn, 8 to 15 seconds each. Replacing it with SGLang, which does continuous batching, took the same PDF to under 5 minutes on 12 June 2026.

One lock, one page at a time

The original handler was a FastAPI app with one GLM-OCR model (0.9B parameters) in GPU memory and a lock around the generation call. It can't use a GPU properly: a synchronous, GPU-blocking handler with no batching serves one page at a time, regardless of how much GPU is idle.

The client polls the RunPod job to a 570-second deadline, then Celery retries it. The retry posts a new job instead of resuming the existing one, so any parse over about 9.5 minutes generated duplicate jobs. A 37-minute document produced a small storm of them.

On our own DGX Spark the same architecture failed worse. The OCR client had max_workers defaulting to 16 in some configurations and 32 in others, firing 16 to 32 concurrent OCR requests at a server that could serve one. The Celery worker command had no concurrency limit, defaulting to the core count, around 20: ten documents queued, all ten started OCR at once. Every worker process also loaded its own copy of the PP-DocLayoutV3 layout model, 1.7 to 1.8 GiB per fork, sharing memory with the language model. The box hit 114 GiB used with 7 GiB free; I stopped the OCR worker before it took the model layer down too. Zero of those ten documents reached completion, and the knowledgebase collected 38 churned failure records.

Sustainable concurrency on that configuration was 2 to 3 documents. It's marketed as an AI appliance.

Why it took months to fix

Every attempt to fix the OCR path over several months concluded GLM-OCR had to run on transformers, because its vision encoder wasn't supported by SGLang or vLLM. That belief was wrong and untested. We traced it to a throwaway comment written on 19 March 2026, saying the model was small enough that transformers was fast enough. It got paraphrased into "the encoder is not SGLang-supported" and copied into deployment configs, the customer bundle's model manifests, and my own project notes. Three sources that looked independent agreed, so every fresh attempt repeated the claim.

SGLang added GLM-OCR support as GlmOcrForConditionalGeneration and shipped it in v0.5.9 on 24 February 2026. The model card pins sglang>=0.5.10. Our own deployment planner document had listed GLM-OCR as running on vLLM, SGLang or Ollama the entire time.

What SGLang changed

In the same week, the RunPod handler moved to SGLang, and I proved the same thing locally on the DGX Spark box, replacing the transformers sidecar with an SGLang server.

python3 -m sglang.launch_server --model-path zai-org/GLM-OCR --served-model-name glm-ocr \
  --host 0.0.0.0 --port 5002 --attention-backend triton --disable-cuda-graph \
  --mem-fraction-static 0.30 --context-length 32768 --max-total-tokens 32768 \
  --chunked-prefill-size 8192 --trust-remote-code

Measured results:

Single-stream, SGLang is no faster than transformers at this size. The entire win is continuous batching: many pages in flight at once, weights read once and used for all of them. Sending one page at a time gets none of it.

The RunPod side ran on an SGLang base image with the client pipeline at max_workers=8, for $0.465 in OCR spend on the 155-page document.

An 18-document run on the DGX box with the SGLang server OCR'd 1,148 pages with zero errors and zero OOM kills.

Four things bit us on the way:

• The stock SGLang image ships openai without its distro dependency, so importing certain SGLang modules dies at startup. pip3 install distro is the whole fix, confirmed by stubbing out imports one at a time against the unmodified image until distro was the only thing missing.
• Handler dependencies must not go into SGLang's system Python: ours live in a virtual environment with access to system packages, and a build-time import smoke test in CI now catches environment breakage before it reaches live workers.
• CUDA version compatibility: the CUDA 12.9 runtime runs on RunPod's 12.8 drivers via CUDA minor-version compatibility. The newer CUDA 13.0 runtime needs driver 580 or newer, or torch silently falls back to CPU instead of failing.
• On the GPU used for the DGX Spark, --attention-backend triton --disable-cuda-graph is required. FlashAttention-3 crashes, FlashInfer produces plausible-looking garbage OCR text, and triton with cuda-graphs hits an illegal memory access.

The health endpoint returns a bare 200 with no JSON body, so the readiness probe accepts both the new response and the old server's response format. A legacy fallback environment variable is configured on the RunPod template but no longer used.

Running it on RunPod serverless

RunPod used to build the Dockerfile itself, hitting its roughly 15-minute build timeout on heavy CUDA images and leaving stale images deployed with nothing to promote. The parser repositories now build in GitHub Actions, push to a private registry under the Certant organisation, and RunPod pulls the resulting tag.

The PP-DocLayoutV3 model, about 500 MB, lives on a RunPod network volume rather than baked into the image: about 7.3 GB instead of roughly 8, no re-download on a handler update, and MinerU and Docling share the volume. If the model isn't there, the startup script logs a warning and downloads it on first request, one to two minutes, then persists for every cold start after.

Cold start is 60 to 90 seconds: five initialisation steps run before the handler accepts a job. Patch the torch distributed module (GLM-OCR imports it but never uses it; NCCL init fails on a single GPU), write a YAML config for the layout model, import the OCR client after those patches, patch two OCR client bugs, then construct the parser with layout enabled. The endpoint sits on a 16 to 24 GB GPU, workers at 0-3, 300-second idle timeout, scaling to zero between documents.

RunPod worker logs aren't public, only the console, so debug output has to travel inside the job response.

The handler fails fast now: the worker exits at startup if the text server doesn't come up, and returns an error per job when the health check is unhealthy. It used to silently emit layout-only output with almost no text. Truncated pages come back flagged with a finish reason, and the per-page token ceiling went from 4096 to 32768 to stop dense pages cutting off mid-clause.

The 20 MiB wall, and Ghostscript

RunPod's request body limit is about 20 MiB. We send PDFs as base64, inflating them by a third; ZIP does close to nothing on a PDF whose bulk is already-compressed images. A 39 MB scan from a customer's system failed "too large even after ZIP compression" no matter the transport.

The fix: shrink embedded images with Ghostscript before the ZIP step, only on the oversized branch (base64 over 19 MiB), in two tiers: 150 DPI, escalating to 100 DPI if still too big. Then ZIP regardless, then check again.

Measured on real files from the system corpus with Ghostscript 10.06.

To check legibility I built a synthetic 300 DPI scan and OCR'd it with tesseract per tier: 150 DPI gives a 97% word match, 100 DPI gives 93%, 72 DPI collapses to 45%. That's why the default stays at 150, with 100 as fallback and 72 never used. The corpus is born-digital, text stored as vectors that downsampling doesn't touch, so only figures shrink; I sampled 120 files and found zero pure scans.

Ghostscript exits 0 on a corrupt or truncated PDF, writing a 2.5 KB page-less stub; the exit code lies. We detect the failure from stdout and stderr markers plus a 50 KB output size floor, and fall back to the original bytes. The downsampler never raises.

This feature is behind a flag, default on, and has been verified locally. It hasn't reached the remote environments yet: those need the base image rebuilt to get the ghostscript package, and without that rebuild the feature silently does nothing, a bad failure mode not yet fixed.

The client was throttling itself

Months after the server was fixed, a separate DGX Spark exercise tried consolidating OCR onto the Gemma4 vision model already serving chat on the same vLLM instance, instead of the SGLang server above. OCR was still the slowest stage of document ingest there, a mean of 166 seconds per document (range 69 to 234), 18 seconds per page, GPU idle between pages, averaging 7 of 64 available batch slots filled.

The cause was in our own client, and it applies to either backend. The client pins the pipeline's maximum workers from an environment variable, set to 1: correct for the old single-stream server, where concurrent requests queued behind the lock and timed out, and wrong against a batching vision server, which sends one page when it could take dozens.

Raising it is environment-only, no code change. Four configurations, measured on a 50-document run:

Throughput rose about 2.5 times, from 87 to 222 documents an hour, and the GPU became the bottleneck instead of the client. Best latency and best throughput are different configurations; optimising a queue means the bottom row, even though each document takes longer in it.

During OCR the GPU runs about 2,052 image-prompt tokens a second on prefill against only about 260 generated tokens a second, since every page image expands into thousands of vision tokens. Vision OCR is prefill-bound and compute-bound; text chat is decode-bound and bandwidth-bound. Decode-side tuning for chat (speculative decoding, larger max-num-seqs) does nothing measurable for OCR throughput. The remaining lever without more GPU is rendering pages at lower resolution, fewer vision tokens to prefill, at some cost to OCR quality I haven't measured.

At 8 workers and 8 OCR processes the box was GPU-saturated, using about 96 GiB of its 121 GiB unified pool, close enough to the edge that I wouldn't push it further on that hardware.

What is still not fixed

The client retry path still posts a new job instead of resuming the existing one when polling times out. At SGLang speed almost nothing takes 9.5 minutes, so it rarely fires, but the fix is persisting the job ID and resuming polling across retries. It's on the list, not done.

The retry hardening lives on the MinerU client, not the GLM-OCR one: from 3 attempts with a 1-second base backoff (about 7 seconds total) to 5 attempts with a 10-second base capped at 120 seconds (about 310 seconds). The schedule is 10, 20, 40, 120, 120 with 10% jitter, retrying only connection errors, timeouts and 502/503/504; a 400 fails immediately.

The Ghostscript downsampling is local-only until the base image is rebuilt on the remote hosts.

And the local development default parser differs from the customer deployment default, a mismatch from when large PDFs couldn't get through the transport. That reason is gone; flipping the default was proposed and hasn't happened.

If you are doing this yourself

For anyone running a vision OCR model behind an API:

# Server: continuous batching, not a lock around one model
python3 -m sglang.launch_server --model-path zai-org/GLM-OCR --served-model-name glm-ocr \
  --port 5002 --attention-backend triton --disable-cuda-graph \
  --mem-fraction-static 0.30 --context-length 32768 --chunked-prefill-size 8192

# Client: stop sending one page at a time
max_workers=8      # pages in flight per document
worker_concurrency=8           # documents in flight
downsample_oversized=true   # 150 DPI, escalate to 100, never 72

Then measure batch occupancy on the server: wall-clock time on the client won't tell you the GPU is idle. Ours read 4 of 64 for months; nobody checked the dashboard.