Skip to content
P300 Papers

Locus: Owner-Drained Chunk Mailboxes for KV-Block Recycling in CPU LLM Inference

authors
Victor Bona
status
independently published
published
cite key
bona2026locus

cat abstract

A technical white paper on Locus, a domain memory pool for CPU LLM serving whose remote-free path is a per-worker lock-free chunk mailbox: a finished request's KV blocks are returned as one atomic push, and the pool owner drains every mailbox off the allocation hot path, so freeing never contends on a shared queue and the design carries zero tuning parameters. Developed under a falsification-first methodology, it is evaluated on LOCUS-EVAL v1, a frozen four-workload suite of deterministic serving-shaped KV traces run against jemalloc, mimalloc, and system malloc on Apple Silicon, where the mailbox ranks first on every workload. A self-critical touch-parity audit corrects the headline: at a one-byte-per-block parity the advantage is 1.6x to 2.7x over mimalloc on the three trace workloads and compresses to about 1.15x over system malloc once full KV writes dominate, with a transient burst-cancellation footprint of up to 1.5x the theoretical peak. The scope is deliberately narrow (a single macOS host, synthetic traces, an allocator-level measurement), with NUMA locality and end-to-end serving integration stated as future work.

ls index-terms/

RustLLM InferenceKV CacheMemory PoolAllocatorsBenchmarks

cite --format=bibtex

@misc{bona2026locus,
  author = {Victor Bona},
  title = {Locus: Owner-Drained Chunk Mailboxes for KV-Block Recycling in CPU LLM Inference},
  year = {2026},
  month = {jul},
  url = {https://blog.victorbona.dev/papers/locus-whitepaper.pdf},
  note = {Independently published}
}

Questions about this paper, or working on something close to it?

victor-bona · terminal

type help for commands · esc to close