patchwork
An LLM serving engine, written from scratch, that runs a Mixture-of-Experts model bigger than the GPU it lives on — weights oversubscribed into system RAM, active experts paged onto a consumer 16 GB card. Fused int4 kernels, speculative decoding with verified-identical output, and a perplexity gate that keeps quality honest — benchmarked openly against llama.cpp on the same machine.
expert offload · fused int4 kernels · speculative decoding · quality-gated benchmarks
| model vs card | 21 GB int4 MoE served on a 16 GB-VRAM GPU — 1.3× oversubscribed |
| decode | 30–37 tok/s warm via speculative decoding — output-identical to greedy, ~1.8× over plain |
| prefill | ~200–245 tok/s on a 1.4k-token prompt |
| quality | ±2% perplexity regression gate — enforced, not eyeballed |
| baseline | ~15× over the naive-offload ~2 tok/s bar · honest gap analysis vs llama.cpp |
