A small, hand-written Qwen3 inference engine for local hardware — C11 core, no frameworks in the hot path, and every GPU kernel gated against a scalar CPU oracle on real hardware before it counts as working.

Most engines try to run everything. Kipp deliberately runs one pinned model family and aims to run it well enough that you can read the whole engine in an afternoon.

How fast

Qwen3-4B on an Apple M5 Max, against llama.cpp on the same host, the same weights, and the same session — the honest comparison, losing rows included (BF16 prefill re-measured 2026-08-08 with panel-flash attention; Kipp quant prefill re-measured 2026-08-09 on the tensor path; decode rows from the 2026-07-28 session, unchanged since — see the ‡ note on the quant rows):

* affine4 (scale+bias, gs32) is Kipp's higher-quality Q4_1-class 4-bit scheme. Kipp also ships scale4_gs32 (v0.0.9) — scale-only, 18-byte group, now format-matched to Q4_0 — which decodes +4.5% over affine4 (same-session A/B, BF16 control valid), narrowing the llama.cpp Q4_0 decode gap to ~1.13×, at prefill parity and a full-vocab NMSE of 4.4e-3. See BENCHMARKS.

† The 12,800-token row is a cool-start median (idle-GPU cooldown before each run, both engines): the M5 laptop chassis cannot hold steady state under sustained long-context prefill. See BENCHMARKS.

‡ v0.0.8 moved the quantized projections to the Metal 4 tensor path — a 2.3× Kipp-internal prefill gain at 2,048 tokens (Q8_0 1,247 → 2,882, affine4 1,250 → 2,868, same-session vs the simdgroup kernel each replaces, MAD ≤0.3%). A stable llama.cpp quant head-to-head is not available: on this laptop chassis llama-bench's quantized prefill swings ~2× across thermal states (Q8_0 -p 2048 measured 2.1k–4.4k tok/s in one session, lacking a steady-state protocol), so the llama figures shown are the last clean-session reference, not a same-session comparison.

The BF16 prefill @12800 row is the v0.0.7 story. At 2,048 tokens prefill is projection-bound and v0.0.6's tensor-ops matmul already carried it, but as context grows attention takes over: at 12,800 tokens the simdgroup flash-attention kernel was ~85% of the prefill wall, running at only ~3 TFLOP/s effective. Panel-flash moves that inner work onto the Metal 4 tensor units — matmul2d over gathered KV panels instead of a simdgroup reduction — lifting it to 20–25 TFLOP/s: +37% over the kernel it replaces at 12,800 tokens (+26% at 6,400, +11% at 2,048), decode untouched, quantized schemes bit-identical. llama.cpp's own flash-attention is still simdgroup-only even on M5, which is why Kipp now leads it 1.54× at long context. v0.0.8 then moved the quantized projections onto the same tensor units — Q8_0 and affine4 prefill both jump 2.3× (dequantize each weight block into a threadgroup tile, then matmul2d), so BF16, Q8_0, and 4-bit now all run the neural accelerators. Every number traces to a committed file in bench/results/, and the measurement protocol explains why they are all same-session medians rather than best-of runs.

Also measured: 175× faster TTFT on a repeated 6,890-token prompt via cross-request prefix reuse, Q8_0 perplexity within 0.02% of BF16, and 4-bit decode at 128 tok/s (affine4) — or +4.5% on the new scale-only scale4_gs32 (v0.0.9), a 2.83 GiB artifact at 4.4e-3 logit NMSE.

What it is

• Readable by design — follow one token from embeddings to logits through the linear, model-specific reference pass in src/kipp.c.
• Native all the way down — C11 core, a narrow Objective-C Metal bridge, and isolated CUDA kernels for NVIDIA.
• No frameworks in the hot path — Python and shell are tooling only; PyTorch, JAX, and similar frameworks never enter inference.
• One model family, deeply — closer to whisper.cpp and antirez/ds4 than a generic runner such as llama.cpp or vLLM.
• Correctness first — every GPU backend must match the CPU oracle on real hardware before it is called working. Kernels that no gate exercises are treated as untested, not as working.
• mmap-backed loading — weights are mapped instead of eagerly copied into RAM wherever the backend permits.
• Registry, not runner — every supported checkpoint is pinned to one revision in src/kipp_checkpoints.h; anything else is rejected at load.

What is in v0.0.8

• Backends: scalar CPU oracle, Metal on Apple M5-class hardware, and CUDA validated on H100 for the current default checkpoints (14B/32B on A100).
• Prefill on the tensor units: BF16, Q8_0, and affine4 projections all run the Metal 4 matmul2d path (quant kernels dequantize each weight block into a threadgroup tile first — v0.0.8, Q8_0/affine4 prefill 2.3×), plus panel-flash attention over gathered KV panels for long context (v0.0.7, +37% at 12,800 tokens). Decode still runs the simdgroup/matvec kernels.
• Weights: BF16, near-lossless Q8_0, and Q4-class affine 4-bit projections — all three on the tensor-ops matmul path for prefill.
• KV cache: paged 32-position blocks, cross-request prefix sharing, and an opt-in Q8_0 KV cache (~1.9× smaller; a memory feature, not a speed one).
• Serving: OpenAI Completions and Chat Completions, SSE, 32-way batched decode, full sampling, Prometheus metrics.
• CLI: greedy or sampled generation, multi-turn chat, perplexity scoring, prompt-lookup and draft-model speculation.

Quickstart

Prerequisites: a C11 compiler, uv for the Python tooling, and Xcode command-line tools for the Metal build. The tooling environment includes PyTorch, because weight conversion reads the original safetensors — inference itself never touches it.

git clone https://github.com/Polished-Snow/Kipp.git
cd Kipp
make quickstart   # downloads + converts Qwen3-0.6B (~1.5 GB), then generates

That is clone to first token in one command. It uses the smallest registered checkpoint so the first run costs a couple of gigabytes rather than the ~16 GB the 4B default needs, and picks Metal on macOS or the CPU path elsewhere.

Or install a prebuilt binary (macOS arm64, Linux x86_64)

Releases ship tarballs with the CLI and server binaries, so inference needs no toolchain at all — the Python tooling is only required to convert weights:

curl -fsSLO https://github.com/Polished-Snow/Kipp/releases/latest/download/kipp-macos-arm64.tar.gz
curl -fsSLO https://github.com/Polished-Snow/Kipp/releases/latest/download/SHA256SUMS
shasum -a 256 --check --ignore-missing SHA256SUMS
tar -xzf kipp-macos-arm64.tar.gz   # kipp, kipp-metal, kipp-server, kipp-server-metal

On Linux, substitute kipp-linux-x86_64.tar.gz (CPU binaries; build the CUDA backend from source with make cuda-generic). The macOS binaries are ad-hoc signed and need macOS 14 or newer. The curl path above just works; if you download through a browser instead, clear Gatekeeper's quarantine flag first with xattr -dr com.apple.quarantine kipp-macos-arm64.

Build the individual binaries directly if you prefer:

make cpu          # session-backed CPU CLI
make metal        # Apple Metal CLI
make server-metal # completions/chat/metrics server on Metal
make test         # hermetic unit tests; no model download (needs uv)

Converting a larger checkpoint is explicit, because it downloads and rewrites multiple gigabytes. CHECKPOINT selects any id from the registry:

make CHECKPOINT=qwen3-4b-base convert   # ~8 GB download, ~7.5 GiB GGUF

Then run a decode — greedy by default, or sampled with a seed:

build/kipp-metal --backend metal \
  --model models/qwen3-4b-base/kipp-qwen3-4b-base-bf16.gguf \
  --prompt "Hi 世界" --decode 8

build/kipp-metal --backend metal \
  --model models/qwen3-4b-base/kipp-qwen3-4b-base-bf16.gguf \
  --prompt "Once upon a time" --decode 64 \
  --temperature 0.8 --top-p 0.95 --seed 7

The real-model correctness gates need a converted checkpoint and its pinned vectors:

make test-cpu-model                          # complete CPU model suite
make test-metal                              # complete Metal suite
make test-server
make CHECKPOINT=qwen3-0.6b-base test-model   # any registry checkpoint

Or serve the OpenAI Completions subset on loopback:

build/kipp-server-metal --backend metal --port 8080 \
  --model models/qwen3-4b-base/kipp-qwen3-4b-base-bf16.gguf

curl -s http://127.0.0.1:8080/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"prompt": "The capital of France is", "max_tokens": 8, "temperature": 0}'

The Metal binary embeds standalone MSL and compiles it through Metal's runtime API. Backend selection is explicit and never falls back to CPU.

More runnable CLI, chat, streaming, and metrics requests live in examples/.

Repository tour

src/kipp.c              strict GGUF loading, tokenizer, sampling, CPU oracle
src/kipp.h              public model, tokenizer, sampling, KV-session C API
src/kipp_backend.h      fixed internal backend boundary + runtime config
src/kipp_checkpoints.h  the supported-checkpoint registry (the growth point)
src/kipp_chat.c         native Qwen3 ChatML rendering
src/kipp_spec.c         prompt-lookup speculative drafting
src/kipp_kv_pool.c      production KV block allocation and prefix sharing
src/kipp_cli.c          greedy/sampled decode CLI with timing metrics
src/kipp_server.c       completions/chat server: batching, SSE, metrics
src/metal/kipp_metal.m  the only Objective-C in the project (Metal bridge)
metal/kipp_kernels.metal standalone, build-time-embedded MSL kernels
src/cuda/kipp_cuda.cu   CUDA bridge, isolated from every other backend
cuda/kipp_kernels.cu    CUDA kernels validated on NVIDIA A100/H100
tests/                  native + server tests, generated reference vectors
tools/                  Python/shell tooling; checkpoints.py mirrors the registry
examples/               copy-pasteable CLI and HTTP requests
docs/                   design docs + the documentation site
AGENT.md                contribution constraints, for humans and AI agents alike

Roadmap

Full details are in the roadmap. Deferred work includes ROCm/HIP, extra model families, and arbitrary-GGUF compatibility.

Documentation

The full documentation lives at polished-snow.github.io/Kipp:

• Architecture — design constraints, backends, and correctness boundaries
• Roadmap — correctness-gated implementation phases
• Model support — what supporting a checkpoint means
• Benchmarks — measurement policy and reference results
• Reproducing — exact build, gate, and benchmark commands
• Releasing — release checklist and hardware policy
• Research — reference repositories and design notes
• Contributing — ground rules for human and AI contributors

Contributing

Read AGENT.md first. It is the source of truth for every contributor: keep a C11 core, isolate backends, preserve mmap-backed loading, test on real hardware before claiming success, and never vendor code from the reference repositories.

Security

Kipp's server is a hand-written C11 HTTP server with its own HTTP and JSON parsers. It binds loopback only and has no authentication or TLS by design — see SECURITY.md for the threat model and for how to report a vulnerability privately.

Citing

If you use Kipp or its measurements in academic work, see CITATION.cff (GitHub renders a "Cite this repository" button from it).

Enjoying Kipp?

Kipp is built in the open by Polished Snow. If you find it useful, star the repository and follow project updates:

License

Kipp is open source under the MIT License.

_{Built by Polished Snow · X · LinkedIn}