Kipp Benchmarks

Kipp has a scalar CPU correctness oracle plus correctness-gated Metal and CUDA backends. Results must not mix unsupported checkpoints, weight schemes, or backend variants.

Required report fields

• Kipp commit and clean/dirty working-tree state
• Model identifier, revision, parameter count, and quantization
• Backend, build target, compiler, and relevant build flags
• Hardware model, memory capacity, and operating-system/driver versions
• Prompt length, generated-token count, batch size, and concurrency
• Warm-up procedure and number of measured runs
• Peak memory, prefill throughput, generation throughput, and latency
• Sampling settings and any cache or scheduler configuration

Benchmark policy

Report medians and a dispersion measure rather than only the best run. Keep prefill and token-generation measurements separate. Comparisons with other engines must use equivalent model weights, context, sampling settings, and hardware, and must link the exact commands used.

tools/bench.py is the canonical subprocess harness. It performs discarded warm-up runs, captures the CLI's separate prefill and decode timers, samples peak process RSS with /usr/bin/time -l, and reports the median, median absolute deviation, minimum, maximum, and each raw run as JSON. bench/spec_bench.py, bench/server_bench.py, bench/prefix_bench.py, bench/load_bench.py, and bench/ppl_bench.py cover speculation, server batching, cross-request prefix reuse, open-loop serving load, and quantization quality; all record the full engine/hardware/model provenance block via bench/_provenance.py.

All numbers are sustained steady-state measurements: Apple-silicon GPU clocks are demand-scaled, so benches run on an otherwise idle machine, back-to-back in one session after a multi-minute GPU warm-up, and a file is trusted only when its recorded dispersion is tight (see bench/README.md, "Measurement protocol").

Apple M5 Max (v0.0.3 measurement campaign, 2026-07-22)

The development machine changed from a base M5 (10-core GPU, 24 GiB) to an M5 Max (40-core GPU, 128 GB); throughput is roughly 4× the sections below, which are retained for the base-M5 configuration. An earlier 2026-07-21 campaign was measured on a build whose Metal matrix kernels had silently failed to compile (a reserved MSL keyword; every correctness gate still passed) — those numbers are superseded, and the harness now refuses to record results from a fallback build. Current reference numbers (Qwen3-4B, Metal, greedy; every value traces to a committed bench/results/*.json):

• Decode tok/s (64 tokens, median of 5, 2026-07-30 session): BF16 59.7, Q8_0 94.4, affine4 127.9. Session-to-session drift on this machine is a few percent in either direction (previous campaigns read 61.2 / 97.4 / 128.8 and 59.8 / 94.3 / 127.9); the controlled back-to-back A/Bs measure decode unchanged through every kernel change — including the tensor path, which decode never takes — and each campaign is re-run in one session so its numbers stay comparable with each other.
• Wikitext-2 perplexity (full test set, 2,048-token windows): BF16 7.731, Q8_0 7.733 (+0.02%), affine4 8.171 (+5.7%) — Q8_0 is effectively lossless; affine4 is Q4-class.
• Prefill tok/s (348-token / 2,048-token prompt, 2026-07-30): BF16 2500 / 3679 on the Metal 4 tensor path ("The tensor-ops path" below; 2.4–2.8× the 1034 / 1312 the simdgroup kernels reach, which stay the shipped path on pre-M5 devices), Q8_0 1065 / 1213, affine4 1080 / 1214 on the simdgroup kernels (quantized projections gained 10–12% at 2,048 tokens from the v0.0.5 shared-dequant rotation; their tensor variants are future work). The quantized kernels dequantize each 32-weight block once per 64-token tile.
• Context scaling (Q8_0): decode 98.4 → 44.7 tok/s from a 3- to a 12,800-token prompt after the split-K long-context path (ctx-*.json).
• Model-size sweep (BF16 decode): 0.6B 269, 4B 60.7, 8B 33.5 tok/s — bandwidth-bound decode scales inversely with streamed weight bytes across the family (8B Q8_0: 57.8 tok/s; 8B prefill 316/331).
• Server aggregate (Q8_0, sampled, 48 tokens/sequence): n = 1/2/4/8 choices → 76.9/96.8/89.8/80.1 tok/s; 1/2/4/8 concurrent connections → 55.9/61.3/78.5/80.8. Aggregate flattens as per-step scheduling gaps let the demand-scaled GPU clocks sag — a consumer-SoC serving effect.
• Cross-request prefix reuse: a 6,890-token prompt sent twice adopts 6,880 tokens on the second request; prefill drops 30.4 s → 174 ms (175× TTFT).
• Speculation (Q8_0, 256-token decode, paired-baseline A/B): adaptive-gated 2.27× on repetitive text, above parity on code (1.24×), and a 0.84× floor elsewhere (ungated: 2.10× / down to 0.27×). Gains are smaller than earlier drafts because the sampling fast path made the plain-decode baseline itself faster.
• llama.cpp A/B: the 2026-07-22 comparison (decode 35.0/56.5, prefill 2,174) proved irreproducible and is superseded — see "The llama.cpp comparison was stale" below for the corrected, much less flattering numbers and what caused the discrepancy. llamacpp-qwen3-4b.json now holds the 2026-07-28 measurements.
• CUDA revalidation (cuda-h100-gates.json): all four default checkpoints pass --model and --phase4-cuda on an ephemeral NVIDIA H100 80GB (worst observed NMSE 5.9e-7 against the CPU oracle).
• Quantized KV cache (--kv-quant q8_0, opt-in; bench/results/qkv/): the KV cache is ~1.9× smaller than BF16 (136 vs 256 bytes per head-row), which is the point — it extends context and concurrent-session count under Apple's single-buffer cap (the server's 0.6B pool drops from ~3,584 to 1,904 MiB). It is a memory feature, not a speed one: in a same-session Metal 4B decode A/B the per-value dequantization slightly outweighs the reduced byte traffic, so Q8_0 KV decode runs at 0.85–0.97× of BF16 across 512–16,384 tokens rather than faster. Quality is near-lossless (NMSE 1.4e-5 CPU / 2.6e-5 Metal vs BF16 KV, arg max identical). A faster decode dequant and an MMA quantized-prefill path are follow-ups.

Prefill round shape (2026-07-26)

Metal prefill was the one axis where Kipp lost to llama.cpp: 504 tok/s against 2,174 at a matched 2,048-token prompt, measured back to back on the same host, weights, and session. Two independent limits caused it, and the arithmetic matters because the intuitive reading is wrong.

Projection weight traffic is set by the in-kernel token tile, not by the round size:

weight bytes per prefill = ceil(total_tokens / matmul_token_tile) x projection_bytes

The dispatch grid is (row groups, token groups) and each threadgroup streams its own weight rows over the full shared dimension, so threadgroups that would share weight rows are never co-resident. Widening the round therefore moves no weight bytes; only the tile does. What the round width fixes is occupancy: the K and V projections have 1,024 rows, so at a 32-token round they dispatched 16 threadgroups onto a 40-core GPU, and the engine sustained only ~255 GB/s, roughly half of what the device can hold.

Both levers were measured separately, one session, idle machine, five runs each (Qwen3-4B BF16, 2,048-token prompt, M5 Max):

Three results worth keeping:

• The round is the dominant lever (+142% with zero byte reduction); the tile adds +7% on top. Weight bandwidth was therefore not the binding constraint, and the widely-assumed fix had the smaller effect.
• Raising the tile first is a 29% regression, because at a 32-token round a 32-token tile halves the grid. Ordering is not cosmetic here.
• Naively widening the round also costs 11% of decode, because one constant sized the split-K partial buffer as well. Separating the limits (see docs/ARCHITECTURE.md, "Activations and logits") recovers that and lands 40% higher prefill than the naive change.

The round size itself was swept rather than guessed (same session, 2,048-token prompt): 256 -> 1191.4, 512 -> 1273.8, 1024 -> 1340.3, 2048 -> 1292.0 tok/s. Prefill peaks near 1024, but decode falls from 56.25 to 54.78 there and to 53.57 at 2048, so 512 is the shipped default: it takes the bulk of the prefill win while leaving decode -- the larger competitive margin -- untouched.

A third limit turned up only when the quantized schemes were measured, and it nearly shipped a large regression. The quantized matmul kernels already stage dequantized weights in ~16.9 KiB of threadgroup memory, so widening their token tile costs far more in occupancy than it saves in weight traffic. Isolated on Q8_0 at a 2,048-token prompt:

The wider round is worth 2.5x for Q8_0 as well, but the wider tile is -88% for it while being +7% for BF16. Tile width is therefore a property of each kernel rather than of the projection layer, and the two quantized kernels keep the narrower tile. Correctness alone would never have caught this: all three schemes passed every gate at either tile.

Back-to-back A/B of the shipped configuration against the previous one, all three weight schemes, same session, after a sustained warm-up (Qwen3-4B, 2,048-token prompt, five runs each):

Decode is unchanged for every scheme. An earlier run of the same comparison measured BF16 prefill at 407 tok/s with a median absolute deviation of 10.5 and affine4 decode 20% low; those readings were thermally polluted and are not used. The rule this reinforces is in bench/README.md: only a back-to-back same-session A/B is meaningful here, because the narrow-round configuration spent proportionally more time in per-round fixed cost and therefore tracked GPU clock state more closely than the new one does.

Every step is bit-exact: the full-logit fingerprint printed by --prefill-metal is unchanged through the refactor, the wider round, and the larger tile, verified by pinning the round back to 32 to separate the refactor from the speedup.

Two further changes were implemented, measured, and reverted rather than shipped, because the repository's standard is that a change earns its place:

• Four query tiles per attention threadgroup, so the simdgroups sharing a KV head walk the same K/V tiles together and three of four reads hit cache. Flat at 2,048 tokens (+0.3%) and -7.3% at 12,800 (532.3 vs 574.0). One layer's KV at these lengths is a few megabytes and was already cache-resident, so the sharing bought nothing while the larger threadgroup allocation (8.3 -> 25.3 KiB) cut resident threadgroups per core.
• A 64-token matmul tile, which would halve weight traffic again: 133.8 tok/s, a 90% regression, from register spilling once the accumulator set reaches 32 fragments.

Remaining prefill cost is therefore neither weight traffic nor attention locality. (Superseded 2026-07-28: the conclusion that "closing the rest needs a threadgroup-staged, K-blocked matmul" was tested and falsified — see the next section. The 2,174 tok/s reference figure it reasoned from was itself stale.)

Provenance note: the shipped-configuration figures in this section are backed by the committed bench/results/4b-*.json records from the release re-run; the intermediate sweep values were measured on the working tree that carried the change under test, per the policy at the top of this file.

What the matmuls are actually bound by, and the llama.cpp comparison was stale (2026-07-28)

Two findings from one measurement day, both of which correct this file.

The llama.cpp comparison was stale. The 2026-07-22 A/B recorded llama.cpp at 2,174 tok/s prefill / 35.0 decode (BF16). Re-running the exact committed command, binary, model, and OS on 2026-07-28 gives ~3,800–4,100 tok/s prefill and ~60–63 decode — roughly 2× on both axes — while Kipp reproduces its own committed numbers to within noise in the same session. The old llama.cpp session was evidently measured in a degraded GPU-clock state (the uniform ~2× across unrelated workloads is the signature; the exact cause is not reconstructable). The head-to-head below replaces it, and the lesson is recorded here precisely because this repository's claims are only as good as their worst measurement: cross-engine numbers are now re-validated in the same session as the Kipp numbers they are compared against.

Where the missing prefill time actually goes. A new isolated instrument (build/kipp_test_metal --mm-bench-metal) runs the live projection-matmul pipelines on every Qwen3-4B shape at a 512-token round, cycling through eight distinct weight buffers (so the system-level cache cannot serve one resident copy) versus reusing one. The result is unambiguous: rotating equals reusing on every shape, effective weight fetch is 17–24 GB/s against ~455 GB/s of demonstrated sustained bandwidth, and the kernels run at 8.7–12.3 TFLOP/s. The matmuls are compute/issue-bound, not traffic-bound, and summing the per-shape times attributes ~82% of 2,048-token prefill wall clock to them. Weight-traffic reduction — the motivation this file previously assigned to a future "K-blocked matmul" — cannot help; that plan is retired.

Consistent with that, explicitly staging the activation tile through threadgroup memory (removing the transposed device-memory gathers from the hot loop) was implemented, measured at −7 to −10% on every shape, and reverted: the fourth time explicit staging has lost to this GPU's implicit cache hierarchy. The simdgroup-matrix BF16 kernel sits at ~80% of the achievable ALU-issue ceiling and stays as is.

What did work: one shared dequant block per quantized threadgroup. The Q8_0/affine4 kernels staged four private 32-row dequantized blocks (16.9 KiB of threadgroup memory) — one per simdgroup, each serving 16 tokens. They now share a single cooperatively-dequantized 32-row block (4.2 KiB) across a 32-row × 64-token threadgroup tile: weight bytes are dequantized once per 64 tokens instead of once per 16, and the 4× smaller staging keeps several threadgroups resident per core. Accumulators, dequant expressions, and FP32 accumulation order are unchanged, so the kernels are bit-exact — the --prefill-metal fingerprints and the --pooled-metal tripwire did not move a digit. Same-session interleaved A/Bs: Q8_0 1137.7 → 1206.0 tok/s, affine4 1139.4 → 1213.9 (two passes each, decode dead flat); against the committed v0.0.4 records the release re-run reads +10.2% (Q8_0) and +11.7% (affine4) at 2,048 tokens. Widening the quant token slice on top of it (16 → 32 tokens per simdgroup) collapses to ~1 TFLOP/s from register spilling — the third confirmation that the 2-fragment slice is that kernel's register ceiling — and was reverted.

The corrected head-to-head (same session, same host and weights, llama.cpp 178a6c449 with Metal, llama-bench -p 2048 -n 256 -r 5; every figure traces to llamacpp-qwen3-4b.json and 4b-*.json, 2026-07-28):

llama.cpp's decode figures are from isolated decode-only runs (-p 0 -n 256, cooled machine): its interleaved -p 2048 -n 256 run heats the GPU enough to depress its own tg readings to 47–69 tok/s, and quoting those would flatter Kipp. The honest decode summary is therefore: llama.cpp is ~3% ahead on the matched schemes (63.1/100.5 vs 61.2/97.4) and clearly ahead on 4-bit decode (scale-only Q4_0 dequantizes more cheaply than scale+bias affine4; the schemes also differ in quality — affine4's perplexity cost is documented above, Q4_0's on the llama.cpp stack was not measured here). The prior "decode 1.7× in Kipp's favor" claim derived entirely from the stale llama.cpp session and is withdrawn.

The decisive prefill variable is the instruction class. On M5, llama.cpp routes GEMM through the Metal 4 tensor API (mpp::tensor_ops, the neural accelerators) by default — it enables that path only on M5/M6/A19/A20-class devices, and its own source notes it is a wash or a loss on M4 and earlier. That path is worth ~2.3–3× on prefill GEMM and nothing on decode (matvec). Forced onto the same simdgroup-matrix instruction class Kipp uses (GGML_METAL_TENSOR_DISABLE=1), llama.cpp's prefill drops to 1,261 / 1,022 / 608 — Kipp leads its own instruction class on every scheme (1.04× / 1.19× / 2.0×).

What follows from this: matching llama.cpp's default BF16/Q8_0 prefill on M5 requires an mpp::tensor_ops matmul path, not further simdgroup tuning — that is the next roadmap item, behind the same runtime-probe-and-fall-back ladder and oracle gates as the simdgroup kernels. (Delivered in v0.0.6 — next section.)

The tensor-ops path: BF16 prefill 2.80×, and where llama.cpp's design was wrong for this engine (2026-07-30)

v0.0.6 ships the mpp::tensor_ops (M5 neural-accelerator) matmul path for BF16 layer projections, and it closes the instruction-class gap the previous section documented:

The committed records (4b-bf16-*.json, 2026-07-30 session) read 3,679 / 2,500 at 2,048 / 348 tokens. That is parity with llama.cpp's own tensor path (its cool-session 3,788 from the 2026-07-28 record; its quant rows are unchanged — Kipp's quantized schemes still run the simdgroup kernels, bit-frozen by fingerprint in both env states). The 12,800-token ratio falling to 1.40× says what the next campaign is: with projections ~5× faster, long-context prefill attention is now the dominant cost.

Three design findings, two of them against the llama.cpp blueprint this port started from:

• Do not stage the weight tile. llama.cpp's tensor kernel stages a 64×32 weight tile through threadgroup memory (it must — its quant types dequantize there). Measured on Kipp's shapes, that staged design reaches only 0.96–2.9× the simdgroup kernels, while feeding both operands as plain device tensors reaches 4.9–6.2× (45–65 TFLOP/s), rotation-immune at 111–128 GB/s of weight streaming. This is the sixth consecutive loss for explicit threadgroup staging on this GPU; for pure BF16 the tensor units stream device memory faster than any hand-staging scheme.
• Consume FP32 activations directly. matmul2d accepts float sources, so the tensor path reads the round's FP32 working buffer instead of the staged-BF16 copy the simdgroup kernel needs. Float activations cost ~12–18% per GEMM, but they delete the entire kipp_bf16_stage pass (four dispatches per layer, ~90 ms per 2,048-token prefill) plus one activation rounding step — a net win end-to-end and strictly better numerics: pooled-vs-oracle NMSE 5.12e-07 against the simdgroup path's 2.09e-06, matrix-vs-vector 2.78e-06 against 1.52e-05.
• matmul2d is bitwise stable where it matters. Its output measured bitwise invariant to the dispatch token count (26/32/96/512) and bitwise deterministic across repeated runs. That is what allows a wholesale kernel-class swap at the same ≥8-token threshold: the paged/pooled bitwise placement gates keep comparing a single class — and now cover the tensor path on M5-class hardware.

Because CI runners cannot run (or even compile) this path, it carries more tripwires than any other kernel class: a runtime compile-and-probe ladder behind a device allow-list, KIPP_METAL_REQUIRE_TENSOR=1 turning silent fallback into a load failure (it caught a real kernel bug during development), a bench harness that refuses degraded-build numbers, an affirmative "tensor path disabled (reason)" line asserted by CI, tensor slots in the shader geometry probe, and an exact-integer operator test at tolerance 0.0 on a ragged 72×133 shape — integer FP32 accumulation is order-independent, so a wrong matmul2d descriptor cannot hide inside a tolerance. Per-kernel-class tripwires (M5 Max): tensor --prefill-metal fingerprint 794f1799c6a7326a / pooled NMSE 5.12341885e-07; the frozen simdgroup values (49e2ada96bce2804 / 2.08738076e-06) remain verifiable with KIPP_METAL_TENSOR_DISABLE=1, and both must be checked on their own path.

Optimized Metal kernels on Apple M5 (v0.0.1)

Measured on 2026-07-13 with Kipp v0.0.1's batched Metal path: one serial compute encoder per command buffer, up to 32 prefill tokens per batch, simdgroup-matrix BF16 projections for batched prefill (16-token tiles, FP32 accumulation, with a vector-kernel fallback on devices without bfloat simdgroup matrices), a simdgroup matvec for single-token decode, and split-K streaming online-softmax attention (eight partial softmaxes per head merged in threadgroup memory). Same hardware, model artifact, build flags, and harness as the baseline below; one discarded warm-up and five measured subprocesses per configuration.

Commands:

python3 tools/bench.py --warmup 1 --runs 5 --decode 8
python3 tools/bench.py --warmup 1 --runs 5 --decode 32 --prompt "<265-token prompt>"

Results (median, with median absolute deviation):

• 3-token prompt, greedy decode of 8: prefill 9.375 tokens/s (MAD 0.939, range 8.436–12.557); decode 13.776 tokens/s (MAD 0.063, range 13.259–13.985)
• 265-token prompt, greedy decode of 32: prefill 147.769 tokens/s (MAD 2.219, range 123.937–149.988); decode 13.072 tokens/s (MAD 0.802, range 12.133–13.969)
• Single spot check at a 1,981-token prompt: prefill 136.6 tokens/s and decode 13.0 tokens/s (decode was 11.0 tokens/s before split-K attention — long contexts are where the partitioned kernel pays).
• Peak process RSS: ~40 MB median in both configurations (see the RSS caveat below; mapped GPU-touched weights are not charged to the process)

Short-prompt prefill is dominated by fixed per-evaluation cost; the 265-token configuration reflects the batched matrix-kernel prefill path. Decode remains memory-bandwidth-bound: at ~13.8 tokens/s the engine streams roughly 115 GB/s of BF16 weights through the M5. Prefill at this revision was limited by per-layer KV re-reads in attention rather than by the projection matmuls. That attribution is specific to the 32-token round this section measured and no longer describes the engine -- see "Prefill round shape" below for the current model and the measurement that corrected it.

Batched decoding through the server (48 sampled tokens per sequence, temperature 0.8, single measurements). Multi-choice requests (n) and independent concurrent clients batch the same way — each step reads the weights once for every active sequence:

• one request, n = 1/2/4/8: 12.7 / 18.9 / 29.7 / 41.4 tokens/s aggregate
• concurrent clients (distinct prompts and seeds), 1/2/4/8 connections: 12.7 / 20.3 / 29.1 / 39.7 tokens/s aggregate

Aggregate throughput approaches compute limits instead of scaling the bandwidth cost with the number of sequences.

Phase 3 Apple M5 baseline (superseded)

Measured on 2026-07-10 with the original one-token-per-command-buffer kernels, on:

• Apple M5, 10-core GPU, 24 GiB unified memory
• macOS 26.4, Apple clang 21.0.0
• dirty development tree at 0eac2059cfa138c8e56ce2664c9213c9e4340261
• Qwen/Qwen3-4B-Base revision 906bfd4b4dc7f14ee4320094d8b41684abff8539
• 8,049,127,680-byte BF16 Kipp GGUF-v3 artifact
• -std=c11 -O2 -Wall -Wextra -Wpedantic -Werror
• prompt Hi 世界 (3 tokens), greedy decode of 8 tokens, batch 1, concurrency 1
• one discarded warm-up and five measured subprocesses

Command:

python3 tools/bench.py --warmup 1 --runs 5 --decode 8

Results:

• Prefill: 6.405 tokens/s median, 0.027 tokens/s median absolute deviation (range 6.379–6.719)
• Decode: 8.193 tokens/s median, 0.004 tokens/s median absolute deviation (range 8.189–8.207)
• Peak process RSS: 47,775,744 bytes median (45.6 MiB), 32,768-byte median absolute deviation

Peak process RSS is not total unified-memory residency. Metal accesses the 7.50 GiB model through a shared no-copy mmap, and macOS does not charge those GPU-touched mapped pages to the CLI's reported peak RSS in the same way as ordinary process allocations. The artifact size and cache layout therefore remain necessary context; the RSS number must not be presented as the total memory needed to run the model.

This was a readable fixed-shape baseline, not an optimized performance claim. At that revision the implementation intentionally excluded fusion, quantization, batching, and private weight copies; later results above supersede those constraints.