CachePin

Where do your prompts split?

Two agents load the same tools in a different order. The second request can lose a reusable prefix. CachePin finds the first change and shows what a consistent order would recover.

Compare two prompts ↓ Run the CLI example ↗

Prefix comparison

Choose a pair or edit the JSON. This browser example compares the serialized model, tools and system fields in UTF-8 before and after normalization. It omits messages. Use the CLI for a full request diagnosis and token estimates.

First difference · before normalization
First difference · after normalization

Compare requests in your MCP client

Open the CachePin bundle in a client that supports MCPB extensions on macOS or Linux (x86-64 or ARM64). Call diagnose_diff with two serialized requests, prev_json and curr_json. The tool runs locally and returns the byte offset where the requests first differ.

Download CachePin 0.2.0 (.mcpb) Package details and SHA-256

Estimate a cache bill

Change the workload and compare two scenarios. The model assumes the normalized workload pays for one prefix write. Actual hits depend on your provider, timing and cache capacity.

Per-Run Cost Reduction
15.4× cheaper
Saved per 100 runs
$3,780
Baseline workload$40.50
One shared prefix write$2.65
Estimated Cache Hit Rate
2.5% → 97.6%
Cold-Start Cache Writes
32 → 1 writes

Assumes 512 fresh uncached tokens/turn per agent. Hit-rate penalties for tool order, timestamps and document order are fixed assumptions in this simulator.

Recorded measurements

42.56 GB/s
AVX2 prefix diff (50.3× vs Node 22)
[LIVE-CPU]
84.2 µs
18,535-byte input, p50 (22.3× Py · 3.45× Node)
[LIVE-CPU]
32 → 1
Cold writes in a 32-request loopback burst
[LIVE-CPU]
15.39×
Synthetic 32-agent cost ratio
[SIM-TRACE]
1.16 µs
Local route decision, 8 backends (p50)
[LIVE-CPU]

8-GPU vLLM Eviction-Regime Routing [SIM-TRACE]

48 shared prefixes across 8 GPUs (20,480 KV tokens/GPU, Zipf-1.25 multi-agent bursts, 160 ms decode residency). Single-home 2×-divert regresses P99 TTFT by +9.8% vs Least-Loaded; cachepin cuts P99 by 37.0% and Mean TTFT by 2.42× at ~1.15 µs/route.

Routing Policy Fleet KV Hit P50 TTFT P99 TTFT Mean TTFT Replications Status
Round-Robin51.1%136.6 ms443.0 ms137.3 ms366KV thrashing
Least-Loaded52.6%106.0 ms376.0 ms119.1 ms355Cache-blind
Consistent-Hash96.1%16.8 ms351.6 ms72.9 ms0Hot-GPU queue
SingleHome-2xDivert (Ranvier-style)66.9%14.4 ms413.0 ms89.7 ms180+9.8% P99 cliff
cachepin Adaptive-Replication87.5%12.8 ms260.0 ms37.1 ms66-37.0% P99 · 1.15 µs

Reproducible Cross-Language Controls [LIVE-CPU]

Measured on identical 18,535-byte multi-agent payloads (benchmarks/compare_controls.py → benchmarks/results.json). Live-hydrated directly from results.json.

Runtime Compile P50 Compile P99 LCP Diff GB/s CLI Cold Start
cachepin (Rust + SIMD)————
cachepin Python FFI (libcachepin.so)————
Node.js 22 (V8 JIT control)————
Python 3.12 (stdlib control)————