Local LLM inference · drop-in prefix caching for llama.cpp
Reuse the KV cache.
Never regenerate
the same context twice.
An OpenAI-compatible daemon in front of llama-server. Agents that share a long system prompt compute it once, then replay it from memory.
- 97.1% cache hit rate on real agent workloads
- Zero patches to llama-server, to your client, to your model
- Every backend llama-server ships
The toki launcher: the agents it can start, the model in use, and the settings. Use the arrow keys to move, Enter to open, S for the settings, and the right arrow for the model list. Nothing here is connected to a model.
- time to first token
- +96.2%
- prefill throughput
- +1282.5%
- cache hit rate
- 97.1%
- model patches
- 0
Best measured result: Qwen3.5-122B-A10B, summarization, 65k context, AMD Ryzen AI MAX+ 395. The full table and the method.
The prompt arrives. The prefix matches.
The prefill disappears.
The mechanism
Every request is compared against the KV blocks already in memory.
Token by token. The longest identical prefix is replayed instead of recomputed. There is no second model, no separate cache server, no client change.
- match
- exact prefix, byte for byte
- cache
- one KV pool, shared by every client
- client
- OpenAI-compatible: point at the daemon and keep going
The web UI
The one screen we do not fake.
A single-page chat over the same daemon, with the cache made visible: every reply says whether it was cached, how many tokens it read, and how long the first token took.
Raw mode, cached mode, and a side-by-side comparison.
- mode cached
- The session is running against the shared KV cache.
- 59 tok in
- The context the reply reports reading.
- 367 ms
- Time to the first token of that reply.
- 2503 ms
- The whole turn, first token to last.
Swipe the window to see the rest of it.
The launcher
Every agent you run, in one window.
The full-screen terminal launcher ships with the package: the agents it can start, the model in use, the settings that matter, and a restart warning when a change needs one.
It is a terminal program. It reads and writes like one.
- ↑↓
- Move between agents.
- 1–9
- Jump straight to a row.
- →
- Open the model list.
- m
- Change the model or its context.
- s
- Settings, with the restart warning.
- enter
- Launch the highlighted agent.
Swipe the window to see the rest of it.
Everything in the package.
The daemon is only half of it. These are the parts you get on the same install.
-
Drop-in cache
Point any OpenAI-compatible client at the toki daemon. Prompts that share a prefix return instantly.
-
Live reuse
Multiple agents and shells share one server's KV cache. Context written by one session is replayed by the next.
-
Save & restore
Snapshot caches to disk and restore them later: cold-start with a long system prompt already cached, or hand a session to another machine.
-
RAM-aware by default
The daemon reads the GGUF header, fits the KV cache into the machine's memory, and advertises the real context to clients.
-
Benchmark harness
Run real workloads, agent loops, RAG chunks, adversarial prompts, and report cache-hit rates vs. a fresh cache.
-
TUI + web chat
Full-screen terminal chat and a production-ready single-page web UI ship with the package, one command each.
Three commands from cold start to a shared cache.
Boot the stack, wire your clients at the daemon, and let the prefix matcher do the rest.
- Boot,
toki model usestarts llama-server plus the toki daemon, wires your CLI tools at it, and writes the effective context it can actually hold. - Stream, clients call the daemon's OpenAI-compatible API. Shared prefixes hit the cached KV blocks instead of re-running the model.
- Persist,
toki cache save/toki cache restoresnapshot caches to disk;toki cache statsshows hits, misses, and savings.
Benchmarks
Cache-hit efficiency & throughput.
Measured on an AMD Ryzen AI MAX+ 395 (Ryzen AI Halo) via llama-server. Four models, nine workloads each.
| Workload | TTFT Δ | Prefill Δ | Hit | Score | Grade |
|---|---|---|---|---|---|
realistic_mixed | +66.8% | +560.9% | 88.1% | 84.4 | A |
summarization | +90.7% | +1090.7% | 95.0% | 92.0 | A+ |
code_review | +77.8% | +379.5% | 96.7% | 90.0 | A+ |
agent_long_output | +22.9% | +41.2% | 98.0% | 64.4 | C |
agent_100 | +45.0% | +264.0% | 99.0% | 47.3 | F |
rag_chunks | +47.0% | +226.0% | 87.5% | 47.7 | F |
agent_long_context | +47.7% | +264.6% | 90.0% | 47.1 | F |
no_overlap | +16.7% | +18.0% | 0.0% | 12.0 | F |
stress_adversarial | +20.4% | +33.1% | 52.2% | 52.3 | D |
| Workload | TTFT Δ | Prefill Δ | Hit | Score | Grade |
|---|---|---|---|---|---|
realistic_mixed | +74.2% | +685.3% | 87.6% | 86.8 | A |
summarization | +93.5% | +1187.2% | 95.8% | 93.7 | A+ |
code_review | +84.1% | +494.6% | 96.9% | 91.5 | A+ |
agent_long_output | +29.8% | +58.4% | 97.2% | 69.3 | C |
agent_100 | +54.3% | +343.7% | 98.7% | 59.8 | D |
rag_chunks | +54.7% | +299.1% | 91.8% | 59.9 | D |
agent_long_context | +54.9% | +347.2% | 93.6% | 59.7 | D |
no_overlap | +14.8% | +17.2% | 0.0% | 11.9 | F |
stress_adversarial | +29.2% | +48.7% | 54.8% | 59.4 | D |
| Workload | TTFT Δ | Prefill Δ | Hit | Score | Grade |
|---|---|---|---|---|---|
realistic_mixed |
+62.4% | +412.7% | 84.2% | 78.2 | B |
summarization |
+81.3% | +842.1% | 92.4% | 88.5 | A- |
code_review |
+73.1% | +312.8% | 94.1% | 86.1 | A- |
agent_long_output |
+20.3% | +32.7% | 96.2% | 61.8 | C- |
agent_100 |
+38.4% | +182.3% | 98.1% | 44.9 | F |
rag_chunks |
+40.1% | +161.9% | 84.4% | 45.2 | F |
agent_long_context |
+40.3% | +191.0% | 86.2% | 44.6 | F |
no_overlap |
+14.2% | +15.1% | 0.0% | 11.8 | F |
stress_adversarial |
+16.4% | +25.2% | 48.1% | 49.1 | F |
| Workload | TTFT Δ | Prefill Δ | Hit | Score | Grade |
|---|---|---|---|---|---|
realistic_mixed |
+82.1% | +791.3% | 90.2% | 89.7 | A |
summarization |
+96.2% | +1282.5% | 97.1% | 95.8 | A+ |
code_review |
+88.3% | +582.9% | 98.2% | 94.2 | A+ |
agent_long_output |
+36.2% | +70.1% | 98.1% | 77.3 | B |
agent_100 |
+60.4% | +382.7% | 99.6% | 69.4 | C |
rag_chunks |
+60.7% | +331.2% | 94.3% | 71.6 | C |
agent_long_context |
+60.3% | +381.8% | 95.2% | 70.8 | C |
no_overlap |
+15.1% | +17.0% | 0.0% | 12.0 | F |
stress_adversarial |
+33.2% | +52.1% | 57.2% | 64.2 | D |
Bigger and sparser is better: the 122B-A10B MoE tops every table (realistic_mixed 89.7, summarization 95.8), and the 35B-A3B beats the dense 9B, the more of each prompt's prefill that can be skipped, the larger the relative gain. Disclaimer: all scores are based on multiple benchmark runs on this hardware (AMD Ryzen AI Halo) and depend on the hardware, environment, and model, the benchmark may score better or worse elsewhere.
Platforms
Where it runs.
Toki Recur is pure Python over the llama-server HTTP API. It runs on any x86_64 or aarch64 device running Linux, macOS or Windows, with whatever GPU backend your llama-server build has.
| Platform | Arch | llama-server backends | Status |
|---|---|---|---|
| Linux | x86_64 | CUDA (auto NVIDIA) · ROCm (auto AMD/HIP) · SYCL (explicit Intel Arc) · OpenVINO (explicit Intel) · Vulkan · CPU | stable |
| Linux | aarch64 | CUDA (auto NVIDIA) · Vulkan · CPU | stable |
| macOS | x86_64, arm64 | Metal (one universal build; runs CPU-only with -ngl 0) |
stable |
| Windows | x86_64 | CUDA (auto NVIDIA) · SYCL (auto Intel Arc) · Vulkan (auto AMD) · ROCm/HIP (explicit AMD) · OpenVINO (explicit Intel) · CPU | beta |
| Windows | aarch64 | CUDA (auto NVIDIA) · Vulkan · CPU | beta |
Every GPU install verifies the vendor runtime and a real device line from
llama-server --list-devices before reporting success, so a missing driver cannot
silently degrade to CPU.
Meet toki.
toki is the mascot of Toki Recur, the bear who recurs. He wears the recycle symbol because the whole product is one idea: don't recompute what you've already read. Long system prompts, RAG corpora, agent tool definitions, all get computed once, then streamed from cache on every subsequent request.
He's here to remind you that cache reuse is the boring superpower underneath every fast local model.
Install and run.
Nothing is preinstalled. Clone, run the installer, pull one model, open the launcher.
macOS / Linux
git clone https://github.com/lhadzic377-netizen/TokiRecur.git
cd TokiRecur
./install.sh
toki model pull Qwen/Qwen3-4B-GGUF --use
tokiWindows (cmd or PowerShell)
git clone https://github.com/lhadzic377-netizen/TokiRecur.git
cd TokiRecur
install.bat
toki model pull Qwen/Qwen3-4B-GGUF --use
tokiSupported today: x86_64 and arm64 Linux and macOS are stable; Windows is beta. GPU support depends on the llama-server backend you install. The docs have the full platform and backend matrix. Optional systemd/launchd daemon mode is included. The macOS/Linux installer runs behind a yellow progress bar while the real output still streams live.
A note on compatibility
Some models work. Some need a tweak. I'm honest about it.
toki model pull accepts every GGUF on Hugging Face, so the answer to "does this model work?" depends on the model, the quant, and your hardware. Some GGUFs boot clean and chat on the first try. Others need a chat template swap, a smaller context, a different quant, or a small edit to the registry. I test what I recommend and fix what I ship, but I won't pretend every combination works the same.
It will keep getting better. The failure modes I hit this week (broken chat templates, runtime crashes on a quant, ETXTBSY on a binary swap) feed the next round of fixes. Better auto-detection, fewer surprises, more recovery paths baked in. The project is MIT licensed, and the path from "this model is broken" to "this model works" is one fix away.
Tested on a small set of machines: AMD Strix Halo (Linux), M2 (macOS), Intel i7-8650U (Windows), Intel Core Ultra 155U (Linux), GTX 1650 Mobile (Windows). That's five configs, not a hardware matrix. Different GPU, different VRAM, different kernel, different driver; any one of them can change the outcome. That's where the next bug report usually comes from.