A local-model experiment reused all 3633 input tokens across session IDs. My negative control passed anyway. How to separate session identity from cache state before trusting a speedup.
LLM · benchmarking · caching6 min readI wrote in my notes: "4 concurrent, ~350 tok/s aggregate, continuous batching works." The next day I looked at completion times and saw four requests finishing one after another, 91.2 tok/s aggregate. This post shows how one division and a table of completion timestamps tells real parallelism apart from a tidy queue, and then works out how far a 98 tok/s single stream really is from the physical ceiling.
llm · benchmark · apple-silicon10 min readA public benchmark claims a 27B dense model hitting 2103.6 tok/s on one consumer card — about 26× faster than normal. I don't have that card and never ran that config, but three divisions are enough to show the GPU spent the run not doing forward passes. You can run all three on your own machine.
llm · benchmark · inference10 min read
Ariaki