The model card says 1M context, and the math says it fits in 64GB. So why did my Mac get stopped at 256K?

I fed it a 262,188-token prompt. Before the model finished reading it, the process got a SIGTERM and exited with code 143.

The thing that killed it was my own watchdog: if free system memory drops to 3%, stop immediately. No answer, no error. I just pinned it down.

Half an hour earlier, right after the 64K run passed, I'd written in my notes: "Based on the 64K data, 256K looks very promising capacity-wise."

The day before that, I'd done an even rosier calculation: with the KV cache compressed to 4-bit, even the full 1M context on the model card would come to 62.6 GB for weights plus cache, leaving this 64GB Mac about 6 GB of headroom.

Three claims, each more optimistic than the last, all wrong. The worst part: the 64K numbers alone were enough to predict that 256K would die. I just didn't run the division.

If you only want the "two measurements predict the limit" division, it's section 4. If you want to know where the paper math went wrong, section 3.

1. The paper math: one multiplication that looks solid

The model is Poolside's Laguna S 2.1, a 118B-parameter MoE (mixture of experts: the network is split into many "experts" and each token only wakes a few of them; here roughly 8B parameters are active per token). I ran a community 3-bit quant on mlx-vlm, on an M1 Max with 64GB of unified memory. After loading, the weights sit at 49.23 GB resident.

What eats memory at long context is the KV cache. For every token the model reads, each attention layer stores a key and a value for it, and every token generated afterwards has to look back through all of them. Read more, store more. The weights are a fixed cost you pay once; the KV cache is the part that scales with how long your prompt is.

This model has a memory-saving design that's becoming common in long-context models: of its 48 layers, only 12 use global attention and have to remember the full history. The other 36 use a sliding window that only looks at the last 512 tokens and overwrites as it goes. So those 36 layers stay at a fixed ~75 MB no matter how long the context is, which is negligible. Only the 12 global layers grow with length.

How much one token costs in those 12 layers comes straight out of the model config:

KV bytes per token:
  global layers × KV heads × head dim × 2 (K and V) × bytes per value
  = 12 × 8 × 128 × 2 × 2 bytes (BF16)
  = 49,152 bytes ≈ 48 KiB

That's just a multiplication. Times 1,048,576 tokens it's 48 GiB. Add 49 GB of weights and you're past 100 GB. A 64GB machine doesn't stand a chance.

So you compress. TurboQuant is a scheme that squeezes the KV cache to roughly 4 bits. Using the overhead formula in mlx-vlm's source, each value goes from 2 bytes to:

4/8 + 2/128 = 0.515625 bytes

That's 3.88x compression. Per token, 49,152 bytes becomes 12,672. Run it again:

1M × 12,672 bytes = 13.3 GB
49.23 + 13.3 + 0.08 (sliding layers) = 62.6 GB
64 GiB = 68.7 GB, leaving 6.1 GB

I even checked that the flag wasn't a no-op: I dumped all 12 global-layer caches at runtime and every one of them was a TurboQuant cache object. On paper the road was open.

2. Three measured lengths

Then I actually fed it. For each length I generated filler text of that size with one line buried in the middle, "the passphrase is NEBULA-COPPER-7319", and after reading I asked for the passphrase. I watched two memory readings the whole time: MLX's self-reported peak memory, and macOS's free system memory percentage. Watchdog rule: kill the process if free memory drops below 3% or free swap drops under 96 MB.

Length	Actual tokens	Passphrase found	Prefill speed	Time to first token	Tail decode	MLX peak	Lowest free memory
64K	65,582	✅	82.4 tok/s	just over 13 min	15.6 tok/s	51.94 GB	16%
128K	131,116	✅	66.96 tok/s	32 min 50 s	10.27 tok/s	54.52 GB	11%
256K	262,188	—	—	never finished reading	—	—	fell below 3%, killed

Prefill is the phase where the model reads the whole prompt before it emits anything. Reading 128K took more than half an hour.

What did the paper math predict for 256K? 49.23 + 262,188 × 12,672 bytes = 52.6 GB. That's 16 GB below the 68.7 GB ceiling. And yet free memory hit the floor.

3. How much does one more token actually cost?

Subtract the MLX peaks of the 64K and 128K runs:

increment per token ≥ (54.52 − 51.94) GB ÷ (131,116 − 65,582) tokens
                    = 2.58 GB ÷ 65,534
                    ≈ 39.4 KB / token

Why "≥": MLX reports a high-water mark since process start. The peak right after model loading was 51.82 GB, and the 64K run's 51.94 is barely above it, so part of that reading is probably the load-time watermark. The true 64K run peak can only be lower than 51.94, which makes this subtraction a lower bound. The 128K reading of 54.52 is well above the load watermark, so that one is a genuine run peak.

Compare it with the paper number:

Compression formula:  12,672 bytes / token
Measured lower bound: 39,400 bytes / token   → 3.1x
Uncompressed BF16:    49,152 bytes / token

In plain terms: the KV cache really is 3.88x smaller, but peak memory growth per token is only 20% short of "no compression at all". Whatever you saved got eaten by something else.

I didn't pin down what ate it. The prime suspect is the temporary buffers during prefill that scale with context length. I fed the prompt in 256-token chunks, and each chunk has to run attention against the entire history so far, so the intermediates grow with how much has been read, and the compressed history very likely gets temporarily decompressed for that. That's a guess. I didn't nail it with a profiler.

For the question "how long a context can this machine take", the root cause doesn't matter: what kills you is the peak, and the steady-state cache is only part of that peak.

Another engine is more extreme. Reading oMLX's source, for a fresh request it runs the whole prefill at full precision and only converts the cache to TurboQuant once reading is done, to avoid quantization error contaminating prefill. On that path, the peak for 1M is the uncompressed 100+ GB. Compression only helps the steady state. It does nothing for the first read.

4. Two measurements tell you where it dies

This is the division I should have done at the time and didn't:

tokens you can still add ≈ remaining headroom ÷ increment per token
usable limit ≈ last measured length + tokens you can still add

Using free system memory (the thing the watchdog watches): lowest free was 16% at 64K and 11% at 128K. That's 5 percentage points, or 3.44 GB, per 65,534 tokens, about 52 KB per token. From 11% down to the 3% red line leaves 8 points, 5.5 GB:

5.5 GB ÷ 52 KB ≈ 105,000 tokens
131,116 + 105,000 ≈ 236,000 tokens

Predicted death around 236K. I fed 262K, and it died. That checks out.

The painful part is that the 64K run alone was enough. At that point I had two numbers: 49.23 GB resident after loading, 51.94 GB peak after 64K. Rough slope: 41.3 KB per token (this mixes in fixed overhead, so it runs high, but the order of magnitude is right). Getting to 256K means reading another 196,606 tokens, about 8.1 GB. At 64K the system had 16% free, about 11 GB. Subtract and you're left with 2.9 GB, a bit over 4%, right on top of the 3% line.

Three lines of arithmetic, and instead I said "very promising". The number in my head was still the 12,672 bytes from section 1. I never swapped in the number I'd just measured.

5. Even if it fits, you can't afford the wait

Say you have a 128GB machine and memory is no longer the problem. Speed still is.

The lower bound on time to first token is a division:

time to first token ≥ prompt length ÷ prefill speed

Prefill speed dropped from 82.4 at 64K to 66.96 at 128K; longer is slower. Even using the already-degraded 128K speed, the floor for 1M is:

1,048,576 ÷ 66.96 ≈ 15,660 s ≈ 4.35 hours

And that's optimistic, because speed keeps falling. If you extrapolate the two points assuming time per token grows linearly with length, 1M takes over 15 hours. An extrapolation is an extrapolation; treat it as an order of magnitude.

Generation speed tells the same story. Three tail-decode points: 30.2 tok/s at 4K, 15.6 at 64K, 10.27 at 128K. Convert to milliseconds per token and fit a line through the 64K and 128K points:

ms per token ≈ 30.8 + 0.508 × context read so far (thousands of tokens)

Check it against the third point: at 4K the line gives 32.9 ms, i.e. 30.4 tok/s. Measured: 30.2. It holds, so the line is usable. Every extra 1,000 tokens of context adds about 0.5 ms to every token generated, because each new token has to look back over the entire history.

Extrapolated: about 6.1 tok/s at 256K and about 1.8 tok/s at 1M. Under two tokens per second, which is roughly watching it type one character at a time.

I'd also estimated speed once before measuring anything, and the way that went wrong is worth spelling out. All I had was someone else's numbers from the model card, on an M5 Max: same 3-bit build, 64K prefill at 624 tok/s. I scaled that by the memory bandwidth ratio of the two machines (546 vs 400 GB/s), which amounts to assuming the M1 Max would do 457, and extrapolated a cold 1M read of about 4.2 hours. The measured 64K speed was 82.4, 5.5x slower than even the scaled number. Several variables were tangled together: online compression, 256-token chunks, the compute gap between two chip generations, and I didn't separate them. One thing is certain: I picked the wrong ruler. Prefill crunches one big block of matrix math at a time and is mostly compute-bound, so scaling it by bandwidth was wrong from the start. As for "about 4.2 hours for 1M" landing so close to the 4.35 hours in this section: pure coincidence. Don't count it as validation.

6. What to ask when someone says "supports 1M"

During this run I had three different "context lengths" in front of me at once:

• The model card's max_position_embeddings = 1,048,576
• The max_kv_size = 204800 I set on the eval server
• The longest prompt I actually read safely: 131,116

The first two are permissions. Only the third is evidence. The model card's 1M means "position encoding is defined up to 1M". It has nothing to do with your hardware. My own 204800 is worse: the eval questions were all short Q&A, nothing long was ever fed in, but that number sits in the config and is easy to misread later as "200K works". I wrote a line in my notes specifically warning against reading it that way, because I knew I would.

So whenever you see "machine X runs model Y at Z context", ask three things first:

1. Did they actually feed a Z-length prompt, or just set a config value to Z?
2. Are they reporting resident memory after loading, or the peak while reading the longest prompt?
3. How long do you wait for the first token at Z?

A post that reports "weights + KV from the formula" without a measured peak can overstate long-context capacity several times over. Mine was off by 3.1x. One that says "it fits" without reporting time to first token can be ignored outright: 1M starts at over four hours of waiting, which for anything interactive means unusable.

7. What's wrong with this experiment

• One run per length, no repeats.
• MLX peak is a high-water mark, and the 64K reading is contaminated by the load watermark, so section 3 can only give a lower bound.
• Free memory percentage is affected by other processes and by macOS memory compression, and the compressor doesn't behave linearly. There was already about 3.3 GB in swap before the runs started.
• 256K was killed by my own 3% threshold. Without it, the run might have limped through on swap, so the accurate statement is "can't be read safely", not "can't be read". I wasn't willing to bet the whole machine on finding out.
• I only used 256-token chunks and didn't sweep chunk size. Larger chunks might be faster, and might also raise the peak.
• The tail decode sample is small: the 64K run only generated 32 tokens. The 4K point came from a smoke test before the main runs, so its settings may not match exactly.
• The passphrase test only shows it can retrieve one line from long text. It says nothing about reasoning quality over long context.
• I didn't locate what's eating the extra ~27 KB per token.
• Every speed number at 1M is an extrapolation.

8. Next time you want to know how much context a model can take on a given machine, go in this order

1. From the model config, read global attention layer count, KV heads and head dim, and multiply out KV bytes per token. Count sliding-window layers separately; they're usually negligible.
2. If the KV cache is compressed, multiply by the compression ratio. Treat the result as a lower bound on per-token memory growth, not an estimate.
3. Read the engine source to see whether prefill compresses as it goes, or reads at full precision and compresses afterwards. If it's the latter, throw out step 2 and size the peak uncompressed.
4. Measure two lengths, say 64K and 128K. Record peak memory and lowest free system memory for each.
5. Subtract the two to get the real per-token increment and compare it to steps 1 and 2. If they're off by several times, stop trusting the paper math.
6. Remaining headroom ÷ per-token increment + last measured length = usable limit. Compute this before deciding whether to try the next length.
7. Time to first token ≥ length ÷ prefill speed at the last measured length. If that's longer than you're willing to wait, fitting in memory doesn't help.
8. Fit generation speed as "ms per token = a + b × length" from two lengths, check it against a third, and only extrapolate if it holds.
9. Before running the biggest length, put a watchdog in place that kills the process when free system memory drops too low.

On my machine the answer is 128K, and each read takes 32 minutes 50 seconds. Enough time to brew two pots of tea.