Ariaki
/
Back to blog

2026.10.17

Three samples, memory didn't budge, so I ruled out a leak. Five weeks later it went from 1.6GB to 22GB. What did those five minutes miss?

A 64GB M1 Max ran out of memory and rebooted. I sampled three times, saw a flat slope, and blamed too many resident processes instead of a leak. Five weeks later an embedding service had climbed almost linearly by 20GB in three and a half days, while a speech-to-text service sat parked at 58GB. This post works out one division: reading resolution over slope tells you the minimum time you have to watch. It also shows that MLX sets its default cache limit to nearly the whole machine, so two processes can put memory on the books at 1.9x what you actually have.

macOSApple SiliconMLXdebugging

Back in mid-September I wrote a rule into my notes: "A large absolute number is not evidence of a leak. The slope is."

I based it on three samples. Over 5 minutes 20 seconds, the memory of five resident processes on that Mac didn't move by a single digit: 21GB, 36GB, 6259MB, 2128MB, 5408MB, identical all three times. Flat slope. So I called it: too much stuff resident, the steady state just fills the box, nothing leaking. I killed the two biggest processes and moved on.

This morning I looked at the same machine again. A text embedding service that sat at 1.6GB 17 minutes after boot is now at 22GB. The points in between line up almost perfectly, gaining 0.23 to 0.26GB an hour. Next to it, the speech-to-text service is parked at 58GB, and 40989MB of the 41984MB swap is already used.

The rule itself was fine. The way I measured the slope was the problem. And the slope was sitting in my own September notes all along; the last paragraph of section 1 shows where.

If you just want the division, go to section 2. If you want to know which of today's numbers you still shouldn't trust, section 5.

1. The machine, and the misdiagnosis

An M1 Max with 64GB of unified memory. Unified memory means the CPU and GPU share one pool, so VRAM is just RAM. It runs a few small resident model services for my own AI assistant: speech-to-text, text embeddings (turning a sentence into a vector of numbers for semantic search), and text-to-speech. The first two run on MLX, Apple's own machine learning framework.

The September crash was a kernel panic with a textbook signature: a system daemon hadn't checked in with the watchdog for 182 seconds, the memory compressor was at 100%, and there was a string of Jetsam events. Jetsam is the macOS mechanism that kills processes when memory runs short. In plain terms, memory pressure got so bad that even system processes couldn't get scheduled, the watchdog decided the machine was dead, and it rebooted.

Everyone's first guess was "Python memory leak," mine included. My first round of sampling used the wrong tool (item 1 of the checklist at the end covers that). I switched tools, sampled three more times, got three flat readings, threw out the leak theory and called it overcommit.

Going back through those notes now, there are two lines I didn't think twice about. Seven minutes before the formal sampling I'd taken a quick reading: speech-to-text at 33GB, embeddings at 1574MB. The first formal point had them at 36GB and 2128MB. In those 7 minutes one grew 3GB and the other grew 554MB. I had the slope in my hands. It just never made it into "the three points."

2. One division: how long you have to watch

minimum observation time to see a slope ≈ reading resolution ÷ slope

It's a division, that's all. Plug in this case:

  • macOS's built-in footprint tool reports large processes in whole gigabytes (I got readings like "21 GB" and "58 GB"). Only processes in the few-GB range get MB precision ("5408 MB"). For the big processes, the resolution was 1GB
  • The embedding service's average slope over three and a half days: 0.235GB per hour
1 GB ÷ 0.235 GB/hour ≈ 4.3 hours
my window: 5 min 20 s
at that slope, 5 min 20 s adds about 21MB, 1/48 of the reading resolution

21MB on top of a reading that says "21 GB" never moves the last digit. Three identical points are perfectly consistent with "it's leaking right now." I read "the instrument can't see it" as "nothing changed."

It gets worse. At the time the embedding service was only at 2128MB, a size where footprint does report MB, so a 21MB change should have shown up. It didn't move either, even though it had grown 554MB in the 7 minutes before. The most likely explanation: memory in this kind of service only grows while it's handling requests, requests come in bursts, and my 5 minutes happened to land in a quiet stretch. I can't prove that one; section 5 explains why.

So the division needs a second half:

measure the slope per request handled, not per wall-clock minute
if the window had no traffic, every leak has a slope of 0

That makes it easy to push back on "I watched it for ten minutes, memory was stable, no leak." Ask what precision the reading had, then ask how much work the service did in those ten minutes. If neither answer is available, the statement carries very little information.

3. Flat means healthy? 58GB looked familiar

The embedding service is climbing. The speech-to-text service does the opposite: 4.9GB at 17 minutes after boot, 57.9GB two days later, still 57.9GB 20 hours after that, 58GB today. By my September rule, its slope is zero, so it goes in the "steady state just fills the box" bucket and counts as normal.

But 58 is a suspicious number. On the same machine, in the Python environment the speech service runs in, I read MLX's defaults:

total memory:                                   64.0 GiB
system-recommended GPU working set limit:       60.8 GiB
MLX default cache limit:                        60.8 GiB

When MLX is done with a chunk of GPU memory, it doesn't hand it back to the OS right away. It parks it in its own cache pool so the next allocation of the same size can reuse it and skip the cost of allocating again. By default, the cap on that pool equals the entire recommended GPU working set: 60.8GiB.

In other words, on a 64GB machine, a single process's cache is allowed by default to grow to 60.8GiB. The speech service sitting at 58GB looks like it grew right up under that ceiling and got held there. It stopped because it hit the cap.

I grepped both services' code. Neither ever calls set_cache_limit or clear_cache.

4. A forty-line probe: when does the cache grow?

Knowing the default isn't enough. I wanted to know under what conditions it grows. I wrote a small script on the same MLX version (0.32.2) with three arms, each doing 120 rounds of "allocate a chunk of GPU memory, compute something, drop it," then sitting idle for 15 seconds:

Arm Cache after 50 Cache after 100 After 15 s idle Process footprint at idle
Random 1–4MB each time 206.1 MB 369.6 MB 403.5 MB 422 MB
Fixed 4MB each time 8 MB 8 MB 16 MB 34 MB
Random 1–4MB + 128MB cap 127.0 MB 126.7 MB 128.6 MB 147 MB

Three things fall out:

  1. Fixed sizes barely grow the cache. Varying sizes make it climb steadily. 403.5MB versus 16MB, a 25x gap. In a real service, every audio clip is a different length and every batch of text is a different length, so the requested sizes change constantly.
  2. Idle doesn't give it back. After 15 seconds of doing nothing, the cache hadn't shrunk by a byte. So "memory doesn't drop when it's idle" proves nothing either way about a leak.
  3. With a cap, it stops at the cap. Cap of 128MB, final cache 128.6MB. Footprint tracks the cache, roughly the cache plus a baseline of a dozen-odd MB.

That shape matches what I saw on the real machine: varying sizes, cache keeps climbing, stops at the cap. The embedding service handles small inputs and may still be climbing; the speech service allocates big chunks and hit the ceiling long ago.

Now do the accounting:

memory on the books = resident processes × per-process cache cap
2 MLX services × 60.8 GiB = 121.6 GiB
physical memory 64 GiB, so memory is on the books at 1.9x

Every process thinks it owns the whole machine. Six minutes before the October 13 reboot, the Jetsam report had two Python processes at 56.8GB and 48.7GB, 105.5GB combined (a books number that includes swapped-out pages; section 5 has the details), 1.6x the machine's total memory. That bill has already come due once.

There was another misleading reading this morning. The system's memory pressure tool said "58% free" while swap was nearly full at the same moment. Idle cache pages had been compressed and swapped to disk, freeing physical memory, so the pressure tool looked optimistic. But those pages still belong to the process, and the day it needs them again they have to be read back from disk. If you judge whether the machine is safe by the "free percentage" alone, this layer will fool you.

5. Which of these numbers you still shouldn't trust

  • I never read the cache size from inside either service process. mx.get_cache_memory() can only be called in-process; you can't attach from outside. "The 58GB is cache" is an inference resting on three pieces of evidence: footprint attributes nearly all of that 58GB to the GPU buffer category; it stopped just under the default cap; and the service code never sets a cap. All three are indirect, and I didn't check whether a dependency sets its own cap internally. If it's actually a real leak somewhere in a library that just happens to stall at 58GB, sections 3 and 4 both need rewriting.
  • The embedding service's straight line has 4 points. The first 3 come from per-process page counts in Jetsam reports, so I didn't pick the timestamps; the last one is from footprint. I cross-checked the two sources at the same moment: speech service 57.9GB vs 58GB, text-to-speech 5.3GB vs 5407MB, so individual processes agree. But I already got burned by those page counts in September: I summed them across all processes, reported 117GB, and the actual physical usage was only 29.9GB. So the 105.5GB in section 4 is a books number only. It includes compressed and swapped-out pages and does not mean that much physical memory was in use at once.
  • The probe is a synthetic load using arrays of ones, not a real model. It shows that this MLX version's cache grows when sizes vary, doesn't give memory back when idle, and respects a cap. It doesn't show that the real services grow exactly this way.
  • The "no requests in those 5 minutes" idea in section 2 is a guess; neither service logs requests.
  • I never confirmed the cause of the October 13 reboot. There's no panic file in the directory where the system keeps crash reports.
  • The fix isn't deployed. Setting a cap on the two services means restarting them, and I haven't done that yet. I also haven't measured whether a cap slows them down through more frequent reallocation.

6. Next time you suspect a memory leak, check in this order

  1. On Apple Silicon, read phys_footprint from footprint, not RSS. GPU buffers don't count toward RSS: in September the speech service showed 3797MB RSS against a 36GB footprint, 9.5x apart; another service showed 427MB RSS against 21GB, 50x apart.
  2. Before you choose an observation window, do the division: reading resolution ÷ the slope you suspect. If it comes out to 4 hours, don't conclude anything from 5 minutes.
  3. Make sure the window contains real requests. Record how many requests were handled and compute the slope per request.
  4. If it stopped growing, check where it stopped. If it's near a default cap (for MLX, the system-recommended GPU working set, 60.8GiB on a 64GB machine), the cache hit the ceiling, and the accounting is resident processes × cap.
  5. For any resident MLX service, call mx.set_cache_limit() explicitly at startup, sized by how many such processes share the machine.
  6. To judge whether the machine is safe, look at how much swap is used, not just the "free memory" percentage.