AriakiI bypassed a hosted AI gateway and called the vendor's API directly. The whole point was to go faster.
Then I wired it into a game and played a round. Median latency: 481 ms. The old gateway route, same API, had a median of 243.
I took a detour and came out twice as slow.
The embarrassing part is that I drew the map for that detour myself. A few days earlier I'd said, with a straight face, "of those 300 ms, the model itself is probably under 50 — the rest is the gateway." I had no direct-route number at all. I'd just split the bill in my head.
If you only want the trick, section 3 is one subtraction that isolates connection overhead. If you want the rate-limit math, go to section 4.
1. Setup: a tiny game that makes decisions several times a second
I run a small game testbed on my own machine: Flappy Bird and friends, where every few moments a model gets asked "flap or wait?" The model returns one option plus probabilities. No text generation, so a single call ought to be fast.
One of the decision engines is a vendor's cloud decision API. It has two front doors: the vendor's own endpoint, and a pass-through behind a hosted AI gateway. The gateway gives you unified auth plus billing metadata and a generation ID for free. The price is one extra hop.
I started on the gateway, and calls felt like roughly 300 ms. For a game that's painful, which is how I ended up saying "most of it is the gateway."
2. The first comparison: the gateway is one third
Once I had a direct API key, I ran a small head-to-head: 24 questions from public datasets, each sent once through each route, alternating AB/BA order so neither route always got to go first. Each route used one persistent HTTP session, no retries, with 2 seconds of rest between pairs.
| Route | Requests | Succeeded | p50 | p95 |
|---|---|---|---|---|
| Direct | 24 | 24 | 165.2 ms | 200.9 ms |
| Gateway | 24 | 16 | 243.4 ms | 296.5 ms |
Pairing the same question in the same time slot and subtracting, the gateway's median overhead was +80.2 ms. In 15 of the 16 pairs the gateway was slower; in the remaining one it was faster by 0.4 ms, which is noise.
So the gateway's share is:
gateway share ≈ (gateway p50 − direct p50) ÷ gateway p50
= (243.4 − 165.2) ÷ 243.4 ≈ 32%
A third. My "the rest is all gateway" pointed in roughly the right direction and was off by a wide margin on size. The direct call alone costs 165 ms, and the gateway owes none of that.
I also checked the answers. All 16 successful pairs picked the same option; the largest probability gap was 0.06, and 13 pairs matched exactly. Same model behind both doors, nothing swapped in transit.
At this point the conclusion looked simple: go direct, save 80 ms.
Then I plugged the direct route into the game.
3. 481 ms: same direct route, different way of calling it
The game runs in a browser, and the browser can't hit that vendor endpoint directly (it answers every browser CORS preflight with a 400). So a small local proxy forwards the calls. I'd written that proxy the lazy way: every incoming request spawns a curl subprocess.
Real browser, one round: 82 requests, 0 errors, p50 481 ms.
The benchmark said direct was 165. Why was it three times that in the game? Three suspects: subprocess startup, slow Python, or something else. Rather than guess, I sent the same request eight times each in four different ways (dropped the first, took the median of the remaining seven):
| How it's called | p50 |
|---|---|
| New curl subprocess, new TLS each time | 368 ms |
| Python, new connection each time | 323 ms |
| Python, one reused connection | 162 ms |
| Old proxy (curl inside) | 368 ms |
That settles it. Python vs curl is 45 ms, so the subprocess isn't the main offender. The real cliff is between "new connection" and "reused connection":
per-request connection overhead ≈ new-conn p50 − reused-conn p50
= 323 − 162 = 161 ms
Those 161 ms are a fresh TCP handshake plus TLS handshake on every call. curl's own timing agrees: 0.18 s had already passed by the time the TLS handshake finished. Put plainly, every "flap or wait?" started with a round of "hello, here's who I am, here's my certificate" before anyone got to the actual question.
It also explains why the section 2 numbers looked so good: the benchmark script used a persistent session; the game proxy used one-shot curl. The experiment measured a direct route with connection reuse. What I plugged into the game was a direct route without it. The experiment wasn't wrong. It just measured something other than what I was actually running.
The fix is boring: the proxy keeps one long-lived connection per upstream, behind a lock, rebuilding and retrying once if it drops. curl is gone. Another round:
125 requests, 0 errors, p50 167 ms, p95 223 ms (was 481 / 605).
That's 5 ms above the raw reused connection at 162. I attribute those 5 ms to the local proxy hop plus the game loop, but that's inference; I didn't measure them separately.
Side note: the proxy code carried an old comment claiming a certain Python stdlib HTTP client "gets 403'd by the CDN," which is why curl was used in the first place. I retried with a custom User-Agent and couldn't reproduce it. One comment nobody ever re-checked, and every request paid for an extra handshake.
4. The gateway's other bill: a rate-limit error that passes the blame
The 8 gateway failures in the section 2 table were all 429s. The response body said:
The upstream provider is currently experiencing high demand. Please retry shortly.
That reads like the model vendor behind the gateway was overloaded. The response headers told a different story:
X-Ratelimit-Limit-Requests: 30
X-Ratelimit-Remaining-Requests: 0
X-Ratelimit-Limit-Tokens: 250000
X-Ratelimit-Remaining-Tokens: 233750
Retry-After: 10
93% of the token quota was still left. What ran out was the gateway's own request cap: 30 requests per 10 seconds. In the same time window the direct route took 24 calls without a single rate limit, and returned no rate-limit headers at all. The error text blamed the upstream; the gate was the gateway's own.
For a closed-loop game, that gate hurts far more than the 80 ms:
sustainable decision rate ≤ request cap ÷ window seconds
= 30 ÷ 10 = 3 per second
Three questions a second, max. A game that wants to ask every 100–200 ms fills the window, hits a 429, and is told to wait 10 seconds. In real-time mode, the bird has long since hit the ground.
Do the same math for yourself: read the request cap and window off the rate-limit headers, divide, and compare with how often you actually call. If it's lower than what you need, good latency won't save you.
5. How to see through someone else's API latency numbers
This taught me to ask one question first: does this number include a handshake?
A lot of "I benchmarked API X" posts are a loop that runs curl over and over and averages the results. Going by the table above, every one of those calls carries roughly 160 ms of connection setup, and more if the server is on another continent. What you get is "connect + serve," not "serve."
It cuts the other way too. A pretty number measured on a persistent session will triple, like mine did, once it lands in code that opens a new connection every time.
So for any latency number, ask two things:
- Was it a fresh connection per call, or a reused one?
- Is it a paired difference from the same client, same time slot, same request — or two p50s from two different runs subtracted blindly?
My original "300 ms, mostly gateway" claim skipped both questions and had no direct baseline. I was splitting a bill against a denominator I'd never measured.
6. Where this experiment is weak
- Small samples. The route comparison is 24 pairs, only 16 of which succeeded; the layer probe has 7 usable samples per method. Trust the order of magnitude, not the decimals.
- The gateway's 243.4 ms p50 is computed over the 16 successes; the 8 429s are excluded. Count the rate-limit waits and the real gateway picture is much worse.
- The in-game 481 ms came from a real browser; the gateway's 243 came from a Python script. The measurement environments aren't equivalent, so "took a detour, came out twice as slow" is an order-of-magnitude comparison, not a strict pairing.
- One location, one time window, and no separate round-trip-time measurement, so I can't split the roughly 150 ms of server-side time into physical distance vs model compute.
- curl's timing only recorded when TLS finished (0.18 s); I didn't keep the DNS and TCP breakdown.
- I also switched a browser extension to the direct route, but never loaded it in a real Chrome, so that part isn't verified.
7. Next time "this API is slow," check in this order
- Measure a direct baseline first, with a reused connection, from the same machine. Without it, any claim about where the time goes is a guess.
- Send the same request several times with a fresh connection each time and several times over one reused connection. Subtract. That difference is your handshake tax per call.
- Check the code you're actually running: does the HTTP client open a new connection every time? Is there a subprocess inside a loop? Does the proxy keep connections alive?
- If there's a gateway in the path, pair direct and gateway calls on the same request in the same time slot and take the median of the paired differences. Don't just subtract two p50s.
- Read the rate-limit headers and compute request cap ÷ window seconds. Compare that with how often you really call.
- When an error says "upstream is busy," look at the
Remainingheaders to see whose gate you actually hit. - Before routing around an old code comment that says "library X doesn't work," reproduce it once.