AriakiThe first line of the README says: about 1000x faster than HuggingFace's tokenizer.
I ran six workloads on a 6-core Ryzen of my own. The best one came out at 26.33x. The worst came out at 0.07x, which means more than ten times slower than the thing it's supposed to replace.
Same library, same tokenizer file, same machine, and the speedup spans more than two orders of magnitude. That's what this post is about: a "N times faster" number is mostly decided by its denominator. And the 20x I measured myself turned out to be padded too. Section 4 covers that.
If you just want the division, jump to section 3. If you want to know which of my numbers not to trust, read section 4.
1. Where the 1000x comes from
Quick context. A tokenizer is the step that chops text into the integer IDs (tokens) a model actually consumes. The library is called GigaToken, written in Rust. Its main trick is a hand-written SIMD version of the "pre-tokenization" split that normally runs through a general-purpose regex engine, plus a cache of how already-seen words get split. It only handles the chopping. The model itself doesn't compute a single token faster because of it.
The README has full tables for three machines, all measured on the same 11.9 GB English web-text corpus. Here's the Qwen 3.5 / 3.6 row from each:
| Machine | GigaToken | HF tokenizers | Speedup |
|---|---|---|---|
| Apple M4 Max (16 cores) | 6.31 GB/s | 6.3 MB/s | 994× |
| Dual-socket EPYC 9565 (144 cores) | 15.49 GB/s | 27.7 MB/s | 558× |
| Ryzen 7 9800X3D (16 threads) | 5.22 GB/s | 51.6 MB/s | 101× |
Same row, and it drops from 994x to 101x. Where does the difference go?
Split it and it's obvious. Compare M4 Max with the 9800X3D:
Numerator (GigaToken) change: 6.31 ÷ 5.22 = 1.21x
Denominator (HF) change: 51.6 ÷ 6.3 = 8.19x
Speedup change: 994 ÷ 101 ≈ 9.8x ≈ 1.21 × 8.19
The speedup moved by almost 10x. The new library accounts for 1.21 of that. The other 8x is HF being unusually slow on the M4 Max. Put bluntly, the "1000x" headline uses the machine where the old implementation is at its worst as the denominator.
To be fair, the README doesn't hide this. The benchmark details say HF only gets the first 100 MB, pre-split by document, while GigaToken encodes the whole file. It also says the HF-compatible mode "won't quite reach 1000x." It's all written down. People just don't scroll that far.
Speedup = new throughput ÷ old throughput. When you see a big multiplier, first go find the conditions the denominator was measured under. No domain knowledge required. It's one division.
2. Moving it to my 6-core machine
My box is a Ryzen 5 5600, 6 cores / 12 threads. GigaToken 0.9.0 (still beta) against HF tokenizers 0.23.1, using the tokenizer file from a local Qwen3.5-27B. Every run was checked token-by-token against HF; a number only counts if the output matched.
| Workload | GigaToken | HF | Speedup |
|---|---|---|---|
| Bundled CLI on a small file of a dozen or so KB | 0.12 MB/s | 1.67 MB/s | 0.07× |
| One 171-byte short text | 24.63 MB/s | 4.08 MB/s | 6.04× |
| 100 short texts (17 KB) | 113.52 MB/s | 23.91 MB/s | 4.75× |
| 10,000 short texts (1.75 MB) | 82.00 MB/s | 31.11 MB/s | 2.64× |
| 256 MB file, split by document separator | 344.85 MB/s | 17.20 MB/s | 20.05× |
| One 1.71 MB long document | 159.43 MB/s | 6.06 MB/s | 26.33× |
The short-text rows use a Chinese sentence plus a bit of JSON, a line of Python and an emoji, meant to look like the mixed text you normally stuff into a prompt.
A few things jump out:
Small files through the CLI are slower. 0.12 vs 1.67, more than ten times slower. I didn't dig into why. Most likely a one-off initialization cost that a tiny file can't amortize. For the "tokenize one prompt per request" use case, this row is closer to reality than 1000x.
More short texts, smaller speedup. 1 text: 6.04. 100 texts: 4.75. 10,000 texts: 2.64. Look at the HF column: it climbs from 4.08 to 31.11, more than 7x. HF's batch API spreads documents across threads, so once there are enough of them it gets going. The denominator grows, and the speedup shrinks.
One long document gives the biggest speedup. At 26.33x, HF is doing only 6.06 MB/s. Roughly the same 1.7 MB, split into 10,000 pieces, runs at 31.11; as one piece it drops to 6.06. A single document can't be parallelized across documents, so HF is at its weakest here.
Denominator again. The new library's numbers wander between 0.12 and 345; the old library's between 1.67 and 31.11. Pick whichever row you like as your headline and you can print whichever multiplier you like.
3. Does the time saved even cover one model call?
However big the multiplier, the real question is how many seconds you save, and what share of the whole job those seconds are.
This is an old formula (Amdahl's law), in plain terms:
time saved = tokenization time × (1 − 1 ÷ speedup)
Plug in a single prompt first. 171 bytes: HF takes 0.042 ms, GigaToken takes 0.007 ms. You save 0.035 ms.
I'd previously measured in my own setup that the fastest possible request to a local model, after warm-up, takes 79 ms round trip. 0.035 ÷ 79: you save less than a thousandth of that one call. Put differently, swapping the tokenizer on per-request paths for chat or agents won't even show up in your measurements.
Now flip it: how much text do you need before the saving is worth one model call?
text needed = round-trip time × HF throughput ÷ (1 − 1 ÷ speedup)
= 0.079 s × 17.20 MB/s ÷ (1 − 1 ÷ 20)
≈ 1.43 MB
So you need to tokenize more than about 1.4 MB in one go before switching libraries saves even one model call's worth of time. A typical prompt is a few KB to a few tens of KB, off by a factor of dozens.
Large corpora are a different story. Using my 256 MB throughput for a rough 100 GB estimate: HF is 100,000 ÷ 17.20 ≈ 5,814 s, about 97 minutes. GigaToken is 100,000 ÷ 344.85 ≈ 290 s, under 5 minutes. That's an extrapolation, I didn't actually run 100 GB, and the next section explains why 344.85 is on the optimistic side. But on order of magnitude, for corpus preprocessing, token counting, and packing training data, this library is the real deal.
My verdict ended up simple: keep it in the toolbox, change nothing on the per-request path, pull it out the day there are hundreds of GB to tokenize.
4. Where my numbers are padded
This section decides whether you should believe section 2.
I built the 256 MB file out of the README itself. I concatenated one README end to end and truncated at 256 MB. The trouble is that a big part of GigaToken's speed is caching words it has already seen, and in this file nearly every word has been seen tens of thousands of times. Real corpora have long-tailed word distributions; cache hit rates won't be anywhere near this high. Treat 20.05x as an upper bound.
The short-text runs cut the same corner. The 100 and 10,000 runs are one sentence with a counter appended. The 1.71 MB long document is that 171-byte text repeated ten thousand times. The top number, 26.33x, comes from exactly the most repetitive input. I'm a little embarrassed writing that down.
My absolute numbers don't line up with the README, and I didn't find out why. The README has the 9800X3D doing 5.22 GB/s on Qwen 3.5; my 5600 managed 344.85 MB/s, a 15x gap. The core count difference doesn't explain 15x, and the methods aren't identical either (I went through its CLI, and HF was only compared on the first 32 MB). I didn't chase it, so I only use my absolute numbers against HF on the same machine, never against other people's machines.
Correctness was only verified on what I tested. Token-by-token checks passed on every run above, plus four hand-picked cases: Chinese, combining characters, code with emoji, and chat special tokens. But at the time the repo had an open issue about Qwen 2.5 producing a different token order on certain Unicode combining sequences. Not hitting it doesn't mean it isn't there.
No end-to-end measurement. Everything I measured is tokenization alone. A real pipeline also reads from disk, decompresses, cleans and writes out. I didn't measure what share tokenization takes, so the "tokenization time" in section 3's formula is something you have to profile yourself.
I didn't break down the CLI's 0.07x. I can't tell you whether it's a one-time init or a fixed cost paid on every run.
5. Next time you see "N times faster," ask in this order
- What's the denominator, and on what machine. Change the machine and the old implementation's throughput can swing 8x.
- Did both sides get the same data, split the same way. Whole file versus the first 100 MB is already a different problem.
- Is the test data repetitive. Anything with a cache will look absurdly good on repeated input.
- What does your workload look like. One small request at a time, tens of thousands of short texts, one long document: the speedup ranged from 0.07 to 26.
- Compute the absolute number with
time saved = time of that step × (1 − 1 ÷ speedup), then compare it to the most expensive step in the whole job. - Before switching, check token-by-token on your own real data, and keep the old implementation as a fallback.
I spent an afternoon benchmarking a tokenizer and concluded I don't need it day to day. The day I have 100 GB to chop, it'll give me back an hour and a half.