AriakiTen runs, sorted by accuracy. The real fly brain came seventh:
shufw_s3 0.8056 ← weights shuffled, fake graph
shufw_s1 0.7917
fullrand_s3 0.7847 ← fully random wiring
degpres_s3 0.7847
fullrand_s2 0.7778
shufw_s2 0.7639
connectome 0.7639 ← the real brain, 7th place
degpres_s2 0.7569
degpres_s1 0.7500
fullrand_s1 0.7222
Pool the nine fakes and compare: the real brain is -0.0069 behind them, p=0.85. No advantage at all, and the sign points the wrong way.
I spent three days on this, hoping to show the opposite. The worse part is that I can't use the loss either — I later worked out my own measurement precision and found the instrument can't resolve the gap I was hunting for.
Want the judgement rule? Chapter 2, it's one division. Want to know what actually does break this network? Chapter 5.
1. The bet
First, what's being measured. A connectome is what you get when you slice a fly's brain, image it under an electron microscope, and reconstruct every neuron and every wire between them. The public FlyWire female whole-brain release has 139k neurons and 54.5M synapses (Nature 2024). I used the 630 version that ships with the modelling paper: 127,400 neurons, 14,687,178 edges, 52,793,639 synapses, running in Brian2 on my own machine, each neuron the simplest possible leaky integrate-and-fire unit.
There's a tempting idea here: could this graph work as a reservoir? Reservoir computing goes like this — you take a huge messy recurrent network, don't train it at all, push your input through it, and train only a trivial linear readout on the output. The big network's job is to blow the input up into very high dimensions so that things which weren't linearly separable become separable.
The key detail: in the standard recipe that big network is randomly generated. The fly's was shaped by evolution.
So the question is clean: at the same scale, with the same degree distribution, does wiring tuned by evolution make a better reservoir than random wiring?
I measured one thing that made the bet look even better. In the real connectome, edges where A→B and B→A both exist make up 26.55% of connections. At its connection density (0.0905%), random wiring would give you about 0.09%; my degree-matched random graphs measured 0.56%. That's a 47x gap.
Reciprocal connections are local echo loops, and echo is exactly the physical basis for a reservoir holding onto "what just happened". So this looked like a lock: evolution left a pile of echo loops in there, random graphs have none, the connectome should win on a memory task.
I searched around and couldn't find anyone who'd published this comparison. Questions that sound like somebody must have done them already sometimes really haven't been done.
2. How coarse is one tick on the ruler: a division
This is the most useful paragraph here, and it's just a division.
My first setup: 6 classes, 8 repeats each = 48 samples, stratified holdout, 3 per class held out — 18 test points total.
Classification accuracy is discrete. With 18 test points you can only get 0 right, or 1, or ... 18. So:
smallest step in accuracy = 1 ÷ number of test points
18 test points → 1/18 = 0.0556
How big is the effect I'm looking for? In the reservoir literature this kind of structural advantage usually runs a few percentage points. My pre-registered rule was difference > 0.02 counts as a win.
A 0.0556 tick, measuring a 0.02 thing. One tick on the ruler is 2.8x bigger than the object. Under those conditions any "difference" you measure is just "got one more question right or one fewer", and it has nothing to do with whether the connectome is better.
When I spotted this my first instinct was to add more seeds and rerun. That wouldn't help — seeds add repeats, not test points. With holdout, your resolution is set by test-set size, no matter how many times you run it.
The fix is a different cross-validation scheme: switch to leave-one-out, so every sample takes a turn as the test point. 6 classes × 24 repeats = 144 samples = 144 test points:
1/144 = 0.0069
Now 2.8x finer than the 0.02 I'm hunting. That's the point where the experiment is allowed to start.
Go one layer deeper and it gets bleaker. To make a difference in proportions hold up statistically, the per-group sample size is roughly:
n ≈ p(1-p) × (1.96 × √2 ÷ d)²
p is the accuracy level, d the difference you want to resolve. Plug in p=0.76, d=0.02: 3,464 samples per group. What I actually ran was 144 per group for the real brain and 432 per group for each random family (three seeds pooled).
Which means: even if the connectome genuinely has a 2-point advantage, this experiment cannot see it. Single-group standard error is 0.0354, larger than the effect. So the "loss" in chapter 4 strictly only supports "no advantage larger than about 5 points". Nothing else follows from it.
You can lift this rule straight out: before any A/B, work out the smallest step your metric can take and compare it to the difference you're hunting. If the first number is bigger, the experiment is dead before it starts.
3. Why 100% accuracy is worthless data
Ruler fine enough. Second trap: the exam has to discriminate.
The task I designed: stimulate the network at 6 different spatial patterns of sites, then try to recover which pattern it was from the whole-brain firing response. First run, all four groups scored 1.000. Perfect.
A perfect score means no ranking is possible. Real brain 100%, fully random graph 100% — the experiment measured nothing. The cause was that my 6 patterns each lit up 300 non-overlapping sites, a difference so large any network could separate them.
So I built a difficulty ladder, turning three knobs at once: how many sites the patterns share (overlap, higher is harder), drive strength (lower means weaker, noisier responses), and readout window length (shorter means less information). The rule was pre-registered: accuracy must land inside (chance+0.1, 0.95) to count as discriminating. Chance for 6 classes is 0.1667, so the window is (0.267, 0.95).
| overlap | drive | window | accuracy | shuffled labels | verdict |
|---|---|---|---|---|---|
| 0.00 | 250 | 200ms | 1.000 | 0.211 | ceiling |
| 0.70 | 250 | 200ms | 1.000 | 0.200 | ceiling |
| 0.90 | 250 | 200ms | 1.000 | 0.233 | ceiling |
| 0.90 | 60 | 200ms | 1.000 | 0.189 | ceiling |
| 0.95 | 60 | 100ms | 0.833 | 0.211 | usable |
| 0.98 | 60 | 50ms | 0.278 | 0.178 | near floor |
One rung out of six is usable. Cranking overlap from 0 to 0.9 and cutting drive to a quarter left accuracy pinned at a perfect score; it took 0.95 before anything moved, and one rung further dropped it almost to guessing. The usable window is alarmingly narrow, and I only hit it on the fifth try.
That "shuffled labels" column is the other guard rail: shuffle the labels and rerun, and accuracy must fall back to chance. Across ten runs it averaged 0.159 against a chance level of 0.1667. If it didn't, my classifier would be peeking at labels and the whole round would be void.
4. Ten groups, and the gap that wasn't there
With difficulty fixed at overlap=0.95, drive 60, 100ms window, I ran ten groups: the real connectome, plus three random families at three seeds each.
The families: shufw shuffles weights across existing edges (shape untouched, only which weight sits where), degpres rewires while preserving every neuron's in- and out-degree, fullrand rewires completely at random.
The ranking is the table at the top. Pooled by family, Welch's t-test:
| group | accuracy | n | vs connectome | p |
|---|---|---|---|---|
| connectome | 0.7639 | 144 | — | — |
| shufw | 0.7870 | 432 | -0.0231 | 0.57 |
| degpres | 0.7639 | 432 | +0.0000 | 1.00 |
| fullrand | 0.7616 | 432 | +0.0023 | 0.96 |
That degpres row is +0.0000. An exact zero. The real brain and the degree-matched random graphs got precisely the same number of questions right across 144 test points.
The 47x reciprocal-edge advantage bought nothing on this task.
I should be precise about what this result is: not "connectomes are useless as reservoirs", but "at this one task, this one operating point, this sample size, I saw no gap". By the arithmetic in chapter 2, all this experiment can rule out is an advantage above roughly 5 points. A 2-point advantage is invisible to it.
5. What actually does move this network
A null result invites a wrong inference: that the network is insensitive to everything, so it doesn't matter what you change. Not so. In a separate ablation study it's very sensitive — just not where I expected.
Using a fixed readout neuron (the proboscis-extension motor neuron from the modelling paper, baseline 84.6 Hz) as a probe, two kinds of perturbation, three seeds each:
flip the excitatory/inhibitory sign on 20% of edges → 0.6 / 2.8 / 61.6 Hz
randomly delete 20% of edges → 47.2 / 34.8 / 80.8 Hz
Same 20% of edges touched either way, and flipping signs drives this neuron down to 1% and 3% of baseline while deleting edges does essentially nothing.
I tripped over something here worth calling out separately. My original rule said "silenced means firing rate == 0". By that rule, 20% sign-flipping produced 0/3 silenced — because 0.6 and 2.8 aren't zero. But 0.6 Hz is 0.7% of baseline; functionally that neuron is dead. A rule written as "equals zero" scores "1% left" as healthy. The correct threshold is below 10% of baseline.
Rung by rung the shape is clearer:
flip 1% → 81.8 / 78.4 / 93.0 Hz nothing
flip 5% → 83.0 / 61.2 / 85.2 Hz fine
flip 10% → 74.2 / 58.8 / 110.4 Hz one run above baseline
flip 20% → 0.6 / 2.8 / 61.6 Hz collapse
10% is the cliff edge. Below it you can flip freely — one run even jumped to 110 Hz, above baseline. Between 10 and 20 it falls off.
Why this matters: the wires and synapse counts in a connectome are measured — you can count them in the electron micrographs. Whether an edge is excitatory or inhibitory is inferred from a neurotransmitter type predicted by a classifier (Eckstein et al. 2024). So the one field in the whole graph that isn't a measurement happens to be the field that's most destructive to perturb. People say a connectome gives you "discounted weights"; the discount lands precisely on the worst spot.
One more finding that matters for anyone wiring a connectome to a robot: under a 20% sign flip, 82.5% of neurons brain-wide are still firing while the readout neuron is down to 1-3%. Your global metrics look fine and the hand you connected to those descending neurons has already stopped moving. The readout end is far more fragile than the population.
6. How to tell whether a "connectome simulation" demo is real
There have been a lot of these lately — fly brain hooked to a virtual body, walking, dodging shadows, grooming. An eLife reviewed preprint (2026-08-17, The digital sphinx: Can a worm brain control a fly body?) dismantled the genre in one move: the authors took the C. elegans connectome, hooked it to a fly body, and trained a mapping layer with deep RL in between.
The result is highly realistic fly walking. From the abstract:
The resulting digital sphinx produces highly realistic fly walking—yet it is biologically meaningless. ... behavioral fidelity is achievable without biological fidelity, making such models easy to overinterpret.
Looking right doesn't mean the thing inside is right. So for any connectome demo, ask one question: was that interface layer trained? If it was, however good the walking looks, all it proves is that the interface learned the job. The connectome isn't doing the work.
A few real examples, checked against their own READMEs and blog posts rather than secondhand write-ups. One widely shared desktop fly turns out to be a 668-neuron subgraph, 0.5% of the brain; its README states outright that the escape reaction time is not a live measurement, and that the cross-individual interface synapses don't exist in the data at all and were modelled in. A more serious whole-brain-plus-body project states that its visual activation is "somewhat decorative" and doesn't materially drive behavioural output.
Every layer admits its own simplifications; it's the secondhand retelling that deletes the caveats. When you see "X simulated Y", go read the limitations section of the primary README. Authors are usually an order of magnitude more honest than the people summarising them.
The same shape shows up in graph metrics. I computed the graph-reachable set from the 21 input neurons: 99.4% of the brain is reachable. Run the same stimulus through the actual simulation and only 0.33% of neurons fire. A factor of 301.
graph-reachable ÷ actually firing = 99.4% ÷ 0.33% ≈ 301
Pure graph analysis can prove "impossible"; it cannot predict "will it happen". Next time you see "there's a pathway between this region and that one", ask whether that was computed on the graph or observed in a simulation.
A more specific self-inflicted wound: I once wrote down "the pathway in the paper is A→B→C→readout neuron". Later checking showed that claim came from another project's README, not the paper, and I'd been calling it the paper's pathway. In both dataset versions that final hop has zero direct edges. The actual dominant driver is a neuron that appears in no secondhand description at all.
7. Where this experiment can't be trusted
Every number above came off my own machine, but they're not equally trustworthy, so here they are separately.
The ranking and p-values only have the power to exclude large effects. As computed earlier, 144 points per group can only resolve gaps above roughly 5 points. The hardest claim in this whole piece stops at "no large advantage".
Only one operating point was tested. The difficulty ladder swept 6 rungs, but the four-way comparison only ran at the selected rung. Whether a different overlap or window flips the conclusion, I don't know.
Only 3 seeds per random family. Group accuracies range from 0.7222 to 0.8056, an 8-point spread that clearly exceeds any between-group difference. Three seeds don't pin down that variance.
The classification task may not require memory at all. This is the one that worries me most. A reservoir earns its keep by retaining past inputs in the current state, and "identify which pattern was just stimulated" might be solvable from the instantaneous response alone. If so, I measured this network's instantaneous separability, which has nothing to do with evolution-tuned echo loops — the bet in chapter 1 may never have been tested at all.
An earlier memory-capacity round is void, though it also pointed down. I first ran a proper memory-capacity metric: the real connectome scored 0.000, while the weight-shuffled fakes scored 0.447 to 0.515. That looks like a rout, but every group in that round had a current-input correlation of 0.98-0.99, meaning the whole network was just an echo of the present input — the network was saturated, around 55 spikes per neuron per second, and every group's memory was crushed to zero. Invalid measurement. The root cause was that I twice used "fraction of active neurons" as my operating-point criterion, and activity only counts whether a neuron fired, not how hard it's firing. That round is not evidence, but it didn't help the connectome either.
Regularisation strength was selected on all the data. The outer leave-one-out loop is clean, but the inner choice of ridge penalty used the full sample. That treats every group identically, so it can't bias the comparison, but it does inflate every group's absolute accuracy slightly.
Until these are closed out, I won't state the conclusion firmly.
8. Next time you run a comparison like this, in this order
- Compute your metric's smallest step first.
1 ÷ number of test points. If it's coarser than the effect you want, don't run — redesign. - Then compute the sample size you need.
n ≈ p(1-p)(1.96√2/d)². If that's unaffordable, either raise the effect size you're willing to detect or admit up front that you can only exclude large effects. - Sweep a difficulty ladder on the experimental group only. Accuracy must land inside (chance+0.1, 0.95) before you compare anything. Perfect scores and chance-level scores both produce no ranking.
- Run a shuffled-label control every round. It has to fall back to chance. If it doesn't, your classifier is peeking and the round is void.
- Never write a threshold as "equals zero". Write "below X% of baseline". 0.6 Hz and 0 Hz differ statistically and are identical functionally.
- Healthy global metrics don't mean a healthy readout. With 82.5% of the brain still firing, the neuron you actually care about can be at 1%. Watch the thing you're actually reading.
- Read the paper before citing "the paper says". I called a README's claim "the paper's pathway"; the final hop of that pathway doesn't exist in the data.
- Prove your operating point is valid before interpreting anything. When the network is saturated, every group's memory is zero — that's a failed measurement dressed up as a finding.
As for whether a connectome makes a good reservoir, all I can say right now is that at my precision I can't tell. Seeing it would take about three thousand samples per group, probably a dozen-odd hours of compute. I'll get there after I work out whether the task needs memory at all — otherwise three thousand samples just measures the wrong thing more precisely.