Three checks lied to me in one day. Two said I had failed.

A check script told me that none of the candidate parameter settings qualified. The experiment could not proceed.

I bought it. I started changing the experimental design.

Later, I took the gate's own criterion and worked through the sixty rows it had saved. Fifteen qualified. The criterion evaluated to True. The script had marked them False. No error, no warning. It had quietly crossed out fifteen correct answers.

I hit two more that same day. One script reported that everything passed, but I had read the wrong column. The numbers for sixty-six channels all belonged to their neighbors. Another said I had data leakage. Its check was looking at a field that meant something quite different from what I thought it meant.

Two of the three were false fails: the check rejected something that was actually fine.

I used to guard against only one direction. Most writing about testing and validation focuses on that direction too: watch out for checks that pass too easily. But the bill for a false fail is harder to reckon with. It makes you abandon a valid direction. Then you redesign around the wrong diagnosis and break something that was working.

If you just want the self-check code, skip to section 4. It is four lines.

1. The first kind: the wrong column, and a pass

I was analyzing a public dataset of fruit-fly olfactory receptors. I wanted to know how many odors had been tested against each receptor.

The CSV had a trap: 78 columns in the header, 79 in each data row. The first cell of every data row held the odor's chemical identifier. The header had no name for that cell.

So when I used row[j] to get the value for receptor j, I actually got the value for receptor j-1. Everything was shifted by one. All 66 affected channels were wrong.

The script finished normally. I sent out the results. I even singled out two "beautiful anchors":

Or22a  I reported 497 odors  →  actual 225
Or67d  I reported 161        →  actual 0

Or67d belongs to the fruit fly's sex-pheromone pathway. I had specifically described it as "the most thoroughly studied one." That channel did not have a single populated cell in this dataset. I had assigned its neighbor's numbers to it.

Worse, I had a chance to catch this on the spot. The dataset contained 691 odors in total. I was reporting that Or22a had been tested on 497 of them. A single receptor supposedly covered more than seventy percent of the odor library. That number looked unreasonable. I saw it and kept going.

During the fix, I added one line:

assert 0 <= coverage <= len(odor_rows), f'{r} coverage {coverage} exceeds total odor count; column index may be off by one again'

The assertion costs nothing. It can catch every such shift: after a one-column shift, some channels' counts necessarily fall outside the legal range.

Before selecting columns by position, align the header against one real output. That is familiar advice. The other half deserves emphasis: add an order-of-magnitude assertion to each key metric. Header alignment depends on remembering to do it this time. The assertion needs no prediction about what might go wrong. It puts a physical boundary around the data itself.

2. The second kind: the field meant something else

I was testing whether a learning model was cheating. The standard check is to shuffle the labels and retrain. If the model can still perform well, information is leaking.

The check returned 0.94 to 1.00. Those are disastrous numbers. A score of 96 after the labels have been shuffled can only mean leakage.

But that conclusion did not fit my understanding of the code. I opened the implementation and found two saved fields:

shuffled_training_recall   0.94~1.00
original_label_accuracy    0.46~0.58

The first means: train on shuffled labels, then test recall against those same shuffled labels. It reproduces what it learned. That number should be high. It measures the model's capacity to memorize, rather than label leakage.

The second is the leakage check: train on shuffled labels, then test against the original, correct labels. Recovering the real pattern from corrupted training would count as leakage. The measured accuracy was 0.46 to 0.58, right at guessing level. The check had passed cleanly.

The criterion had been attached to the first field. A cleanly passing self-check was declared a failure. The conclusions of the entire experimental round were thrown out.

This mistake is hard to spot because both fields were treated as some kind of "shuffled something." Both were legitimate diagnostics. Both were computed and saved. Choosing the wrong one produced no signal. Neither quantity was itself a bug. They answered different questions.

The criterion for writing a gate: put a one-sentence comment beside the criterion explaining which question the field answers. If you cannot write that sentence, you have not yet worked out what you are measuring.

3. The third kind: the gate added a requirement

This was the expensive one.

I needed a difficulty gate. A candidate parameter setting had to put the average accuracy for each of two input types between 0.55 and 0.95. Above that range, the task was too easy to reveal a difference. Below it, everything failed, which also concealed any difference.

The implementation added another condition: every individual cell also had to exceed 0.55.

An average and every cell sound close. Their effects on eligibility are very different. A parameter setting can have a qualifying average of 0.62 while containing a cell at 0.53. It passes the criterion I asked for. The implemented criterion kills it.

Out of sixty candidates, fifteen died this way. The script concluded that "this sub-experiment cannot achieve discriminating power under these settings." That sounded like a fundamental problem with my experimental design. In fact, suitable operating points had been sitting in the saved data all along.

I found it by doing one extra thing: recalculate the gate's own criterion from the saved data, then compare the result with its recorded flag, row by row.

Qualified on recalculation: 15 rows
Recorded as qualified: 0 rows
Disagreements: 15/60

Fifteen disagreeing rows. Zero exceptions thrown.

4. A four-line self-check

All three mistakes have something in common: a gate is code, code has bugs, and a gate's bugs need not raise errors. It can quietly issue a judgment that disagrees with the data.

So the gate needs a check of its own. The method is plain, but four lines are enough:

rows = json.load(open('calibration.json'))
mismatch = [r for r in rows
            if bool(gate_expr(r['measured'])) != bool(r['gate_flag'])]
assert not mismatch, f'{len(mismatch)}/{len(rows)} rows disagree between criterion and flag — the gate implementation has a bug'

The important part is to write gate_expr again, independently. Do not import the function from the gate. You are checking whether the implementation departed from the intent. Reusing that implementation would make it its own proof.

Run this when the gate reports FAIL. After a PASS, you look at the results. That gives you a natural opportunity to notice something wrong. After a FAIL, you go straight to changing the design. The mistaken judgment may never get reviewed.

These four lines cost a few seconds. My mistake cost an entire experimental round, plus a valid direction I nearly abandoned.

5. What the experiment eventually found

The checks served a specific question. In the circuit responsible for olfactory learning in the fruit-fly brain, the mushroom body, does the particular wiring pattern contribute anything?

I replaced the real connections with degree-preserving rewiring. Each neuron kept the same number of edges. Only who connected to whom changed. Everything else stayed fixed. Then I checked whether performance dropped.

I ran six rounds. The first five were ties. Each time, I could find an excuse: the task was too easy, or the inputs were synthetic data I had made up.

The sixth round closed off those excuses. It used public, real olfactory receptor-response data: measured responses to 70 real odors across 34 receptor channels. I used the literature to map receptors to anatomically correct projection neurons. The mapping covered 44 of 51 glomeruli, or 86%. Learning used the biological local dopamine rule, rather than gradient descent. Difficulty calibration confirmed that performance had moved off the ceiling.

The results:

Real odors, real wiring vs degree-preserving rewiring (after multiple-comparison correction)
  Memory capacity           +0.0457   not significant
  Similarity discrimination -0.0333   not significant
  Noise robustness          +0.0100   not significant

Only 1 of 39 tests was significant, and it was unrelated to the real topology

Still a tie.

This does not mean biological connectomes are useless. It means the marginal contribution of that particular wiring pattern was not measurable in this model, with this learning rule, on this task. That is the most honest statement I can make. Anything stronger goes beyond the experiment.

6. Where this round is not trustworthy

The olfactory data are consensus values integrated across laboratories, rather than raw measurements from a single experiment. Recording conditions differ between laboratories. Combining their results brings assumptions of its own.

The receptor-to-glomerulus mapping came from the literature. It was not an annotation supplied with the connectome dataset. I have not independently verified every mapping.

Of 686 projection neurons, only 155 were actually driven by real data. I set the other 531 to zero. That is a modeling choice, not a biological fact.

There were five paired seeds. A parametric test with n=5 is fragile. Multiple-comparison correction further reduces detection power. Nonsignificance does not establish equivalence.

The most important limitation is the last: I wrote these check scripts. That very day, I had just demonstrated that I could write them wrong. The four-line self-check in section 4 only tests whether the implementation departed from the intent. It cannot tell whether the intent itself was right.

7. Next time, write the gate in this order

1. Wherever columns are selected by position, align the header against one real output first. Record the column order in a comment.

2. Add an order-of-magnitude assertion to every key metric. Coverage cannot exceed the total. Accuracy cannot exceed 1. Latency cannot be negative. This lasts longer than header alignment because it does not depend on whether you counted correctly this time.

3. Beside each criterion, write one sentence explaining which question its field answers. If you cannot write it, you have not thought it through yet.

4. When implementing the criteria, read them against the requirements one by one. Check that you have neither added nor omitted a condition. An extra floor and a missing ceiling both turn the gate into something different from what was requested.

5. When the gate reports FAIL, recalculate the criterion from the saved data using an independently rewritten expression. Compare it with the recorded flags. Four lines of code.

6. Do the same when it reports PASS. Give it slightly lower priority, but do not skip it. A PASS at least has downstream results that can help catch the mistake.