A Green Light That Proves Nothing

A write returned 2xx. Zero rows were written.

No error. No warning. No exception anywhere in the stack. The client asked for an insert, the server replied with success, and nothing at all had happened.

What was going on

This was PostgREST over Postgres with row-level security. On a fixture whose policies had been switched off and back on by hand, an insert was being silently reduced to zero rows by policy, and the response was still a success.

I want to be careful about the claim. I did not fully characterise that tampered state and I am not asserting a cause for it. A fixture rebuilt cleanly from the schema behaves correctly.

What generalises is narrower and, I think, considerably more useful:

A success response is not proof of a write. A store is proven by reading its own writes back, and by nothing else.

The cascade this caused

The part that actually cost time was not the bug. It was what the bug did to my reasoning.

I had written a verification script: eight checks that ask a backend, before anything is wired to anything, whether it is correctly configured.

Table 1. Each check names the specific setting to change if it fails. A diagnostic that does not tell you what to do is a complaint.

The first version of that script trusted check 4. It saw 2xx and moved on.

So when check 5 failed, the script was reasoning from a false premise, and it produced three cascading misdiagnoses in a row. Each internally consistent. Each confidently worded. At one point it blamed the append-only triggers for what was an insert problem, which is a completely plausible hypothesis given what it believed, and I nearly acted on it.

That is the real damage. Not the wrong answer. The confident wrong answer, delivered by an instrument I had built specifically to be trusted.

The script now stops at the read-back. The write is not believed until its result has been read back, event for event. Eight of eight against a real fixture, with real Postgres, real schema, real policies and triggers.

The general failure, which is not about databases

The database is incidental. The shape is everywhere:

A check that confirms the request was accepted, and reports it as though the effect had occurred.

// Proves the function was called
expect(saveSpy).toHaveBeenCalledWith(item)

// Proves something happened
const saved = await store.get(item.id)
expect(saved).toEqual(item)

// Proves the function was called
expect(saveSpy).toHaveBeenCalledWith(item)

// Proves something happened
const saved = await store.get(item.id)
expect(saved).toEqual(item)

// Proves the function was called
expect(saveSpy).toHaveBeenCalledWith(item)

// Proves something happened
const saved = await store.get(item.id)
expect(saved).toEqual(item)

Once you are looking for it, the first pattern is in most test suites. And it produces the specific category of test I have learned to be most suspicious of: the one that would pass even if the implementation were wrong.

The question I now ask of every assertion

If the implementation were broken, would this test actually fail?

If I cannot name a concrete wrong implementation that this assertion would catch, the assertion proves nothing. And its green tick is worse than no test at all, because it occupies the space where a real test would have gone, and it does so in the reassuring typography of a pass.

The ones I have caught in my own work

Written out because they are embarrassing and therefore useful.

• Asserting that something is absent in a state where absence was already guaranteed by construction.

• Comparing two states that turn out to be identical, so the comparison could never fail.

• Checking a count without checking what was counted.

• A test that passed identically whether a feature was enabled or disabled.

• An assertion that passed for a reason other than the one it claimed. A whitespace guard was never exercised because the input threw earlier in the function, so the guard had never run once in production or in test.

That last one is my favourite. The test was green for weeks and the behaviour it claimed to protect had never been executed.

The instrument that actually works

There is only one method I have found that reliably distinguishes a real suite from a decorative one, and it is not a coverage number.

Break the implementation on purpose and require that a specific test catches it.

INVERSION HARNESS

  for each known invariant:
    1. introduce a specific, realistic break
    2. run the suite
    3. REQUIRE at least one test to fail
    4. record which test caught it
    5. restore, and verify the restore

INVERSION HARNESS

  for each known invariant:
    1. introduce a specific, realistic break
    2. run the suite
    3. REQUIRE at least one test to fail
    4. record which test caught it
    5. restore, and verify the restore

INVERSION HARNESS

  for each known invariant:
    1. introduce a specific, realistic break
    2. run the suite
    3. REQUIRE at least one test to fail
    4. record which test caught it
    5. restore, and verify the restore

The project currently holds 81 such inversions.

The interesting statistic is not 81. It is that in the most recent batch, four were missed on the first run, and all four were my test's fault rather than the code's. The breaks were real. The tests that should have screamed stayed green, and had I not been deliberately breaking things, I would never have known.

Four out of nine. That is roughly the rate at which my own assertions are decorative, measured rather than guessed, and it is the single most useful number I have about my own work.

Why test counts are worse than useless

This is why I do not report test counts as an achievement, and get slightly irritated when others do.

A count tells you how much machinery you have. It says nothing about whether any of it would notice a failure. Two suites of 300 tests can differ by a factor of ten in how many real defects they would catch, and nothing visible from outside distinguishes them.

The only number that means anything is how many deliberate breaks your suite catches, and the only honest way to know that is to break things on purpose and watch what happens.

One more restraint worth mentioning. The restore step has to be verified rather than assumed. A harness that breaks a file and restores it from memory can leave the repository in a state nobody noticed, and then every subsequent result is about a codebase that does not exist anywhere. The restore writes its undo to disk before touching anything, because signal handlers do not run when a process is killed outright.

2xx means the message arrived. It does not mean anything happened. Everything above is a consequence of taking that sentence seriously.

Related reading

• Three Days Ago, Not the Twelfth of September

• Never Create a User Action So You Can Measure It

• Two Bugs That Only Korean Files Could Find

• The Thing You Can't Reconstruct

More from the Moc Mallen Studio journal