Evidence Is Only Valid for the Build It Measured

Two answers to one question

The question was ordinary: what commit is the running service built from? There were two places to look, and they disagreed.

The container's environment carried a build variable naming one commit. The image itself carried a revision label naming a different one. Nothing warned that these had diverged, because nothing was comparing them; each was written by a different step and read by a different person.

We took the environment variable, because it was the one we happened to look at first, and it was wrong. The label was right. The gap between them was roughly ten commits of work by somebody else: sensor fields, harness gauges, connector provenance, a merged ontology page.

We never established why they had drifted, and that is worth saying rather than tidying away. What mattered was not the mechanism. It was that a running system offered two answers about its own identity and nothing in the process treated that as a defect.

What saved it was refusal

A fix was written on the wrong base and offered for deployment. The deploy script rejected it.

It does one useful thing: before deploying, it compares the revision label on the image it is about to ship against the checkout it was asked to build, and refuses when they differ. It also refuses an image stamped unknown, an image built outside the script, whose contents nothing can attest to.

So instead of a deployment that quietly reverted ten commits, there was a message saying the running service was built from a commit this checkout did not contain. The fix was rebased onto the commit that was actually deployed, and shipped from there.

The valuable property is that the check is mechanical and pessimistic. It does not ask whether the difference looks important. It refuses on any difference, including the ones a human would have waved through, which are precisely the ones that turn out to matter.

The second failure: evidence with no configuration attached

The first failure cost nothing, because it was caught. The second was ours and we reported it before catching it.

Having deployed a fix, we verified it: measure a workspace, restart the service, measure again, confirm the numbers no longer move. They did not move. We wrote that up as proof the fix worked.

While that test was running, another session built and deployed a different image over the top. The container was recreated underneath the measurement. The readings were real, but they were taken from a build we had not identified, and possibly from one that did not contain the fix at all. The result proved nothing, and we had already presented it as though it did.

The repair is not more care. It is recording the configuration as part of the measurement: read the image revision immediately before the test and immediately after, and treat the result as void if they differ. Restated, and it is uncomfortable how obvious it sounds:

Test evidence that does not name the configuration it was taken from is not evidence.

Why this is a systems engineering problem

This reads like a deployment story and it is not. It is the same failure that a verification programme exists to prevent, appearing in the infrastructure rather than in the product.

A verification activity produces a result about a configuration. Detach the two and the result is worthless: a passed test against an unnamed build cannot be used to close a requirement, because nobody can say what was tested. Every standard that governs this work says so, and every one of us believes it right up until the thing being configuration-controlled is a container rather than a product.

The parallels are exact.

• An image label is a configuration item identifier. Two of them disagreeing is a configuration control failure, not a nuisance.
• Refusing to deploy an unstamped image is refusing to accept an unidentified article.
• A test whose configuration changed mid-run is a test that has to be repeated, not explained.

A team that would never accept a test report without a build number will happily accept "we deployed it and it looked fine" about the tool producing the test reports.

The cheap version of all of this

None of the fixes are sophisticated, which is the argument for doing them.

Stamp the artefact, not the environment around it: an identity travelling with the thing cannot be left behind when the thing moves. Have exactly one authoritative answer to "what is running", and if a second appears, treat the disagreement itself as the bug rather than picking a winner. Make the deployment path refuse rather than warn, because a warning in a build log is read once and never again. And record the identity on both sides of every measurement.

The whole set is perhaps thirty lines. The alternative is a class of failure where everything reports success: the deployment succeeds, the service comes up healthy, the test passes, and none of it is about the code you think it is about.

What to take from it

• One source of truth for what is running. When two exist, the disagreement is the finding. Do not adjudicate it, remove it.
• Bind identity to the artefact. Anything asserted alongside a build rather than attached to it will eventually describe a different build.
• Refuse, do not warn. The check that saved ten commits worked because it stopped, not because it printed something.
• Stamp your measurements with the configuration. Read the build identity before and after, and void the result if it moved. Otherwise you will eventually report a passing test against a build you cannot name.
• Assume concurrency. If two people can deploy the same service, they will, and one of them will do it in the middle of your verification.