Our pricing rules live in a shared Java library that four services depend on. Between releases its main branch publishes 2.3.0-SNAPSHOT to our Nexus repository, and the services' main branches depend on that snapshot so they pick up rule changes without waiting for a release. That arrangement had worked for three years.
In October an engineer working on a branch that changed how pallet surcharges round ran mvn deploy from his laptop, to try the change in a test environment. It published as 2.3.0-SNAPSHOT, because that is what the branch's pom said. Nexus accepted it, as it accepts every snapshot deploy from anyone with write access.
A snapshot version is not a build. It is a name for whichever timestamped build was deployed last. Maven resolves it by reading the version's maven-metadata.xml in the remote repository and taking the newest entry, and nothing in that file records which branch a build came from. The next morning the order service's main build resolved the branch build, passed its tests, because none of them covered surcharge rounding, and went to staging.
It was not even consistent. Maven's default update policy checks the remote for a newer snapshot once a day, and two of our six build agents still had the previous day's copy in their local repository. Services built on those agents that morning got the main branch code. The same commit of the same service behaved differently depending on which agent picked it up.
A tester noticed staging quotes a few pence away from production for a third of pallet orders. It took most of a day to trace, because every pom we opened said the same version.
Snapshots now publish only from the library's main branch, by the CI user. Developers lost deploy rights to the snapshot repository and test their branches with a version that carries the branch name. Builds run with -U, so every agent asks the repository rather than trusting a day old cache, and every service release runs the enforcer plugin's requireReleaseDeps rule, so nothing reaches production depending on a snapshot at all.
A version that anyone can overwrite tells you the name of the code, not which code it is.
– Sergey Shinder
Top comments (0)