The day a minor dependency patch took down a client's checkout
About eight months ago we updated a payments library on an online shop for a client that sells kids’ clothes. It wasn’t even a major version bump: just an npm update, 4.2.1 to 4.2.3, two patch releases. The changelog said “bug fixes and performance improvements,” that phrase that tells you nothing. We shipped it on a Thursday afternoon without thinking twice, because we’d been updating that way for months with no issues.
A minor dependency patch can take production down without the changelog ever warning you, especially when it touches timeouts or internal response formats the library doesn’t consider part of its “public interface” — which is why we now treat updates to payments, auth, or anything that sends data outward with more care than a UI component. That’s exactly what happened to us: Friday at ten in the morning the client wrote to us because the “Complete purchase” button was spinning forever. Not on every order, only the ones paying with a specific mobile payment method. It took us almost three hours to find that the patch had silently changed the format of an internal timeout, and on slow connections the SDK died before it got a confirmation back. None of that was in the changelog. None of it showed up in our tests either, because our checkout tests hit a mocked gateway that answers in 40 milliseconds, not the 4 real seconds you get on a phone with bad coverage out in some small town.
We changed the policy after that, though it took me a while to admit we needed to. Well, more than a while, it took me time to accept the problem wasn’t bad luck, it was the process. Now any library that touches payments, auth, or sends data to a third party gets updated by hand, tested on a real device with network throttling turned on, and waits until Monday. Everything else (UI components, small utilities, things you’d notice at a glance if they broke) still goes through the usual automated flow.
That’s not a popular stance among people who insist you should always run the latest version for security reasons. And they’re not entirely wrong: sitting on patches has its own cost too, especially once the delay stretches from days into months. But four years of keeping client sites running has convinced me that “it’s in the changelog” and “we’ll notice” are two very different things, and that what decides how fast something gets updated should be what breaks if it fails, not how easy it is to automate.
What we still haven’t figured out is what to do about transitive dependencies, the ones that don’t even show up in our package.json, pulled in by some second or third level library. Dependabot doesn’t even open a PR for those half the time, and the failure shows up with no warning, same as when something breaks in production and you find out from an alert instead of the changelog.