Me: "It's only a one line code change." Release Manager: "OK, we'll push it through."
This was one of the most memorable statements of my early career. I was confident in my fix for this urgent customer-facing issue. There was pressure to get it out quickly, so as a means of accomplishing that goal, I threw out the infamous statement above. It worked. Management approved the fix, and it went out the door. What I didn't know at the time was that my upcoming free weekend was going along for the ride.
Yes, my fix addressed the customer's issue. Worked great -- for that particular scenario. Unfortunately, this was code within a complicated billing system that handled a wide variety of business rules. Unfortunately, my change disturbed the handling of a very important business rule. As a result, thousands of customer utility bills went out incorrectly over the course of the next few days.
Had we taken the extra day of sending this through our standard set of tests, the problem would have been caught. It was only one line of code, so why bother, right? In fact, if my memory of the situation is correct, it was only a one character code change.
The end result was as follows:
- A corrected fix had to be made
- The testing that I initially avoided had to be performed anyways
- Not only did I have to work the weekend, but others were brought in as well.
- Once the problem was fixed, I had to spend quite a bit more time fixing data and verifying that that data fix didn't make it worse.
What I learned:
- Changes being sent out as hot fixes need to have a gatekeeper.
- The gatekeeper has to not accept the "It is one-line code change" answer as justification for releasing the change. They should ask: What has changed? What problem does it solve? How was it tested? What could this change break -- what can go wrong? If the answers satisfy the gatekeeper, then let the change go out.
- Risks of any change should be weighed and appropriate remediation steps need to be performed. If a critical area is being touched, then some testing of that area needs to occur to provide at least some comfort that we aren't about to make things worse.
- If a one-line change can do this much damage, imagine the potential of bigger changes. When in a hot-fix environment, keep change to a minimum.
There is a lot of pressure to fix every urgent issue as quickly as possible. It is easy to send a fix through just to get the urgency to go away. It only takes one of these one-liners to ruin your weekend and change your perspective on how to go about changing software.