Balancing Technical Debt with Business Constraints

– The toughest task for the CTO?

In software engineering, balancing technical debt with business constraints is possibly the toughest job for the  Chief Technology Officer (CTO) or VP of Engineering.  On one side is the natural desire and logic to perfect the app before releasing it to the market, while on the other side is the accumulating pressure to launch, meet deadlines, get the cash flowing in.  As a CTO, I need to find the balance, and that is not anywhere near as simple as it sounds.  Sometimes it can feel like that scene on Star Wars where the heroes find themselves stuck in a giant garbage compacter with some kind of creature that wants to eat them…  Pressure is coming from everywhere!

Ralf Westphal says, “There is no such thing as technical debt.”  Bliss on Twitter says, “Ha! there’s no such thing as a “technical debt-free code base!”. Sadly, Mr. Westphal’s argument turns out to be a “straw man” argument and the Bliss quote is correct. Technical debt IS a reality.  Business constraints are a reality.  Both of them often seem to be huge walls closing in from either side.

Let’s take a look at a couple of definitions.  What is technical debt and what are the constraints that I, as the CTO have to work within? The constraints are a little more obvious.  Deadlines, money, time, pressure from sales directors, shareholders, upper management, etc., all put a lot of pressure on a CTO to get the product out where it will generate revenue.  On the other side, the coders keep saying it’s not ready.  (What they really mean is, it’s not perfect… more on that later). Technical debt is a little harder to define.  That’s not so much because it’s so complicated, but more because so many people have so many ideas of what it means. Basically, any debt is something you take on so that you can buy or do something before you have the means on hand.  In other words, you borrow against the future, banking on your ability to pay for it when the time comes.  Furthermore, debt is something that has to be paid back, just like in the financial world.

Technical debt, in a nutshell, is what the coders need to do but don’t do before releasing an application.   Or, as Lev Lesokin so eloquently put it, “Technical debt refers to the accumulated costs and effort required to fix problems that remain in code after an application has been released.”  The reality, this is in every industry, but they just call it by a different name.  In the automotive industry, they call it  the warranty period.  Basically, when automotive manufacturers send out a car, they have already set an expected warranty failure rate, and in fact, they send out product KNOWING it has a problem, but decide they can live with the cost of fixing it later in the field, more than they can live with a delayed launch.

Furthermore, potentially catastrophic technical debt is built right in if there is a bad product design due to a lack of vision and understanding of the purpose of the product, even if the coding is excellent.  Finally, it must be mentioned that a CTO often finds himself saddled with the legacy of old and terribly outdated product(s) left behind by his predecessor. This isn’t always immediately obvious, but is something that must be anticipated and looked for.

As a CTO, my job is to balance this technical debt with the constraints put on me, as outlined above. In a small software company with only a handful of coders, where everyone knows one another, it can be a lot easier to control technical debt and keep everyone happy.  When everybody knows everybody else, from the coders to the CEO, the lines of communication are more open, and even more important, the coders each know what’s going on.  In many organizations, however, there’s a lot more going on, and the pressure on the CTO is huge.  According to CAST Research Lab, the technical debt of an average-sized applications is about one million dollars!  Remember, debt has to be paid back! If technical debt gets ignored, as with any debt, the day of reckoning will come, when the debt is called in.  This can be catastrophic.

As an example, in my last experience, I had to deal with a 20-year old product including 6,000 bugs (for 7 million lines of code). Most of them were light, but some were very serious. I couldn’t stop to deliver new versions to customers, so we did a campaign to gain a clear status of the possible debt by mapping in a matrix the debt based on the features of the product. For each cell, we defined the risks (probability of occurrence, gravity). Then quoted the main issues using the 80/20 rule. We shared this status with all those involved (sales, support,etc.) and decided together on a strategy to fix it, taking into account constraints from sales and support teams.

The problem is, as Ward Cunningham said, “The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a standstill under the debt load of an unconsolidated implementation, object-oriented or otherwise”. Technical debt increases, not only because the “bugs” and “not-quite-right” code is building up, but also because, at the same time, software and the platforms it runs on are constantly evolving. I must make sure that there is cash flow for my company, or it can’t continue.  I must make sure that product deadlines are met as closely as humanly possible.  This means getting code to market in a timely manner, and that can sometimes mean shipping it before it’s fully tested in order to meet deadlines or to get some desperately needed cash in order to continue with other projects.

In other words, I am justified in telling my crew to do what it takes to get the job done by the deadline, because they’re going to have to fix things and update things on an ongoing basis, anyway.  Furthermore, they need to know that no matter how good they think they are, there ARE going to be things they didn’t think of that show up in the field, and spending endless hours double checking everything might serve only to make the software out-of-date before it hits the real world.

On the other hand, if I push them to just get it out there without giving them time to at least run automated testing to eliminate as many defects and bugs before shipping, it’s going to come back and bite me, the CTO.  Not only am I going to have angry customers, but the coders are going to be yelling at me and telling me it’s my fault, and my boss (and maybe the shareholders) are going to be yelling at me for taking shortcuts (even though they put pressure on me to “git ‘r done”).

So, there are constraints on both sides.  The first ones are the deadlines, usually created by the sales staff when they promised delivery by such and such a date. Then there’s the issue of cash flow.  Overhead is expensive, and investors want a return on their money.  There is a lot of pressure on a CTO to ship.

In smaller companies, I could deal with this to a certain extent due the high visibility of everything.  Since everyone knew everyone, we could talk about what was happening, make suggestions, pick up the pace where necessary, prioritize, and so on.  But in a larger company, how do you scale visibility into your technical debt?  Think again about the fact that the technical debt of an AVERAGE-SIZED application is one million dollars.  That translates into a LOT of coders at work.  How can I, as the CTO, be sure that everything is going to be ready in time, that it’s all going to work on time, and that when it ships, it’s not totally top heavy with technical debt that going to bite me, later?

I need to think of myself less as a manager, and more as a referee.  Upper management, shareholders, sales manager, etc., are the constraints on one side, while coders with tunnel vision, perfectionist tendencies, slow fingers, etc., are the constraints on the other side.  I need to make sure both sides get along and play nice.

In conclusion, technical debt is a reality (inherited or not), something that I, as the CTO, have to expect, live with, and also deal with in a timely and efficient manner.  The main thing that’s still the main thing that all of us from coders, to me to the sales teams,upper management and shareholders must remember is, we all work together to make the customer happy.  When we achieve that, everything else will work out, and I have successfully achieved the balance between technical debt and business constraints.

Jean-Christophe (Jay C)

PS. If you want to read more about the technical debt, I’ve found a very nice post by Michal Dymel, who talks about it from a perspective of developer.