Managing Tech Debt in the Modern Development Shop

Tech debt exists in any modern development shop. No project is budgeted or given a timeframe to afford to release a product with absolutely no technical debt. It just doesn’t happen. Time to market and cost to develop almost always are the priorities over code perfection. But that doesn’t mean you release a smelly mess. It means you manage the technical debt.

Just like financial debt, the technical debt must be managed, and that is what I am going to talk about in this article.

What Is Tech Debt?

Technical debt is something that accumulates when design or implementation decisions are made that sacrifice code perfection in order to fit into time or cost constraints.

Before I was a web developer I was a project manager for over 20 years, so I am quite familiar with the challenges of managing against the triple constraints of quality, time, and cost. In my view, developers tend to really struggle with this.

Managing quality against time and cost doesn’t mean you produce bad work. It means you work smartly. You understand the requirements and scope and ensure you aren’t over-delivering. You avoid recreating wheels that don’t need to be recreated. Most importantly, you concede that every piece of code can’t be a candidate to publish as a textbook example of your coding brilliance.

Here is the formal definition: https://en.wikipedia.org/wiki/Technical_debt

Why Is Tech Debt Always Present?

In a perfect world, none of us would have mortgages on our homes, but we do. Why? Let’s first ponder “what if mortgages didn’t exist?” Wouldn’t the cost of homes be lower? The answer is yes.

Even if you personally do not believe in the concept of a mortgage, because the majority of society is willing to sacrifice a good chunk of their future income on a mortgage, that drives housing costs up, right?

Because the mortgage is embraced by the ecosystem, it means if you want to buy a home you have to mortgage your future.

Similarly, if the ecosystem of developers embraces the concept of accumulating technical debt to fit into the time and cost constraints, can you really be the developer that stands firm and says “schedule and budget be darned, this code is done when I say it is done?” Probably not.

Let me be clear, I am not saying it’s a necessary evil. Well, I guess I sort of am saying exactly that. The key point of clarification is this — it doesn’t mean bad code gets released into production!

So what does it mean? It means we all have to understand what “good code” and “bad code” mean in the context of an ecosystem where technical debt isn’t avoided at all costs, and instead is managed effectively. Let’s talk about what that means.

Awareness & Decision Points

First, development teams need to be aware of when they are potentially accumulating technical debt. When we are at a point of making a decision, we should consider the two possibilities:

• If we were to develop to textbook standards and to an infinite scale, how much time would that take and what would be gained by that time?
• If we take a more “pragmatic” approach, what do we gain and what does that do to future work on this product?

The team must quantify the pros and cons of each path, and then the decision comes easily.

Technical Debt-Driven Development (TDDD)

I will create a buzzword right here and now, and I will call it Technical Debt-Driven Development”, or TDDD for short. In TDDD, we manage a bucket of technical debt which has accumulated, and we are aware each time we put more debt in that bucket and we have considered the trade-offs. Most importantly, when we have to modify a software product we are aware of the technical debt affecting each component and we take that into consideration.

Debt Inventory

In TDDD, we take inventory of the technical debt that we have inherited and as we continue to rack up more debt, so we understand our technical debt posture and can make better decisions. Perhaps that legacy c#/.Net app that was developed 8 years ago and still is used by the business. Or that wen app riding on PHP 5.6 carries some technical debt.

Yes, we have to take inventory and understand everything we have inherited and the associated technical debt. It isn’t just about time and cost. For example, I just said something that should have peaked some brows — a PHP 5.6 web app in 2022? Is it public facing? Why yes it is.

For those that aren’t aware, PHP 5.6 went end-of-life (EOL) about 6 years ago. That means the app is riding on a tech stack that likely has hundreds of security vulnerabilities which will never be patched.

How does this PHP 5.6 example fit into the TDD model? It’s a security risk, so how is that OUR debt? The answer is simple….. have we made that security risk known to the right people? If not, it’s OUR debt until we do, and it is debt that can cost a lot of jobs.

Poorly Managed Tech Debt

This PHP 5.6 web app example is a good example of poorly managed technical debt. It should have come onto the organization’s risk radar when PHP 5.6 was approaching end-of-life. It was a lifecycle management failure if at that point it was not considered a business risk.

And if it was our development shop that released that product 8 years ago, we have a hand in failing to manage the lifecycle.

Tech Debt Impact Changes Over Time

As the PHP 5.6 example shows, technical debt can change over time. This is another reason why it is a good idea to maintain an inventory of your technical debt and re-assess the impact as time goes on. It was not the same problem in 2016 when PHP end-of-life was on the horizon.

Tech Debt: Internal vs External

The PHP 5.6 example is a good example of technical debt that starts out internally, within the development team, but then grows eventually into a business risk that rises to the highest levels.

While technical debt can rise from internal to external, it should be a development shop’s goal to contain technical debt before it grows into an external risk.

So how did the PHP 5.6 business risk originally start out as an internal technical debt? It may have not even been a technical debt item at all. It wasn’t a shortcut. It may have made good sense to develop on PHP, but for some reason, the development shop made a conscious decision that PHP was no longer the preferred platform, and hence they did not invest time to migrate it to PHP 7 or another tech stack.

At that point in time, whether they realized it or not, they accumulated a big item of technical debt and one which would certainly grow to become a business risk. Any time end of life comes into play, and no one is actively managing the lifecycles of past products, this will be the end result.

An Example of Internal Tech Debt: Algorithm Development

Let us talk about another example and one that doesn’t necessarily evolve into a huge business risk. Say, for example, a business requirement emerges to develop a complex algorithm. For the sake of simplicity, let’s take an example every developer knows: the algorithms that are typically part of pre-hire testing. Let’s assume that the business requires something that is a variant of the “number of islands” problem.

In these algorithms, and probably all of them, there are multiple ways to get the desired end result. Generally speaking, there will be a number of “brute force” solutions, but one really jazzy and elegant “optimal” solution.

We are at the point where we need to decide whether are we looking for the optimal solution, or will a brute-force solution does the trick? Why wouldn’t we just always say that optimal is always the goal?

Well, one reason is that minimizing development time is a priority over computing resources. Other factors will be how readable and understandable the code is, which will impact future modifications.

Let us also assume that this specific algorithm hasn’t already been solved. Because in the real world, the first thing we would do is look for libraries and other similar problems and solutions. That goes to the “avoid creating wheels that don’t need to be created” statement I made earlier. Let’s just say that there are no libraries or similar problems/solutions to be found.

It is rare that one is going to just conjure up the optimal solution, and very likely that a brute force method will come first. After that first solution is discovered, and tested/validated to work, you are then at a point to decide “am I done here“.

You should be able to look at the algorithm and understand its runtime in terms of “big O”. If O is 1, or logarithmic, you might very well decide that you are done. But if O is exponential, your instinct will probably tell you there is optimization to be discovered if more time is invested.

It is at this point you are making a debt decision, and if you decide not to invest that time, you just took on some technical debt. You probably inserted a comment into the code “optimization probably possible, but will be involved and not feasible at this point”.

That is a technical debt decision being made, and one that won’t likely grow to affect anyone beyond your team. It may come back as an issue down the road when business volume requires the solution to scale, and that brute force algorithm can’t work within the time constraints at that scale.

Tech Debt Team Etiquette

If I am the developer that just made that technical debt decision, to call it “done”, how should that be handled with my team? The answer is, “it depends”. It depends on how your team views the matter, and what the “technical debt appetite” is of the team.

In my view, developers should be empowered to make decisions of this nature, as it is not feasible to convene the team to ponder each possible path.

Teams can grind to a halt if no one is empowered to make decisions of this nature. That said, each team member should be aware and document that a technical debt item was assumed. Perhaps standardizing on comments beginning with “// TDDD: brute force is fine for now, but could be an issue for future volumes/scalability.”

Perhaps a technical debt register is maintained, and each developer adds to that register, and then other team members can be aware, and assert themselves if they disagree.

Documenting Tech Debt

As technical debt is racked up and appears on a register, that helps the team understand everything that is “out there” which can impede future work. A lot of “it’s fine for now but may not scale” debt might warrant further investigation.

That said, technical debt isn’t necessarily bad. We want to be sure that we develop good solutions for the volume of today and the near future, but we don’t overdevelop to scales which will never happen. It’s a variation on the “you ain’t gonna need it” YAGNI principle.

Reconciling Tech Debt

Just like financial debt, technical debt should be reconciled periodically and “balanced”. In the context of technical debt, balancing means understanding how your total debt will affect future work.

If your debt register is full of items that are a variation of “this is fine but may not scale”, that might impact your time estimation when asked “change these features, especially if the volume is increasing and the algorithm is intensive.

Failing to Reconcile Tech Debt

On the other hand, if technical debt is undocumented, developers are not really aware when they are making these decisions, etc. the result is that you just don’t know all the potential boat anchors at bay. Thus a time estimate is given which turns out to be unrealistic.

Team & Developer Impacts of Mismanaged Tech Debt

The impact on the team is plainly obvious when unknown technical debt exists and impossible timeframes are set which positions the team for difficult future modifications. Impossible timeframes are complicated by stumbling upon seemingly lazy or incomplete code, and the end result is the organization develops a “crunch culture” where you are expected to plough through whatever you have to.

Over time, team members experience poor work/life balance, and they detach and are not personally committed or passionate about the team’s success.

Worse, is that good developers might be blamed for the issues that surface when re-visiting code. It may very well have happened that anyone developing the first cut of code 18 months ago might have done the same. But when it is causing complications today, that person is going to be blamed.

Team Members are Protected When Tech Debt is Well Managed

Contrast that with a team that understands how to manage technical debt and follows the advice in this article. The team could go back to 18 months ago and understand why that algorithm was brute-forced, and more importantly establish that anyone on the team could have known and objected.

Having a technical debt registry and documenting the justification will remove a lot of politics from the battlefield. The team can all go back and understand the thinking, and realize that any of them could have spoken up — at that time. But they, presumably, de-prioritized it because it wasn’t more important than what they were working on at the time.

At the end of the day, well-managed technical debt sets the stage for teamwork and empathy instead of individualization and blaming.

Every single member of the development team certainly understands that the reality is every piece of code can’t be developed for infinite use cases and scales.

Chipping Away At Significant Tech Debt

In the preceding discussion, I spoke about dealing with technical debt in the future as being rolled up into work around future business requirements. This is appropriate when the technical debt is not overwhelming.

If the technical debt is overwhelming, what can you do? No, the answer isn’t “tell the business you have to re-write the thing”. By the way, that is the mindset of many developers that inherit technical debt.

A better idea is to at least de-couple and loosens the adhesion between other parts of the app and the high-debt pieces. How do we do that?

How Microservices Architecture Reduces Tech Debt

One of the benefits microservices brings is the ability to break up a monolithic app into more manageable pieces. In my experience, legacy code that needs serious re-work is almost always tightly coupled and bonded to other components of the app.

If you cannot remedy the legacy code as you would prefer, you can at least compartmentalize it and make it easier to surgically repair down the road. While this doesn’t completely remedy the debt item, it will improve the debt posture.

For example, regarding the example algorithm of which I spoke earlier, let’s assume that was part of a monolithic app, and that component is called directly by the user interface, by reports and pages, etc.

As a step one, it may make sense to bring that algorithm into its own self-contained microservice, with an interface such as REST which other components use to access that service.

But that seems like a lot of code, so why not just fix the algorithm instead of adding complexity? Because it isn’t a lot of work and complexity, because you can likely generate REST the client interface code leveraging swagger/openApi, and you end up with that service being loosely coupled and not entirely bonded to the services that use it.

When you do decide to re-design that service, you won’t need to touch those calling services again and you won’t necessarily even have to bring them down at all.

This makes it easier to address in the next visit, easier to test, and easier to potentially roll back.

In addition, you could potentially re-write that service in a language that is better suited for what it does, or perhaps you are trying to de-emphasize the language/frameworks of the legacy code and move to another world.

There are many reasons to embrace microservices architecture, and in my view, it is technical debt to not have already embraced a microservice-oriented architecture.

Using that PHP 5.6 app example again, if I was not able to get the business to fund completely re-writing it or porting it to PHP 7, I would at least start transitioning the most business-critical pieces to microservices that are in a modern tech stack, so I can at least be sure there are not security vulnerabilities in those microservices.

And I would build those microservices such that there is no presumed trust in the parts that remain in PHP 5.6.

Summary

We talked about a lot of good stuff here! What is technical debt, why is it something you need to expect to encounter, and how do you manage it. We talked about the need to make it something we consciously are aware of, so we document it, track it, and understand it.

We talked about how that understanding positions us to make better timeframe and level of work estimations, and how mismanaging technical debt can lose jobs, destroy a team, and cause developer turnover.

Finally, we talked about how embracing microservices architecture positions us to lower our technical debt posture, make us more nimble and make it easier to address an app down the road.

More from Thomas Carlisle

Log4j Vulnerability Underscores a Negative Side Effect of Open Source