Estimates and Tech Debt
If you ask most software engineers what some of the worst parts of their jobs are. Then chances are estimates will fall somewhere in the top 3. Followed closely by tech debt.
So why are estimates so difficult?
And why do most well-intentioned projects, accumulate so much tech debt?
Table of Contents
The estimated elephant in the room
Estimates are notoriously difficult to get right for all but the simplest of features. And to make matters worse, we tend to come under pressure from project managers or other parts of the businesses we work in. We come under pressure to trim estimates down to fit the expectations of others. Even when those expectations are not justifiably grounded.
And astonishingly, we often oblige and cave to that pressure.
People want to know when a piece of software will be ready. That is a perfectly reasonable thing to ask of us.
But here’s the catch. We currently have no real reliable way of forecasting how long a piece of work will take, and often we loosen our definition of ready to suit the narrative.
The underlying problem
The problem which underpins all of this, is the fact that most modern systems are highly complex. This in itself, is at least partially our fault. We do not manage complexity in the ways we ought to.
As well was this, we have to contend with the fact that most people if not all are not geniuses.
We do not understand each and every little detail about the systems they are working with. This is also true of the problems that those systems are trying to solve.
Generally, the more experienced engineers are better placed to provide more accurate estimates. But, naturally they also cannot foresee every eventuality.
And that is absolutely fine. It is to be expected.
So how do businesses address this? Well there are 2 main ways in which this is approached:
- We expect people to actually be geniuses, to hold all the details in our minds, to have the ability to foresee every eventuality. This is primarily what the Waterfall model is built on.
- Or we expect that we will not know about the breadth of the problem until we begin developing the solution. In this case, we should have the freedom and confidence to make informed decisions quickly based on the results of our development efforts. We accept that we do not understand the entirety of the problem until we are actually exploring it. This is what the Agile methodology is based on.
Would you be surprised if I said that most organisations probably still impose option 1? How shocked would you be if I said that most businesses who do so, also say they are agile too?
Cutting estimates
The businesses which employ us do so for the expertise that we provide. And as engineers, we have a duty of care to the systems we build.
When we are asked if we can cut estimates. This should be a warning sign. To reduce the time to completion, certain aspects of our solutions would have to be sacrificed. In most situations like this, testing, quality of code, modularity and good designs are the things which are omitted.
But the fact is, these are things ensure that development of our systems don’t slow to a crawl in the future. These are the things which allow us to develop with confidence. These are exactly the sort of things which enhance and speed up our development over time.
We have a tendency to think that by removing these, we can build things quicker. And that may be true to an extent. But the reality is, that over any significant period of time, this will theory will be invalidated pretty clearly.
Without those aspects, our systems become buggy and difficult to change. If our systems become more and more difficult to make changes to, then how can we reliably cast estimates?
The engineer’s duty of care
In other engineering professions, the duty of care of the engineer is transcribed in guidance and legislation.
Most engineers who are tasked with creating physical products, like structural or electrical engineers are responsible for the solutions they design. They must ensure that the products they create are safe, reliable and efficient. They have to deliver fail-safe mechanisms, whilst also allowing for extensibility.
Excuse the broad-speaking terminology, but do those things ring a bell?
These are some of the all-encompassing attributes which our systems also aim to exhibit.
When we trim development down to the bare-bones. Our development experience worsens. We lose confidence in our systems, and it becomes progressively more difficult to add features and make changes.
Our industry has even attached a word to this phenomenon. We call it tech-debt.
It should be noted that there is a place for being pragmatic and moving quickly. Especially in startups and early stage companies. But allowing the net build-up of tech debt is essentially the antithesis of moving quickly.
If we fall into the trap of trimming these out, then we are failing in our duty of care to the systems we are responsible for.
Well, how quickly can you really do it?
Businesses employ us for our expertise. When a non-technical person asks us the dreaded question of well, how quickly can you really do it?, we should never resort to forgoing quality. This will inevitably lead to a reduction in quality of our codebases.
Ultimately, we are the ones who care about the quality of our systems. We have to maintain a high level of confidence in our systems. If we do this, we can deliver features faster. Which impacts our businesses dramatically. This is the crux of what the people around us want from us.
The non-technical person pressuring us for a smaller estimate is not doing so maliciously (for the most part). And for the record, I am not advocating for any kind of abrasive response to this. But the reality is that they do not understand the repercussions associated with reducing the quality of our output. And it’s not their job to! If anything, we are the ones who have that level of appreciation and understanding.
It is also our duty of care to advise those around us of this.
A not-so-real conversation about estimates
Forgive the exaggeration, but let’s imagine the following conversation:
Product Manager: Hey, how long do you think feature X will take?
Engineer: As a rough estimate I’d say 2 sprints. Assuming we don’t uncover any severe issues.
Product Manager: Okay, well, how quickly can you really do it?
Engineer: You can have this feature in less than the original estimate. But it’s going to cost you. Every one of your next feature requests is going to take 10% longer. Oh, that that has a compounding effect, it will be more than 10% in 2 months time.
Obviously, I’ve plucked these numbers out of thin air so forgive me again. But what do you think our Product Manager would say that?
People often give up information in layers. Primarily this is what the other person is trying to get to when they ask this question.
Fundamentally, it is our responsibility to ensure that our systems allow us to sustainably deliver features in a safe manner. If we cannot do this, then we have failed in our duty of care.
This compounding drag effect is what we do not communicate to the companies we work for. But this is the all too real result of allowing tech debt to accumulate.
The mental model
When we trim our estimates, we are essentially modelling our development like the following:
We are conceding on the parts of our development which are vitally important to ensuring the quality of our systems. We are saying that we can stop and finish halfway. We have compartmentalised the model of our development as such. And reduced the importance of the things which really matter to us.
Now let’s say we modelled our development as below:
Now, we cannot possibly hand over a solution which is half complete. You might also recognise this model. It illustrates the Test-Driven-Development model albeit on a higher level.
If we modelled development like this, quality would be baked in to our solutions. In the same way quality is baked in to systems built with TDD.
If we did this we would not be able to reduce estimates by trading quality because our solutions would not be ready. Our testing and refactoring phases happen within periods of development. They become intrinsically linked, and therefore cannot be displaced or removed.
Arguably this might feel like an extreme idea. But, this is our responsibility as engineers.
The caveat
Of course, there will be plenty of dissenting opinions. And that is perfectly okay!
Ultimately, there may be situations where we may be forced into going against our better judgement. Because the real-world is not perfect and there are other aspects to balance.
What is important is that we, as engineers, communicate and signal the impact of tech debt effectively.
If we end up with the more permanent mindset that moving fast means always leaving tech debt behind, then this is most likely misguided.
Soon enough, we will end up moving much more slowly.
That debt needs to be paid at some point. The longer we leave that debt in place, the more interest we accrue. This means it becomes more costly to rectify.
Summary
Cutting back on quality to deliver an initial feature will ultimately slow us down. This is also the exact opposite of what the businesses we work for want from us.
So it’s our responsibility to make sure this does not happen.
That being said, we have to be pragmatic about tech debt. We may still face pressures to deliver things faster if it means picking up tech debt along the way.
But we have to be vigilant about paying that debt back as soon as we can.