Observations After a Decade Making Software
Forewords
The waves towered above our ship. Each time we reached a crest, I felt as light as a feather, freed from gravity's grasp, if only for a moment. The sea happily accepted our descent. I was cushioned by a mattress in my bunk each time we dropped. My port window showed nothing but water, a strange sight given that my cabin was above sea level at our departure.
I did not envy the crew. They had plenty on their hands with containers were stacked intimidatingly high - their weight distribution appeared slightly concerning to me. But who was I to judge? I was just a passenger. The crew quickly confined us to our cabins - which was probably a good idea. We would only get in the way - after all, I was travelling home with my newly minted degree in computer science!
Ten years have passed - and a lot has happened. I'm married to a beautiful wife. I have two great kids. All of which I'm eternally grateful for. More relevant to this post is my professional career. I've been fortunate to be a part of massive projects on a scale where one part of the project has no clue what the other is doing. I've been a part of tiny teams scrambling to keep things running while also adding features. I've been heading development and building teams. I've taken part in architecting critical systems for not just the business but for society at large. I've made decisions that will impact companies for decades to come.
I've made a good chunk of observations so far in my career, and I share some of them in this post. I won't flesh each point out in great detail (some topics can fill a book) nor go out of my way to pose concrete solutions. These are just high-level observations - nothing more, nothing less.
Most Developers Don't Fully Understand Their Responsibilities
Many developers claim their main duty is to "maximise business value", a notion as fundamental as "the sky is blue". But we're not unique in this; it's an organizational-wide goal. Beyond this, we also need to craft software that won't be a liability, covering security, bugs, maintainability, and legality—much like an accountant's duty over financial records.
The pressures are real: management, deadlines, and other obstacles in the way of great craftsmanship. But we need to balance short-term gains, like meeting deadlines, and long-term ones, like creating a sustainable system. Over-prioritizing short-term wins can lead to problematic systems requiring replacement, trapping companies in a costly rewrite cycle.
Many developers feel they can't address technical debt due to a lack of authority. But if managing technical debt is our responsibility, we must also have the authority to make decisions on it—this is our domain. If we limit ourselves to simply be pushers of "business value" then we don't get to complain about technical debt and terrible solutions. If we accept a wider set of responsibilities, then we'll have much more freedom and influence.
With great responsibility comes great power.
Perfect is the enemy of good; good is the enemy of great
I often stress the importance of focusing on great craftsmanship, and I often face pushback - my ideas are labelled utopic, only workable in a perfect world. The argument is that "completed and good is better than perfect and unfinished". This argument - taken at face value - is entirely valid. Its validity ends when we circle back to the word "great" in "great craftsmanship".
I've come to loathe the word "pragmatic". Not because I'm against being pragmatic. I've come to loathe "pragmatic" because it is used as an excuse to implement subpar solutions. One common example in microservice solutions is when a specific service contains all the important data that some new feature needs, but that feature doesn't fit the domain. The word "pragmatic" is often invoked, forcing the feature into the existing service. Two years later, the service is now bloated, slow, and difficult to manage.
There are times when accruing some short-term technical debt is okay. Technical debt should have a limited lifetime and should not be the norm. There are healthy ways of practising pragmatism - using it as a justification for poor craftmanship is not.
My arguments do not aim for perfection. Such a thing doesn't exist. There's a place between "good" and "perfect" which we should be aiming for.
Pareto's Influence on Developer Teams: Craftsmen, Resistors, and the Neutral Sixty
It's okay to view coding as just a job; not everyone needs to live and breathe it. Committing all your spare time to mastering new technologies isn't necessary, as I believe in a work-life balance, not a live-to-work philosophy. When I assert that only two out of ten developers genuinely care about their craft, I'm encouraging a personal commitment to doing the best job possible within one's capacity, recognising that "best" varies for everyone.
Wanting to excel might demand occasional learning outside work hours. While this may sound paradoxical, I endorse occasional personal investment to become a better developer. As posed in the book "Software Craftsman,": "Who is responsible for your career?" The answer is straightforward—you are. Self-learning enhances current job performance and invests in our future careers.
On the flip side, two out of ten developers resist change. They oppose new processes, technologies, or ideas, not due to their efficacy but simply because they represent change. These individuals aim to maintain the status quo, evading extra effort. While some changes might indeed be unwarranted, a general openness to trial and potential improvement is crucial.
What about the remaining 60%? They fall somewhere between the two groups previously mentioned. Rarely initiating or advocating for change, they continue their work without significantly steering the course. This kind of developer isn't a detriment to development, but they do need good guidance as they will follow whatever engineering culture the organisation has - for better or worse.
We keep repeating basic mistakes despite all the shared knowledge within our industry
In our field, resources abound. Countless books, courses, articles, talks, and podcasts cover every imaginable technology and process. Many talented professionals share their knowledge daily. Despite this, we seem stuck in a cycle of repetition, failing to learn and evolve.
We understand that automated testing is crucial for any healthy system. We have recognised the pitfalls of waterfall project management and the importance of rapid feedback for user satisfaction. The problem extends to our technical solutions. We've realised certain architectures are outdated and that a distributed monolith typically combines the worst elements of both monoliths and microservices.
Yet, despite this knowledge, we keep making the same mistakes. The root cause remains elusive. My best guess involves a blend of education, lack of passion for the craft, and failure to take responsibility for personal growth.
Regardless of the core issue, the truth is that despite all the information out there, we keep making the same mistakes repeatedly. To prove my point: I talked to a developer about being late on a project the other day. He really wanted to meet this deadline. His suggested solution? Get more developers on the project - completely ignoring "The Mythical Man-Month", originally released in 1975.
Team Performance Remains a Black Box to Most Leaders
What baffles me is the number of leaders oblivious to their teams' performance. Many teams function like black boxes—given a list of priorities, they deliver code with little to no information about their progress. Are they improving, deteriorating, or stagnating?
As I've mentioned earlier, with great responsibility comes great power. But such power must not be unchecked. The business has a right to know its progress and understand the team's status. Sharing this information builds trust between the team and stakeholders and helps identify underperforming teams.
There are myriad ways to expose this data—DORA metrics, source code static analysis, production metrics, and more. These can provide crucial performance indicators.
However, when I propose teams reveal such metrics, they often react apprehensively. It's a valid concern, given management's history of metric misuse. But my intent isn't surveillance—it's to provide information the team can ideally leverage themselves or at least offer the business enough context to initiate necessary discussions.
Unfortunately, it's rare to see teams track, analyse, and act upon this data, let alone share it with the broader business. This opacity fosters a black box-like team where only the team are privy to their performance (albeit often ignored). Black box teams allow for dysfunctions to persist unnoticed - even by themselves.
Complacency Often Cripples Innovation in Numerous Development Team
Development teams should change over time. Original team members can get stuck in old habits. Their early architecture, coding styles, and processes can become outdated. When new developers join the team, they often follow the old guard's lead, reinforcing these outmoded practices and resisting change. It is difficult to notice the absence of change. Therefore it is no surprise that leaders overlook this issue.
This leads to a team that's stuck in its ways. If there's been no significant change in how they work for over six months, it's a clear sign. Instead of innovating, the team just cycles through routine tasks using old methods. Successful teams, on the other hand, continually experiment and improve how they work.
Teams that get stuck in their ways often feel they need to redo their system completely. The irony is, those asked to fix problems that stem from technical debt often allowed it to become a problem in the first place.
This is a pattern I've seen across multiple teams. As someone who tends to have strong opinions, I ask why something is the way it is. The response is often "It's something we decided on ten years ago" or "I don't remember". Teams that are really struggling tend to whip out the ol' "We've always done it that way" and shrug off any suggested alternatives.
Architecture is Misunderstood and Misapplied
Every sizable organisation usually has "architects" who steer the technological vision. But it's not uncommon to find these architects being veterans rather than experts in architecture. Don't get me wrong; I'm not questioning their competence or monopolising the title. However, their architectural knowledge tends to be confined to what they're acquainted with, which can be limiting.
Here's a familiar snag: sticking to the 3-layer architecture like glue. It's all well and good until the codebase explodes in complexity. Despite the growing pains due to more complex features, teams keep clinging to the old framework, shuffling things around within the layers in a futile game of musical chairs. They whip up a tangled web of service classes, each with its own role, trying to piece them together for some grand purpose. On more than one occasion, I've heard developers refer to this patchwork as "architecture." I beg to differ (but that is a post for another day).
A particular gripe of mine is with "accidental architecture." It’s as if one person made a structural choice that caught on, spreading like an architectural flu across the teams. Fast forward a few years, and the whole organisation has adopted this design, with nobody really understanding why. You'd be hard-pressed to find a company that can trace back why they adopted a specific architectural style or what advantages it initially offered.
In a nutshell, we need to up our game in architecture. We have to improve how we shape architects and broaden their horizons. I'm not trying to gatekeep the "architect" title, but I firmly believe we must pair it with a well-rounded understanding of the ever-evolving architectural field.
Rounding off
After a decade, it might sound like I have a pretty negative outlook on the industry. There's much room for improvement. But this doesn't mean I look at the industry negatively. I do believe things are improving for the better - albeit slowly.
If we circled back to the person in his bunk somewhere between England and Denmark a decade ago - he'd probably be bummed to read this post. However, with a decade of growth, I'm more excited. A list of negatively loaded observations can also be a list of potential. It is a list of exciting challenges and discoveries. It is a list of learning and growth.
I do not dread the decade to come. The industry is still young and is trying to figure things out. Most leaders have not yet really understood the difference between physical and digital goods - at least not from the view of production and maintenance.
Even the job of a "programmer" has changed drastically in the last few decades - and it is still changing. Before the internet, software was a "ship-and-forget" kind of deal - now, it is a perpetual thing that must be kept alive. Simply knowing how to code is not enough - we need to know about developer processes and agile methods. We need closer relationships with what operations are doing through DevOps. We must be great communicators as we're expected to work more closely with users and customers. We have more responsibilities when it comes to larger projects. There's a lot to being a developer these days (or maybe it always was?)
Here's to another decade of change, learning and growth!
