🇬🇧🇺🇸 Why do we overengineer?

The most common sin a software developer tends to do is overengineering. I have been guilty of doing this more often than I'd like to admit, and I am still prone to doing it from time to time.

Why do we, as software engineers, tend to overengineer?

The answer is quite simple, and I am pretty certain will upset a few: we tend to believe we're way smarter than we actually are, which combined with a little too much of being too overzealous, can result in overly complicated code that either does a simple thing in a complicated way, either does a bunch of many imaginary problems we think will appear in the future.

Let's start with the beginning: why do we think we are way smarter than we actually are? I am trying to say all software developers are stupid? Of course not, but we tend to either over-simplify the situation (and not foresee all the possible fail cases), or over-analyze it and take into consideration way more possible use-cases than needed. We, as software developers, more than often are not in direct contact with the clients or the business side, and we don't have the context of the things we are tasked to do.

"Hey Vlad, we need to extract the images from this link, and save them in our app, so that when we will need them, we will need to give them to the clients"

Cool, we have the requirements but we don't have enough info to actually do something useful, because we don't really know the use case for this, the "why"s, the "how"s and the "what"s of the problem. What images are we interested in? (do we just blindly get all the img elements from the page? Or do the users not care about the images in the headers or in the sidebars? Or do we only need the ones from the article content?), where do we store them? (in an cloud object storage service? On an FTP server? Local because they are temporary?), when are we going to be asked for the images? (3 minutes from when we download the images or one week later?) How do the clients need to get them? (Bundled in a zip archive sent on email? In an excel file with links to download the image? Do we need to process them in some way before, like make sure they all have the same aspect-ratio or resolution?).

All these questions should come as part of the business requirements, but more than often, such in-depth details go not communicated. On one side, the business part doesn't know the technical jargon or the different possibilities that exist with such vague requirements (ex. that there are images that are not needed in the webpage, how to deliver the images: excel file, uploading them on an FTP server, etc). On the other side, the developer doesn't really know what to ask because they are not presented the use-case of the client, or the business value of this feature. And in a lot of organizations, this is doubled down by a fear of asking too many questions, coming from the fear of being perceived either as stupid, or not independent enough.

So, as a developer, knowing all these possibilities, we try to produce solutions for all: this downloading of images should be configurable and flexible enough so later on, when the clients will want to use it, they will be able to cover all their possible use-cases. We start to craft all kinds of clever sub-systems, abstractions that allow us to implement ten different concrete implementations of various components, write tests for all weird cases that might happen if the wrong planets align at the wrong time.

We end up spending too much time building too many things, with way more flexibility and being way more "future proof" that it will ever be needed. Only to find out those clients moved on because we took too long to deliver... Now scrape all of that!

Or an even worse situation: with all the requirements and context on the table, some developers still choose this path because they want to do the code as "future proof" as possible, so later on when some requirements change that will affect that area, they won't have to go back to it: "It already is implemented! You just have to configure this whole thing in this way and voila!".

Too avoid all these mistakes, there is one simple trick (the business folks hate it!): communcation on both ends! Try to put the clarifying questions to uncover the desired outcome for the clients, and then design and implement the fastest way to get there. If the business folks get overwhelmed with the tech jargon you throw at them or don't have the answers, help them either come up with them, or instruct them what to ask the clients. And if the client doesn't know, propose them a simple way to achieve their goals. After all, that's what pays the bills: the client's problems that get solved by you.