🇬🇧🇺🇸 Top-down building

Through my programming career, I went through multiple staging and through multiple ways of thinking about the code.

As a juniors, I used to focus on the code too much and too little on the problem. That made me write code that was hard to integrate with the other components of the project, and that was very hard to test. Actually, as a junior I barely wrote any tests. All the quality assurance process relied heavily on manually testing on some very specific use case.

You can imagine that this way of doing things wasn't going to lead very far. The code I wrote was hard to debug, when a new input with some new specifications was encountered, extending my code to also cover that case was a pain: with no proper quality assurance methodology in place, modifying the code for the new cases often broke the old cases.

Programming modules that were slightly more complex was resembling more and more a game of whack-a-mole.

Let's take the example of writing a XML parser for this API https://www.reddit.com/r/romania/.xml that will extract the images with the purpose to send them to an image crawler.

Juniors are focusing too much on the implementation

As a junior, I would have wanted to be able to call my code object-oriented, so I would just create a huge class, with a method parse and just have some convoluted logic based on tags, attributes, and children nodes, enough so I would get the required information and return it.

The code would look something like this (Python pseudo-code-ish)

class XmlParser:
    def parse(self, content):
        root = etree.ElementTree.fromstring(content)
        results = []
        for item in root:
          if item.tag == 'entry':
            for entry_child in item:
              if entry_child.tag == '{http://search.yahoo.com/mrss/}thumbnail':
                  results.append(entry_child.attrib['url'])
        return results

Straight forward, everything works fine for the given example, and it gives the correct results.

But everything is obscured in node manipulations and hardcoded strings, and as a reader, I would have no idea what exactly is happening there without having an example or spend some time trying to figure out exactly what happens at each step.

Of course, this example is overly simplified and somewhat easy to understand, but in real-life, code can become way more convoluted than this.

My personal rule is that if I can't figure out what a function/method does in less than 5 seconds, it is too complex and it needs to be refactored (broken down, better names, simplified the core algorithm, etc).

For somebody that sees the code for the first time and having no context about the structure of the input XML, a lot of questions arise:

class XmlParser:   # seems like a generic XmlParser. Can I use it to parse all kinds of XML files?
    def parse(self, content):  # parse what? Not clear what it returns.
        root = etree.ElementTree.fromstring(content)
        results = []  # what results? What am I digging for inside the XML?
        for item in root:
          if item.tag == 'entry':  # ok I am getting some kind of entries...
            for entry_child in item:  # or am I?
              if entry_child.tag == '{http://search.yahoo.com/mrss/}thumbnail':  # ok, some thumbnails in a weird namespace
                  results.append(entry_child.attrib['url'])  # and finall the URL
        return results

Ok, so I am getting some entry -> thumbnail -> url hierarchy, but still what does that even mean? What am I doing exactly. What were the requirements? How does the input look like?

Those questions are not unreasonable. There are just too many unknowns.

Writing good code never forces you to dig into the implementation details. It offers a high level documentation of the business logic and what is required.

This is the result of down-to-top development. You analyze the business logic, and then formulate the following pseudocode in your mind:

• I need to iterate the root and find all the entry nodes
• for each entry I need to find the thumbnails
• for each thumbnail I need to extract the url.

That is the implementation. We dig into the XML nodes and attributes and write our code based on that. But the requirements are different: we need to extract the image urls from the posts, from a Sub-Reddit feed.

Starting with the details is confusing, because everybody knows how to read code, but understanding it is a whole 'nother problem.

I often hear less experiences developers, when I ask them to tell me what their code does, starting reading the actual code out loud: I iterate this list with item, for each item I iterate its children, for each children I check if its tag is thumbnail, and if that's the case, I extract the url attribute . Gee, thanks. I also know how to read code. I want to hear the business logic and high-level details.

Communicating high level details is a skill in itself that will make you able to communicate technical details to less technical people (or the business side). It is probably the hardest thing to do in a tech company, because this job is at the intersection of the business side (which focuses on clients, cashflow, strategy) and the development side (which focuses on the technologies, design patterns, systems, databases, etc).

Starting from high level and going to low level

Systems are built starting from the high level requirements. Business side will never be able to provide details such as "we need to iterate through this node and find its children with the tag entry ". And that's not their job. Or anybody's job.

If I have to give you this level of details as part of the task description, I should be basically writing the code myself instead of relying on you. All the requirements will be high level, maybe with some direction or implementation hints at best.

The way to approach new features is to start from the high-level requirements, and go deeper and deeper.

This way of doing things requires doing multiple levels of abstraction, writing more code and some juniors might say "That's counterproductive. Why write more code to do the same thing if you can get it done in fewer lines of code?" .

But what they don't realize is that the time saved taking shortcuts now will be spent tenfold by your colleagues trying to figure out what the Hell did you try to achieve there?

If a programmer isn't able to quickly tell what a piece of code does in at most a minute, I blame it on the person who wrote the code (not really blame. I am not a fan of blaming people for stuff. We are all learning, and if some work is done in a suboptimal way, it is a learning opportunity. I would call the person who wrote that code and refactor it together to both see another better way of achieving the same thing).

So how would the code from above look like written in a top-down approach?

First, I would start with the proper naming of things:

class SubredditXmlContentParser:  # signal that we work with pre-fetched XML content
  def __init__(self, content):
    self.content = content

  def extract_thumbnail_urls(self):
    thumbnail_urls = []
    # the business logic will come here
    return thumbnail_urls

The general purpose is clearly established. This class will work with XML content and will extract the thumbnail urls.

Next steps, is to outline the business logic for doing this: the "journey" from the full XML document to the actual URLs. But the trick is to not dig into the implementation details yet, but outline the strategy and concepts first.

class SubredditXmlContentParser:  # signal that we work with pre-fetched XML content
  def __init__(self, content):
    self.content = content

  def extract_thumbnail_urls(self):
    thumbnail_urls = []
    for post in self.get_posts():  # signals that we have posts (the post being a business object)
      if self.has_image(post):  # and posts may have images (the image being a business object)
          thumbnail_urls.append(
            self.get_thumbnail_url_from_post(post)
          )
    return thumbnail_urls

At this point, I have no concrete implementation, but for a reader, it contains enough information to grasp the whole process: we have posts on a page, and we have images in a post. The implementation details are the next step (in a lot of real-life scenarios, we would have multiple levels of "business logic" before going to the implementation details. But this being a simplified example, we don't really have that complicated business logic to require that).

After adding the implementation details, we end up with

class SubredditXmlContentParser:
  def __init__(self, content):
    self.content = content

  def extract_thumbnail_urls(self):
    thumbnail_urls = []
    for post in self.get_posts():
      if self.has_image(post):
          thumbnail_urls.append(
            self.get_thumbnail_url_from_post(post)
          )
    return thumbnail_urls

  def get_posts(self):
    root = etree.ElementTree.fromstring(self.content)
    for element in root:
      if root.tag == 'entry':
        yield element

  def has_image(self, post);
    for child in post:
      if child.tag == '{http://search.yahoo.com/mrss/}thumbnail'
        return True
    return False

  def get_thumbnail_url_from_post(self, post);
    for child in post:
      if child.tag == '{http://search.yahoo.com/mrss/}thumbnail'
        return child.attrib['url']
    return None

We see some duplicated code in the has_image and get_thumbnail_url_from_post . We can refactor the business logic a little to make it more efficient:

class SubredditXmlContentParser:
  def __init__(self, content):
    self.content = content

  def extract_thumbnail_urls(self):
    thumbnail_urls = []
    for post in self.get_posts():
      image = self.get_image_if_any(post)
      if image:
          thumbnail_urls.append(
            self.get_thumbnail_from_image(image)
          )
    return thumbnail_urls

  def get_posts(self):
    root = etree.ElementTree.fromstring(self.content)
    for element in root:
      if root.tag == 'entry':
        yield element

  def get_image_if_any(self, post);
    for child in post:
      if child.tag == '{http://search.yahoo.com/mrss/}thumbnail'
        return child
    return None

  def get_thumbnail_url_from_image(self, image);
    return image.attrib['url']

And here we go. In 99% of the time, the readers of the will not need to dig into the implementation details because the code from extract_thumbnail_urls tells the whole story: what concepts we have in the XML document, how we work with them and what exactly we are working with.

Another benefit of doing things this way is that we can also extend the code to support the JSON feed of the subreddit (if needed, of course). Having the core algorithm not dependent on the implementation details, we can extract it in an abstract class, and then leave the implementation details to the concrete implementations.

The JSON variant of the feed can be found here: https://www.reddit.com/r/romania/.json

class SubredditContentParser(abc.ABC):
  def __init__(self, content):
    self.content = content

  def extract_thumbnail_urls(self):
    thumbnail_urls = []
    for post in self.get_posts():
      image = self.get_image_if_any(post)
      if image:
          thumbnail_urls.append(
            self.get_thumbnail_from_image(image)
          )
    return thumbnail_urls

  @abc.abstractmethod
  def get_posts(self):
    pass

  @abc.abstract_method
  def get_image_if_any(self, post):
    pass

  @abc.abstractmethod
  def get_thumbnail_fro_image(self, image):
    pass


class XmlSubredditContentParser(SubredditContentParser):
  def get_posts(self):
    root = etree.ElementTree.fromstring(self.content)
    for element in root:
      if root.tag == 'entry':
        yield element

  def get_image_if_any(self, post);
    for child in post:
      if child.tag == '{http://search.yahoo.com/mrss/}thumbnail'
        return child
    return None

  def get_thumbnail_url_from_image(self, image);
    return image.attrib['url']

class JsonSubredditContentParser(SubredditContentParser):
  def get_posts(self):
    return self.content['data']['children']

  def get_image_if_any(self, post);
    return post.get('thumbnail')

  def get_thumbnail_url_from_image(self, image);
    return image

Even though we ended up writing more code, now we can easily see the two distinct parts that our code is made from: the business logic/requirements and the actual implementation details. When bugs occur or requirements change, we can do the changes accordingly easier.

Business requirement changes will result in changes in the core algorithm (in the abstract class), and bugs/unexpected changes (eg. the URL is now in another attribute, or the posts have now another parent) will result in implementation detail changes.

Of course, it's not all black and white, there are changes that will require tweaks in both sides (eg. posts to have more images, and we need to fetch all of them).

Conclusion

Hope you enjoyed this post, I usually don't write code examples (or at least not that many) because it tends to make the post longer than I like them. But sometimes, various ideas and concepts are better explained through practical examples, and I really believe that writing good readable code should be the main focus of developers, rather than getting things done quicker.

The time saved from taking shortcuts today will be spent tenfold in the future by our colleagues. And if we care about the project we work on and our colleagues, we will spend that extra time to make their life easier and enable everybody to move fast.

But you need to care to do that, and there are people out there who do not care, all they do is click in and clock out, do the bare minimum to get by and if they get called out they play the victim card and say that the employer is unfair (overly specific example, I know, but that's a topic for another time).

'Till next time, remember to start writing the business logic first, and then build down to the implementation details. If a colleague reads your function/method and can't tell quickly what it does, it's too complex/convoluted and needs to be refactored. That's why pair programming is powerful.

Keep having fun writing maintainable code!