My Python projects structure

I just saw a question on the Python subreddit, asking about how to structure a Python project and I wanted to elaborate a little bit on this subject here. It's a question a lot of newer people that come to Python get confused about and getting some things right from the start helps a lot in the learning process. Packaging, especially in the Python ecosystem, is a delicate topic, having some caveats and some aspects that are not that obvious especially to newbies if you don't even know what question to ask.

In the past few years, some alternatives to pip appeared out of some frustration of the default approach and some inconsistencies pip has, such as poetry, but in this post I will focus on working with pip.

I will describe my usual project structure which is enables me to work and iterate fast. Over the time I came to a set of personal rules-of-thumb that help me move fast and be consistent between projects. The more you develop, muscle memory will play a non-negligible part in your development speed so it's important to develop some personal workflows that will boost your productivity.

My usual project structure

When working a Python project, there are two important things I consider important to achieve: having a good development and test workflow and having a good publishing workflow. Usually, developing libraries means developing reusable pieces of code that can be easily included and integrated in other projects. To achieve that, the most handy way we have is packing in such a way that other users can install it with pip.

So, the structure I usually go for has the following components

  • <libname> + <libname>/__init__.py - a folder where all the sources will live. We include the __init__.py file because it's important for other users to be able to import our package as import libname
  • setup.py is the most important file that enables our library to be packaged, published and used by other people. I will give a simple minimal example for its content that works:
from setuptools import setup, find_packages

# this is where the magic happens
setup(
	# the name of the project. When users do pip install libname, the name is taken from here
	name='libname',
    # the semantic version. Most used standard for versioning 
    version='0.0.1',
    # instruct pip to include certain files when packaging
    # the find_packages shortcut allows us 
    packages=find_packages(exclude=['tests', 'docs', 'examples']),
    # here we specify the dependencies
    install_requires=[
    	'django>=3'  # this is an example
    ],
    # defining extras so that we can install extra dependencies
    # with pip install -e .[dev]
    extras_require={
    	'dev': ['pytest', 'sphinx', 'pycodestyle', 'twine']
    }
)

We can also include more metadata for the project such as short and long descriptions, author, author email, repository URL, etc. But for now, these are the most important keys in setup.py that make your project publishable and usable by others.

  • MANIFEST.in - add this if you need to include files with extensions other than .py. This took me a while to figure out. This is an example for this file with the most basic use cases:
include README.md
recursive-include assets/images/ *.png
include *.txt
  • Makefile if you are on a unix sytem so that you can create various shortcuts that will speed up different processes. I have always available the make start command which starts the example project, make test which runs the unit tests and make publish which builds and publishes the current version.
  • setup.cfg is a file where you can put various configurations for 3rd party tools such as pytest or bump2version.

If the I work on an application rather than a library, I also usually have around:

  • requirements/ - a directory to keep all the requirements. I have a base.txt with the dependencies absolutely required by the app to run, dev.txt which extends the base requirements file (first line is -r base.txt) which adds the development requirements such as testing, documentation and build dependencies and prod.txt which contains the production requirements such as raven (sentry's error reporting library), gunicorn, etc. If you have a setup.py file you can include the dev and production requirements as extras and install them with pip install -e .[dev] and pip install -e .[prod].
  • docker/ - a directory with the Docker-related files such as various Dockerfiles for local development and staging/production, auxiliary files needed for the project to run in a certain environment (eg. different logging.conf files for different environments)
  • docker-compose.yaml - I heavily use docker compose when doing local development. The projects I usually work on have more moving parts that interact with each other (for eg. caches, databases, more microservices) and using docker compose saves a lot of trouble and helps me keep my environment clean.

This sums up the structure I usually go for when doing development. Of course, in some cases, some files do not make sense to have, so this list is more of a guideline, not a hard list of files and folders to have no matter what. For example, when working with Django, you won't need a setup.py because all interaction with the project is done via the manage.py script Django provides.

Show Comments