Four Early Lessons from Working on Machine Learning Projects

Some high-level reflections from working on a Computer Vision project in PyTorch.

Be Practical and Dive In

When beginning a project, it is necessary to dive into the codebase in a very practical manner.

• Make a copy of the codebase (if you’re starting from an existing one) on a parallel branch (e.g. called original; origin/original) that you won’t touch and instead keep for reference.
• Begin by running the code and putting breakpoints into functions or the top-level script(s) to interrogate objects as they flow through the model: e.g. the dataset, the augmentations, the training loop, and most importantly the forward pass.
• Write detailed but succinct comments throughout the script to keep track of the information you find, so that you do not have to repeat these checks too many times.

Make Experimenting Easy for Yourself

Reduce the dataset size and training time for experimentation over the bulk of the project’s lifetime.

• Replicate the baseline results (e.g. of an existing paper or model) and once this has been successfully done (for example as a validation of the dataset and the codebase to show it reproduces the published results, reduce the size of the dataset you are working with if it is onerous to work with or training / evaluation times are longer than an hour or a few hours.
• Getting results more quickly to show potential improvements is more important than running each experiment on the whole dataset.
• You will run a lot of experiments, at least if you are doing your job right!
• Successful ideas and their experiments can then be run on the whole dataset to produce results that are comparable against benchmarks, the baseline or the state of the art for publication or training models.
• Importantly, this approach relies on a good and representative sampling strategy that allows you to experiment on a reduced (but still sufficiently large given the model capacity) dataset with lower computation cost but infer that the same approach is genuinely effective.

Maintain Diverse Perspectives

Use Git, Docker and other tools to make life easier for yourself, but don’t forget to manually keep track of the project in your mind as well. This will also help with reasoning about the project’s weaknesses and strengths, for example different experiments that tell you something about each other.

• Git allows you to create multiple branches for experiments, but it is also important to keep track of the branches and ideally integrate as much functionality as possible into a single main branch.
• This allows you to take modifications forward easily when spinning up new experiments and models, for example concretely, writing better or optimised class methods (like the forward pass) of a model will automatically be pulled into new experimental branches if the model on main includes them.
• You can use additional arguments to specify which model type and options should be invoked and specify defaults to retain the base model behaviour and API, e.g. setting a new argument to False by default and only using (or even creating) a method conditionally on it being set to True when invoking training.

Understand Your Model

Investigate your models and results early through visualisations, metrics and diagnostics

• If you propose a model, there is often some underlying supposition of why it ought to work.
• Create hypotheses about the corollaries of these assertions, or even about your assumptions, and log metrics and visualisations early in the model design process to validate these.
• In the case that your expectations about the model’s behaviour are not met, this model interrogation becomes useful as a means of debugging your model at a high level.
• For example, concretely, when using image segmentation algorithms, visualise the segmentations that are actually fed as input to and used by the model in the forward pass. It is useful to keep track of what is genuinely going through your model, and not what a function or piece of code does in isolation or in another context.

Concluding Remarks

The most important thing is to be practical.

Do not spend too much time trying to understand theory, do not be a perfectionist, do not worry about breaking the model and take a few smart precautions to allow you to do that without stress.

Most of all, immediately begin working practically with the code or problem and, in parallel, accompany this work with constant consultations of the theory and background. In this way, you will build a more concrete understanding of the models and theory in tandem by creating a tighter feedback loop, where the theory is seen in practice and the practical elements anchor your theoretical understanding.