Software Engineering Patterns for Machine Learning

Have you ever talked to your Front-end or Back-end engineer peers and noticed how much they care about code quality? Writing legible, reusable, and efficient code has always been a challenge in the software development community. Endless conversations happen every day across Github pull requests and Slack threads around this topic.

How to best adapt SOLID principles, how to make use of effective software patterns, how to give the most appropriate names to functions and classes, how to organize code modules, etc. All these discussions might be simple and naive at first glance, but their implications are high and deeply known by senior developers. Cost to refactor, performance, reusability, legibility, or, more simply put, technical debt can hinder a company’s capacity to grow in a sustainable way.

This situation is not different in the ML world. Data Scientists and ML Engineers typically write lots and lots of code. There’re very different sets of codebases these profiles work with. From writing code for doing exploratory analysis, experimentation code for modeling, ETLs for creating training datasets, Airflow (or similar) code to generate DAGs, REST APIs, streaming jobs, monitoring jobs, etc.

All of them have very different objectives, some are not production-critical, some others are, most likely (and honestly), never going to be read again by another developer, some might not break production directly but have very subtle and risky implications on the business, and obviously, some others can cause harsh impact on the end user or product stakeholder.

In this listicle of articles, I will go through all these different types of codebases from a very honest and pragmatic point of view, trying to give advice and tips to produce high-quality ML production code. I will put real-world examples from my own experience working at different type of companies (big corporates, start-ups) and from different domains (banking, retail, telecommunications, education, etc).

Best practices for exploratory notebooks

Effective use of Jupyter Notebooks for business insights

Understand the strategic utilization of Jupyter Notebooks from a business and product insights perspective. Uncover techniques to boost their impact on analyses.

Crafting purposeful notebooks for analysis

Learn the art of tailoring Jupyter Notebooks for exploratory and ad-hoc analysis. Refine your notebooks to include only essential content that offers the clearest insights into the posed questions.

Adapting language for diverse audiences

Consider the audience (technical or business-savvy) in your notebook endeavors. Utilize advanced terminology when appropriate, but balance it with a straightforward executive summary that communicates key conclusions effectively.

Optimizing notebook layout for clarity

Discover a suggested layout for structuring notebooks that enhances clarity and comprehension. Organize your content to guide readers through the analysis logically.

Reproducibility tricks for reliable insights

Explore tactics to ensure the reproducibility of your notebook-based analyses. Uncover tricks and strategies that contribute to maintaining the reliability of your findings.

Best practices for building ETLs for ML

The significance of ETLs in machine learning projects

Exploring a pivotal facet of every machine learning endeavor: ETLs. These combinations of Python code and SQL play a crucial role but can be challenging to keep them robust for their entire lifetime.

Building a mental model for ETL components

Learn the art of constructing a mental representation of the components within an ETL process. This understanding forms the foundation for effective implementation and will let you understand pretty quickly any open source or third-party framework (or even build your own!).

Embracing best practices: standardization and reusability

Discover essential best practices around standardization and reusability. Implementing these practices can enhance the efficiency and consistency of ETL workflows.

Applying software design principles to data engineering

Dive into the integration of concrete software design principles and patterns within the realm of data engineering. Explore how these principles can elevate the quality of your ETL work.

Directives and architectural tricks for robust data pipelines

Gain insights into an extensive array of directives and architectural strategies tailored for the development of highly dependable data pipelines. These insights are specifically curated for machine learning applications.

Best practices for building training and inference algorithms

The nature of training in machine learning

Training is often seen as an engaging and imaginative aspect of machine learning tasks. However, it tends to be relatively straightforward and brief, especially when developing the initial model iteration. The complexity may vary based on the business context, with certain applications requiring more rigorous development than others (e.g., risk models vs. recommender systems).

Foundational patterns for simplified training

To streamline the training process and reduce repetitive code, foundational patterns can be established. These patterns serve as a basis to avoid excessive boilerplate coding for each training procedure. By adopting these patterns, data scientists can dedicate more attention to analyzing the model’s impact and performance.

Transition to production and challenges

After constructing the machine learning model, the next step is transitioning it into a production environment. This step introduces a range of challenges, such as ensuring the availability of features, aligning features appropriately, managing inference latency, and more. Addressing these challenges in advance is crucial to successful deployment.

Holistic design for ML systems

To mitigate potential issues during production deployment, a holistic approach to machine learning system design is recommended. This involves considering the entire system’s architecture and components, including training, inference, data pipelines, and integration. By adopting a comprehensive perspective, potential problems can be identified and resolved early in the development process.

The role of experimentation in machine learning

Delve into the fundamental role of ML experimentation. Explore how it shapes the process of refining models and optimizing their performance.

neptune.ai is an experiment tracker for ML teams that struggle with debugging and reproducing experiments, sharing results, and messy model handover.

It offers a single place to track, compare, store, and collaborate on experiments so that Data Scientists can develop production-ready models faster and ML Engineers can access model artifacts instantly in order to deploy them to production.

Optimizing models through offline experiments

Discover the realm of offline experiments, where model hyperparameters are systematically varied to enhance key metrics like ROC and accuracy. Uncover strategies for achieving optimal results in this controlled setting.

Navigating online experimentation: A/B testing and beyond

Explore the dynamic domain of online experimentation, focusing on A/B testing and its advanced iterations. Learn how these techniques allow for real-world evaluation of model performance tailored to user behavior.

Bridging the gap: offline metrics to product impact

Understand the crucial connection between the Data Science team’s efforts to enhance model metrics and the ultimate impact on product success. Learn strategies to effectively correlate improvements in offline metrics with real-world product outcomes.

Techniques for alignment: model enhancements and product metrics

Delve into techniques and approaches that facilitate the alignment of iterative model improvements with tangible product metrics, such as retention and conversion rates. Gain insights into achieving a harmonious synergy between data-driven enhancements and business objectives.

What’s next?

We’ve already seen that in ML, code quality is just as crucial as in traditional software development. Data Scientists and Machine Learning Engineers work with various codebases, each serving different purposes and with varying degrees of impact on the business and end users. In this listicle, we’ve explored the key aspects of producing high-quality ML production code, covering everything from exploring data sets to implementing experimentation tools.

With these articles, we aim to provide you with an end-to-end perspective, sharing valuable insights, advice, and tips that can elevate your ML production code to new heights. Embrace these best practices, and you’ll be well-equipped to overcome challenges, minimize technical debt, and help your team grow.

So, whether you’re an aspiring ML practitioner or an experienced professional, get ready to enhance your coding expertise and ensure the success of your machine learning projects. Dive into the next article in the series talking about best practices for exploratory notebooks and elevate your MLOps strategy to unprecedented levels!