Setting Up a Machine Learning Project (ML4Devs, Issue 5)
Set up your ML project for success: integrate early and iterate often.
In the Why Machine Learning Projects Fail issue, we saw that the learnings from software development can be applied in ML too to improve chances of success:
Consolidate Ownership: Cross-functional team responsible for the end-to-end project.
Integrate Early: Implement a simple (rule-based) model and develop product features around it.
Iterate Often: Build better models and replace the simple model, monitor, and repeat.
ML applications, like any software applications, are consumed in two ways:
Pull: End-users triggers a request for “pulling” results from an ML model. For example, when you are looking at a product on an e-retailer website, its web app requests an ML product recommendation service to find what else you are likely to buy. Such ML applications are deployed as microservices.
Push: Results are “pushed” as notification or alerts to the end-user. For example, a financial monitoring system may continuously analyze trading events, and raise alert upon discovering any anomaly.
The pull mechanism is more common than the push. In either case, the ML is just a part of the system serving a business need.
The Modeling and Engineering silos can be avoided by setting up a cross-functional team of product managers, developers, data engineers, and data scientists that is responsible for the feature end-to-end.
It improves communication. Data Scientists are in tune with business needs and production constraints, and developers get to know the nuances in utilizing the model.
Do not jump into making an ML model. First, make a skeleton application end-to-end. For the ML component, just implement a dummy baseline rule-based system. It is okay if it is not accurate.
Seeing how your ML model will be consumed in a barely functioning system is a superpower.
For the ML component, design application-centric APIs. Even if you have a strong sense of what model you are going to use, resist spilling the model specifics into the APIs. A nice encapsulation will make it easy to swap and experiment with the models.
Integrating early with a dummy baseline model also decouples ML from the development of other parts. While some developers can iterate over and enrich the skeleton application, ML engineers can work on models and always have an end-to-end system to test and experiment with.
This integration also gives you a baseline for benchmarking evaluation metrics. There are four important metrics:
Business Success: Impact of the product feature on the business (e.g. rate of recommended product being bought). Either implicitly through user actions, or designing a way for users to give feedback.
Model Performance: Benchmark the effectiveness of the model.
Latency: Time taken in model inference. A perfect model that tests users’ patience is a bad model.
Data Drift: Used after the deployment to monitor if data distribution of data encountered in wild is shifting.
With careful design, it is possible to have a high product experience despite not-do-high model performance. For example, showing search or recommendation results only when confidence is above a certain threshold, or showing top 3 suggestions instead of one, can lead to higher user satisfaction. Careful product design plays a huge role in ML success.
Start with a simple data processing and model training pipeline containing:
Data Ingestion: Collect and consolidate data from multiple sources.
Data Preprocessing: Clean, curate, and extract features.
Model Training: Train model alternatives.
Model Evaluation: Track performance metrics.
Result Logs: Record all details for comparison and reproducibility.
A detailed article on organizing machine learning projects.
A lesson in Full Stack Deep Learning course on Setting up ML projects.
For pipeline on Apache Spark: ML Pipelines in Spark MLlib.
If you are comfortable with ML modeling, this is a highly technical lesson on setting up a baseline.
ML4Devs is a weekly newsletter for software developers with the aim:
To curate and create resources for practitioners to design, develop, deploy, and maintain ML applications at scale to drive measurable positive business impact.