---
title: "Machine Learning"
description: "622 posts about Machine Learning, summarised, each linking to the original."
---

# Machine Learning
> 622 posts about Machine Learning, summarised, each linking to the original.

## Articles

### [Hyperparameter Search with Transformers and Ray Tune](https://yomu.fyi/post/hyperparameter-search-with-transformers-and-ray-tune.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: system
- Published: Nov 2, 2020
- Type: Tutorial

Hugging Face Transformers models often require hyperparameter tuning to achieve high performance, but practitioners frequently rely on basic grid search or bypass tuning entirely. The Transformers 3.1 release integrates Ray Tune into the Hugging Face Trainer to provide advanced hyperparameter optimization without losing framework customizability. By invoking hyperparameter\_search with the Ray backend, users can configure trial budgets, parallelize runs across multiple GPUs, and plug in search algorithms like HyperOptSearch alongside schedulers like ASHAScheduler. Experimental benchmarks on a BERT model using the RTE dataset show that Population-based Training reached 70.5% test accuracy in 48 GPU minutes, outperforming Grid Search at 65.4% test accuracy. The integration also supports experiment tracking tools like Weights and Biases out of the box.


### [Transformer-based Encoder-Decoder Models](https://yomu.fyi/post/transformer-based-encoder-decoder-models.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Patrick von Platen
- Published: Oct 10, 2020
- Type: Explainer

Natural language generation tasks map input sequences to target sequences whose lengths cannot be known in advance and vary by content. Standard deep neural networks struggle with these variable mappings because their inputs and targets require fixed-dimensional vector representations. While recurrent neural networks addressed this challenge by generating target sequences auto-regressively from a compressed context state, transformer-based encoder-decoder architectures became the standard paradigm in modern natural language processing. The framework processes sequence-to-sequence problems by encoding source sequences and auto-regressively decoding target sequences token by token using conditional probability distributions. A step-by-step implementation demonstrates greedy decoding with Hugging Face Transformers, passing cached encoder hidden states alongside previously generated target tokens to iteratively generate German translations from English inputs.


### [How to Use Quasi-experiments and Counterfactuals to Build Great Products](https://yomu.fyi/post/how-to-use-quasi-experiments-and-counterfactuals-to-build-great-produc.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 28, 2020
- Type: Explainer

Correlation does not establish causation, so the post presents causal inference as a way to determine why product metrics change and to validate or invalidate business strategies. The proposed evidence ladder places randomized A/B tests first, quasi-experiments such as fixed-effects regression and difference-in-differences second, and counterfactual estimation third, with descriptive statistics providing no direct causal evidence. It explains that quasi-experiments use non-random treatment and control divisions, while difference-in-differences depends on a parallel-trends assumption and fixed-effects regression assumes relevant dividing factors were collected. For a security update released to everyone, the team trained a time-series model on unaffected feature usage and global activity trends, then found no lift versus actual usage, indicating no negative effect on usage. Because observational methods bring greater uncertainty and frequent false positives, the post recommends stating assumptions, relaxing them through robustness checks, and using DAGs and Dagitty to test causal hypotheses.


### [Block Sparse Matrices for Smaller and Faster Language Models](https://yomu.fyi/post/block-sparse-matrices-for-smaller-and-faster-language-models.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: François Lagunas
- Published: Sep 10, 2020
- Type: Announcement

Full dense layers in neural networks are often unnecessarily large and can be pruned without sacrificing precision, but available sparse algebra tools and PyTorch native implementations lack efficiency. To resolve this issue, the pytorch\_block\_sparse extension introduces BlockSparseLinear as a drop-in replacement for standard linear layers alongside BlockSparseModelPatcher for on-the-fly model modifications. The library uses NVIDIA CUTLASS C++ CUDA templates derived from Yulhwa Kim's cutlass tilesparse proof of concept. Although baseline sparse operations remain roughly twice as slow as cuBLAS-optimized dense counterparts, performance scales with sparsity, making a 75% sparse matrix approximately two times faster than a dense layer while reducing memory consumption by four times. Future updates will target dynamic sparsity optimization and NVIDIA Ampere Tensor Core primitives.


### [The Reformer - Pushing the limits of language modeling](https://yomu.fyi/post/the-reformer-pushing-the-limits-of-language-modeling.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Patrick von Platen
- Published: Jul 3, 2020
- Type: Explainer

Standard transformer models hit memory bottlenecks on long sequence modeling tasks due to the quadratic asymptotic memory complexity of global self-attention and oversized positional embedding matrices. The Reformer architecture overcomes these constraints to train sequences of up to half a million tokens using under 8GB of RAM. It re-engineers transformer operations using local and Locality Sensitive Hashing self-attention, chunked feed forward layers, reversible residual layers, and axial positional encodings. In empirical benchmarks using google/reformer-crime-and-punishment, axial positional encodings reduce the model parameter count from over 136 million to approximately 2.58 million by factorizing the positional dimensions. This architectural change cuts inference memory consumption from 959 MB down to 447 MB for evaluated benchmark workloads.


### [Categorizing Products at Scale](https://yomu.fyi/post/categorizing-products-at-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Apr 30, 2020
- Type: Explainer

Shopify describes a product-categorization system for more than one billion products across the Google Product Taxonomy, which contains over 5,000 hierarchical categories. The model uses product titles, descriptions, collections, tags, vendors, and merchant-provided types, with text preprocessing and a fixed-length term-frequency HashingTF featurizer in PySpark rather than image features or vocabulary-based methods such as Word2Vec. Kesler’s Construction transforms multiclass training into a binary dataset by embedding each candidate class into feature tokens, while logistic regression provides a single, interpretable classifier that can use taxonomy relationships and reduce computational load. The resulting baseline supports more than 20 teams and downstream experiences, while schematized Kafka events and an internal annotation platform provide human feedback; the post identifies class imbalance, multilingual text, and image features as improvement areas.


### [The Evolution of Kit: Automating Marketing Using Machine Learning](https://yomu.fyi/post/the-evolution-of-kit-automating-marketing-using-machine-learning.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 4, 2020
- Type: Explainer

Kit began as a heuristic, rules-based virtual employee that simplified Facebook advertising for Shopify merchants by selecting products and budgets while defaulting other settings to best practices. The post describes its evolution into an artificially intelligent assistant that uses merchant spending behavior and store signals such as traffic and sales to recommend marketing actions. Kit's machine-learning design combines regression to predict likely spending with classification to estimate whether a budget and store state are likely to generate sales; training uses feature engineering, Apache Spark, Google Cloud ML Engine, and monitored model metrics. For real-time predictions, Bigtable stores engineered features, while a Tornado web container delegates model computation to TensorFlow Serving. The resulting system powered one third of Kit's marketing campaigns.


### [How to generate text: using different decoding methods for language generation with Transformers](https://yomu.fyi/post/how-to-generate-text-using-different-decoding-methods-for-language-gen.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Patrick von Platen
- Published: Mar 1, 2020
- Type: Tutorial

Auto-regressive language generation relies heavily on decoding methods to convert predicted next-token probability distributions into coherent text. Greedy search selects the single most probable token at each timestep, but it frequently leads to repetitive phrasing and misses high-probability sequences hidden behind low-probability intermediate words. Beam search alleviates this issue by maintaining multiple parallel hypotheses and can be paired with n-gram penalties to reduce immediate repetitions. Sampling approaches, including top-K and top-p sampling, offer alternative paths that can yield more fluent, varied open-ended generation. Using the Hugging Face transformers library, practitioners can configure these strategies via parameters on the model generate function.


### [How to train a new language model from scratch using Transformers and Tokenizers](https://yomu.fyi/post/how-to-train-a-new-language-model-from-scratch-using-transformers-and.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Julien Chaumond
- Published: Feb 14, 2020
- Type: Tutorial

Recent library enhancements simplify pretraining custom language models from raw text corpora. A small 84-million-parameter RoBERTa architecture comprising six layers, twelve attention heads, and a hidden dimension of 768 was pretrained on a 3 GB Esperanto text dataset. Training began by fitting a 52,000-vocabulary byte-level Byte-pair encoding tokenizer, which natively handled Esperanto diacritics and reduced average sequence lengths by approximately 30 percent compared to GPT-2 tokenization. The model was trained using masked language modeling, evaluated using fill-mask pipelines, and subsequently fine-tuned for part-of-speech tagging via standard token classification scripts. Final model artifacts and configurations were uploaded for community reuse via pre-trained model interfaces.


### [Using Grab’s Trust Counter Service to Detect Fraud Successfully](https://yomu.fyi/post/using-grab-s-trust-counter-service-to-detect-fraud-successfully.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Chao Wang
- Published: Oct 21, 2019
- Type: Problem & solution

Grab's Trust Platform team built the Counter service to detect fraud across business verticals like transportation, food, and payments. The platform replaces manual, multi-week engineering cycles with a self-service UI where data analysts can define and experiment with counters independently. Operating on an asynchronous ingestion and synchronous transaction model, the architecture evaluates incoming stream data, enriches it via internal services, and persists aggregated signals to ScyllaDB through Grab-Stats. A multi-bucket strategy partitions queries into fifteen-minute, hourly, and daily granularities to maintain low-latency query aggregations across wide time ranges under strict SLAs.


### [Making Grab’s Everyday App Super](https://yomu.fyi/post/making-grab-s-everyday-app-super.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Justin Bolilia
- Published: Jul 3, 2019
- Type: Explainer

Grab manages an expanding superapp ecosystem comprising ride-hailing, food delivery, payments, and partner content surfaced through the Grab Feed. As content volume grows, the platform risks overwhelming users with irrelevant information. To address this, Grab built a recommendation engine that ranks cards using signals across user profiles, content metadata, and contextual factors such as time and location. The system employs multiple recommendation strategies—including popularity metrics, user favorites, collaborative filtering, habitual patterns, and cross-platform deep embeddings—which are selected or aggregated. Recommendation quality is evaluated via offline metrics like Recall@K and NDCG alongside online engagement experiments.


### [Catwalk: Serving Machine Learning Models at Scale](https://yomu.fyi/post/catwalk-serving-machine-learning-models-at-scale.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Nutdanai Phansooksai
- Published: Jul 2, 2019
- Type: Problem & solution

As machine learning adoption expanded at Grab, individual teams created fragmented model serving solutions that duplicated engineering effort and required data scientists to handle underlying infrastructure. To resolve these inefficiencies, Grab developed Catwalk, a self-service machine learning model serving platform. The system runs TensorFlow Serving containers across a managed Kubernetes cluster integrated with Grab's observability stack. Data scientists deploy or update models simply by saving files using the tf.saved\_model API to dedicated Amazon S3 buckets, while Kubernetes automates orchestration, ingress routing, and pod autoscaling. Catwalk abstracts server management away from data scientists, shortens deployment timelines, and provides high availability during model version rollouts.


### [Guiding You Door-to-Door via Our Superapp!](https://yomu.fyi/post/guiding-you-door-to-door-via-our-superapp.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Neeraj Mishra
- Published: Apr 12, 2019
- Type: Announcement

Grab addressed passenger navigation challenges at large Southeast Asian venues such as airports and shopping centers. Satellite signals weaken through concrete and steel, creating GPS inaccuracies that caused the rendezvous distance between passengers and drivers at large venues to exceed twice the average. While introducing Entrances previously mapped over 120,000 green dots to lower rendezvous distances, passengers still struggled to locate specific pickup spots indoors. The company launched Venues, an in-app feature delivering turn-by-turn text and photo directions to designated pickup points. To support this system, operations teams surveyed sites with cameras and scanners to capture landmarks, after which in-house teams masked faces and vehicle license plates.


### [How We Harnessed the Wisdom of Crowds to Improve Restaurant Location Accuracy](https://yomu.fyi/post/how-we-harnessed-the-wisdom-of-crowds-to-improve-restaurant-location-a.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Pravin Kakar
- Published: Apr 2, 2019
- Type: Problem & solution

Grab discovered that abnormally short driver wait times often indicated restaurants registered at incorrect coordinates due to moves or onboarding errors. To fix this, Grab used driver-partner GPS pings, timestamps, and order status updates to infer true food collection locations. The system cleans the data by filtering low-quality GPS pings and isolating the longest temporal streak a driver spends within a predefined radius of the venue. Clusters of inferred pick-up points are then ranked by order volume, the proportion of off-target pick-ups, and median distance errors before routing to mapping operations for verification. This periodic correction workflow achieved a fivefold reduction in order cancellations caused by unfound merchant locations.


### [Recipe for Building a Widget: How We Helped to “Peak-Shift” Demand by Helping Passengers Understand Travel Trends](https://yomu.fyi/post/recipe-for-building-a-widget-how-we-helped-to-peak-shift-demand-by-hel.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Lara PuReum Yim
- Published: Mar 7, 2019
- Type: Problem & solution

Transport demand spikes during regular commuting hours often outpace driver availability, resulting in passenger wait times and fare surges. To mitigate these imbalances, Grab created the Travel Trends Widget for its mobile feed to redistribute ride requests toward off-peak windows. The widget uses machine learning forecasting to present historical supply-demand patterns alongside pricing trends for the upcoming two hours. To handle anticipated high query rates across millions of database entries, engineers periodically load precomputed trend data into an in-memory data structure rather than querying the database per request. The feature rolled out to feeds in Singapore and Jakarta within four weeks of initial development.


### [Understanding Supply & Demand in Ride-hailing Through the Lens of Data](https://yomu.fyi/post/understanding-supply-demand-in-ride-hailing-through-the-lens-of-data.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Aayush Garg
- Published: Feb 20, 2019
- Type: Explainer

Grab measures ride-hailing supply and demand across space and time to resolve geo-temporal allocation mismatches between moving drivers and ride-seeking passengers. The analytics pipeline defines supply as idle online drivers and demand as passengers checking fares within brief time slots, aggregating locations into geohashes. Each driver is mapped across neighbouring demand units and inversely weighted by straight-line distance, which yields the effective supply, supply-demand ratio, and supply-demand difference for each geographic polygon. Grab uses these aggregated metrics to identify marketplace imbalances, deploying driver heatmaps to shift excess supply and passenger travel trend widgets to defer time-insensitive ride requests.


### [How Shopify Uses Recommender Systems to Empower Entrepreneurs](https://yomu.fyi/post/how-shopify-uses-recommender-systems-to-empower-entrepreneurs.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Dora Jambor
- Published: Jan 25, 2019
- Type: Explainer

Shopify uses recommender systems to help merchants find relevant insights, actions, apps, themes, services, and other resources across its commerce platform. Its approach represents implicit signals such as purchases, installations, clicks, and views as user-item interactions, then applies binary classification and linear methods including SLIM and LRec to learn an item-item similarity matrix and rank unseen items. The framework supports task-parallel training, interpretable coefficients, user-feature enrichment, and cold-start handling for new users or items, while evaluation considers precision alongside diversity, novelty, serendipity, and business-specific goals. In initial A/B tests, personalized app recommendations produced a 50% higher install rate, home feeds were up to 12% more likely to be rated useful, expert matching increased response and collaboration, and theme recommendations made merchants over 10% more likely to launch stores.


### [Grab Senior Data Scientist Liuqin Yang Wins Beale-Orchard-Hays Prize](https://yomu.fyi/post/grab-senior-data-scientist-liuqin-yang-wins-beale-orchard-hays-prize.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Yang Liuqin
- Published: Jul 20, 2018
- Type: Announcement

Grab Senior Data Scientist Dr. Liuqin Yang, Professor Defeng Sun, and Professor Kim-Chuan Toh received the 2018 Beale-Orchard-Hays Prize for their research paper introducing SDPNAL+. The software employs a majorised semismooth Newton-CG augmented Lagrangian method to solve large-scale semidefinite programming problems with nonnegative constraints. While traditional methods struggled beyond matrix dimensions of 2,000 and 5,000 constraints, SDPNAL+ successfully scales to matrix dimensions of 9,261 and over 12 million constraints. In benchmark testing, the software solved a problem on a desktop PC in 1.5 hours that required 122 hours on a 56-core CPU and 128-GPU cluster using a traditional solver. Grab implements these optimisation techniques to accelerate its passenger-driver allocation algorithms by hundreds of times.


### [The Data and Science Behind GrabShare Part I: Verifying Potential and Developing the Algorithm](https://yomu.fyi/post/the-data-and-science-behind-grabshare-part-i-verifying-potential-and-d.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Tang Muchen
- Published: Oct 20, 2017
- Type: Problem & solution

Expanding from point-to-point dispatch services to dynamic carpooling requires matching independent passenger requests traveling in similar directions without causing unacceptable delays. Grab evaluated the feasibility of its GrabShare service by analyzing historical trip data with DBSCAN clustering on coordinates projected into a Universal Transverse Mercator system. This analysis demonstrated that 35% to 46% of rides across typical daytime windows fell into tight geographic clusters with near-identical pickup and drop-off coordinates. The resulting matching framework adapts the baseline dispatch flow by searching for in-transit drivers and enforcing real-time seat reservation constraints. Route assignment decisions subsequently evaluate detour times, trip angles, and expected arrival times to ensure driver utilization improves while total driving distance decreases.


### [How to Go from a Quick Idea to an Essential Feature in Four Steps](https://yomu.fyi/post/how-to-go-from-a-quick-idea-to-an-essential-feature-in-four-steps.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Da Huang
- Published: May 16, 2017
- Type: Problem & solution

Grab engineered an in-app messaging platform, GrabChat, to help drivers and passengers coordinate pickups across Southeast Asian markets characterized by weak 2G connectivity and high packet loss. The team developed an in-house TCP messaging architecture consisting of a TCP gateway named Gundam and a message dispatcher named Hermes connected to internal backend services over HTTPS. To protect backend server resources from resend loops during poor connection states, the communication protocol adopts a "server only push once" model that delegates retry handling to the client. Data science evaluations using a pre-trained cancellation prediction model confirmed that GrabChat adoption correlated with reduced booking cancellations. Following early usage feedback, the team further iterated on the feature by introducing pre-written message templates to reduce driver distraction on the road.


[Newer posts](https://yomu.fyi/topic/machine-learning/page/30.md) · [Older posts](https://yomu.fyi/topic/machine-learning/page/32.md)