The Age of Machine Learning As Code Has Arrived

Summary

Recent findings from the 2021 State of AI Report and Kaggle State of Machine Learning and Data Science Survey indicate that machine learning is expanding into critical infrastructure while Transformers become general-purpose architectures across text, vision, and audio. In response, organizations face questions about scaling infrastructure, team composition, and engineering maturity. Rather than treating machine learning as isolated sandbox experiments or hiring solely data scientists, teams benefit from adopting established software engineering and DevOps principles like versioning, testing, automation, and continuous deployment. Furthermore, the rise of pre-trained Transformer architectures enables practitioners to fine-tune existing off-the-shelf models rather than training from scratch, reducing compute costs and training duration. Tools from platforms such as Hugging Face streamline model deployment, latency optimization, and infrastructure abstraction.

Context

The 2021 State of AI Report and Kaggle survey indicate rapid growth of machine learning in mission-critical infrastructure, raising questions about whether industry maturity and engineering workflows can keep pace.

Approach / What changed

Apply proven software engineering and DevOps practices—such as automation, testing, versioning, and cloud infrastructure abstraction—while adopting pre-trained Transformer models and off-the-shelf tooling from Hugging Face instead of training deep learning models from scratch.

Takeaways

• According to the Kaggle survey, 75% of respondents use cloud services and over 45% use enterprise ML platforms, led by Amazon SageMaker, Databricks, and Azure ML Studio.
• Transformers are expanding beyond NLP into computer vision, audio, and point clouds, setting benchmarks with models like Vision Transformer and CoAtNet while requiring fewer training compute resources.
• Decade-old software engineering practices—including versioning, reusability, testability, automation, and monitoring—must be applied to machine learning systems to move beyond sandbox experiments into production.

Related reading

• huggingface.co · Jan 18, 2021

  How we sped up transformer inference 100x for 🤗 API customers

  Deploying large transformer models in production poses severe latency and scaling hurdles for machine learning engineers. Hugging Face achieved a cumulative 100x inference acceleration on its Accelerated Inference API by pairing library-level algorithmic adjustments with low-level hardware compilation. The initial 10x improvement reduces forward-pass computations, such as restricting attention calculations to the latest token in GPT generation tasks, while utilizing Rust-based tokenizers and caching mechanisms. The subsequent 10x speedup requires custom static graph compilation targeted to selected CPU or GPU hardware profiles, applying unused flow removal, instruction-specific layer fusion, and precision quantization via ONNX Runtime. These coordinated techniques enable real-time prediction latency across varied model sizes and demand profiles without sacrificing task accuracy.

  Nicolas Patry

• huggingface.co · Sep 14, 2021

  Introducing Optimum: The Optimization Toolkit for Transformers at Scale

  Running Transformer models fast and efficiently at production scale presents a significant challenge due to the complex compatibility between model architectures, acceleration techniques like quantization or sparsity, and specific silicon hardware features. Hugging Face introduced Optimum, an open-source library built to abstract hardware acceleration complexity and optimize training and inference on target devices through direct collaborations with hardware partners. Using quantization as a primary example, engineering teams often struggle with eager-mode model graph modifications, kernel compatibility checks, calibration parameter tuning, and acceptable accuracy loss trade-offs. Optimum addresses these challenges by integrating tools such as Intel Neural Compressor, which supports post-training quantization, quantization-aware training, and dynamic quantization driven by user-defined YAML configurations. Through these hardware-tailored integrations, the library aims to democratize production performance and reduce the aggregate energy consumed by machine learning workloads.