Deploy Hugging Face models easily with Amazon SageMaker

Summary

Hugging Face and Amazon introduced an inference solution integrating Hugging Face Transformers with Amazon SageMaker. The SageMaker Hugging Face Inference Toolkit and specialized Deep Learning Containers enable deployment of trained models or publicly available Hub models to managed production endpoints. Users configure endpoints with minimal code using the SageMaker Python SDK, standard pipelines, or custom inference scripts that override default behaviors. The architecture supports models stored in Amazon S3 as well as direct references via HF_MODEL_ID and HF_TASK environment variables. Deployments benefit from native AWS infrastructure features, including built-in monitoring, Identity and Access Management permission controls, and Virtual Private Cloud connectivity.

Context

Deploying transformer-based deep learning models to scalable, secure production endpoints often involves complex environment configuration, serving stack maintenance, and custom pre- and post-processing code.

Approach / What changed

Hugging Face created Inference Deep Learning Containers and a SageMaker Inference Toolkit that leverage transformers pipelines for zero-code deployments from Amazon S3 or the Hugging Face Model Hub, while also supporting custom inference code.

Takeaways

• The SageMaker Hugging Face Inference Toolkit uses transformers pipelines to enable zero-code inference deployments without custom pre- or post-processing code.
• Models can be deployed directly from the Hugging Face Model Hub by supplying HF_MODEL_ID and HF_TASK environment variables.
• Deployments can be instantiated using the SageMaker Python SDK, boto3, AWS CLI, Terraform, or CloudFormation templates.

Related reading

• huggingface.co · Mar 23, 2021

  The Partnership: Amazon SageMaker and Hugging Face

  Hugging Face and Amazon formed a strategic partnership establishing Amazon Web Services as Hugging Face's preferred cloud provider to simplify training and deploying Transformer models. To facilitate this collaboration, the teams introduced Hugging Face Deep Learning Containers optimized for PyTorch and TensorFlow in Amazon SageMaker. The integration provides an extension to the SageMaker Python SDK, enabling data science teams to configure distributed training jobs across single-node and multi-node GPU clusters. Workflows integrate directly with SageMaker Automatic Model Tuning, SageMaker Studio, and Amazon S3 for managed experiment tracking and artifact storage. The open-source containers allow trained models to be exported for inference via SageMaker or published directly to the Hugging Face Model Hub while utilizing EC2 Spot instances to reduce compute expenses.

  Philipp Schmid

• huggingface.co · Apr 8, 2021

  Distributed Training: Train BART/T5 for Summarization using 🤗 Transformers and Amazon SageMaker

  Hugging Face and Amazon SageMaker introduced optimized Deep Learning Containers alongside a SageMaker Python SDK estimator to streamline distributed transformer model training. This workflow demonstrates fine-tuning the 400-million-parameter facebook/bart-large-cnn model for dialogue summarization on the 16,000-conversation samsum dataset. Using SageMaker Data Parallelism enabled in the distribution configuration, training executes across two ml.p3dn.24xlarge compute instances equipped with sixteen total GPUs and a cumulative batch size of 64. The HuggingFace estimator pulls the run_summarization.py script directly from GitHub, enables mixed precision with fp16, and tracks evaluation performance through ROUGE scores. Following completion, the unzipped model artifacts and a generated model card are pushed to the Hugging Face Hub using the huggingface_hub library for hosted inference.