# Deploy GPT-J 6B for inference using  Hugging Face Transformers and Amazon SageMaker

huggingface.co · Philipp Schmid · Jan 11, 2022

**Type:** Tutorial

## Summary

Deploying EleutherAI's 6 billion parameter GPT-J model for production inference presents latency hurdles due to large memory footprints and lengthy startup times. Loading the model via standard methods takes up to several minutes, conflicting with strict real-time response limits such as Amazon SageMaker's 60-second threshold. To overcome this limitation, the model is serialized using PyTorch's native save mechanisms, packaged into a compressed archive with supporting assets, and stored on Amazon S3. This alternative loading workflow reduces GPT-J load times down to 7.7 seconds. An Amazon SageMaker real-time endpoint is then deployed on an NVIDIA T4 GPU instance using the Hugging Face Inference Toolkit.

## Context

EleutherAI's 6 billion parameter GPT-J model has a ~24GB memory footprint in float32 and requires significant RAM to load. Default loading methods take 1 minute and 23 seconds from disk or over 3 minutes over network, exceeding Amazon SageMaker's 60-second response limit for real-time inference.

## Approach / What changed

Convert the float16 model using torch.save to serialize the entire module, packaging it into a model.tar.gz archive on S3 alongside tokenizers. Deploy this artifact to an Amazon SageMaker real-time endpoint on an ml.g4dn.xlarge instance using HuggingFaceModel, and send parameterized inference payloads.

## Takeaways

- Using torch.save and torch.load drops GPT-J loading time from 83 seconds to 7.7 seconds, representing an approximately 10.5x speedup.
- Serializing modules via PyTorch binds data to specific classes, requiring matched PyTorch and Transformers library versions across saving and loading environments.
- SageMaker real-time endpoints require responses within a 60-second window, making fast initial model loading critical for endpoint reliability.

**Tags:** [AWS](https://yomu.fyi/topic/aws), [Deployment](https://yomu.fyi/topic/deployment), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/gptj-sagemaker)