My Journey to a serverless transformers pipeline on Google Cloud

Summary

The post documents Maxence Dominici’s effort to deploy a Transformers sentiment-analysis pipeline on Google Cloud for classifying customer reviews from Discord. After considering AI Platform Prediction and App Engine, he chose Cloud Run with a Docker image because custom checkpoint handling, dependencies, and resource requirements made the other options unsuitable. The implementation uses a PyTorch DistilBERT model fine-tuned on SST-2, Flask, Gunicorn, and a GET request protected by an optional API key; the shown handler retrieves a review but passes the hard-coded text “that was great” to the classifier. Cloud Run is configured with 4 GB of memory, one worker, and one thread to limit memory pressure and billing. Requests take under five seconds including model loading, while cold starts may add about ten seconds, and the author reports that the pretrained model’s accuracy was already impressive.

Takeaways

1. 1

   Cloud Run was selected after AI Platform Prediction encountered beta-service issues and App Engine had TensorFlow installation problems and limited PyTorch concurrency.

2. 2

   The container uses PyTorch DistilBERT fine-tuned on SST-2, with 4 GB of memory and a single worker and thread to accommodate model memory usage and limit billing exposure.

3. 3

   Measured request handling took less than five seconds including pipeline loading and prediction, while cold starts could add roughly ten seconds; loading the model at startup was identified as a possible improvement.