---
title: "Python"
description: "252 posts about Python, summarised, each linking to the original."
---

# Python
> 252 posts about Python, summarised, each linking to the original.

## Articles

### [Introducing Optimum: The Optimization Toolkit for Transformers at Scale](https://yomu.fyi/post/introducing-optimum-the-optimization-toolkit-for-transformers-at-scale.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Morgan Funtowicz, Ella Charlaix, Michael Benayoun, Jeff Boudier
- Published: Sep 14, 2021
- Type: Announcement

Running Transformer models fast and efficiently at production scale presents a significant challenge due to the complex compatibility between model architectures, acceleration techniques like quantization or sparsity, and specific silicon hardware features. Hugging Face introduced Optimum, an open-source library built to abstract hardware acceleration complexity and optimize training and inference on target devices through direct collaborations with hardware partners. Using quantization as a primary example, engineering teams often struggle with eager-mode model graph modifications, kernel compatibility checks, calibration parameter tuning, and acceptable accuracy loss trade-offs. Optimum addresses these challenges by integrating tools such as Intel Neural Compressor, which supports post-training quantization, quantization-aware training, and dynamic quantization driven by user-defined YAML configurations. Through these hardware-tailored integrations, the library aims to democratize production performance and reduce the aggregate energy consumed by machine learning workloads.


### [Welcome spaCy to the Hugging Face Hub](https://yomu.fyi/post/welcome-spacy-to-the-hugging-face-hub.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Omar Sanseviero, Ines Montani
- Published: Jul 13, 2021
- Type: Announcement

Hugging Face has integrated spaCy pipelines into the Hugging Face Hub to simplify the distribution, testing, and deployment of natural language processing models across the community. More than 60 canonical models from the spaCy 3.1 release are hosted in the spaCy organization, alongside discoverable community pipelines. Users can package and upload models directly through the command line using the spacy-huggingface-hub extension, which auto-generates required metadata and model cards. Hosted pipelines offer direct pip installation from live wheel URLs, interactive browser exploration, and production-ready HTTP endpoints through the Inference API. The platform provides immediate interactive widget support for named entity recognition components, with upcoming support planned for text classification and part-of-speech tagging pipelines.


### [Deploy Hugging Face models easily with Amazon SageMaker](https://yomu.fyi/post/deploy-hugging-face-models-easily-with-amazon-sagemaker.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Philipp Schmid
- Published: Jul 8, 2021
- Type: Announcement

Hugging Face and Amazon introduced an inference solution integrating Hugging Face Transformers with Amazon SageMaker. The SageMaker Hugging Face Inference Toolkit and specialized Deep Learning Containers enable deployment of trained models or publicly available Hub models to managed production endpoints. Users configure endpoints with minimal code using the SageMaker Python SDK, standard pipelines, or custom inference scripts that override default behaviors. The architecture supports models stored in Amazon S3 as well as direct references via HF\_MODEL\_ID and HF\_TASK environment variables. Deployments benefit from native AWS infrastructure features, including built-in monitoring, Identity and Access Management permission controls, and Virtual Private Cloud connectivity.


### [Using & Mixing Hugging Face Models with Gradio 2.0](https://yomu.fyi/post/using-mixing-hugging-face-models-with-gradio-2-0.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Abubakar Abid
- Published: May 25, 2021
- Type: Announcement

The Hugging Face Model Hub hosts more than 10,000 user-submitted machine learning models spanning natural language processing, image classification, and audio processing. Machine learning developers increasingly need to demo models to interdisciplinary collaborators or non-programmers to uncover biases and failure points. The Gradio 2.0 library enables users to create graphical user interfaces and load almost any Hugging Face model using a single line of code. By default, these interfaces query Hugging Face's hosted Inference API without requiring an API key, though users can also run computations locally via the transformers library. Gradio interfaces can be customized through the Interface class and composed to load multiple models in parallel or chain them in series.


### [Elixir at Ramp 🧪](https://yomu.fyi/post/elixir-at-ramp.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Pablo Meier
- Published: May 24, 2021
- Type: Explainer

Ramp engineer Pablo describes why Elixir suited several critical services, while arguing that technology choice is only one part of engineering success. He contrasts BEAM, Erlang’s virtual machine, and OTP’s abstractions for reliable, concurrent systems with the adoption and developer-experience advantages of Elixir, including familiar syntax, Mix, iex, hot code reloading, doctests, macros, and sigils. Immutability and aliased module calls are presented as safeguards against hidden state changes and circular-import problems in growing codebases, though the post also notes that Erlang and Elixir lag behind bolted-on type checkers. Ramp uses Elixir for its real-time Authorization engine and Risk analysis platform; the Authorization service is described as exceptionally stable and flexible for rapid feature development. The company’s largest backend remains Python, and the conclusion prioritizes sleep, trust, ownership, and psychological safety over any specific stack.


### [Hooking Up Fly Metrics](https://yomu.fyi/post/hooking-up-fly-metrics.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: May 20, 2021
- Type: Tutorial

Fly apps include built-in Prometheus instrumentation, exposing automatically collected metrics through a public Prometheus endpoint and allowing applications to export custom metrics. The walkthrough connects an organization’s Fly Prometheus URL to Grafana with a Fly token, then explains dashboard variables and PromQL queries using counters, gauges, histograms, labels, irate(), rate(), increase(), sum(), and regional grouping. Examples cover network traffic, TCP connections, HTTP response rates by region, and response-time heatmaps built from histogram buckets ranging from 5ms to 120s. Application instrumentation is shown for Go, Python Flask, and Node.js, with scraping configured in fly.toml through a metrics port and path. The post concludes that metrics are currently free, with some tier free forever, although a future pricing plan is expected for ambitious usage.


### [Scaling-up BERT Inference on CPU (Part 1)](https://yomu.fyi/post/scaling-up-bert-inference-on-cpu-part-1.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Morgan Funtowicz
- Published: Apr 20, 2021
- Type: Benchmark

Scaling BERT-like model inference on CPUs presents practical deployment challenges as transformer architectures proliferate in production environments. Using a reproducible Hydra-based benchmarking framework, inference workloads were evaluated on an Amazon Web Services c5.metal instance featuring an Intel Xeon Platinum 8275 processor with AVX512 and VNNI support. Out-of-the-box evaluations showed PyTorch 1.8.1 delivering lower latency than Google TensorFlow 2.4.0, likely due to PyTorch relying on OpenMP and Intel MKL/oneDNN rather than Eigen. Implementing multiple inference streams by pinning parallel model instances to dedicated CPU cores yielded near-linear throughput scaling. Matching instance core allocations to specific problem sizes also demonstrated that smaller workloads can achieve optimal latency on fewer cores, enabling up to a sixfold reduction in cloud compute costs.


### [Introducing 🤗 Accelerate](https://yomu.fyi/post/introducing-accelerate.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Sylvain Gugger
- Published: Apr 16, 2021
- Type: Announcement

🤗 Accelerate enables PyTorch developers to execute raw training loops across CPUs, multi-GPU setups, and TPUs with minimal boilerplate modifications. Standard distributed training typically demands manual device assignments, DistributedSampler configurations, and divergent code paths that break single-device portability. By wrapping models, optimizers, and dataloaders through a unified prepare call, Accelerate automatically manages hardware placement, mixed precision operations, and sampler batch slicing without requiring custom distributed samplers. Distributed evaluation is streamlined using a gather utility that consolidates prediction tensors across active worker processes. Furthermore, a dedicated command-line interface provides an interactive questionnaire to store runtime configuration defaults and orchestrate local or AWS SageMaker runs.


### [Distributed Training: Train BART/T5 for Summarization using 🤗 Transformers and Amazon SageMaker](https://yomu.fyi/post/distributed-training-train-bart-t5-for-summarization-using-transformer.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Philipp Schmid
- Published: Apr 8, 2021
- Type: Tutorial

Hugging Face and Amazon SageMaker introduced optimized Deep Learning Containers alongside a SageMaker Python SDK estimator to streamline distributed transformer model training. This workflow demonstrates fine-tuning the 400-million-parameter facebook/bart-large-cnn model for dialogue summarization on the 16,000-conversation samsum dataset. Using SageMaker Data Parallelism enabled in the distribution configuration, training executes across two ml.p3dn.24xlarge compute instances equipped with sixteen total GPUs and a cumulative batch size of 64. The HuggingFace estimator pulls the run\_summarization.py script directly from GitHub, enables mixed precision with fp16, and tracks evaluation performance through ROUGE scores. Following completion, the unzipped model artifacts and a generated model card are pushed to the Hugging Face Hub using the huggingface\_hub library for hosted inference.


### [Understanding BigBird's Block Sparse Attention](https://yomu.fyi/post/understanding-bigbird-s-block-sparse-attention.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Vasudev Gupta
- Published: Mar 31, 2021
- Type: Explainer

Transformer-based models face severe computational bottlenecks due to quadratic time and memory complexity, making sequences longer than 512 tokens impractical. BigBird addresses this limitation by approximating full attention with block sparse attention, scaling context lengths up to 4096 tokens at a lower computational cost. Instead of attending to all tokens, BigBird combines sliding attention for local context, global tokens for long-range relationships, and random tokens to accelerate information transfer. The model is integrated into HuggingFace Transformers, enabling fine-tuning for tasks like long-document summarization and extractive question answering. When sequences are shorter than 1024 tokens, using original full attention remains recommended.


### [The Partnership: Amazon SageMaker and Hugging Face](https://yomu.fyi/post/the-partnership-amazon-sagemaker-and-hugging-face.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Philipp Schmid
- Published: Mar 23, 2021
- Type: Announcement

Hugging Face and Amazon formed a strategic partnership establishing Amazon Web Services as Hugging Face's preferred cloud provider to simplify training and deploying Transformer models. To facilitate this collaboration, the teams introduced Hugging Face Deep Learning Containers optimized for PyTorch and TensorFlow in Amazon SageMaker. The integration provides an extension to the SageMaker Python SDK, enabling data science teams to configure distributed training jobs across single-node and multi-node GPU clusters. Workflows integrate directly with SageMaker Automatic Model Tuning, SageMaker Studio, and Amazon S3 for managed experiment tracking and artifact storage. The open-source containers allow trained models to be exported for inference via SageMaker or published directly to the Hugging Face Model Hub while utilizing EC2 Spot instances to reduce compute expenses.


### [Fine-Tune Wav2Vec2 for English ASR in Hugging Face with 🤗 Transformers](https://yomu.fyi/post/fine-tune-wav2vec2-for-english-asr-in-hugging-face-with-transformers.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Patrick von Platen
- Published: Mar 12, 2021
- Type: Tutorial

Wav2Vec2 is a pretrained Automatic Speech Recognition model that learns speech representations from over 50,000 hours of unlabeled audio using a contrastive objective and masked feature vectors. The model can be fine-tuned end-to-end on labeled speech datasets using Connectionist Temporal Classification without requiring an external language model. To demonstrate this process, a base-sized checkpoint is fine-tuned on the Timit dataset, which contains five hours of training data. Data preparation involves normalizing transcriptions to lowercase, removing special punctuation characters, and configuring both a specialized feature extractor and tokenizer. Evaluating the fine-tuned acoustic model without padding on the Timit test dataset yields a word error rate of 22.1%.


### [Simple considerations for simple people building fancy neural networks](https://yomu.fyi/post/simple-considerations-for-simple-people-building-fancy-neural-networks.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Victor Sanh
- Published: Feb 25, 2021
- Type: Explainer

Building and training neural networks often presents frustrating debugging challenges because implementations can contain subtle errors while still running without crashes and achieving decent performance. The author shares a mental framework derived from natural language processing research to systematically construct and debug deep learning workflows. Practitioners should begin with thorough qualitative and quantitative dataset analysis before implementing simple baseline models to establish performance benchmarks. When coding the network, developers should verify implementation correctness by overfitting a tiny batch without regularization and tracking training dynamics such as losses, parameters, and gradients. Hyperparameter optimization should prioritize understanding underlying component behaviors through targeted random grid searches rather than launching hundreds of blind runs.


### [Retrieval Augmented Generation with Huggingface Transformers and Ray](https://yomu.fyi/post/retrieval-augmented-generation-with-huggingface-transformers-and-ray.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: system
- Published: Feb 10, 2021
- Type: Problem & solution

Huggingface Transformers added the Retrieval Augmented Generation architecture to combine standard sequence-to-sequence models with external document retrieval for knowledge-intensive tasks. Scaling RAG fine-tuning across multiple GPUs previously created bottlenecks because the torch.distributed implementation forced the single rank 0 worker to handle all retrieval index lookups. To resolve this synchronization bottleneck, Ray was integrated into the contextual document retrieval mechanism using stateful actor abstractions. Dedicated Ray processes separate from the training workers now load the large index and process retrieval requests concurrently. This architectural separation delivers up to a 2x speedup per retrieval call and removes the strict dependency on PyTorch for training.


### [Hugging Face on PyTorch / XLA TPUs](https://yomu.fyi/post/hugging-face-on-pytorch-xla-tpus.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: ds, Lysandre
- Published: Feb 9, 2021
- Type: Announcement

PyTorch / XLA integrates Cloud TPU training into Hugging Face transformers while preserving the standard Trainer interface. The implementation adds an xla device type and uses lazy execution to trace intermediate representation graphs before lowering them to XLA Higher Level Operations for compilation. Parallel data loading through MpDeviceLoader overlaps CPU tracing with TPU execution, while xm.optimizer\_step coordinates gradient consolidation across data-parallel replicas. Checkpoints use xm.save to transfer tensors via CPU storage and restrict disk writes to the master process. Benchmarks on a Cloud TPU v3-8 system training bert-large-uncased on WikiText-103 demonstrate completion times of 178.4 minutes in FP32 precision and 106.4 minutes in BF16 precision.


### [Fit More and Train Faster With ZeRO via DeepSpeed and FairScale](https://yomu.fyi/post/fit-more-and-train-faster-with-zero-via-deepspeed-and-fairscale.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Stas Bekman
- Published: Jan 19, 2021
- Type: Benchmark

Machine learning model sizes frequently outpace consumer GPU memory, preventing practitioners from loading or fine-tuning large architectures on standard hardware. To address these resource constraints, the Hugging Face Trainer introduced experimental support for Zero Redundancy Optimizer features from DeepSpeed and FairScale starting in transformers v4.2.0. In dual-GPU benchmarks with a t5-large model, FairScale and DeepSpeed integrations expanded allowable batch sizes and shortened training durations compared to baseline DistributedDataParallel runs. Furthermore, DeepSpeed permitted single-GPU fine-tuning of a t5-3b model on a 24GB RTX-3090 card at batch size 20, whereas the baseline execution crashed immediately with an out-of-memory error. These optimizations manage memory allocation internally to curb memory fragmentation while requiring changes only to training scripts rather than model code.


### [Xendit Built a Counter-Fraud Watchlist for the Fintech Industry](https://yomu.fyi/post/xendit-built-a-counter-fraud-watchlist-for-the-fintech-industry.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Published: Dec 2, 2020
- Type: Problem & solution

Payment processor Xendit needed a fast, cost-effective counter-fraud watchlist system to screen transactions against international sanctions lists in real time without relying on expensive providers like Worldcheck or Refinitiv. Leveraging their existing familiarity with Postgres, the team loaded parsed sanctions lists from the United Nations and the Indonesian government into Supabase. They implemented full-text search using the Postgres Trigram extension, enabling relevance scoring across queried records. A dedicated database function handles the search queries directly via Xendit's Python clients, allowing the entire solution to reach production in under one week. The deployment has operated reliably in production for nine months without issues, with potential future iterations planned around machine learning techniques.


### [How to Build a Production Grade Workflow with SQL Modelling](https://yomu.fyi/post/how-to-build-a-production-grade-workflow-with-sql-modelling.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 19, 2020
- Type: Problem & solution

Shopify’s Starscream data platform handled 76,000 jobs and wrote 300 terabytes daily, but its PySpark workflow was slow and cumbersome for commonplace reporting. An audit found that about 70 percent of PySpark jobs were full batch queries that did not require generalized computing, while users often wrote SQL before translating it into Python. The team built Seamster with dbt and Google BigQuery, adding source and model structures, ownership metadata, Python-based unit tests with fixed mock inputs, CI validation, and warehouse rules. A base layer isolates raw-source changes, while models are organized into base, application-ready, and presentation layers and defined through SQL, schema, README, and test files. After a multiweek beta, most users reported shipping models in days rather than weeks, although dbt’s incremental support could not safely handle some late-arriving-data, key-resolution, rebuild, and very large-model cases.


### [Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models](https://yomu.fyi/post/leveraging-pre-trained-language-model-checkpoints-for-encoder-decoder.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Patrick von Platen
- Published: Nov 9, 2020
- Type: Tutorial

Pre-training sequence-to-sequence transformer models incurs massive computational costs, limiting development primarily to large institutions. To mitigate these expenses, encoder-decoder architectures can be warm-started using existing pre-trained checkpoints from encoder-only or decoder-only models like BERT and GPT-2. This walkthrough details the methodology and implementation of warm-starting sequence-to-sequence architectures using Hugging Face Transformers. By utilizing the EncoderDecoderModel framework alongside Seq2SeqTrainer, practitioners can construct and fine-tune models such as BERT2BERT on datasets like CNN/DailyMail. The resulting fully trained BERT2BERT model achieves a ROUGE-2 score of 18.22 on the full evaluation set, matching competitive sequence generation baselines at a fraction of standard pre-training costs.


### [Porting fairseq wmt19 translation system to transformers](https://yomu.fyi/post/porting-fairseq-wmt19-translation-system-to-transformers.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Stas Bekman
- Published: Nov 3, 2020
- Type: Tutorial

The fairseq WMT19 news translation system was ported to the Hugging Face transformers library under the FSMT architecture. Development began with an initial proxy notebook emulating the transformers API before implementing native configuration, modeling, tokenization, and checkpoint conversion scripts. The author first targeted English-Russian models to handle complex dual-vocabulary requirements and directly evaluate output quality during testing, later extending the implementation to merged-vocabulary language pairs. Although model ensembling was omitted due to library limitations, removing fairseq optimizer states reduced the final model download footprint from 13GB to 1.1GB. The implementation was validated locally against converted fairseq checkpoints and merged into the main codebase after two weeks of iterative review.


[Newer posts](https://yomu.fyi/topic/python/page/11.md) · [Older posts](https://yomu.fyi/topic/python/page/13.md)