How to Move Independent Work Through an asyncio Pipeline
A practical asyncio workflow for moving independent work through typed events, queues, concurrent tasks, and awaited API calls. Learn where validation and fallback belong.
Loading…
Topic
Python
248 posts about Python. Every summary links to the original.
- 248 posts
- 28 companies
- 1 explainer
See the idea
Python explainers
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Elixir at Ramp 🧪
Ramp engineer Pablo describes why Elixir suited several critical services, while arguing that technology choice is only one part of engineering success. He contrasts BEAM, Erlang’s virtual machine, and OTP’s abstractions for reliable, concurrent systems with the adoption and developer-experience advantages of Elixir, including familiar syntax, Mix, iex, hot code reloading, doctests, macros, and sigils. Immutability and aliased module calls are presented as safeguards against hidden state changes and circular-import problems in growing codebases, though the post also notes that Erlang and Elixir lag behind bolted-on type checkers. Ramp uses Elixir for its real-time Authorization engine and Risk analysis platform; the Authorization service is described as exceptionally stable and flexible for rapid feature development. The company’s largest backend remains Python, and the conclusion prioritizes sleep, trust, ownership, and psychological safety over any specific stack.
Hooking Up Fly Metrics
Fly apps include built-in Prometheus instrumentation, exposing automatically collected metrics through a public Prometheus endpoint and allowing applications to export custom metrics. The walkthrough connects an organization’s Fly Prometheus URL to Grafana with a Fly token, then explains dashboard variables and PromQL queries using counters, gauges, histograms, labels, irate(), rate(), increase(), sum(), and regional grouping. Examples cover network traffic, TCP connections, HTTP response rates by region, and response-time heatmaps built from histogram buckets ranging from 5ms to 120s. Application instrumentation is shown for Go, Python Flask, and Node.js, with scraping configured in fly.toml through a metrics port and path. The post concludes that metrics are currently free, with some tier free forever, although a future pricing plan is expected for ambitious usage.
Scaling-up BERT Inference on CPU (Part 1)
Scaling BERT-like model inference on CPUs presents practical deployment challenges as transformer architectures proliferate in production environments. Using a reproducible Hydra-based benchmarking framework, inference workloads were evaluated on an Amazon Web Services c5.metal instance featuring an Intel Xeon Platinum 8275 processor with AVX512 and VNNI support. Out-of-the-box evaluations showed PyTorch 1.8.1 delivering lower latency than Google TensorFlow 2.4.0, likely due to PyTorch relying on OpenMP and Intel MKL/oneDNN rather than Eigen. Implementing multiple inference streams by pinning parallel model instances to dedicated CPU cores yielded near-linear throughput scaling. Matching instance core allocations to specific problem sizes also demonstrated that smaller workloads can achieve optimal latency on fewer cores, enabling up to a sixfold reduction in cloud compute costs.
Introducing 🤗 Accelerate
🤗 Accelerate enables PyTorch developers to execute raw training loops across CPUs, multi-GPU setups, and TPUs with minimal boilerplate modifications. Standard distributed training typically demands manual device assignments, DistributedSampler configurations, and divergent code paths that break single-device portability. By wrapping models, optimizers, and dataloaders through a unified prepare call, Accelerate automatically manages hardware placement, mixed precision operations, and sampler batch slicing without requiring custom distributed samplers. Distributed evaluation is streamlined using a gather utility that consolidates prediction tensors across active worker processes. Furthermore, a dedicated command-line interface provides an interactive questionnaire to store runtime configuration defaults and orchestrate local or AWS SageMaker runs.
Distributed Training: Train BART/T5 for Summarization using 🤗 Transformers and Amazon SageMaker
Hugging Face and Amazon SageMaker introduced optimized Deep Learning Containers alongside a SageMaker Python SDK estimator to streamline distributed transformer model training. This workflow demonstrates fine-tuning the 400-million-parameter facebook/bart-large-cnn model for dialogue summarization on the 16,000-conversation samsum dataset. Using SageMaker Data Parallelism enabled in the distribution configuration, training executes across two ml.p3dn.24xlarge compute instances equipped with sixteen total GPUs and a cumulative batch size of 64. The HuggingFace estimator pulls the run_summarization.py script directly from GitHub, enables mixed precision with fp16, and tracks evaluation performance through ROUGE scores. Following completion, the unzipped model artifacts and a generated model card are pushed to the Hugging Face Hub using the huggingface_hub library for hosted inference.
Understanding BigBird's Block Sparse Attention
Transformer-based models face severe computational bottlenecks due to quadratic time and memory complexity, making sequences longer than 512 tokens impractical. BigBird addresses this limitation by approximating full attention with block sparse attention, scaling context lengths up to 4096 tokens at a lower computational cost. Instead of attending to all tokens, BigBird combines sliding attention for local context, global tokens for long-range relationships, and random tokens to accelerate information transfer. The model is integrated into HuggingFace Transformers, enabling fine-tuning for tasks like long-document summarization and extractive question answering. When sequences are shorter than 1024 tokens, using original full attention remains recommended.
The Partnership: Amazon SageMaker and Hugging Face
Hugging Face and Amazon formed a strategic partnership establishing Amazon Web Services as Hugging Face's preferred cloud provider to simplify training and deploying Transformer models. To facilitate this collaboration, the teams introduced Hugging Face Deep Learning Containers optimized for PyTorch and TensorFlow in Amazon SageMaker. The integration provides an extension to the SageMaker Python SDK, enabling data science teams to configure distributed training jobs across single-node and multi-node GPU clusters. Workflows integrate directly with SageMaker Automatic Model Tuning, SageMaker Studio, and Amazon S3 for managed experiment tracking and artifact storage. The open-source containers allow trained models to be exported for inference via SageMaker or published directly to the Hugging Face Model Hub while utilizing EC2 Spot instances to reduce compute expenses.
Fine-Tune Wav2Vec2 for English ASR in Hugging Face with 🤗 Transformers
Wav2Vec2 is a pretrained Automatic Speech Recognition model that learns speech representations from over 50,000 hours of unlabeled audio using a contrastive objective and masked feature vectors. The model can be fine-tuned end-to-end on labeled speech datasets using Connectionist Temporal Classification without requiring an external language model. To demonstrate this process, a base-sized checkpoint is fine-tuned on the Timit dataset, which contains five hours of training data. Data preparation involves normalizing transcriptions to lowercase, removing special punctuation characters, and configuring both a specialized feature extractor and tokenizer. Evaluating the fine-tuned acoustic model without padding on the Timit test dataset yields a word error rate of 22.1%.
Simple considerations for simple people building fancy neural networks
Building and training neural networks often presents frustrating debugging challenges because implementations can contain subtle errors while still running without crashes and achieving decent performance. The author shares a mental framework derived from natural language processing research to systematically construct and debug deep learning workflows. Practitioners should begin with thorough qualitative and quantitative dataset analysis before implementing simple baseline models to establish performance benchmarks. When coding the network, developers should verify implementation correctness by overfitting a tiny batch without regularization and tracking training dynamics such as losses, parameters, and gradients. Hyperparameter optimization should prioritize understanding underlying component behaviors through targeted random grid searches rather than launching hundreds of blind runs.
Retrieval Augmented Generation with Huggingface Transformers and Ray
Huggingface Transformers added the Retrieval Augmented Generation architecture to combine standard sequence-to-sequence models with external document retrieval for knowledge-intensive tasks. Scaling RAG fine-tuning across multiple GPUs previously created bottlenecks because the torch.distributed implementation forced the single rank 0 worker to handle all retrieval index lookups. To resolve this synchronization bottleneck, Ray was integrated into the contextual document retrieval mechanism using stateful actor abstractions. Dedicated Ray processes separate from the training workers now load the large index and process retrieval requests concurrently. This architectural separation delivers up to a 2x speedup per retrieval call and removes the strict dependency on PyTorch for training.
Hugging Face on PyTorch / XLA TPUs
PyTorch / XLA integrates Cloud TPU training into Hugging Face transformers while preserving the standard Trainer interface. The implementation adds an xla device type and uses lazy execution to trace intermediate representation graphs before lowering them to XLA Higher Level Operations for compilation. Parallel data loading through MpDeviceLoader overlaps CPU tracing with TPU execution, while xm.optimizer_step coordinates gradient consolidation across data-parallel replicas. Checkpoints use xm.save to transfer tensors via CPU storage and restrict disk writes to the master process. Benchmarks on a Cloud TPU v3-8 system training bert-large-uncased on WikiText-103 demonstrate completion times of 178.4 minutes in FP32 precision and 106.4 minutes in BF16 precision.
Fit More and Train Faster With ZeRO via DeepSpeed and FairScale
Machine learning model sizes frequently outpace consumer GPU memory, preventing practitioners from loading or fine-tuning large architectures on standard hardware. To address these resource constraints, the Hugging Face Trainer introduced experimental support for Zero Redundancy Optimizer features from DeepSpeed and FairScale starting in transformers v4.2.0. In dual-GPU benchmarks with a t5-large model, FairScale and DeepSpeed integrations expanded allowable batch sizes and shortened training durations compared to baseline DistributedDataParallel runs. Furthermore, DeepSpeed permitted single-GPU fine-tuning of a t5-3b model on a 24GB RTX-3090 card at batch size 20, whereas the baseline execution crashed immediately with an out-of-memory error. These optimizations manage memory allocation internally to curb memory fragmentation while requiring changes only to training scripts rather than model code.
Xendit Built a Counter-Fraud Watchlist for the Fintech Industry
Payment processor Xendit needed a fast, cost-effective counter-fraud watchlist system to screen transactions against international sanctions lists in real time without relying on expensive providers like Worldcheck or Refinitiv. Leveraging their existing familiarity with Postgres, the team loaded parsed sanctions lists from the United Nations and the Indonesian government into Supabase. They implemented full-text search using the Postgres Trigram extension, enabling relevance scoring across queried records. A dedicated database function handles the search queries directly via Xendit's Python clients, allowing the entire solution to reach production in under one week. The deployment has operated reliably in production for nine months without issues, with potential future iterations planned around machine learning techniques.
How to Build a Production Grade Workflow with SQL Modelling
Shopify’s Starscream data platform handled 76,000 jobs and wrote 300 terabytes daily, but its PySpark workflow was slow and cumbersome for commonplace reporting. An audit found that about 70 percent of PySpark jobs were full batch queries that did not require generalized computing, while users often wrote SQL before translating it into Python. The team built Seamster with dbt and Google BigQuery, adding source and model structures, ownership metadata, Python-based unit tests with fixed mock inputs, CI validation, and warehouse rules. A base layer isolates raw-source changes, while models are organized into base, application-ready, and presentation layers and defined through SQL, schema, README, and test files. After a multiweek beta, most users reported shipping models in days rather than weeks, although dbt’s incremental support could not safely handle some late-arriving-data, key-resolution, rebuild, and very large-model cases.
Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models
Pre-training sequence-to-sequence transformer models incurs massive computational costs, limiting development primarily to large institutions. To mitigate these expenses, encoder-decoder architectures can be warm-started using existing pre-trained checkpoints from encoder-only or decoder-only models like BERT and GPT-2. This walkthrough details the methodology and implementation of warm-starting sequence-to-sequence architectures using Hugging Face Transformers. By utilizing the EncoderDecoderModel framework alongside Seq2SeqTrainer, practitioners can construct and fine-tune models such as BERT2BERT on datasets like CNN/DailyMail. The resulting fully trained BERT2BERT model achieves a ROUGE-2 score of 18.22 on the full evaluation set, matching competitive sequence generation baselines at a fraction of standard pre-training costs.
Porting fairseq wmt19 translation system to transformers
The fairseq WMT19 news translation system was ported to the Hugging Face transformers library under the FSMT architecture. Development began with an initial proxy notebook emulating the transformers API before implementing native configuration, modeling, tokenization, and checkpoint conversion scripts. The author first targeted English-Russian models to handle complex dual-vocabulary requirements and directly evaluate output quality during testing, later extending the implementation to merged-vocabulary language pairs. Although model ensembling was omitted due to library limitations, removing fairseq optimizer states reduced the final model download footprint from 13GB to 1.1GB. The implementation was validated locally against converted fairseq checkpoints and merged into the main codebase after two weeks of iterative review.
Hyperparameter Search with Transformers and Ray Tune
Hugging Face Transformers models often require hyperparameter tuning to achieve high performance, but practitioners frequently rely on basic grid search or bypass tuning entirely. The Transformers 3.1 release integrates Ray Tune into the Hugging Face Trainer to provide advanced hyperparameter optimization without losing framework customizability. By invoking hyperparameter_search with the Ray backend, users can configure trial budgets, parallelize runs across multiple GPUs, and plug in search algorithms like HyperOptSearch alongside schedulers like ASHAScheduler. Experimental benchmarks on a BERT model using the RTE dataset show that Population-based Training reached 70.5% test accuracy in 48 GPU minutes, outperforming Grid Search at 65.4% test accuracy. The integration also supports experiment tracking tools like Weights and Biases out of the box.
How to Build an Experiment Pipeline from Scratch
The post describes building an email experimentation pipeline at Shopify after an existing framework for web properties and Shopify admin experiences could not support tests in external channels. The motivating case was a personalized blog-post recommendation algorithm whose incremental benefit needed comparison with generic blog emails, while local randomization and text-file storage created risks around unsubscribed users, shop-email relationships, and overlapping audiences. After requirements gathering, system mapping, mock-table queries, stakeholder review, and technical design review, the author implemented three pipeline phases in PySpark: experiment definitions, consolidation, and filtering with deduplication and exclusion rules. Production output supported ongoing randomization, backfills, category tags, sample limits, and integration with Shopify’s existing analysis framework; the tool was subsequently used across multiple data teams for email experiments.
Transformer-based Encoder-Decoder Models
Natural language generation tasks map input sequences to target sequences whose lengths cannot be known in advance and vary by content. Standard deep neural networks struggle with these variable mappings because their inputs and targets require fixed-dimensional vector representations. While recurrent neural networks addressed this challenge by generating target sequences auto-regressively from a compressed context state, transformer-based encoder-decoder architectures became the standard paradigm in modern natural language processing. The framework processes sequence-to-sequence problems by encoding source sequences and auto-regressively decoding target sequences token by token using conditional probability distributions. A step-by-step implementation demonstrates greedy decoding with Hugging Face Transformers, passing cached encoder hidden states alongside previously generated target tokens to iteratively generate German translations from English inputs.
Block Sparse Matrices for Smaller and Faster Language Models
Full dense layers in neural networks are often unnecessarily large and can be pruned without sacrificing precision, but available sparse algebra tools and PyTorch native implementations lack efficiency. To resolve this issue, the pytorch_block_sparse extension introduces BlockSparseLinear as a drop-in replacement for standard linear layers alongside BlockSparseModelPatcher for on-the-fly model modifications. The library uses NVIDIA CUTLASS C++ CUDA templates derived from Yulhwa Kim's cutlass tilesparse proof of concept. Although baseline sparse operations remain roughly twice as slow as cuBLAS-optimized dense counterparts, performance scales with sparsity, making a 75% sparse matrix approximately two times faster than a dense layer while reducing memory consumption by four times. Future updates will target dynamic sparsity optimization and NVIDIA Ampere Tensor Core primitives.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.