How to Move Independent Work Through an asyncio Pipeline
A practical asyncio workflow for moving independent work through typed events, queues, concurrent tasks, and awaited API calls. Learn where validation and fallback belong.
Loading…
Topic
Python
250 posts about Python. Every summary links to the original.
- 250 posts
- 28 companies
- 1 explainer
See the idea
Python explainers
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Securing GitOps pipelines
Grab's real-time data platform team transitioned from an Atlantis-driven Terraform workflow to an in-house GitOps platform called Khone to manage streaming infrastructure resources like Kafka topics and Flink pipelines. The earlier setup suffered from coarse-grained access controls, required manual merge request comments, and lacked flexible validation capabilities within native configuration files. Khone derives environment parameters directly from standardized directory paths and uses Python with the python-hcl2 library to inspect and validate resource definitions before executing Terraform stages in parallel. To prevent configuration tampering in merge requests, CI/CD pipeline definitions and execution scripts are isolated in a separate administrative repository and fetched during job runs using shallow Git clones.
Monte Carlo Simulations: Separating Signal from Noise in Sampled Success Metrics
Sampled success metrics can be difficult to interpret when the underlying population is too expensive or time-consuming to measure completely. The method presents a Monte Carlo simulation for estimating how sample size and an assumed trend affect the reliability of Shopify’s sampled “shop issue rate,” which estimates requirement violations for an average installed app. Its workflow establishes simulation metrics such as 1mDO, 2mDO, and 1qDO, measures MAPE, defines a sampling distribution, and runs repeated simulations across parameter combinations using Python and pandas. For the example, the issue count is modeled with a Poisson distribution, while sample size, lambda, and monthly decrease are varied to quantify uncertainty and collection cost. With 100 audits per month, an initial mean of 10 issues, and a 5-percent monthly decrease, the simulations predict monthly decreases 83 percent of the time and decreases over two months 97 percent of the time.
Supabase Beta November 2022
Supabase’s November 2022 update previews Launch Week 6 while reporting a smaller set of product releases and community activity. Remix Auth Helpers are now available for server-side authentication, updated for supabase-js V2 and usable with TypeScript; the post also introduces the Edgy Edge Functions YouTube series and three examples covering OG image generation, Oak API servers, and Puppeteer screenshots. Product fixes address auth.signOut cookies and session removal, magic links can be customized with {{ .TokenHash }}, functions support redirects, and Supabase can be installed through Conda. The update also announces a Fireship course building a Reddit-inspired app with Supabase, PostgreSQL, and React, alongside a broad list of community resources and projects.
How to Export Datadog Metrics for Exploration in Jupyter Notebooks
Datadog dashboards can constrain metric analysis because they offer a limited set of visualizations and lack tooling for complex work such as statistical modeling. They also aggregate data over wider time ranges: the examples contrast one-second metrics across 15 minutes with two-hour intervals across 30 days, potentially hiding interesting events. The guide uses Datadog’s REST API and Python in Jupyter notebooks, requiring an API key, an APP key, and a metric query such as CPU utilization over time. It defines a time range, splits it into buckets whose width is controlled by time_delta, requests each window in a loop, appends the results, and converts them into a dataframe. The resulting data can be examined with tools such as seaborn; a KDE plot is used to inspect the distribution of system CPU utilization, with the exported data offering greater granularity than the dashboard example.
What is a Full Stack Data Scientist?
Shopify defines full stack data science as an ownership-oriented, end-to-end approach to solving business problems across the data science lifecycle. Rather than specializing in one stage, practitioners participate in discovery and analysis, acquisition, data modeling, pipeline development, and production delivery. The role combines business acumen, communication, Python and SQL programming, exploratory data analysis, data engineering, and machine learning, while treating machine learning as a tool rather than a default and starting simple before adding complexity. Shopify describes the model as generalist with T-shaped development: broad capability is paired with deeper expertise in a specific area, enabling individual strengths within teams to address complex challenges.
Slack Consolidate: a slackbot built with Python and Supabase
Slack Consolidate is a Python Slackbot that watches configured Slack channels and reposts messages to destination channels according to routing rules stored in Supabase. The tutorial sets up Slack app permissions from a manifest, installs the Python SDKs, configures environment variables for Supabase and Slack tokens, and creates slack_channels and slack_watcher PostgreSQL tables. Its polling loop loads channel mappings, fetches up to 20 recent messages, converts Slack timestamps to ISO-compatible values, inserts messages into the watcher table, builds message links, and posts through a bot client; it waits 1.4 seconds between channel processing and rescans channel configuration hourly. Public-channel posts contain a link, while private-channel posts include the message text as well. The project also provides a Google Colab notebook for trying the example.
Policy Gradient with PyTorch
Policy gradient algorithms optimize reinforcement learning policies directly without learning intermediate action-value functions. Instead of assigning discrete Q-values to actions, these methods parameterize a stochastic policy and apply gradient ascent on an objective score function. This setup eliminates manual exploration tuning, resolves perceptual aliasing in identical states, and naturally accommodates continuous or high-dimensional action spaces. The Reinforce Monte Carlo policy gradient algorithm operationalizes this approach by collecting full episodic trajectories and adjusting parameters along the gradient of the log-action probabilities scaled by return. Practitioners implement this algorithm using PyTorch to evaluate agent robustness across environments like CartPole-v1, PixelCopter, and Pong.
Liftoff! How to get started with your first ML project 🚀
Beginners in machine learning frequently face difficulties when selecting a framework and defining the scope of their initial hands-on project. Sentence Transformers provides an accessible starting point by computing dense vector representations for sentences, paragraphs, and images, enabling semantic search through cosine similarity. A structured four-step strategy guides practitioners through listing library capabilities, identifying compelling datasets, selecting a familiar secondary tool, and brainstorming project concepts. Applying this framework, a song lyrics dataset was paired with Gradio Blocks to build a prompt-based playlist generator. Developing this application required evaluating pre-trained models, hosting generated embeddings on Hugging Face Spaces, and utilizing multi-processor support to accelerate embedding generation.
Accelerate Large Model Training using DeepSpeed
Training large models on hardware with limited GPU memory frequently causes out-of-memory errors when using standard Distributed Data Parallel. To solve this bottleneck, Hugging Face Accelerate integrates DeepSpeed ZeRO data parallelism to shard optimizer states, gradients, and model parameters across workers. For a 900-million-parameter DeBERTa model, ZeRO Stage 2 increased the maximum per-device batch size from eight to forty while achieving a 3.5-fold training speedup over DDP without degrading accuracy or F1 score. Advanced configurations allow sequence-to-sequence chatbot finetuning and full ZeRO Stage 3 CPU offloading for models like the 1.5-billion-parameter GPT-XL, enabling batch size 16 training where DDP fails. Accelerate enables these memory optimizations through simple configuration files and the accelerate launch command with minimal or zero code modifications.
Getting Started With Embeddings
Embeddings represent unstructured information such as text and images as numerical vectors in a shared semantic space. To demonstrate their utility, a simple semantic search engine is built over US Social Security Medicare frequently asked questions. The system generates 384-dimensional vector representations for thirteen FAQ entries by dispatching POST requests to the Hugging Face Inference API using the sentence-transformers/all-MiniLM-L6-v2 model. Incoming user queries are converted into matching vector representations and evaluated against stored dataset vectors using the util.semantic_search function from the Sentence Transformers library. By calculating cosine similarity scores, the system retrieves and ranks the five most semantically relevant questions without requiring custom keyword rules or massive labeled training sets.
Convert Transformers to ONNX with Hugging Face Optimum
Exporting Hugging Face Transformers models to the ONNX format can be accomplished through three different abstraction levels. The low-level approach utilizes torch.onnx.export, requiring manual specification of dummy inputs, input names, output names, opset versions, and dynamic axes configurations. At the intermediate level, the transformers.onnx package simplifies the conversion process by relying on FeaturesManager and prebuilt configuration objects to handle dynamic axis definitions automatically. The high-level method uses Hugging Face Optimum classes such as ORTModelForSequenceClassification by setting the from_transformers flag to True inside from_pretrained. This Optimum export leverages transformers.onnx internally and produces a model ready for immediate inference execution or integration into pipelines.
Python data loading with Supabase
Supabase offers a PostgreSQL-based backend-as-a-service platform alongside an open-source Python SDK for automating CRUD operations and data-intensive tasks. After spinning up a new project in the Supabase dashboard, developers can configure relational schemas in the Table Editor by establishing foreign key constraints between tables like Vendor and Product. Authentication for the SDK relies on passing project URL parameters and API keys, which can be securely stored and loaded through environment variables. Programmatic data population utilizes the supabase client library combined with the Faker package to generate realistic synthetic entries and push them directly to database tables using insert and execute calls. Populated records can subsequently be examined in real time through the dashboard Table Editor interface to verify schema integrity and data persistence.
Intel and Hugging Face Partner to Democratize Machine Learning Hardware Acceleration
Intel has joined Hugging Face's Hardware Partner Program to accelerate Transformer training, fine-tuning, and inference on Intel platforms. Large Transformer models introduce latency bottlenecks in production workloads like search and chatbots, where hardware-level optimization typically requires tedious trial and error. To streamline model optimization, the collaboration introduces Optimum Intel, an open-source library integrating the Intel Neural Compressor for automated quantization, pruning, and distillation. A demonstration applies post-training dynamic quantization to a fine-tuned DistilBERT classification model using a CPU-only PyTorch setup. The quantized model converted 38 Linear and 2 Embedding operators to 8-bit integers, reducing evaluation duration by 1.34x while keeping the accuracy drop within a 5% threshold.
8 Data Conferences Shopify Data Thinks You Should Attend
Shopify’s data scientists and engineers curated eight upcoming data conferences for 2022, covering virtual, hybrid, and in-person formats. The selection supports learning and development by helping practitioners hear from peers about applications, techniques, use cases, and opportunities in the wider data community. Hybrid options include Data + AI Summit, Transform, RecSys, and ODSC West, with topics spanning analytics, machine learning, open source technologies, recommender systems, MLOps, natural language processing, and big data. In-person choices—PyData London, KDD, Re-Work Deep Learning Summit, and Crunch—feature tutorials, research, business applications, AI challenges, data-team practices, and machine learning at scale, with locations including London, Washington D.C., Toronto, and Budapest.
The Annotated Diffusion Model
Denoising Diffusion Probabilistic Models generate data samples by learning to reverse a discrete-time forward diffusion process that incrementally adds Gaussian noise to inputs. The forward process adds noise across a predefined schedule of variance parameters up to a fixed number of steps, transforming data into an isotropic Gaussian distribution. Because the true reverse conditional probability distribution is intractable, a neural network optimizes the variational lower bound to approximate the distribution mean at each step while keeping the variance fixed. Sampling subsequently proceeds by iteratively denoising pure Gaussian noise through successive network evaluations to reconstruct a clean data sample. Although diffusion models demonstrate high generation quality, their primary drawback remains the requirement of multiple forward passes during inference compared to models like generative adversarial networks.
Automated Experiment Analysis - Making experimental analysis scalable
Manual ad-hoc analysis of online controlled experiments at Grab introduced operational inefficiencies, inconsistent quality control, and scalability barriers across teams. To resolve these issues, Grab extended its GrabX experimentation platform with an Automated Experiment Analysis system that standardises metrics and automates statistical evaluations. The architecture stores experiment configurations and metric definitions from Cosmos DB into Azure Data Lake as bronze datasets, uses Spark on Databricks via Azure Data Factory to process subjects into silver datasets, and applies an internal Python Decision Engine to generate final gold results. These gold datasets are stored in star-schema fact and dimension tables and presented directly in the GrabX interface using embedded Power BI visualisations. The automation eliminates repetitive data pipeline construction for analysts, ensures reproducible findings aligned with initial hypotheses, and accelerates product launch decisions.
Efficient Table Pre-training without Real Data: An Introduction to TAPEX
Conventional language model pre-training relies on massive natural text corpora and general-purpose objectives like masked language modeling, leaving a substantial gap when applied to structured table tasks. To bridge this gap efficiently, TAPEX (Table Pre-training via Execution) pre-trains a language model, such as BART, by learning a neural SQL executor over a synthetic corpus. The system pairs web tables with systematically sampled executable SQL queries and gathers target execution outputs via an off-the-shelf SQL engine. During downstream fine-tuning, the model consumes natural language questions and flattened tables to predict answers. TAPEX achieved new state-of-the-art results across four benchmark datasets, including WikiTableQuestions and TabFact, while achieving a pre-training speedup of nearly 50 times over TaBERT.
Double Entry Transition Tables: How We Track State Changes At Shopify
Shopify needed a reliable way to count merchants using Shopify Balance over time, where usage requires both an active Shopify Balance account and an active Shopify account. The data team built accounts_transition_facts with double entry transition tables, a format that represents each state change as a -1 row for the previous state and a +1 row for the new state. Individual attribute tables use account_id, transition_at, and an index that breaks timestamp ties; the pipeline merges attributes, carries forward missing values, and generates net_change rows. Because net_change is additive, queries can sum it while filtering statuses, group results by date, or feed daily aggregates, while additional attributes can be added without rewriting existing SQL or PySpark. The initial build required substantial effort but produced a reporting structure intended to scale with Shopify Balance’s growing complexity.
Gradio 3.0 is Out!
Gradio 3.0 introduces a ground-up redesign of the Gradio library to improve machine learning demo development and browser performance. The frontend has been rebuilt using modern technologies such as Svelte, resulting in smaller payloads, faster page load times, and cleaner visual designs suitable for embedding in blog posts. Component updates include revamping the Dataframe component to support CSV drag-and-drop, introducing new UI elements like Gallery, and adding the TabbedInterface class to group demos under distinct tabs. Developers also gain access to Gradio Blocks, a low-level Python API enabling custom layouts, multi-step interfaces, dynamic component property modifications, and complex data flows. A competition called the Gradio Blocks Party has been organized through the end of May to encourage community adoption.
Accelerated Inference with Optimum and Transformers Pipelines
Transformer-based models are moving into production for large-scale workloads, but default implementations remain slow and resource-intensive compared to traditional machine learning algorithms. Hugging Face Optimum addresses this issue by integrating performance optimization tools and accelerated runtimes like ONNX Runtime directly into standard Transformers pipelines. Developers replace standard model classes with Optimum equivalents to convert models to ONNX, run graph optimizations such as operator fusion, and apply quantization. In an evaluation on an AWS EC2 m5.xlarge CPU instance, optimizing and quantizing a RoBERTa question-answering model cut average latency from 117.61 ms to 64.94 ms while retaining 99.61 percent accuracy. Known limitations include a 2GB remote model size limit, lack of seq2seq support, omitted past key values in causal language models, and no local caching for optimized ONNX models.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.