---
title: "Python"
description: "252 posts about Python, summarised, each linking to the original."
---

# Python
> 252 posts about Python, summarised, each linking to the original.

## Articles

### [Deploying LangChain to Fly.io](https://yomu.fyi/post/deploying-langchain-to-fly-io.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: May 8, 2023
- Type: Tutorial

Deploying a minimal LangChain application to Fly.io, using Flask, requires a small Python project that accepts a place and asks an OpenAI-backed model for three places to eat there. The example uses LangChain’s OpenAI wrapper, PromptTemplate, temperature 0.9, and a default Berlin value, while Flask routes requests at / or / and renders the model’s split response in HTML. Local setup includes a virtual environment, pinned Flask, gunicorn, langchain, openai, and python-dotenv dependencies, with OPENAI\_API\_KEY stored in a local .env file; production uses gunicorn rather than Flask’s development server. Fly launch configures the app name, organization, and deployment region, and fly deploy builds and publishes an image before provisioning machines. The article concludes that more complex, user-dependent requests require LangChain agents and tools, such as Google Places API, because language models lack real-time data and internet browsing.


### [Launch Week 7 Community Highlights](https://yomu.fyi/post/launch-week-7-community-highlights.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: paul\_copplestone
- Published: Apr 14, 2023
- Type: Announcement

Supabase reports community highlights from the past three months, covering ecosystem partnerships, self-hosting, Postgres extensions, client libraries, utilities, upstream work, and contributors. New collaborations include AWS RDS work on pg\_tle and Trusted Language PL/Rust, Clickhouse bidirectional Postgres integration, LangChain vector-store and hybrid-search contributions, a Vercel and OpenAI GPT-search template, Doppler secret syncing, Redwood’s Prisma Quick Start, Dart Edge, and T3 Turbo. The update also describes self-hosting support for Logs and Edge Functions, new Digital Ocean and Stackgres entrants, and extensions including pgvector, HypoPG, pg\_tle, PGroonga v3.0, LZ4 compression, and ICU support. It records new Swift, Python, and C# client libraries, cache helpers, Firebase-to-Supabase supashim, Postgres and OrioleDB patches, and recognition for contributors, while noting more than 48,000 GitHub stars.


### [Securing GitOps pipelines](https://yomu.fyi/post/securing-gitops-pipelines.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Thang Le
- Published: Mar 1, 2023
- Type: Problem & solution

Grab's real-time data platform team transitioned from an Atlantis-driven Terraform workflow to an in-house GitOps platform called Khone to manage streaming infrastructure resources like Kafka topics and Flink pipelines. The earlier setup suffered from coarse-grained access controls, required manual merge request comments, and lacked flexible validation capabilities within native configuration files. Khone derives environment parameters directly from standardized directory paths and uses Python with the python-hcl2 library to inspect and validate resource definitions before executing Terraform stages in parallel. To prevent configuration tampering in merge requests, CI/CD pipeline definitions and execution scripts are isolated in a separate administrative repository and fetched during job runs using shallow Git clones.


### [Monte Carlo Simulations: Separating Signal from Noise in Sampled Success Metrics](https://yomu.fyi/post/monte-carlo-simulations-separating-signal-from-noise-in-sampled-succes.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jan 3, 2023
- Type: Tutorial

Sampled success metrics can be difficult to interpret when the underlying population is too expensive or time-consuming to measure completely. The method presents a Monte Carlo simulation for estimating how sample size and an assumed trend affect the reliability of Shopify’s sampled “shop issue rate,” which estimates requirement violations for an average installed app. Its workflow establishes simulation metrics such as 1mDO, 2mDO, and 1qDO, measures MAPE, defines a sampling distribution, and runs repeated simulations across parameter combinations using Python and pandas. For the example, the issue count is modeled with a Poisson distribution, while sample size, lambda, and monthly decrease are varied to quantify uncertainty and collection cost. With 100 audits per month, an initial mean of 10 issues, and a 5-percent monthly decrease, the simulations predict monthly decreases 83 percent of the time and decreases over two months 97 percent of the time.


### [Supabase Beta November 2022](https://yomu.fyi/post/supabase-beta-november-2022.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: ant\_wilson
- Published: Dec 7, 2022
- Type: Announcement

Supabase’s November 2022 update previews Launch Week 6 while reporting a smaller set of product releases and community activity. Remix Auth Helpers are now available for server-side authentication, updated for supabase-js V2 and usable with TypeScript; the post also introduces the Edgy Edge Functions YouTube series and three examples covering OG image generation, Oak API servers, and Puppeteer screenshots. Product fixes address auth.signOut cookies and session removal, magic links can be customized with {{ .TokenHash }}, functions support redirects, and Supabase can be installed through Conda. The update also announces a Fireship course building a Reddit-inspired app with Supabase, PostgreSQL, and React, alongside a broad list of community resources and projects.


### [How to Export Datadog Metrics for Exploration in Jupyter Notebooks](https://yomu.fyi/post/how-to-export-datadog-metrics-for-exploration-in-jupyter-notebooks.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 29, 2022
- Type: Tutorial

Datadog dashboards can constrain metric analysis because they offer a limited set of visualizations and lack tooling for complex work such as statistical modeling. They also aggregate data over wider time ranges: the examples contrast one-second metrics across 15 minutes with two-hour intervals across 30 days, potentially hiding interesting events. The guide uses Datadog’s REST API and Python in Jupyter notebooks, requiring an API key, an APP key, and a metric query such as CPU utilization over time. It defines a time range, splits it into buckets whose width is controlled by time\_delta, requests each window in a loop, appends the results, and converts them into a dataframe. The resulting data can be examined with tools such as seaborn; a KDE plot is used to inspect the distribution of system CPU utilization, with the exported data offering greater granularity than the dashboard example.


### [What is a Full Stack Data Scientist?](https://yomu.fyi/post/what-is-a-full-stack-data-scientist.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 15, 2022
- Type: Explainer

Shopify defines full stack data science as an ownership-oriented, end-to-end approach to solving business problems across the data science lifecycle. Rather than specializing in one stage, practitioners participate in discovery and analysis, acquisition, data modeling, pipeline development, and production delivery. The role combines business acumen, communication, Python and SQL programming, exploratory data analysis, data engineering, and machine learning, while treating machine learning as a tool rather than a default and starting simple before adding complexity. Shopify describes the model as generalist with T-shaped development: broad capability is paired with deeper expertise in a specific area, enabling individual strengths within teams to address complex challenges.


### [Slack Consolidate: a slackbot built with Python and Supabase](https://yomu.fyi/post/slack-consolidate-a-slackbot-built-with-python-and-supabase.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Rodrigo Mansueli
- Published: Aug 9, 2022
- Type: Tutorial

Slack Consolidate is a Python Slackbot that watches configured Slack channels and reposts messages to destination channels according to routing rules stored in Supabase. The tutorial sets up Slack app permissions from a manifest, installs the Python SDKs, configures environment variables for Supabase and Slack tokens, and creates slack\_channels and slack\_watcher PostgreSQL tables. Its polling loop loads channel mappings, fetches up to 20 recent messages, converts Slack timestamps to ISO-compatible values, inserts messages into the watcher table, builds message links, and posts through a bot client; it waits 1.4 seconds between channel processing and rescans channel configuration hourly. Public-channel posts contain a link, while private-channel posts include the message text as well. The project also provides a Google Colab notebook for trying the example.


### [Policy Gradient with PyTorch](https://yomu.fyi/post/policy-gradient-with-pytorch.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Thomas Simonini
- Published: Jun 30, 2022
- Type: Tutorial

Policy gradient algorithms optimize reinforcement learning policies directly without learning intermediate action-value functions. Instead of assigning discrete Q-values to actions, these methods parameterize a stochastic policy and apply gradient ascent on an objective score function. This setup eliminates manual exploration tuning, resolves perceptual aliasing in identical states, and naturally accommodates continuous or high-dimensional action spaces. The Reinforce Monte Carlo policy gradient algorithm operationalizes this approach by collecting full episodic trajectories and adjusting parameters along the gradient of the log-action probabilities scaled by return. Practitioners implement this algorithm using PyTorch to evaluate agent robustness across environments like CartPole-v1, PixelCopter, and Pong.


### [Liftoff! How to get started with your first ML project 🚀](https://yomu.fyi/post/liftoff-how-to-get-started-with-your-first-ml-project.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Nima Boscarino
- Published: Jun 29, 2022
- Type: Tutorial

Beginners in machine learning frequently face difficulties when selecting a framework and defining the scope of their initial hands-on project. Sentence Transformers provides an accessible starting point by computing dense vector representations for sentences, paragraphs, and images, enabling semantic search through cosine similarity. A structured four-step strategy guides practitioners through listing library capabilities, identifying compelling datasets, selecting a familiar secondary tool, and brainstorming project concepts. Applying this framework, a song lyrics dataset was paired with Gradio Blocks to build a prompt-based playlist generator. Developing this application required evaluating pre-trained models, hosting generated embeddings on Hugging Face Spaces, and utilizing multi-processor support to accelerate embedding generation.


### [Accelerate Large Model Training using DeepSpeed](https://yomu.fyi/post/accelerate-large-model-training-using-deepspeed.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Sourab Mangrulkar, Sylvain Gugger
- Published: Jun 28, 2022
- Type: Tutorial

Training large models on hardware with limited GPU memory frequently causes out-of-memory errors when using standard Distributed Data Parallel. To solve this bottleneck, Hugging Face Accelerate integrates DeepSpeed ZeRO data parallelism to shard optimizer states, gradients, and model parameters across workers. For a 900-million-parameter DeBERTa model, ZeRO Stage 2 increased the maximum per-device batch size from eight to forty while achieving a 3.5-fold training speedup over DDP without degrading accuracy or F1 score. Advanced configurations allow sequence-to-sequence chatbot finetuning and full ZeRO Stage 3 CPU offloading for models like the 1.5-billion-parameter GPT-XL, enabling batch size 16 training where DDP fails. Accelerate enables these memory optimizations through simple configuration files and the accelerate launch command with minimal or zero code modifications.


### [Getting Started With Embeddings](https://yomu.fyi/post/getting-started-with-embeddings.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Omar Espejel
- Published: Jun 23, 2022
- Type: Tutorial

Embeddings represent unstructured information such as text and images as numerical vectors in a shared semantic space. To demonstrate their utility, a simple semantic search engine is built over US Social Security Medicare frequently asked questions. The system generates 384-dimensional vector representations for thirteen FAQ entries by dispatching POST requests to the Hugging Face Inference API using the sentence-transformers/all-MiniLM-L6-v2 model. Incoming user queries are converted into matching vector representations and evaluated against stored dataset vectors using the util.semantic\_search function from the Sentence Transformers library. By calculating cosine similarity scores, the system retrieves and ranks the five most semantically relevant questions without requiring custom keyword rules or massive labeled training sets.


### [Convert Transformers to ONNX with Hugging Face Optimum](https://yomu.fyi/post/convert-transformers-to-onnx-with-hugging-face-optimum.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Philipp Schmid
- Published: Jun 22, 2022
- Type: Tutorial

Exporting Hugging Face Transformers models to the ONNX format can be accomplished through three different abstraction levels. The low-level approach utilizes torch.onnx.export, requiring manual specification of dummy inputs, input names, output names, opset versions, and dynamic axes configurations. At the intermediate level, the transformers.onnx package simplifies the conversion process by relying on FeaturesManager and prebuilt configuration objects to handle dynamic axis definitions automatically. The high-level method uses Hugging Face Optimum classes such as ORTModelForSequenceClassification by setting the from\_transformers flag to True inside from\_pretrained. This Optimum export leverages transformers.onnx internally and produces a model ready for immediate inference execution or integration into pipelines.


### [Python data loading with Supabase](https://yomu.fyi/post/python-data-loading-with-supabase.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Ant Wilson
- Published: Jun 17, 2022
- Type: Tutorial

Supabase offers a PostgreSQL-based backend-as-a-service platform alongside an open-source Python SDK for automating CRUD operations and data-intensive tasks. After spinning up a new project in the Supabase dashboard, developers can configure relational schemas in the Table Editor by establishing foreign key constraints between tables like Vendor and Product. Authentication for the SDK relies on passing project URL parameters and API keys, which can be securely stored and loaded through environment variables. Programmatic data population utilizes the supabase client library combined with the Faker package to generate realistic synthetic entries and push them directly to database tables using insert and execute calls. Populated records can subsequently be examined in real time through the dashboard Table Editor interface to verify schema integrity and data persistence.


### [Intel and Hugging Face Partner to Democratize Machine Learning Hardware Acceleration](https://yomu.fyi/post/intel-and-hugging-face-partner-to-democratize-machine-learning-hardwar.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Julien Simon
- Published: Jun 15, 2022
- Type: Announcement

Intel has joined Hugging Face's Hardware Partner Program to accelerate Transformer training, fine-tuning, and inference on Intel platforms. Large Transformer models introduce latency bottlenecks in production workloads like search and chatbots, where hardware-level optimization typically requires tedious trial and error. To streamline model optimization, the collaboration introduces Optimum Intel, an open-source library integrating the Intel Neural Compressor for automated quantization, pruning, and distillation. A demonstration applies post-training dynamic quantization to a fine-tuned DistilBERT classification model using a CPU-only PyTorch setup. The quantized model converted 38 Linear and 2 Embedding operators to 8-bit integers, reducing evaluation duration by 1.34x while keeping the accuracy drop within a 5% threshold.


### [8 Data Conferences Shopify Data Thinks You Should Attend](https://yomu.fyi/post/8-data-conferences-shopify-data-thinks-you-should-attend.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 10, 2022
- Type: Explainer

Shopify’s data scientists and engineers curated eight upcoming data conferences for 2022, covering virtual, hybrid, and in-person formats. The selection supports learning and development by helping practitioners hear from peers about applications, techniques, use cases, and opportunities in the wider data community. Hybrid options include Data + AI Summit, Transform, RecSys, and ODSC West, with topics spanning analytics, machine learning, open source technologies, recommender systems, MLOps, natural language processing, and big data. In-person choices—PyData London, KDD, Re-Work Deep Learning Summit, and Crunch—feature tutorials, research, business applications, AI challenges, data-team practices, and machine learning at scale, with locations including London, Washington D.C., Toronto, and Budapest.


### [The Annotated Diffusion Model](https://yomu.fyi/post/the-annotated-diffusion-model.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Niels Rogge, Kashif Rasul
- Published: Jun 7, 2022
- Type: Tutorial

Denoising Diffusion Probabilistic Models generate data samples by learning to reverse a discrete-time forward diffusion process that incrementally adds Gaussian noise to inputs. The forward process adds noise across a predefined schedule of variance parameters up to a fixed number of steps, transforming data into an isotropic Gaussian distribution. Because the true reverse conditional probability distribution is intractable, a neural network optimizes the variational lower bound to approximate the distribution mean at each step while keeping the variance fixed. Sampling subsequently proceeds by iteratively denoising pure Gaussian noise through successive network evaluations to reconstruct a clean data sample. Although diffusion models demonstrate high generation quality, their primary drawback remains the requirement of multiple forward passes during inference compared to models like generative adversarial networks.


### [Automated Experiment Analysis - Making experimental analysis scalable](https://yomu.fyi/post/automated-experiment-analysis-making-experimental-analysis-scalable.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Albert Cheng
- Published: May 30, 2022
- Type: Problem & solution

Manual ad-hoc analysis of online controlled experiments at Grab introduced operational inefficiencies, inconsistent quality control, and scalability barriers across teams. To resolve these issues, Grab extended its GrabX experimentation platform with an Automated Experiment Analysis system that standardises metrics and automates statistical evaluations. The architecture stores experiment configurations and metric definitions from Cosmos DB into Azure Data Lake as bronze datasets, uses Spark on Databricks via Azure Data Factory to process subjects into silver datasets, and applies an internal Python Decision Engine to generate final gold results. These gold datasets are stored in star-schema fact and dimension tables and presented directly in the GrabX interface using embedded Power BI visualisations. The automation eliminates repetitive data pipeline construction for analysts, ensures reproducible findings aligned with initial hypotheses, and accelerates product launch decisions.


### [Efficient Table Pre-training without Real Data: An Introduction to TAPEX](https://yomu.fyi/post/efficient-table-pre-training-without-real-data-an-introduction-to-tape.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Qian Liu
- Published: May 23, 2022
- Type: Problem & solution

Conventional language model pre-training relies on massive natural text corpora and general-purpose objectives like masked language modeling, leaving a substantial gap when applied to structured table tasks. To bridge this gap efficiently, TAPEX (Table Pre-training via Execution) pre-trains a language model, such as BART, by learning a neural SQL executor over a synthetic corpus. The system pairs web tables with systematically sampled executable SQL queries and gathers target execution outputs via an off-the-shelf SQL engine. During downstream fine-tuning, the model consumes natural language questions and flattened tables to predict answers. TAPEX achieved new state-of-the-art results across four benchmark datasets, including WikiTableQuestions and TabFact, while achieving a pre-training speedup of nearly 50 times over TaBERT.


### [Double Entry Transition Tables: How We Track State Changes At Shopify](https://yomu.fyi/post/double-entry-transition-tables-how-we-track-state-changes-at-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 18, 2022
- Type: Explainer

Shopify needed a reliable way to count merchants using Shopify Balance over time, where usage requires both an active Shopify Balance account and an active Shopify account. The data team built accounts\_transition\_facts with double entry transition tables, a format that represents each state change as a -1 row for the previous state and a +1 row for the new state. Individual attribute tables use account\_id, transition\_at, and an index that breaks timestamp ties; the pipeline merges attributes, carries forward missing values, and generates net\_change rows. Because net\_change is additive, queries can sum it while filtering statuses, group results by date, or feed daily aggregates, while additional attributes can be added without rewriting existing SQL or PySpark. The initial build required substantial effort but produced a reporting structure intended to scale with Shopify Balance’s growing complexity.


[Newer posts](https://yomu.fyi/topic/python/page/8.md) · [Older posts](https://yomu.fyi/topic/python/page/10.md)