How to Move Independent Work Through an asyncio Pipeline
A practical asyncio workflow for moving independent work through typed events, queues, concurrent tasks, and awaited API calls. Learn where validation and fallback belong.
Loading…
Topic
Parallelism
6 posts about Parallelism. Every summary links to the original.
- 6 posts
- 6 companies
- 1 explainer
See the idea
Parallelism explainers
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine
Production-scale mixture-of-experts (MoE) training must handle uneven, data-dependent expert loads, creating ragged tensors and expensive dispatch, combine, and all-to-all communication. Dropless MoE processes every selected token, preserving model quality but requiring kernels that support variable token counts without padding or dropping. The post presents JAX and NVIDIA Transformer Engine optimizations centered on group-aware MXFP8 quantization, grouped GEMM, and fused expert-parallel (EP) dispatch and combine using NCCL EP. Grouped GEMM computes each expert’s actual workload in one kernel path, while EP reduces movement overhead and can deduplicate tokens crossing networks. For DeepSeek-V3 training on NVIDIA GB200, the reported throughput increased from 103 to 1,068 TFLOPS/GPU, a 10.4x improvement; the post also reports roughly 10x throughput improvement and 97% scaling efficiency at 1,024 GPUs.
Python 3.14 free-threaded build is now available in RHEL
Red Hat Enterprise Linux 9.8 and 10.2 now provide a free-threaded Python 3.14 build alongside the regular interpreter, enabling developers to test parallel CPU execution with threads. The build disables the GIL, allowing Python threads to execute on multiple CPU cores in one process, while built-in dict, list, and set operations retain internal locking for individual operations. Developers install the python3.14-freethreading package from the CodeReady Linux Builder repository; it provides the separate python3.14t executable and commands to verify free-threading support and GIL status. Existing code may expose race conditions, and unsupported C-extension dependencies can re-enable the GIL, so the source recommends testing, dependency checks, explicit synchronization, and workload measurement. The build carries a stated 5%-10% single-threaded performance overhead and mainly targets CPU-bound threaded workloads.
GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model
Meta’s Generative Ads Recommendation Model (GEM), a hybrid foundation model for ads recommendations on Instagram and Facebook, trains across several thousand latest-generation GPUs and combines trillions of sparse embedding parameters with billions of dense parameters. The post frames its engineering problem around recommendation-specific jagged and asymmetric sequences, memory-bound operations, numerical sensitivity, communication overhead, activation pressure, and load skew that make conventional LLM training techniques insufficient. Its co-design approach combines custom kernels such as Jagged Flash Attention (JFA), BlockAttention, and Generalized Dot-Product Attention (GDPA), mixed ultra-low-precision training including MXFP8 attention and MLP, topology-aware 5D parallelism, SM-free collectives, and memory and load-balancing techniques. The reported result is a doubling of end-to-end training efficiency to 20–25% Model FLOPs Utilization (MFU) while total training FLOPs increased fourfold over 12 months; Base Batch Shuffling (BBS) alone delivered a reported 4% efficiency gain, with 4% QPS improvement and 4% peak memory reduction.
Building with Modal and the OpenAI Agents SDK
The post walks through constructing a custom agent harness with the OpenAI Agents SDK and Modal Sandboxes, starting from a minimal coding agent that executes shell commands directly on the host. To improve safety, the agent is moved into isolated Modal Sandboxes, which are secure Linux environments that can be equipped with GPUs for compute-intensive tasks like training models. Memory is added via Sessions to accumulate context across turns, and subagents are introduced to handle focused subtasks with fresh context windows, preventing context bloat in the orchestrator while allowing long-horizon work. An async subagent pool enables parallel execution, with quotas to limit GPU spend, and filesystem snapshots allow checkpointing and deduplication of setup work across subagents, saving time and offloading memory. The harness is applied to the Parameter Golf challenge, where an orchestrator delegates parallel training experiments across different ML frameworks on GPUs, demonstrating autonomous research capabilities with composable skills.
Scaling Large Language Models for e-Commerce: The Development of a Llama-Based Customized LLM
eBay developed e-Llama, customized 8B and 70B language models, by adapting Meta’s Llama 3.1 models to e-commerce rather than training entirely from scratch. The approach uses continued pretraining on filtered public listings, product reviews, classifier-selected open-source examples, replay data, and 10% non-English general-domain data to add domain knowledge while limiting catastrophic forgetting. Training ran on 480 NVIDIA H100 GPUs using Megatron-LM, 3D parallelism, flash-attention-2, a 1:1 general-to-e-commerce sampling ratio, a learning-rate maximum set to 10% of the original, and one trillion tokens. The 70B model required about one month and 340,000 GPU-hours. E-commerce benchmarks improved approximately 25% in English and 30% in non-English, while general-domain NLU degraded 1% for e-Llama 70B; subsequent instruction tuning added safety and instruction-following guardrails.
The Curious Case of the Table-Locking UPDATE Query
The post recounts a PostgreSQL performance incident in which metadata processing stopped as jobs accumulated and an UPDATE targeting a single UUID remained active for days. Diagnostics showed the query consuming more than 80% of database time, while manual execution did not reproduce the delay. A pg_stat_activity and pg_locks query exposed many waiting statements and an AccessExclusiveLock entry whose locktype was tuple, leading the investigation toward PostgreSQL row locking and contention. The eventual cause was threaded code that issued concurrent updates to the same row for each job; removing that concurrency, aborting old backends with pg_cancel_backend, and deploying the change restored processing and began churning through the backlog. A correction notes that the reported lock was on a tuple rather than the table, and that PostgreSQL does not have lock promotion as suggested in the debugging discussion.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.