Loading…
Topic
Serverless
102 posts about Serverless. Every summary links to the original.
- 102 posts
- 13 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Introducing Private Network Gateway
Databricks announced Private Network Gateway (PNG) in public preview on Azure and private preview on AWS, addressing the networking friction that prevented teams from moving workloads to serverless compute. PNG provides a single managed gateway from serverless compute into a customer’s virtual network, eliminating the need for per-resource private endpoints, load balancers, and proxy VMs. Customers configure the gateway via a Network Connectivity Configuration (NCC) and delegate one subnet, with Databricks managing the gateway infrastructure while customers retain control over routing and security. PNG supports use cases such as ingesting from private databases via Lakeflow Connect, connecting AI agents to private MCP servers, using private package repositories, and routing egress through customer firewalls. The gateway offers two traffic modes for precise control or broad egress inspection and works alongside existing private connectivity without disruption.
Modal crosses $1B in Annualized Run Rate and raises $750M Series D
Modal crossed $1B in annualized run rate and raised $750M in Series D funding at a $15.75B post-money valuation, led by Accel. Usage has tripled since the Series C five months ago, with year-over-year growth exceeding 15x, and the company now powers applications from companies including DoorDash, Cognition, and Runway. To avoid the friction of the old stack, Modal built its own file system, container runtime, scheduler, and networking from scratch on a capacity pool spanning over 20 clouds. Sandboxes, launched in 2023 for running LLM-generated code, now represent a $300M run rate product with nearly 3 billion sandboxes spun up and scaling to a million concurrent. New products include multi-node training with a single decorator giving a fully RDMA-connected cluster, Modal servers for HTTP serving, Endpoint Candidates for LLM serving, and generally available virtual machines.
Under the hood, off the hot path: How D1 scales to millions of databases
D1, Cloudflare’s serverless SQLite database, has seen 15x growth in live databases over the past year and scales by keeping query execution on the edge in Durable Objects, separate from the control plane’s metadata operations. The control plane uses an Edge API Worker with Hyperdrive to access Postgres clusters for transactional metadata, while Workers KV caches database-to-Durable-Object mappings for fast edge lookups. Database creation completes in about 1.6 seconds by generating a Durable Object ID, persisting metadata to Postgres, and storing the mapping in Workers KV, with the Durable Object lazily initialized on first query. Storage usage enforcement uses atomic counters in Workers KV that are checked locally on each query and updated asynchronously after execution, avoiding any central database round trip during the query. Precomputed aggregations for billing are stored in ClickHouse, and guiding principles such as eventual consistency and idempotency further reduce reliance on centralized components.
Introducing on-demand CPU and memory profiling with flamegraphs for Workers and Durable Objects
Cloudflare introduced on-demand CPU and memory profiling for Workers and Durable Objects, accessible via the CLI or the Dashboard’s Observability page. Users can generate interactive flamegraphs and download pprof files, with each rectangle’s width representing CPU time or memory usage over a chosen duration. Profiling the R2 binding Worker revealed that genericR2JsonReplacer, called by JSON.stringify, processed nested values multiple times; fixing it made the function 2.7x faster. A different Worker with P999 memory at 133 MB (above the 128 MB limit) was found to have a partially enabled Prometheus code path accounting for 66.7% of allocations; disabling it reduced P999 memory to 118 MB and eliminated out-of-memory errors. The profiling system locates the correct data center and isolate for the Worker or Durable Object, and continuous profiling is being developed to automatically capture samples without manual initiation.
Set Budgets and Alerts for Cloud Data Warehouse Costs
Reactive cost management for cloud data warehouses fails because usage is unattributed, budgets are absent, and alerts arrive too late. Databricks’ proactive framework addresses this through five pillars: cost attribution via warehouse and query tags, layered budgets with email alerts, real-time dashboards, and optimizations. SQL Serverless warehouses eliminate idle compute costs, as Lumen Technologies demonstrated with a 30-40% reduction. Query tags (Public Preview) enable per-query attribution, linking spend to specific dashboards or dbt models. The post provides step-by-step instructions for tagging, budget creation in the Account Console, and tuning auto-stop and statement timeouts to reduce waste.
TypeSafe AI scales Jev to trillions of tokens in three days on Modal
TypeSafe AI’s Jev is a System One Model that outputs structured decisions with probabilities rather than generating text, taking state and questions as input and returning typed values. It uses a custom architecture, a parallel sampler for maximum efficiency, and a training method called Reinforcement Learning for Calibrated Decisions (RLCD). Unlike autoregressive LLMs, Jev produces outputs in parallel, enabling low-latency responses suitable for workloads like large-scale automation, fast browser use, and generative user interfaces. Launched on a Thursday, Jev topped OpenRouter’s leaderboard for contexts between 1K and 10K tokens one week later, handling 15.1% of requests on that platform. To meet demand exceeding one trillion tokens in three days, TypeSafe used Modal’s serverless platform for bursty, high-concurrency inference without managing their own infrastructure, gaining control over networking, routing, and autoscaling needed for ultra-low-latency serving.
Building for good: How civil society organizations are automating on Cloudflare
Cloudflare awarded $7.5 million in credits to 30 civil society organizations to build automation tools using Workers AI and AI Gateway on its global network. LebTown replaced manual editorial workflows with a custom management system, and Kaya Guides built a WhatsApp mental health app processing 500,000 messages monthly with real-time counselor feedback. Larger groups like Freedom House and Article One received volunteer engineering support to build tools for tracking transnational repression and assessing human rights risks from factory audits. Cloudflare’s developer services include automatic DDoS protection and data minimization, addressing privacy and cost barriers that have limited AI adoption in the sector. The program is now open for a second cohort of non-profit organizations to apply for up to $250,000 in credits.
Announcing Cloudflare K2: serverless event streams
Cloudflare launched K2, a serverless event streaming service in public beta, to decouple producers and consumers in distributed systems. K2 stores events as an ordered log, allowing independent consumers to read at their own pace without losing data. Under the hood, it builds a partitioned durable log on top of R2 object storage, using atomic operations for ordering. Produce latency at the 99th percentile is about 1 second due to batching writes to R2 segments. K2 is available now for Workers Paid accounts with a 10GB storage limit and 30 MB/s produce per stream, with no billing during beta.
Introducing Cloudflare Basin: an open, serverless data platform, now generally available
Cloudflare announced the general availability of Basin, a serverless data analytics platform built on Apache Iceberg and R2 Object Storage. Basin consists of three integrated products: Pipelines for ingesting and transforming events via SQL, Catalog for managing Iceberg metadata and automatic table maintenance, and SQL for distributed querying. The platform emphasizes speed with instant resource creation and query execution, scalability through metadata compaction and statistics-driven query planning, and openness by supporting any Iceberg-compatible engine like DuckDB and Snowflake. Basin offers cost efficiency with zero egress fees and usage-based pricing, charging only when data is ingested, processed, or queried. Early adopters have used Basin for use cases including real-time e-commerce optimization, billing metrics storage, and infrastructure telemetry analysis.
Modal Clusters are generally available
Modal announced general availability of Modal Clusters, a serverless multi-node compute primitive for AI workloads that provides up to 6.4 Tbps InfiniBand RDMA per node with PyTorch and NCCL support. A new gang scheduler places N nodes simultaneously by observing the fleet, assigning all pending clusters, and provisioning capacity when needed. RDMA support is fully automated behind a single rdma=True flag, with Modal contributing RDMA support to the gVisor container runtime and upstreaming the changes. Customers like Decagon, 1x, and Runway use Clusters for trillion-parameter fine-tuning, pre-training home robot world models, and multi-node video inference respectively. Clusters are available to all workspaces today through the @modal.clustered decorator, with per-second billing and no reservation required.
VM Sandboxes: Full computers for agents
Modal has launched VM Sandboxes in general availability, providing agents with full Linux VMs accessible via a single flag. The VMs support Docker stacks, local databases, dev servers, and other real-machine features that agents increasingly require, as demonstrated by Ramp’s internal agent driving the entire software development lifecycle. Modal built a custom runtime on the Rust-based Cloud Hypervisor, layering innovations in host filesystems, lazy image loading, memory bursting, and snapshotting to preserve the same APIs and sub-second cold starts as their existing gVisor Sandboxes. Early customers like Linear, Legora, and Snorkel have launched over 20 million VMs, using them for coding sessions, legal agent evaluations, and agent simulations that require multiple containers and real-world environments. Modal recommends using VM Sandboxes when workloads need Docker, FUSE filesystems, or niche kernel features, while gVisor remains the default for compatible workloads.
A practical guide to cost optimization with Lakebase Postgres
Lakebase uses separated storage and compute to reduce costs: branches share storage, autoscaling matches compute to demand, and scale to zero suspends compute after inactivity. Read replicas and high availability add compute without duplicating storage. For synced tables from Lakehouse, sync only the active working set via materialized views and choose the appropriate sync mode—snapshot for volatile data, triggered for incremental, continuous for real-time—and binpack tables into one pipeline to share compute. Right-size initial compute by focusing on the working set, and set PITR retention and snapshot schedules based on actual recovery needs. Snapshot storage is priced at $0.090/GB-month and PITR storage at $0.200/GB-month, both cheaper than regular branch storage. Compute and storage usage can be tracked through system.billing.usage, with separate SKUs for compute, branch storage, PITR, and snapshots.
Lakebase Search: State-of-the-art full text and vector search for Postgres
Traditional OLTP systems like Postgres face challenges meeting the low-latency, high-accuracy search demands of AI agents, often requiring standalone search engines with ETL pipelines. Lakebase introduces lakebase_vector and lakebase_text, two extensions that bring scalable approximate nearest neighbor search and BM25 full-text search directly into Lakebase Postgres. lakebase_vector overcomes pgvector’s memory-bound HNSW index by using hierarchical IVF clustering and binary quantization, achieving 71-millisecond P99 latency at 97 percent recall on the 100-million-vector LAION benchmark. The extensions decouple storage from compute for pay-per-use scaling, enable fast parallel index builds, and support native hybrid search that fuses vector similarity with keyword relevance in a single SQL query. Early adopters like Conexiom reduced infrastructure costs by three times while achieving five times higher throughput, and the service is generally available on AWS and Azure.
Running open-Jev in SQL on Databricks
System One decision models such as Jev produce well-calibrated decisions from discrete options and are fast and cheap for large-scale applications, with potential use cases ranging from analyzing customer support transcripts to applying complex decisions at scale. Open-weight versions like SemIf-OpenJev are available, and Databricks enables serving them on governed data using serverless GPUs via AI Runtime, which provides on-demand GPU compute without managing infrastructure. The workflow is packaged in an importable notebook that downloads the model files, logs and registers the model using Express Deployments, and creates a GPU Model Serving endpoint automatically. The endpoint can be called from SQL via ai_query, which supports arbitrary custom model APIs, allowing structured request formats directly from SQL to return selected classifications and option probabilities as structured columns. The example classifies hotel reviews as good or bad, and users can import the notebook, select Serverless GPU, and run all to get started.
Introducing Worker Previews: isolated preview environments for every change your agent makes
Worker Previews gives every Git branch its own production-like environment with an isolated URL, configuration, state, and observability. Developers deploy a Preview using npx wrangler preview, which creates a separate Worker version with its own variables, secrets, and bindings, distinct from production, on a stable URL that updates with each push. Durable Objects and Containers automatically receive new namespaces per Preview, preventing state contamination across branches during migrations or schema changes. Each Preview supports full observability—logs, errors, metrics, and traces—scoped to that branch, enabling agents to autonomously test, catch failures, push fixes, and verify before production. The feature enables an Agent Development Lifecycle where every change is atomic, independently deployable, and revisable before merging to production.
How CSIRO built scalable, cost-optimized genomic variant querying on AWS
CSIRO built Serverless Beacon (sBeacon), a production-ready implementation of the Global Alliance for Genomics and Health (GA4GH) Beacon protocol for secure genomic variant querying. Its serverless architecture combines Amazon S3, AWS Lambda, Amazon DynamoDB, and Amazon Athena: genomic data remains in its original S3 location, while metadata is indexed in ORC tables and ontology indexes for query processing. Standard VCF files can be queried through API Gateway, with modular Lambda functions resolving ontology terms, querying Athena, and, when needed, issuing HTTP byte-range requests to source buckets so raw VCF bytes do not leave their owning organization. The post reports support for hundreds of millions of individuals and billions of genomic locations, approximately USD 0.40 monthly operation for a 1000 Genomes-scale dataset, roughly five-second real-world queries, and about 18-second generation-to-use onboarding, while noting that whole-genome range queries can time out.
How agentic software development will change databases
AI agents are changing software development from a slow, linear process into rapid evolutionary experimentation, creating many small, ephemeral applications and databases. The post identifies three resulting database requirements: near-zero marginal cost with elastic growth, native branching for rapid variants, and open ecosystems that agents can use reliably. Lakebase addresses branching through O(1) metadata copy-on-write at the storage layer, so agents can branch production or test state without physical data copying and pay only for experiment compute. Its decoupled, serverless compute can scale in sub-second intervals and down to zero, while standard open Postgres page formats in cloud object storage keep data accessible to external engines and tools. Lakebase telemetry reports about 10 branches per project on average, some nested beyond 500 iterations, and roughly four times as many databases created by agents as by humans.
On-Demand Masked Sessions with Twilio Proxy, Voice and Serverless
This tutorial presents a Just-in-Time (JIT) masked session workflow for securely connecting callers when phone numbers cannot be pre-associated because of inventory constraints. The design routes an inbound call to a reserved Twilio number through Twilio Voice, uses an IVR to collect a six-digit tracking code, and dynamically creates a private Twilio Proxy session. It uses five serverless endpoints plus helper functions; because the Out-of-Session Callback lacks keypad digits, a “2-Bounce” flow uses a redirect and a temporary Twilio Sync Document keyed by res- plus CallSid to carry the resolved destination, with a short TTL such as 900 seconds. On the second bounce, the destination is retrieved and deleted, and Proxy receives instructions to auto-create the session and bridge the caller. The result is an on-demand masked connection across standard cellular connections that avoids static phone-number preallocation and preserves the number pool.
Product updates: Sandbox Sidecars, new models, a refreshed dashboard, and more
Modal added day-zero support for Kimi K3, Qwen 3.8, GLM 5.3, and GLM 5.3 Flash on Auto Endpoints, with custom DFlash speculators boosting throughput to 460 tokens per second. New environment-level budgets give Team and Enterprise workspaces per-environment compute limits, while the pricing multiplier for broad region selection dropped from 1.5x to 1.15x. The dashboard got a refreshed navigation, color system, and beta light mode, and Private Environments now support a default No Access role. Sandbox Sidecars are in public alpha, running additional containers via low-latency internal networking, and VM Sandboxes are in public beta with a real Linux kernel for Docker and eBPF. Sandbox directory snapshots and a new filesystem API are now GA, and regional Proxies allow selecting a region for static outbound IPs, while SDK updates improved log APIs and autoscaler reporting.
Hybrid cloud orchestration: Modernizing on-premises infrastructure management with AWS
Managing distributed on-premises infrastructure across hundreds of sites creates inconsistency, manual lifecycle bottlenecks, fragmented visibility, and scalability problems, especially when hardware, operating system, Kubernetes, and application operations span thousands of servers. The proposed design centralizes orchestration in AWS while keeping execution on site, using hybrid connectivity, AWS Lambda, AWS Step Functions, Amazon EventBridge, DynamoDB, AWS Systems Manager, and supporting services. Redfish APIs provide vendor-agnostic bare-metal control for BIOS, firmware, power, and health operations, while EKS Anywhere deploys and operates management and workload clusters on local hardware; orders and inventory records track workflows and resource relationships. DynamoDB Streams also automate DNS, certificates, AWS access, and storage integrations, while centralized observability aggregates server, cluster, and application signals; the post presents architectural patterns and defers detailed implementation workflows and templates to later parts.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.