Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,671 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
How telecom CFOs can make smarter network capex decisions with AI
Telecom CFOs face multi-year, multi-billion-dollar network capex decisions involving spectrum, fiber, and 5G densification amid technology uncertainty and competitive dynamics. The post argues that operators can improve allocation by unifying network quality, customer churn, ARPU, billing, and investment history in a queryable environment. Databricks Genie provides a governed natural-language interface over that enterprise data, allowing finance leaders without SQL or data engineering backgrounds to ask cross-domain questions at geographic granularity and receive answers from systems of record. Example analyses compare ARPU and churn after 5G densification with markets still on the deployment roadmap, while scenario modeling can test accelerated deployment against historical returns. The stated outcome is better evidence for strategic judgment, shifting conversations from generic industry benchmarks toward comparable investments in the operator’s own network.
How we develop pricing and packaging at Fin
Pricing and packaging at Fin is treated as a cross-functional research, forecasting, and strategic problem rather than a finishing step. When new monetization is needed, the process starts with qualitative buyer research to define a pricing model and metric; for Fin, that produced value-based pricing tied to “outcomes,” each defined as the AI Agent successfully handling a customer service query. Quantitative willingness-to-pay studies using Gabor-Granger and Van Westendorp then indicate demand across price points, but do not establish a perfect price. Modeling combines those signals with discounting, usage, segment variation, attach rates, margins, sales capacity, competitor pricing, and finance forecasts to recommend a price alongside ARR and adoption scenarios before executive sign-off. The approach is iterative and applied selectively, while expanding products and capabilities create an ongoing need to review whether the pricing system remains coherent and aligned with delivered value.
How Databricks Genie improves retail personalization
Retail personalization often stalls when CX leaders must wait for analysts to answer questions about segment behavior, loyalty performance, channel preference, or churn risk, narrowing the window for action. Databricks Genie addresses this access gap by letting business users query unified structured and unstructured enterprise data in plain English instead of SQL. Its retail differentiators include identity-resolved queries across channels and devices, lifecycle-stage awareness, campaign response and control-group data integration, and privacy controls enforced through Unity Catalog. The source says Genie reduces routine analyst requests rather than replacing data science teams, while enabling merchandisers, category managers, loyalty marketers, and CX leaders to self-serve operational questions. It cites 7-Eleven’s use of Databricks SQL, Unity Catalog, and AI/BI Genie to launch, refine, and measure personalized offers within a secure, unified platform.
Databricks for Good and Virtue Foundation: Partnering to Connect Medical Volunteers to Critical Health Services in 72 Countries
Virtue Foundation’s VF Match platform connects medical professionals with volunteer opportunities across 72 low and low-middle income countries, but scaling its initial LLM proof of concept into production required a robust data pipeline. The Databricks-based Foundational Data Refresh combines Overture Maps and Bright Data, processes more than 25 million web pages with targeted GPT extraction steps, and uses Spark, Photon, and Lakeflow Jobs to orchestrate checkpointed, modular workloads. Splink resolves duplicate facilities and nonprofits across messy sources; Photon reduced the slowest entity-resolution partition from 30 minutes to approximately two minutes, a 15x improvement. A prototype VF Agent adds natural-language access through LangGraph, Databricks Model Serving, AI Search, and Genie, while the resulting system provides an up-to-date global view of healthcare infrastructure and volunteer needs.
Introducing Arrow UDFs in PySpark: A Faster, Leaner Replacement for Pandas UDFs
Apache Spark’s Pandas UDFs improve Python UDF throughput through Arrow-based serialization and batching, but Pandas/Arrow conversion can add copies, limit zero-copy behavior, and weaken support for complex datatypes. Native Arrow UDFs, introduced in Databricks Runtime 18.0, operate directly on Arrow data without converting inputs to Pandas or NumPy objects, preserving columnar layout and enabling vectorized processing through Arrow’s compute and memory model. The new @arrow_udf decorator supports scalar, aggregate, and table functions, while type-hinted @udf definitions, mapInArrow, and applyInArrow extend the interface to DataFrame and grouped operations; iterator-based modes can amortize initialization costs. In a comparison, an Arrow UDF ran about 10% faster than a Pandas UDF and used about 40% less memory, with better complex-datatype support.
Scaling reinforcement learning at Applied Compute
Applied Compute, founded by former OpenAI Codex and o1 researchers, builds custom AI agents for enterprises through a post-training approach they call Specific Intelligence. Their core training mechanism is Reinforcement Learning, where agents repeatedly attempt tasks in replayable environments and are scored against reward functions to refine model weights. For clients like DoorDash and Cognition, this means training specialized agents for merchant onboarding and bug-catching that improve from continuous use. The company evaluated multiple infrastructure platforms before selecting Modal, which provides ephemeral sandboxes for complex environment simulation, sub-second cold starts to keep GPUs busy, and serverless fan-out for parallel grading. Modal’s primitives allow each phase of the RL loop—rollouts, evals, and inference—to operate with appropriate resources while sharing state efficiently.
Improvements to Cursor Automations
Cursor Automations are now available in the Agents Window as well as the existing Automations interface, allowing users to create and manage them alongside agents in the same workspace. The release also lets an automation attach multiple repositories, so agents can use context across codebases to deliver, test, and verify tasks, or run with no repository when the work is not code-centered. For no-repo use cases, five Cursor Marketplace templates cover Slack digests, product analytics, FAQ drafting from Slack and documentation context, finance reporting from providers such as Stripe, and customer health monitoring across tools including Granola, Slack, and Databricks. Newly created automations receive a 50% discount on all agent runs for the next seven days, while the release directs users to the documentation for more information.
Six Steps to Improve Your Sender Reputation
Sender reputation is the largely private assessment that inbox service providers such as Gmail, Outlook, and Yahoo make before deciding where to deliver email. It reflects infrastructure compliance, sending history and consistency, list hygiene, mail frequency, and recipient engagement, with providers increasingly emphasizing domain reputation over IP reputation. The guide recommends six recovery steps: audit SPF, DKIM, and DMARC; warm up new domains gradually; identify provider-specific blocks or blocklists; clean and segment lists; make unsubscribing easy; and send valuable, mobile-friendly content. It also advises using double opt-in, monitoring engagement, bounce and spam responses, throttling, and DMARC reports, while tightening policies or requesting provider reviews after fixes. Because providers continually reevaluate senders, reputation management requires repeated observation, testing, and adjustment rather than a one-time repair.
Welcoming Tatira Andrade, our new Executive Assistant
Tatira Andrade is joining Resend as an Executive Assistant, bringing nearly 16 years of experience in executive support, complex logistics, and process optimization. Her background includes working with companies such as Nubank, where she helped leadership teams streamline daily operations and scale workflows efficiently, while her interests in travel, geography, languages, and organization align with the role. Andrade says she is motivated by making processes and experiences more seamless; she favors Gemini, uses Command + Shift + 4 for screenshots, and currently names Montreal in autumn as a favorite destination. Her advice to ambitious executive assistants is to keep learning, build technology and language skills, stay updated, and remain genuinely curious about their companies and colleagues.
Optimizing Our Build Times by Migrating from Webpack to Rspack
As Yelp's JavaScript monorepo expanded over time, Webpack build speeds became a significant engineering bottleneck. To resolve this without rewriting complex configurations, the team selected Rspack for its direct compatibility with existing Webpack setups and plugins. Yelp implemented an adapter pattern to generate Rspack configurations dynamically from original Webpack files, enabling a staged opt-in rollout across development tools and CI environments. Following the initial migration, engineers addressed performance issues in barrel files using automated codemods to replace star re-exports and import-then-export patterns with named re-exports. Combined with Rspack's portable persistent caching, the migration reduced integration build times by approximately 52 percent and warm cache builds by up to 80 percent.
Automate Data & KPI Monitoring with SQL Alerts
Databricks SQL Alerts is generally available as an automated alternative to manually checking dashboards and rerunning monitoring queries for business KPIs, pipeline health, and data quality. Teams define a SQL query, evaluation condition, schedule, and notification destinations, then receive alerts when results cross configured guardrails; standalone alerts can also run within Lakeflow Jobs. GA supports authoring in the SQL editor, notifications through email, Slack, PagerDuty, Microsoft Teams, or webhooks, and production management through Git, Declarative Automation Bundles, APIs, Terraform, and SDKs. The post’s revenue example compares yesterday’s result with a seven-day average and triggers when revenue_pct_change is below -5%, while the SQL Alert task can expose OK, TRIGGERED, or ERROR to downstream pipeline logic.
How to Build Real-Time Fraud Detection using Spark Real-Time Mode and Lakebase
Card fraud operates in seconds, while organizations need to score and block suspicious transactions before settlement without adding a separate streaming stack. This Solution Accelerator presents an open-source, end-to-end implementation on Databricks, covering transaction ingestion, Spark Real-Time Mode processing, fraud scoring, Lakebase feature serving, and a Databricks Apps monitoring dashboard. RTM processes streaming workloads in milliseconds and targets sub-300ms latency, while Lakebase supplies a managed, serverless Postgres serving layer with sub-millisecond reads. The design keeps batch ETL, training, streaming, and governance within the same platform. Deployment progresses from a synthetic rate-source quick start to Kafka-based parsing, stateful per-card velocity tracking with TTL, enrichment, weighted scoring, routing, and optional MLflow-trained RandomForest scoring, followed by a Streamlit app for live decisions.
How Databricks Genie improves supply chain visibility with real-time AI analytics
Supply chain leaders often have extensive operational data yet still respond reactively because predictive signals such as supplier lead-time trends, inventory velocity, weather, and commodity prices remain siloed and difficult to synthesize. Databricks Genie is presented as a plain-language intelligence layer that lets leaders interrogate operational and external data in the flow of work rather than relying on analyst-led BI sessions. Users can ask questions combining supplier tiers, lead-time changes, inventory coverage, contract terms, and production schedules, then follow with what-if queries about financial exposure if conditions worsen. The post says Genie returns answers in seconds, supports near-real-time monitoring and proactive alerting, and enables shared, governed answers for procurement, operations, and finance, shifting decisions from reactive reporting toward earlier evidence-based action.
OlmoEarth v1.1: A more efficient family of Earth observation models
OlmoEarth v1.1 is a new family of Earth observation models designed to reduce the compute required for large-scale satellite imagery predictions. Compute dominates the lifecycle from data export through preprocessing, inference, and post-processing, motivating a focus on transformer token sequence length. For Sentinel-2, the models collapse three resolution-specific tokens into one, reducing token counts threefold; this initially caused a 10 ppt drop on the m-eurosat kNN benchmark, so the pre-training regimen was modified. OlmoEarth v1.1 runs up to three times cheaper than v1 while maintaining similar performance across a mix of research benchmarks and partner-built tasks, although some regressions remain, and the same dataset is used to isolate methodological effects.
A CFO’s guide to managing value-based care financial performance
Value-based care changes healthcare finance by tying payment to outcomes, cost efficiency, and equity rather than service volume, while shifting greater financial risk to providers under ACO, bundled, and capitated contracts. A CFO must therefore track attributed populations, PMPM cost trends, utilization, clinical outliers, quality thresholds, and contract performance against benchmarks. Databricks Genie is presented as a conversational interface over integrated attribution, claims, clinical, and benchmark data, allowing questions such as PMPM divergence among high utilizers and DRG drivers to be answered in real time. The source also identifies incorrect attribution and risk-adjustment coding errors as major threats, and describes predictive risk stratification and quality-gap forecasting as ways AI can support earlier intervention.
Scaling Airbnb’s identity graph with a unified knowledge graph infrastructure
Airbnb migrated its massive identity graph from a third-party graph database vendor to a unified, in-house knowledge graph platform to solve long-tail latency and scaling bottlenecks. Built on JanusGraph with AWS DynamoDB for persistence and OpenSearch for indexing, the new architecture decouples storage operations from graph traversal logic. The migration improved P99 query latency, eliminated routine instance reboots, and supported a tenfold increase in write throughput.
Stop rogue AI: How Unity Catalog secures your agent actions
Agentic AI systems connected to external tools can take destructive production actions while operating within delegated authority, yet without restrictions on tool selection or records of their calls. Unity Catalog now governs MCP tools with fine-grained access control, service policies, and payload logging, while Unity AI Gateway enforces decisions in real time. Policies are SQL functions receiving actor and context information; they can allow, deny, or require user consent, inspect arguments and caller properties, and block calls before execution. Every tool call is recorded in a Unity Catalog-managed Delta table with the tool name, arguments, result, user identity, and allow-or-deny status, making the activity queryable with SQL. The service policies and payload logging are available as a Gated Beta, with early access offered through Databricks account teams.
Ready for your busiest day: How we scale
Fin describes how its customer service platform is designed to absorb high-volume events while preserving customer isolation, availability, and operational control. At daily peaks, it handles over 150,000 customer requests per second and more than 70,000 asynchronous requests per second; its source-of-truth data spans 128 Vitess shards, while Elasticsearch supports over 650TB, 1.7 trillion documents, and peaks above 40,000 requests per second. The architecture favors AWS primitives, Vitess managed by PlanetScale, online and incremental migrations, fair queues, feature flags, heartbeat metrics, and automatic rollback, with roughly 250 production ships per workday and merge-to-production time under 10 minutes. The stated conclusion is that scale depends less on headline capacity than on clear scaling levers, tenant fairness, customer-outcome monitoring, reversible change, and routine infrastructure improvement without planned customer-impacting downtime.
Databricks context engineer associate: the industry’s first certification for reliable AI agent systems
Databricks announces the Databricks Certified Context Engineer Associate, described as the industry’s first certification purpose-built for context-aware AI systems. It assesses whether practitioners can design, assemble, and govern the information AI agents receive at inference time, including system prompts, retrieval, memory, external tools, context-window management, and governance. Candidates work with Databricks technologies such as AI Search, Lakebase, MLflow, MCP, and Unity Catalog, while advanced coverage includes multi-agent systems, long-horizon workflows, and evaluation of context decisions. The beta exam will be available free to Data + AI Summit attendees, who may take it once; beta results are expected in 6–8 weeks.
Scaling Real-Time Traffic Forecasting with a Graph-Aware Transformer
DeepETT is Uber’s real-time traffic forecasting system for estimating traversal times on roughly 100 million road segments, supporting routing and arrival-time estimates. It replaces a legacy blend of recent observations and historical patterns with a deep-learning model designed for rapidly changing conditions, sparse roads, and long trips. To keep deployment predictable at global scale, the system uses fixed-size computation over pre-aggregated segment, graph, region, hourly, weekly, and event views rather than a dynamic graph, while predicting segment ETTs independently of routing. Training targets segment-level mean squared error, while trip-level mean absolute error is evaluated offline after routing sums segment forecasts; calibration and continual real-time correction address downstream error. In a global A/B test, long-trip arrival-time accuracy improved 6%, variance explained rose 19%, navigation defects fell 2.73%, and mobility gross bookings increased by an estimated $100 million annually, with forecasts exceeding 2 million per second.
Top topics
- Architecture834
- Performance816
- LLMs660
- Machine Learning622
- Open Source563
- AI Agents553
- AI470
- Developer Experience448
Weekly digest
The best engineering reads, once a week.