Loading…
Testing
42 posts about Testing. Every summary links to the original.
Github ·
Your alt text passes automated checks. That doesn’t mean it’s any good.
Automated accessibility checkers reliably flag missing alt text attributes but frequently miss unhelpful descriptions like raw filenames or repetitive labels. To evaluate image description quality without generating excessive false positives, GitHub built an alt text plugin for the GitHub Accessibility Scanner. The tool combines five deterministic, zero-credential rules that evaluate strings and visual layout spacing with an opt-in vision model check for subjective context. Page context including headings and surrounding prose is extracted alongside images to guide the model using structured outputs and explicit anti-nitpick instructions. While deterministic checks catch unwritten text, the model-driven rule serves as an opt-in prompt for human review rather than an absolute verdict.
Taarik AshenafiWhy AI-Generated Code Is Easy but Engineering Trust Is Hard
At Salesforce, engineering teams discovered that AI coding agents could produce internally consistent code and passing test suites that still failed to meet actual requirements. To ensure correctness before writing code, the team developed an agentic Spec-Driven Development workflow centered on explicit specifications and gated verification. The system separates evidence-based repository lookups handled by agents from subjective judgment calls escalated to human engineers. Implementation plans must cite repository evidence reviewed by a Skeptic Agent, while a Compliance Matrix traces each success criterion directly to executable test proof. Finally, a multi-agent review system evaluates the implementation under an asymmetric rule allowing an independent judge to downgrade passing checks but never override failed gates.
Scott NybergPyCharm for AI-assisted Django Workflows
According to the 2026 Django Developers Survey, ninety percent of respondents incorporate artificial intelligence into their weekly or daily workflows. While agents generate code rapidly, developers remain accountable for understanding, evaluating, and shipping changes within their applications. PyCharm addresses these demands by integrating support for external agents, local models via Ollama and LM Studio, and customizable agent skills across codebases. The IDE maintains version-specific framework assistance, including auto-completion for Django 6.0 template partials, alongside architectural inspection tools and visual diffs. Furthermore, developers can trace application components, inspect API endpoints, execute HTTP requests, and query database migrations directly through built-in tooling.
Will VincentHow to Evaluate Production AI Agents: Measure System Outcomes, Not Conversations
Evaluating AI agents solely on conversational transcripts fails when agents must invoke tools to modify external production systems. Language models can convincingly narrate that an action occurred while completely omitting the underlying tool call, leaving database or billing records unchanged. Outcome-based evaluation addresses this divergence by treating external system mutations rather than generated dialogue as the actual source of truth. Frameworks such as CRMAgentBench implement persistent, stateful environments with strict all-or-nothing validation, verifying tool arguments, execution sequence, final record modifications, and the total absence of unauthorized actions. Furthermore, measuring repeated success across independent executions using pass^k metrics exposes operational variance and inconsistency that standard single-shot scores and pass@k metrics conceal.
Scott NybergHow Agentforce-Powered AI Security Workflows Accelerate Incident Response
Salesforce engineers expanded Security Center from a basic conversational interface into an Agentforce-powered stateful investigations platform for incident response. Managing enterprise security incidents required handling long-running lifecycles, disparate telemetry formats, and finite LLM context windows that risk reasoning pocket collapse. To prevent overflowing context limits, the architecture partitions telemetry into contextual segments and uses AI-driven summarization pipelines before feeding data to the agent. Because non-deterministic LLM outputs break traditional deterministic testing, the team implemented AI-driven evaluation pipelines using simulated customer interactions to evaluate response behavior rather than exact wording. Ongoing work focuses on grounding public foundation models with curated, Salesforce-owned security knowledge to improve platform-specific remediation guidance.
Scott NybergGrab ·
Grab Bench: Evaluating AI on Grab-shaped production work
AI models evaluated on internal tasks often produce subtly plausible errors, such as altering metric definitions in SQL queries or missing hidden stateful invariants in code patches. To address this, Grab built Grab Bench, a configurable evaluation harness that executes model providers through specialized task plugins. The platform evaluates synthetic or redacted datasets that preserve production constraints across domains like query generation, tool use, passenger-profile reasoning, and agentic coding. By recording granular row-level outputs with deterministic scorers and LLM judges, the harness exposes failure modes rather than relying solely on aggregate leaderboards.
Christian CoffrantAirbnb ·
Eval-driven development: Lessons from evaluating GenAI at scale
Airbnb outlines its framework for eval-driven development (EDD), treating GenAI evaluation as a continuous engineering discipline rather than an afterthought. The strategy combines programmatic checks, calibrated LLM-as-a-judge evaluators, and human review to detect subtle quality regressions. By inspecting intermediate agentic execution traces and real-world failure modes, teams ensure generative models remain reliable and aligned with product requirements.
Rohit GirmeAirbnb ·
From weeks to a day: how we made LLM evaluation fast enough to iterate on
Airbnb built a four-layer LLM infrastructure framework to reduce iteration and evaluation turnaround from weeks to a single day. By using per-sample caching for generated references and judge scores, they established a deterministic evaluation foundation that separates genuine model drift from measurement noise. This setup enables rapid, bounded model hotfixes using micro LoRA adapters and comprehensive end-to-end validation across component boundaries.
Baharak SaberidokhtDropbox ·
How we used DSPy to turn AI evaluations into better responses in Dash chat
Dropbox improved its Dash chat agent by establishing an automated optimization loop powered by DSPy and LLM-as-judge evaluations. Engineers first calibrated their LLM judges against human-annotated interaction traces, then used those judges to systematically optimize the agent's system prompts via offline counterfactual replay. This automated workflow reduced incomplete responses by 26% while decreasing overall token consumption.
Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy MeyerzonGrab ·
Scaling out Distroless adoption With AI
Grab is transitioning its microservices to Distroless base images to eliminate unnecessary binaries and reduce vulnerability risks, but the migration risks runtime failures from missing shared objects and system utilities. To safely validate container execution in continuous integration without staging dependencies, the team relied on medium tests that run containerized services alongside internal dependencies managed by Testcontainers. Because hundreds of services lacked this test harness, Grab implemented an agentic workflow using Claude Code and Model Context Protocol integrations to inspect repositories, generate test boilerplate, and resolve configuration errors. Once test baselines are established, an automated patch-test-compare pipeline updates Dockerfiles, constructs multi-stage builds for necessary dynamic libraries, and creates draft merge requests for human approval.
Jia Yee ChongNetflix ·
The Data Canary: How Netflix Validates Catalog Metadata
A manual mitigation action during an incident corrupted a data feed for a subset of titles, causing playback issues and catalog service failures that existing code canary systems failed to catch. To protect streaming reliability, Netflix built an automated data canary system that validates transformed catalog metadata prior to publication. The architecture utilizes a dedicated orchestrator alongside permanent baseline and canary service clusters to coordinate validation using real production traffic. By leveraging custom chaos experiment thresholds, sticky session affinity, and Starts Per Second playback metrics, the system detects regressions in under ten minutes and blocks publication automatically. Controlled failure injection experiments routing approximately 0.2% of global traffic confirmed that issues could be identified in 2.5 to 4 minutes.
Netflix Technology BlogSlack ·
Agentic Testing: Where Agents Fit in the E2E Testing Stack
Traditional end-to-end tests validate rigid user journeys, whereas agentic tests verify whether broad goals can be achieved by adapting actions dynamically. To evaluate agentic testing tradeoffs, researchers executed over 200 runs across Playwright Model Context Protocol (MCP), Playwright CLI, and agent-generated Playwright tests using Claude models. Playwright MCP demonstrated high reliability with failure rates of 0% on simple thread replies and approximately 12% on complex search discovery flows. Playwright CLI and generated code struggled more on complex workflows, exhibiting failure rates of approximately 20% and 48% respectively. Although generated tests were faster with average runtimes of roughly three minutes, agentic testing provides a distinct exploratory layer atop deterministic CI test suites.
Sergii GorbachovDropbox ·
Beyond code generation: rethinking engineering productivity in the age of AI agents
Dropbox shares how widespread AI code generation shifts software development bottlenecks downstream into code review, CI infrastructure, and validation pipelines. To adapt, they built Nova, an internal coding agent platform that safely automates scoped tasks such as migrations and flaky test remediation. They also evolved their developer productivity framework to measure end-to-end customer impact and code quality rather than simple pull request throughput.
Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki OkumuraDropbox ·
Introducing Nova, our internal platform for coding agents
Dropbox developed Nova, an internal platform that runs AI coding agents in isolated cloud environments integrated with their Bazel monorepo. The platform supports both interactive developer workflows and autonomous background tasks, such as automated CI debugging, flaky test remediation, and codebase-wide migrations. By pairing code generation with automated validation and strict execution guardrails, Nova ensures generated fixes are tested and reproducible.
Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin AltschulerGrab ·
Cursor at Grab: Adoption and impact
Following a multi-tool AI strategy, Grab integrated the AI coding assistant Cursor into its engineering toolkit in late 2024 to accelerate software development. Technical staff adoption reached 98% monthly active usage with a 50% suggestion acceptance rate, supported by custom monorepo indexing and preconfigured rules aligned with internal coding conventions. Engineers frequently apply the tool to unit test generation, code refactoring, cross-repository navigation, and routine API scaffolding, with over a third of merge requests incorporating Cursor. The rollout also encompasses non-technical personnel and product designers who, after receiving Git training, submit direct production UI fixes. Statistical evaluations using fixed-effects regression indicate a dose-response relationship between Cursor usage intensity and measurable productivity gains.
Akshay MisraGrab ·
Demystifying user journeys: Revolutionizing troubleshooting with auto tracking
Tracking user journeys across hundreds of mobile UI components via manual clickstream instrumentation frequently caused data gaps and prolonged issue resolution. Grab designed AutoTrack, an SDK that automatically records application lifecycle states, launch triggers, user interactions, and screen contexts. The system monitors transitions across foreground, background, and inactive states alongside entry points like deep links and push notifications. On the client side, AutoTrack captures complete screen hierarchy metadata from native Android activities, fragments, and iOS view controllers, redacting sensitive data to comply with privacy regulations. This automated trace enables engineering teams to reproduce elusive bugs, route incidents to domain owners, and generate automated UI test cases from real-world usage patterns.
Alex ProkofievGrab ·
A Decade of Defense: Celebrating Grab's 10th Year Bug Bounty Program
Grab's bug bounty program has operated for a decade in partnership with HackerOne, expanding from an initial cohort of 23 researchers to over 850 active participants across global regions. The program's scope broadened between 2023 and 2024 to encompass artificial intelligence systems, Indonesian financial services, and a dedicated bounty table for mobile-specific security issues. Grab extended external testing coverage through live hacking appearances at ThreatCon 2023 and DEFCON 32, as well as invite-only anniversary campaigns with regional clubs in Germany, Morocco, and India. Internal cybersecurity teams manage vulnerability reports by emphasizing rapid triage times, direct communication, and payouts upon triage. Over the decade, reported vulnerabilities transitioned from foundational flaws toward more sophisticated and emerging threat categories.
Pei Shan YapGrab ·
How we evaluated the business impact of marketing campaigns
Grab needed a scalable, centralised way to evaluate the business impact of millions of marketing messages sent across expanding campaigns. Manual analysis by marketing analysts produced inconsistent results and failed to scale, leading to the creation of multi-touch attribution and controlled experiment methodologies. The team designed a Marketing attribution model (MAM) that credits touchpoints based on relevance and recency, while using stratified hold-out groups to assess single campaign lift against baseline metrics. Statistical significance calculations, implemented using the jStat library based on the central limit theorem, ensure observed business metric differences like gross merchandise value are not random. The system provides campaign managers with direct access to impact analyses while the team works on challenges surrounding smaller sample sizes and hold-out group constraints.
Jie ZhangGrab ·
Grab Experiment Decision Engine - a Unified Toolkit for Experimentation
Grab's Experimentation platform tests thousands of experimental variants weekly, but analyses were largely handled manually using disparate tools unsuited to multi-sided marketplace nuances. To standardize workflows and eliminate reliability issues, Grab developed the GrabX Decision Engine, an internally open-sourced Python toolkit integrated with its Automated Experiment Analysis application. The framework includes a Trusted Advisor component that automates power analysis and sample size estimation while adjusting for clustered standard errors across metric data types. It also bundles a post-experiment statistical toolbox executing default and custom evaluations, ranging from non-parametric tests to regression models that control for confounders and fixed effects. This unified platform institutionalizes experimentation best practices across teams, streamlining collaboration and ensuring consistent statistical inference.
Ruike ZhangGrab ·
Championing CyberSecurity: Grab's bug bounty programme in 2023
Grab reviewed the performance and milestones of its security bug bounty programme for 2023, marking eight years of running the initiative. Operating quarterly campaigns through HackerOne alongside dedicated internal teams, the company processed approximately 1,000 vulnerability submissions over the year, including 400 reports during Q1 alone. Cumulative payouts to external security researchers surpassed $700,000 since the programme's 2015 launch. To adapt to an expanding service portfolio, Grab broadened its scope to encompass joint ventures and acquisitions while introducing diversified financial and recognition-based rewards.
Nathaniel Callens