---
title: "How to test that a dozen MongoDB drivers, in a dozen languages, conform to a dozen specs"
description: "MongoDB develops a dozen native client library drivers in a dozen languages, each idiomatic to its language but needing consistent API behavior. Instead of relying on English specs or Cucumber tests, engineers created a YAML-based domain-specific test language interpreted by runners in each driver. The Unified Test Format, built on this YAML language, now contains nearly 1,500 files testing conformance to specs covering retries, transactions, client-side encryption, and more. A natural experiment showed four of five drivers had lower nonconformance bug rates after adopting YAML tests; Node.js improved after fixing its test runner. The IEEE ISSRE 2026 accepted a paper on this approach, which has saved over 22,000 lines of test-runner code across seven drivers."
---

# How to test that a dozen MongoDB drivers, in a dozen languages, conform to a dozen specs

[MongoDB](https://yomu.fyi/company/mongodb) · Sep 23, 2026

**Type:** Problem & solution

## Summary

MongoDB develops a dozen native client library drivers in a dozen languages, each idiomatic to its language but needing consistent API behavior. Instead of relying on English specs or Cucumber tests, engineers created a YAML-based domain-specific test language interpreted by runners in each driver. The Unified Test Format, built on this YAML language, now contains nearly 1,500 files testing conformance to specs covering retries, transactions, client-side encryption, and more. A natural experiment showed four of five drivers had lower nonconformance bug rates after adopting YAML tests; Node.js improved after fixing its test runner. The IEEE ISSRE 2026 accepted a paper on this approach, which has saved over 22,000 lines of test-runner code across seven drivers.

## Context

MongoDB develops native drivers in a dozen languages, making them idiomatic but creating a need to ensure consistent behavior and API conformance across all implementations.

## Approach / What changed

A custom YAML-based domain-specific test language interpreted by test runners in each driver, supporting shared test suites and the Unified Test Format. Tests use fail points, entity maps, and assertions for both API and hidden internal behaviors.

## Takeaways

- MongoDB uses a shared YAML-based test language interpreted by per-driver runners to enforce consistent behavior across native drivers in different languages.
- The Unified Test Format lets tests assert hidden internal behaviors like retry logic and connection pool clearing, not just API surface results.
- Adopting YAML conformance tests reduced nonconformance bug rates in four of five drivers; Node.js initially saw an increase due to insufficiently rigorous test assertions.

## FAQ

### How do YAML test runners work across different languages?

Drivers implement a test runner that interprets YAML test files. The runner maps generic YAML operations (like updateOne) to the driver's own API (e.g., update\_one in Python, mongoc\_collection\_update\_one in C), and executes assertions against the driver's behavior.

### How do YAML tests verify invisible internal behavior like retries?

Tests use a failPoint to force the server to close the connection on a specific command. The driver must then reconnect and retry. The test asserts the driver sent exactly two update commands, marked the server type Unknown, and cleared its connection pool, verifying retry logic that is invisible from the API.

### Why did Node.js bug rate go up after adopting YAML tests?

The Node.js driver's test runner did not assert the contents of BulkWriteResult and BulkWriteError classes rigorously enough, so inconsistencies with the spec went undetected for years. The team fixed the runner and changed field names over two major releases.

**Tags:** [API](https://yomu.fyi/topic/api), [MongoDB](https://yomu.fyi/topic/mongodb), [Testing](https://yomu.fyi/topic/testing), [Unified Test Format](https://yomu.fyi/topic/unified-test-format), [YAML](https://yomu.fyi/topic/yaml)

- Source: [MongoDB](https://www.mongodb.com/company/research/how-to-test-mongodb-drivers-conform-to-a-dozen-specs)
- Source URL: https://www.mongodb.com/company/research/how-to-test-mongodb-drivers-conform-to-a-dozen-specs
- Ingested by Yomu: 2026-09-23T18:00:18.640Z

[Read original post](https://www.mongodb.com/company/research/how-to-test-mongodb-drivers-conform-to-a-dozen-specs)