Standard text-to-speech (TTS) models perform well on short interactions but fail to produce natural audiobook narrations because they cannot reliably change speaking style between characters. This creates difficulty for listeners in distinguishing narration from dialogue and identifying which character is speaking. Audible’s technology team developed a controllable TTS system that uses natural-language speech descriptions (e.g., “speak in an enthusiastic, high-pitched voice”) to control dialogue delivery, leveraging LLMs for deep textual understanding. The approach includes a novel data synthesis pipeline that generates speech data with prosodic captions, and BookLLM, a system that automatically profiles characters with 97% accuracy and captions dialogue lines. In MUSHRA tests, the system improved dialogue differentiation by 3.7 points across English and Spanish, with statistically significant improvement for eight out of ten voices.
Context
Conventional TTS models are inadequate for long-form audiobook narration because they cannot distinguish between characters or maintain consistent voices across chapters, motivating the development of a controllable TTS system with LLM integration.
Approach / What changed
The system comprises a data synthesis pipeline that uses expert models and a speech captioner to generate training data with prosodic descriptions, and BookLLM, which uses LLMs to analyze book text, identify characters, assign voice profiles from a curated catalogue, and generate per-dialogue captions for expressivity.
Takeaways
1
Controllable TTS uses natural-language speech descriptions (e.g., ‘whispering nervously’) to control dialogue delivery, enabling the model to produce distinct speaking styles for different characters in an audiobook.
2
BookLLM achieves 97% accuracy in identifying which character speaks which lines in English books (94% in Spanish), including distinguishing narration from dialogue, by processing the entire book text with LLMs.
3
Combining a fixed voice profile per character (assigned once at book level) with per-line captions for moment-to-moment expressivity solves the consistency drift problem that plagues general-purpose TTS over long-form content.
FAQ
How does the system ensure character voice consistency across an entire book?
Each character is assigned a fixed voice profile from a curated catalogue at the book level. At synthesis time, every unit of text passed to the TTS model includes this profile, ensuring the character sounds consistently from beginning to end while per-line captions add expressivity.
What evaluation was conducted on the BookLLM system?
BookLLM was evaluated on 25 English and Spanish books across mystery, sci-fi, and romance genres. With two human annotators providing ground truth, it achieved 97% accuracy in English and 94% in Spanish for identifying which character speaks each line.
How was training data with prosodic descriptions generated at scale?
The team used a human-in-the-loop approach for initial recordings and captioning, then scaled with synthetic data via an expert-model pipeline (extracting pitch, rate, emotion features) and a speech captioner model that generates descriptions from audio, creating an iterative synthetic-data flywheel.
A tradeoff map for synthetic audiobooks: natural-language style control buys character differentiation, but spends captioned data, tokens, and compute. See which lever to pull first and when plain narration is good enough.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.