Efficient Table Pre-training without Real Data: An Introduction to TAPEX

Summary

Conventional language model pre-training relies on massive natural text corpora and general-purpose objectives like masked language modeling, leaving a substantial gap when applied to structured table tasks. To bridge this gap efficiently, TAPEX (Table Pre-training via Execution) pre-trains a language model, such as BART, by learning a neural SQL executor over a synthetic corpus. The system pairs web tables with systematically sampled executable SQL queries and gathers target execution outputs via an off-the-shelf SQL engine. During downstream fine-tuning, the model consumes natural language questions and flattened tables to predict answers. TAPEX achieved new state-of-the-art results across four benchmark datasets, including WikiTableQuestions and TabFact, while achieving a pre-training speedup of nearly 50 times over TaBERT.

Takeaways

1. 1

   TAPEX pre-trains models by learning neural SQL execution over synthetic table, SQL query, and execution output pairs rather than standard language modeling on natural text.

2. 2

   Compared to TaBERT, TAPEX achieved a 2% downstream improvement using only 2% of the pre-training corpus, yielding a speedup of nearly 50 times.

3. 3

   TAPEX achieved new state-of-the-art results on WikiSQL (89.6%), TabFact (84.2%), SQA (74.5%), and WikiTableQuestions (57.5%).