# Efficient Table Pre-training without Real Data: An Introduction to TAPEX

huggingface.co · Qian Liu · May 23, 2022

**Type:** Problem & solution

## Summary

Conventional language model pre-training relies on massive natural text corpora and general-purpose objectives like masked language modeling, leaving a substantial gap when applied to structured table tasks. To bridge this gap efficiently, TAPEX (Table Pre-training via Execution) pre-trains a language model, such as BART, by learning a neural SQL executor over a synthetic corpus. The system pairs web tables with systematically sampled executable SQL queries and gathers target execution outputs via an off-the-shelf SQL engine. During downstream fine-tuning, the model consumes natural language questions and flattened tables to predict answers. TAPEX achieved new state-of-the-art results across four benchmark datasets, including WikiTableQuestions and TabFact, while achieving a pre-training speedup of nearly 50 times over TaBERT.

## Context

A dramatic gap exists between general-purpose pre-training objectives like language modeling and downstream structured table tasks such as table question answering. Obtaining performance gains with standard domain-adaptive pre-training historically requires extremely large volumes of pre-training data, limiting efficiency.

## Approach / What changed

TAPEX synthesizes a non-natural pre-training corpus by taking tables from the web and systematically sampling executable SQL queries, then running them through an off-the-shelf SQL executor such as MySQL to obtain execution outputs. A sequence-to-sequence language model (such as BART) receives the concatenation of the SQL query and flattened table as encoder input, supervised by the execution result at the decoder to mimic a neural SQL executor. During fine-tuning, natural language questions replace SQL queries to generate labeled answers.

## Takeaways

- TAPEX pre-trains models by learning neural SQL execution over synthetic table, SQL query, and execution output pairs rather than standard language modeling on natural text.
- Compared to TaBERT, TAPEX achieved a 2% downstream improvement using only 2% of the pre-training corpus, yielding a speedup of nearly 50 times.
- TAPEX achieved new state-of-the-art results on WikiSQL (89.6%), TabFact (84.2%), SQA (74.5%), and WikiTableQuestions (57.5%).

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/tapex)