Efficient Table Pre-training without Real Data: An Introduction to TAPEX

Summary

Conventional language model pre-training relies on massive natural text corpora and general-purpose objectives like masked language modeling, leaving a substantial gap when applied to structured table tasks. To bridge this gap efficiently, TAPEX (Table Pre-training via Execution) pre-trains a language model, such as BART, by learning a neural SQL executor over a synthetic corpus. The system pairs web tables with systematically sampled executable SQL queries and gathers target execution outputs via an off-the-shelf SQL engine. During downstream fine-tuning, the model consumes natural language questions and flattened tables to predict answers. TAPEX achieved new state-of-the-art results across four benchmark datasets, including WikiTableQuestions and TabFact, while achieving a pre-training speedup of nearly 50 times over TaBERT.

Context

A dramatic gap exists between general-purpose pre-training objectives like language modeling and downstream structured table tasks such as table question answering. Obtaining performance gains with standard domain-adaptive pre-training historically requires extremely large volumes of pre-training data, limiting efficiency.

Approach / What changed

TAPEX synthesizes a non-natural pre-training corpus by taking tables from the web and systematically sampling executable SQL queries, then running them through an off-the-shelf SQL executor such as MySQL to obtain execution outputs. A sequence-to-sequence language model (such as BART) receives the concatenation of the SQL query and flattened table as encoder input, supervised by the execution result at the decoder to mimic a neural SQL executor. During fine-tuning, natural language questions replace SQL queries to generate labeled answers.

Takeaways

• TAPEX pre-trains models by learning neural SQL execution over synthetic table, SQL query, and execution output pairs rather than standard language modeling on natural text.
• Compared to TaBERT, TAPEX achieved a 2% downstream improvement using only 2% of the pre-training corpus, yielding a speedup of nearly 50 times.
• TAPEX achieved new state-of-the-art results on WikiSQL (89.6%), TabFact (84.2%), SQA (74.5%), and WikiTableQuestions (57.5%).

Related reading

• huggingface.co · Jul 3, 2020

  The Reformer - Pushing the limits of language modeling

  Standard transformer models hit memory bottlenecks on long sequence modeling tasks due to the quadratic asymptotic memory complexity of global self-attention and oversized positional embedding matrices. The Reformer architecture overcomes these constraints to train sequences of up to half a million tokens using under 8GB of RAM. It re-engineers transformer operations using local and Locality Sensitive Hashing self-attention, chunked feed forward layers, reversible residual layers, and axial positional encodings. In empirical benchmarks using google/reformer-crime-and-punishment, axial positional encodings reduce the model parameter count from over 136 million to approximately 2.58 million by factorizing the positional dimensions. This architectural change cuts inference memory consumption from 959 MB down to 447 MB for evaluated benchmark workloads.

  Patrick von Platen

• huggingface.co · Mar 31, 2021

  Understanding BigBird's Block Sparse Attention

  Transformer-based models face severe computational bottlenecks due to quadratic time and memory complexity, making sequences longer than 512 tokens impractical. BigBird addresses this limitation by approximating full attention with block sparse attention, scaling context lengths up to 4096 tokens at a lower computational cost. Instead of attending to all tokens, BigBird combines sliding attention for local context, global tokens for long-range relationships, and random tokens to accelerate information transfer. The model is integrated into HuggingFace Transformers, enabling fine-tuning for tasks like long-document summarization and extractive question answering. When sequences are shorter than 1024 tokens, using original full attention remains recommended.