Efficient Table Pre-training without Real Data: An Introduction to TAPEX
📰 Hugging Face Blog
TAPEX introduces a new approach to table pre-training using synthetic data and a neural SQL executor, improving efficiency and reducing the need for large amounts of real data
Action Steps
- Understand the concept of table pre-training and its challenges
- Learn about the TAPEX approach and its use of synthetic data and a neural SQL executor
- Explore the application of TAPEX in improving the efficiency of table pre-training and its potential impact on downstream tasks
Who Needs to Know This
This article is relevant to AI engineers, data scientists, and researchers working on natural language processing and table question answering tasks, as it provides a new approach to pre-training and fine-tuning language models
Key Insight
💡 TAPEX uses synthetic data and a neural SQL executor to improve the efficiency of table pre-training, reducing the need for large amounts of real data
Share This
💡 Introducing TAPEX: a new approach to table pre-training using synthetic data and a neural SQL executor #AI #NLP #TablePretraining
Key Takeaways
TAPEX introduces a new approach to table pre-training using synthetic data and a neural SQL executor, improving efficiency and reducing the need for large amounts of real data
Full Article
Published Time: 2022-05-23T00:00:00.087Z
# Efficient Table Pre-training without Real Data: An Introduction to TAPEX
[Hugging Face](https://huggingface.co/)
* [Models](https://huggingface.co/models)
* [Datasets](https://huggingface.co/datasets)
* [Spaces](https://huggingface.co/spaces)
* [Buckets new](https://huggingface.co/storage)
* [Docs](https://huggingface.co/docs)
* [Enterprise](https://huggingface.co/enterprise)
* [Pricing](https://huggingface.co/pricing)
*
*
* * *
* [Log In](https://huggingface.co/login)
* [Sign Up](https://huggingface.co/join)
[Back to Articles](https://huggingface.co/blog)
# [](https://huggingface.co/blog/tapex#efficient-table-pre-training-without-real-data-an-introduction-to-tapex) Efficient Table Pre-training without Real Data: An Introduction to TAPEX
Published May 23, 2022
[Update on GitHub](https://github.com/huggingface/blog/blob/main/tapex.md)
[- [x] Upvote 1](https://huggingface.co/login?next=%2Fblog%2Ftapex)
* [](https://huggingface.co/SivilTaram "SivilTaram")
[](https://huggingface.co/SivilTaram)
[Qian Liu SivilTaram Follow](https://huggingface.co/SivilTaram)
guest
* [Overview](https://huggingface.co/blog/tapex#overview "Overview")
* [Pre-training](https://huggingface.co/blog/tapex#pre-training "Pre-training")
* [Fine-tuning](https://huggingface.co/blog/tapex#fine-tuning "Fine-tuning")
* [Experiments](https://huggingface.co/blog/tapex#experiments "Experiments")
* [Comparison to Previous Table Pre-training](https://huggingface.co/blog/tapex#comparison-to-previous-table-pre-training "Comparison to Previous Table Pre-training")
* [Conclusion](https://huggingface.co/blog/tapex#conclusion "Conclusion")
* [Take Away](https://huggingface.co/blog/tapex#take-away "Take Away")
In recent years, language model pre-training has achieved great success via leveraging large-scale textual data. By employing pre-training tasks such as [masked language modeling](https://arxiv.org/abs/1810.04805), these models have demonstrated surprising performance on several downstream tasks. However, the dramatic gap between the pre-training task (e.g., language modeling) and the downstream task (e.g., table question answering) makes existing pre-training not efficient enough. In practice, we often need an _extremely large amount_ of pre-training data to obtain promising improvement, even for [domain-adaptive pretraining](https://arxiv.org/abs/2004.02349). How might we design a pre-training task to close the gap, and thus accelerate pre-training?
### [](https://huggingface.co/blog/tapex#overview) Overview
In "[TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://openreview.net/forum?id=O50443AsCP)", we explore **using synthetic data as a proxy for real data during pre-training**, and demonstrate its powerfulness with _TAPEX (Table Pre-training via Execution)_ as an example. In TAPEX, we show that table pre-training can be achieved by learning a neural SQL executor over a synthetic corpus.
[](https://huggingface.co/blog/assets/74_tapex/tapex-overview.png)
> Note: [Table] is a placeholder for the user provided table in the input.
As shown in the figure above, by systematically sampling _executable SQL queries and their execution outputs_ over tables, TAPEX first synthesizes a synthetic and non-natural pre-training corpus. Then, it continues to pre-train a language model (e.g., [BART](https://arxiv.org/abs/1910.13461)) to output the execution results of SQL queries, which mimics the process of a neural SQL executor
# Efficient Table Pre-training without Real Data: An Introduction to TAPEX
[Hugging Face](https://huggingface.co/)
* [Models](https://huggingface.co/models)
* [Datasets](https://huggingface.co/datasets)
* [Spaces](https://huggingface.co/spaces)
* [Buckets new](https://huggingface.co/storage)
* [Docs](https://huggingface.co/docs)
* [Enterprise](https://huggingface.co/enterprise)
* [Pricing](https://huggingface.co/pricing)
*
*
* * *
* [Log In](https://huggingface.co/login)
* [Sign Up](https://huggingface.co/join)
[Back to Articles](https://huggingface.co/blog)
# [](https://huggingface.co/blog/tapex#efficient-table-pre-training-without-real-data-an-introduction-to-tapex) Efficient Table Pre-training without Real Data: An Introduction to TAPEX
Published May 23, 2022
[Update on GitHub](https://github.com/huggingface/blog/blob/main/tapex.md)
[- [x] Upvote 1](https://huggingface.co/login?next=%2Fblog%2Ftapex)
* [](https://huggingface.co/SivilTaram "SivilTaram")
[](https://huggingface.co/SivilTaram)
[Qian Liu SivilTaram Follow](https://huggingface.co/SivilTaram)
guest
* [Overview](https://huggingface.co/blog/tapex#overview "Overview")
* [Pre-training](https://huggingface.co/blog/tapex#pre-training "Pre-training")
* [Fine-tuning](https://huggingface.co/blog/tapex#fine-tuning "Fine-tuning")
* [Experiments](https://huggingface.co/blog/tapex#experiments "Experiments")
* [Comparison to Previous Table Pre-training](https://huggingface.co/blog/tapex#comparison-to-previous-table-pre-training "Comparison to Previous Table Pre-training")
* [Conclusion](https://huggingface.co/blog/tapex#conclusion "Conclusion")
* [Take Away](https://huggingface.co/blog/tapex#take-away "Take Away")
In recent years, language model pre-training has achieved great success via leveraging large-scale textual data. By employing pre-training tasks such as [masked language modeling](https://arxiv.org/abs/1810.04805), these models have demonstrated surprising performance on several downstream tasks. However, the dramatic gap between the pre-training task (e.g., language modeling) and the downstream task (e.g., table question answering) makes existing pre-training not efficient enough. In practice, we often need an _extremely large amount_ of pre-training data to obtain promising improvement, even for [domain-adaptive pretraining](https://arxiv.org/abs/2004.02349). How might we design a pre-training task to close the gap, and thus accelerate pre-training?
### [](https://huggingface.co/blog/tapex#overview) Overview
In "[TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://openreview.net/forum?id=O50443AsCP)", we explore **using synthetic data as a proxy for real data during pre-training**, and demonstrate its powerfulness with _TAPEX (Table Pre-training via Execution)_ as an example. In TAPEX, we show that table pre-training can be achieved by learning a neural SQL executor over a synthetic corpus.
[](https://huggingface.co/blog/assets/74_tapex/tapex-overview.png)
> Note: [Table] is a placeholder for the user provided table in the input.
As shown in the figure above, by systematically sampling _executable SQL queries and their execution outputs_ over tables, TAPEX first synthesizes a synthetic and non-natural pre-training corpus. Then, it continues to pre-train a language model (e.g., [BART](https://arxiv.org/abs/1910.13461)) to output the execution results of SQL queries, which mimics the process of a neural SQL executor
DeepCamp AI