T5

T5, or the Text-to-Text Transfer Transformer, is a large language model developed by Google Research that treats every natural language processing task as a text-to-text problem. Instead of needing separate model architectures or output heads for classification, translation, summarization, or question answering, T5 converts all tasks into the same format: input text goes in, and output text comes out.

The Text-to-Text Framework

The key insight of T5 is its unified framework. For example, translation tasks become "translate from English to French: ..." followed by the source text. Classification tasks become "sentiment: this movie is fantastic" with the model outputting the label string. This simplicity allowed the researchers to apply the same architecture, loss function, and hyperparameters across a wide range of tasks.

Model Architecture and Training

T5 uses a standard Transformer encoder-decoder architecture, with improvements like relative position embeddings. The model was pre-trained on the Colossal Clean Crawled Corpus (C4), a massive curated dataset of web text, using a denoising objective where spans of text are masked and the model learns to predict them. After pre-training, T5 is fine-tuned on specific tasks with task-specific text prefixes.

Impact and Influence

T5 achieved state-of-the-art results on many benchmarks, including GLUE, SuperGLUE, and SQuAD, and its open-source release made it a popular foundation for research and applications. Its text-to-text abstraction influenced later models like InstructGPT and prompted the idea that language models can be unified through a simple input-output interface.

Why T5 Matters

T5 demonstrated that a single model architecture could handle virtually any NLP task with only minor changes to the input format. This shifted the field toward more unified and transferable approaches, paving the way for the rise of instruction-tuned and large-scale generative models.