The Definitive Large Language Models Survey: From GPT to LLaMA and Beyond

Why a Large Language Models Survey Matters in 2025

The pace of advancement in LLM research has been extraordinary. Since the release of GPT-3 in 2020 and ChatGPT in late 2022, the field has seen an explosion of new architectures, training methods, and applications. The original survey paper by Zhao et al. has been updated multiple times to keep pace, reaching its 18th version as of early 2026. This large language models survey is essential reading because it provides a systematic framework for understanding the interconnected components of modern LLMs.

The survey covers four fundamental pillars: pre-training, adaptation tuning, utilization, and capacity evaluation. Each pillar represents a critical stage in the lifecycle of an LLM, from initial training on massive corpora to fine-tuning for specific tasks, deploying in real-world applications, and benchmarking performance against standardized tests. Understanding these pillars is essential for anyone working with or building upon large language models.

For organizations evaluating AI strategy, this survey provides crucial context. It helps distinguish genuine capabilities from hype, identifies the most promising research directions, and reveals the practical limitations that still constrain even the most powerful models. As the original paper on arXiv documents, the landscape of LLMs is far more nuanced than popular narratives suggest.

The Evolution of Language Models: From Statistical to Neural to Large-Scale

Understanding the current generation of LLMs requires appreciating the four major development stages that preceded them. Statistical language models (SLMs), developed in the 1990s, built word prediction systems using Markov assumptions — predicting the next word based on recent context. While effective for information retrieval and basic NLP tasks, these n-gram models suffered from the curse of dimensionality.

Neural language models (NLMs) introduced distributed word representations and neural network architectures including multi-layer perceptrons and recurrent neural networks. The concept of word embeddings — dense vector representations that capture semantic relationships — was revolutionary and laid the groundwork for modern approaches.

Pre-trained language models (PLMs) like BERT and GPT-2 demonstrated the power of pre-training transformer architectures on large corpora before fine-tuning for specific tasks. These models showed that general language understanding could be transferred across tasks, dramatically reducing the data requirements for downstream applications.

The transition to large language models occurred when researchers discovered that scaling model parameters beyond a certain threshold — typically into the tens or hundreds of billions — produced qualitatively different behaviors. Models like GPT-3 (175 billion parameters) exhibited in-context learning, the ability to perform tasks from just a few examples provided in the prompt, without any parameter updates. This discovery sparked the current era of LLM research and development.

Key Architectures in the Large Language Models Landscape

The survey identifies three primary architectural families that dominate the LLM landscape, each with distinct strengths and design philosophies:

Decoder-Only Models: GPT Series

The GPT family from OpenAI exemplifies the decoder-only transformer architecture. GPT-3 demonstrated that a sufficiently large decoder-only model trained with autoregressive language modeling could perform remarkably well across diverse tasks. GPT-4 extended this further with multimodal capabilities and significantly improved reasoning. The decoder-only approach has become the dominant paradigm for generative LLMs due to its simplicity and effectiveness at scale.

Open-Source Alternatives: LLaMA and Its Ecosystem

Meta’s LLaMA models challenged the assumption that larger models are always better. LLaMA demonstrated that smaller models trained on more data could match or exceed the performance of much larger models. The release of LLaMA weights catalyzed an entire open-source ecosystem, including Alpaca, Vicuna, and numerous fine-tuned variants. This democratization of LLM technology has been one of the most significant developments in the field.

Google’s Contributions: PaLM and Gemini

PaLM (Pathways Language Model) from Google DeepMind demonstrated the benefits of efficient distributed training using Google’s Pathways system. PaLM showed particularly strong performance on reasoning tasks, especially when combined with chain-of-thought prompting. The subsequent Gemini models extended this work into natively multimodal architectures, processing text, images, audio, and video within a unified framework.

Other notable architectures covered in the survey include Anthropic’s Claude series, which pioneered constitutional AI approaches; Mistral’s efficient models that maximize performance per parameter; and various Chinese LLMs including Qwen, Baichuan, and GLM that demonstrate the global nature of LLM development.

Pre-Training: Building the Foundation of Large Language Models

Pre-training is the most resource-intensive phase of LLM development, and the survey dedicates significant attention to the key decisions involved. The process involves training a model on massive text corpora using self-supervised objectives, primarily next-token prediction for decoder-only models.

Data Collection and Curation

The quality and composition of pre-training data profoundly impacts model capabilities. The survey documents how leading labs curate their training mixtures, typically combining web crawl data (CommonCrawl, C4), books, academic papers, code repositories, and curated knowledge sources like Wikipedia. Critical preprocessing steps include deduplication (removing near-identical documents), quality filtering (removing low-quality web pages), and toxic content removal.

Scaling Laws and Compute Optimization

One of the most important findings documented in the survey is the existence of scaling laws — predictable relationships between model size, dataset size, compute budget, and model performance. The Chinchilla scaling laws from DeepMind showed that many early LLMs were undertrained: given a fixed compute budget, optimal performance comes from training a smaller model on more data than previously assumed. This insight directly influenced the design of LLaMA and subsequent efficient models.

Training Infrastructure and Techniques

Training LLMs at scale requires sophisticated distributed computing strategies. The survey covers data parallelism, tensor parallelism, pipeline parallelism, and expert parallelism (for mixture-of-experts models). Mixed-precision training using bfloat16 or float16 formats reduces memory requirements while maintaining training stability. Techniques like gradient checkpointing, Flash Attention, and ZeRO optimization further improve efficiency.

Emergent Abilities: When Scale Creates Qualitative Change

Perhaps the most fascinating aspect of the large language models survey is its analysis of emergent abilities — capabilities that appear suddenly as models scale beyond certain thresholds. These abilities are not predictable from the performance of smaller models and represent genuine qualitative shifts in what the models can do.

In-Context Learning

In-context learning (ICL) allows LLMs to perform tasks by conditioning on a few demonstration examples provided in the prompt, without any gradient updates. First documented in GPT-3, this ability enables rapid adaptation to new tasks at inference time. The survey discusses various theories for why ICL emerges, including implicit Bayesian inference and the formation of task-specific computational circuits within the model.

Chain-of-Thought Reasoning

Chain-of-thought (CoT) prompting enables LLMs to solve complex reasoning problems by generating intermediate reasoning steps before arriving at a final answer. This technique has proven particularly effective for mathematical, logical, and multi-step problems. The survey documents how CoT capabilities improve dramatically with model scale and how various extensions — including self-consistency, tree-of-thought, and automatic CoT — have further enhanced reasoning performance.

Instruction Following

Larger models demonstrate a markedly improved ability to follow natural language instructions describing a task, even without explicit examples. This capability forms the foundation of the chat-style interaction paradigm that has made LLMs accessible to general users through products like ChatGPT and Claude. The ability to understand and execute complex, multi-part instructions emerges reliably at scale and improves with instruction tuning.

Understanding these emergent abilities is crucial for organizations evaluating AI capabilities. For more on how AI capabilities map to real-world applications, see our McKinsey State of AI 2024 analysis.

Adaptation Tuning: From General to Specialized Models

Once a base model is pre-trained, adaptation tuning refines it for specific applications. The survey covers two primary approaches: instruction tuning and alignment tuning, each serving distinct purposes in making LLMs more useful and safe.

Instruction Tuning

Instruction tuning involves fine-tuning a pre-trained model on a collection of tasks formatted as natural language instructions paired with desired outputs. Datasets like FLAN, T0, and InstructGPT’s training data have proven effective at improving zero-shot generalization. The survey documents how instruction-tuned models consistently outperform their base counterparts on held-out tasks, demonstrating genuine capability transfer.

Alignment Tuning and RLHF

Alignment tuning addresses a more fundamental challenge: ensuring that model outputs align with human values and intentions. The dominant approach, Reinforcement Learning from Human Feedback (RLHF), involves three stages: collecting human preference data, training a reward model, and optimizing the LLM using reinforcement learning (typically Proximal Policy Optimization). The survey thoroughly documents the RLHF pipeline used by OpenAI for InstructGPT and ChatGPT, as well as alternatives like Direct Preference Optimization (DPO) that simplify the training process.

Anthropic’s constitutional AI approach represents another important alignment strategy, where models are trained to self-critique and revise their outputs according to a set of principles. For a deeper exploration of AI alignment methodologies, visit our AI alignment taxonomy guide.

Parameter-Efficient Fine-Tuning

Given the enormous cost of full fine-tuning, the survey covers parameter-efficient methods including LoRA (Low-Rank Adaptation), prefix tuning, prompt tuning, and adapter layers. LoRA has become particularly popular, enabling task-specific adaptation by training only low-rank update matrices while keeping the base model frozen. This approach reduces training costs by orders of magnitude while preserving most of the performance gains from full fine-tuning.

Utilization: Prompting Strategies and Practical Deployment

The survey’s coverage of LLM utilization is particularly relevant for practitioners. It systematically categorizes prompting strategies and deployment techniques that maximize model performance in real-world applications.

Prompting Fundamentals

Effective prompting is both an art and a science. The survey categorizes prompting into zero-shot (task description only), few-shot (task description plus examples), and chain-of-thought (reasoning steps included) approaches. Each has distinct strengths: zero-shot is most convenient, few-shot provides more reliable formatting, and CoT enables complex reasoning. The survey also discusses prompt engineering best practices including role-playing prompts, structured output formatting, and multi-turn conversation design.

Advanced Techniques: Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) combines LLMs with external knowledge retrieval systems, addressing key limitations including knowledge cutoffs and hallucination. The survey documents how RAG systems retrieve relevant passages from a knowledge base and incorporate them into the model’s context, enabling more factual and up-to-date responses. This approach has become standard practice for enterprise LLM deployments where accuracy is critical.

Tool Use and Agentic Systems

An increasingly important utilization pattern is enabling LLMs to use external tools — calculators, search engines, code interpreters, and APIs. The survey covers how models like GPT-4 and Claude have been trained to recognize when tool use would be beneficial and to format appropriate tool calls. This capability forms the foundation of AI agent systems that can plan and execute multi-step tasks autonomously, a trend that is reshaping how organizations think about automation.

Capacity Evaluation: Benchmarking Large Language Models

Rigorous evaluation is essential for understanding and comparing LLM capabilities. The survey provides a comprehensive overview of evaluation methodologies and benchmarks used to assess different aspects of model performance.

Standard Benchmarks

Key benchmarks discussed include MMLU (massive multitask language understanding), covering 57 academic subjects; HellaSwag and ARC for commonsense and scientific reasoning; HumanEval and MBPP for code generation; GSM8K and MATH for mathematical reasoning; and TruthfulQA for factual accuracy. The survey notes the rapid pace of benchmark saturation — as LLMs improve, benchmarks that were once challenging become insufficient for differentiating model capabilities.

Evaluation Challenges

The survey highlights several persistent evaluation challenges. Data contamination — where benchmark test sets inadvertently appear in training data — threatens the validity of benchmark results. The sensitivity of results to prompt formatting means that small changes in evaluation prompts can produce significantly different scores. Additionally, many important capabilities like creativity, nuance, and ethical reasoning are difficult to capture through automated benchmarks, requiring human evaluation at scale.

Beyond Benchmarks: Real-World Evaluation

Increasingly, the community recognizes that benchmark performance does not always correlate with real-world usefulness. The survey discusses emerging evaluation frameworks that focus on practical deployment scenarios, including arena-style comparisons (like LMSYS Chatbot Arena) where models are evaluated through direct human preference on open-ended tasks. These complementary approaches provide a more holistic picture of model capabilities. For a broader perspective on technology trends driving these developments, see our CB Insights Tech Trends 2025 analysis.

Safety, Ethics, and Responsible AI Development

The survey dedicates significant attention to the safety and ethical considerations surrounding LLM development and deployment. As these models become more capable and widely deployed, understanding and mitigating their risks becomes increasingly critical.

Hallucination and Factual Accuracy

LLMs can generate plausible-sounding but factually incorrect content — a phenomenon known as hallucination. The survey documents various approaches to reducing hallucination, including RLHF training that rewards accuracy, retrieval augmentation that grounds responses in verified sources, and self-consistency methods that check for internal contradictions. Despite progress, hallucination remains one of the most significant challenges for deploying LLMs in high-stakes applications.

Bias and Fairness

LLMs inherit and can amplify biases present in their training data. The survey discusses how these biases manifest across demographic dimensions including gender, race, religion, and nationality. Mitigation strategies include data curation to reduce biased content, debiasing during fine-tuning, and post-processing techniques that filter or adjust model outputs. The survey emphasizes that bias mitigation is an ongoing process rather than a one-time fix.

Environmental Impact

Training large language models requires enormous computational resources, with corresponding energy consumption and carbon emissions. The survey documents estimates of training costs for major models and discusses techniques for reducing environmental impact, including more efficient architectures, improved training algorithms, and the use of renewable energy sources in data centers. The development of smaller, more efficient models like LLaMA and Mistral represents a promising trend toward reducing the environmental footprint of frontier AI.

The Open-Source LLM Revolution

One of the most significant developments documented in the survey is the rise of open-source LLMs. Meta’s release of LLaMA weights in early 2023 catalyzed an explosion of open-source development that has fundamentally altered the competitive landscape of AI.

The open-source ecosystem now includes models competitive with proprietary offerings across many tasks. LLaMA 2 and 3, Mistral, Falcon, and Qwen have demonstrated that open development can produce models rivaling closed-source alternatives, particularly for specialized applications. The proliferation of fine-tuned variants — Alpaca, Vicuna, WizardLM, and hundreds of others — shows how community-driven development accelerates innovation.

This democratization has profound implications for the AI industry. Organizations can now deploy capable LLMs on their own infrastructure, maintaining data privacy and control. Researchers can study model internals, probe for biases, and develop new techniques without being constrained by API access limitations. The survey argues that this openness is essential for building trust in AI systems and ensuring that the benefits of LLM technology are broadly distributed.

However, open-source LLMs also raise unique safety challenges. When model weights are publicly available, it becomes impossible to prevent misuse through access controls alone. The survey discusses the ongoing debate about the appropriate balance between openness and safety, noting that the research community is still working to develop norms and technical safeguards that enable beneficial open-source development while mitigating risks.

Future Directions and Open Research Questions

The survey concludes with a thoughtful analysis of the most important open research questions facing the LLM community. These include:

• Multimodal integration: How can LLMs be extended to natively process and generate across modalities — text, images, audio, video, and structured data — with unified architectures rather than bolted-on components?
• Long-context understanding: Current models struggle with very long documents despite increasing context windows. Research into efficient attention mechanisms, hierarchical processing, and retrieval-augmented approaches continues to push the boundaries of what LLMs can comprehend.
• Reasoning and planning: While chain-of-thought prompting has improved reasoning, LLMs still fall short on complex logical reasoning, mathematical proofs, and multi-step planning. New architectures and training strategies are needed to close this gap.
• Efficiency and accessibility: Reducing the computational requirements of LLMs through quantization, distillation, pruning, and architectural innovations remains critical for broader accessibility and environmental sustainability.
• Trustworthiness and reliability: Building LLMs that know what they don’t know, that can express calibrated uncertainty, and that reliably refuse harmful requests without being overly cautious is an active area of research.
• Agentic capabilities: The evolution from passive question-answering to autonomous AI agents that can plan, use tools, and execute multi-step tasks represents the next frontier for LLM applications.