Embodied AI: How Multimodal LLMs and World Models Are Building Intelligent Robots

What Is Embodied AI and Why It Matters for Robotics

Embodied AI represents a fundamental shift in artificial intelligence research — from systems that process information in disembodied digital environments to agents that perceive, reason, and act within the physical world. Published by researchers at Tsinghua University, this comprehensive survey maps the landscape of embodied AI from its cognitive science foundations to the latest architectures combining multimodal large language models (MLLMs) and world models (WMs) for autonomous robotic behaviour.

The significance of embodied AI lies in its potential to bridge the gap between the remarkable language and reasoning capabilities demonstrated by models like GPT-4, Gemini, and LLaMA, and the physical dexterity and situational awareness required for real-world robotic tasks. While an LLM can describe how to make coffee, an embodied AI system must actually navigate a kitchen, identify objects, manipulate tools, and handle unexpected situations — all in real-time. This integration of perception, cognition, and action defines the frontier of AI research today.

The survey identifies two key paradigms driving progress: multimodal LLMs that bring semantic understanding and high-level planning, and world models that provide physics-grounded prediction and safe exploration. Understanding how these paradigms converge is essential for anyone working in robotics, AI research, or enterprise automation. For a broader perspective on AI governance frameworks shaping this field, see our interactive guide on the Chatham House AI Global Governance Framework.

Embodied AI Foundations: Active Perception, Cognition, and Interaction

The survey decomposes embodied AI into three core subsystems, each addressing a fundamental challenge of operating in the physical world. Active perception encompasses the sensory capabilities that allow an agent to understand its environment, including visual SLAM (Simultaneous Localisation and Mapping) systems like ORB-SLAM and LSD-SLAM, 3D scene understanding models such as OpenScene and Lexicon3D, and active exploration strategies that direct attention to task-relevant information.

Embodied cognition — the reasoning layer — handles task planning, memory management, and decision-making. This is where the revolution in large language models has had the most dramatic impact. Chain-of-thought prompting, episodic memory, and self-reflection mechanisms (as seen in systems like Reflexion and AutoAct) enable agents to decompose complex instructions into executable sub-goals, learn from mistakes, and adapt strategies over time. The integration of foundation models with cognitive architectures represents a qualitative leap in what embodied agents can reason about.

Dynamic interaction covers the execution side: action control policies that translate plans into physical movements, behaviour modelling for predicting and responding to environmental changes, and multi-agent collaboration frameworks for scenarios requiring coordination between multiple robots. Reinforcement learning algorithms — from DQN and PPO to SAC and RLHF — provide the training paradigms for these interaction modules, while emerging methods like GRPO (Group Relative Policy Optimisation) offer improved sample efficiency.

The key insight from the survey is that these three subsystems are deeply interdependent. Active perception must be guided by cognitive goals (you look where you need to look), cognition must be grounded in perceptual reality (you can only plan with what you see), and interaction must be constrained by both perception and cognition (you can only do what’s physically possible and strategically sound). This tight coupling is what makes embodied AI fundamentally different from — and harder than — solving any subsystem in isolation.

Multimodal LLMs in Embodied AI: Semantic Reasoning for Robots

The integration of multimodal LLMs into embodied systems represents one of the most transformative developments in robotics. Models like PaLM-E (a 562-billion parameter vision-language model from Google), RT-2 (Robotic Transformer 2), and EmbodiedGPT demonstrate that large pre-trained models can serve as effective “brains” for robotic systems, providing instruction understanding, visual reasoning, and action generation in unified architectures.

The survey categorises MLLM contributions to embodied AI along three dimensions. First, semantic grounding: MLLMs can interpret natural language instructions (“pick up the red cup next to the laptop”) and ground them in visual observations, identifying relevant objects and spatial relationships. This eliminates the need for hand-coded object detectors and rule-based planners that limited previous generations of robotic systems.

Second, task decomposition: complex multi-step tasks can be broken down into executable sub-tasks through chain-of-thought reasoning. Systems like SayCan combine LLM-generated task plans with learned affordance functions that assess which actions are physically feasible in the current state, preventing the generation of plans that look linguistically reasonable but are physically impossible (such as “move the table through the doorway” when the table is too wide).

Third, cross-embodiment transfer: large pre-trained models can potentially generalise across different robot morphologies and environments. The Octo and OpenVLA models demonstrate that training on diverse robotic datasets enables zero-shot or few-shot transfer to new robots, reducing the engineering burden of deploying AI across different hardware platforms. This cross-embodiment capability could dramatically accelerate the deployment of capable robots across industries.

World Models for Embodied AI: Physics-Aware Prediction and Planning

While MLLMs excel at semantic reasoning, they fundamentally lack an understanding of physical dynamics. A language model may know that “dropping a glass causes it to break,” but it cannot predict the trajectory, impact forces, or shattering pattern. This is where world models — neural networks that learn to simulate environment dynamics — become essential. The survey traces the evolution of world models from the Recurrent State Space Model (RSSM) in the Dreamer family through to modern architectures including JEPA (Joint Embedding Predictive Architecture), diffusion-based world models, and transformer-based variants.

DreamerV3 represents a milestone in world model research, demonstrating that a single general-purpose algorithm can achieve human-level performance across diverse domains — from Atari games to continuous control tasks — by learning compact latent representations of environment dynamics. The model learns to “imagine” future trajectories in its learned latent space, enabling planning without requiring actual physical interaction. This imagination-based planning is transformative for robotics, where real-world trial-and-error is expensive, slow, and potentially dangerous.

JEPA, championed by Yann LeCun at Meta AI, proposes a different approach: learning to predict abstract representations of future states rather than pixel-level predictions. This abstraction makes the model more robust to irrelevant visual details and better at capturing the causal structure of the world. The survey identifies JEPA as a particularly promising direction for embodied AI because it naturally handles the multi-scale, multi-modal nature of physical interaction.

The emergence of video generation models like Sora as implicit world models represents an exciting frontier. These models demonstrate an understanding of 3D geometry, object permanence, and physical interactions that emerges purely from training on internet-scale video data. Whether this understanding is deep enough for safe robotic control remains an open question, but the potential for leveraging vast video datasets to bootstrap world understanding for robots is significant.

Integrating LLMs and World Models: Joint Architectures for Autonomy

The central thesis of the survey is that the most promising path to robust embodied AI lies in the integration of MLLMs and world models into joint architectures. MLLMs provide the “what” and “why” — understanding goals, decomposing tasks, and reasoning about abstract relationships — while world models provide the “how” and “what if” — predicting physical consequences, simulating action outcomes, and ensuring plans are physically feasible.

The proposed joint MLLM-WM architecture addresses several fundamental limitations of each approach used in isolation. MLLMs alone generate linguistically plausible but physically impossible plans. World models alone lack the semantic understanding to interpret high-level goals or generalise to novel tasks. Together, the MLLM generates candidate plans, the world model simulates their outcomes, and the system iteratively refines plans that are both semantically appropriate and physically grounded.

The survey highlights several concrete implementations of this joint paradigm. EvoAgent uses an evolutionary approach where an MLLM generates diverse action candidates and a world model evaluates their predicted outcomes. Other systems use the world model as an “inner simulator” that the MLLM can query during planning, creating a feedback loop between symbolic reasoning and physical prediction.

The data flow in these joint architectures is bidirectional: the MLLM informs the world model about task-relevant features and attention priorities, while the world model grounds the MLLM’s reasoning in physical constraints and predicted outcomes. This bidirectional flow mirrors how human cognition integrates linguistic/conceptual reasoning with embodied physical intuition — a connection to cognitive science that the survey makes explicit.

Vision-Language-Action Models: Bridging Perception and Control

Vision-Language-Action (VLA) models represent a specific architectural paradigm within embodied AI that aims to create end-to-end systems mapping visual observations and language instructions directly to motor actions. This approach eliminates the traditional separation between perception, planning, and control modules, instead training a single neural network to handle the entire pipeline.

Google’s RT-2 demonstrated that a large vision-language model fine-tuned on robotic data can directly output motor commands as text tokens, effectively treating robot control as a language generation problem. This elegant formulation leverages the vast knowledge encoded in pre-trained VLMs and transfers it to robotic manipulation. Similarly, PaLM-E showed that incorporating robotic sensor data directly into a large language model’s input stream enables grounded reasoning about physical manipulation tasks.

The survey also covers more recent developments like CoT-VLA (Chain-of-Thought VLA), which adds explicit reasoning steps between observation and action, and PerAct, which combines perceiver-based architectures with 3D voxel representations for precise manipulation. These models achieve increasingly impressive results on benchmarks but face challenges in generalisation to novel objects, environments, and tasks outside their training distribution. For related insights into how AI models handle complex language tasks, see our analysis of Carnegie’s AI Governance Regime Complex.

Embodied AI Hardware: Accelerators, Compilers, and Edge Deployment

A critical but often overlooked dimension of embodied AI is the hardware challenge. Running large multimodal models on resource-constrained robotic platforms requires significant advances in model compression, domain-specific accelerators, and hardware-software co-design. The survey dedicates substantial attention to this practical dimension, recognising that algorithmic breakthroughs are meaningless without deployable systems.

Model compression techniques — including quantization (reducing numerical precision from 32-bit to 8-bit or 4-bit), pruning (removing redundant parameters), operator fusion, and loop tiling — can reduce model size and inference latency by orders of magnitude. The challenge is maintaining sufficient accuracy for safety-critical robotic applications while achieving the sub-10ms inference times required for real-time control. The survey notes that different compression techniques have different accuracy-latency trade-offs, and optimal combinations depend on the specific model architecture and deployment hardware.

Domain-specific accelerators, from Google’s TPUs to FPGAs and custom ASICs, offer significant performance improvements for the specific computation patterns found in neural network inference. The survey highlights the emerging field of Coarse-Grained Reconfigurable Arrays (CGRAs) as particularly promising for robotic applications, offering a balance between the flexibility of FPGAs and the efficiency of ASICs. Compiler toolchains like TVM bridge the gap between high-level model definitions and optimised hardware execution.

The hardware-software co-design perspective is essential: the survey argues that embodied AI systems should be designed holistically, with algorithmic architectures informed by hardware constraints and hardware designs optimised for target algorithmic patterns. This co-design approach could enable a new generation of robot-optimised AI accelerators that balance the diverse computational needs of perception, reasoning, and control.

Benchmarks and Sim-to-Real Evaluation for Embodied AI Systems

Rigorous evaluation is critical for advancing embodied AI, and the survey provides a comprehensive overview of the benchmarking landscape. Major simulation platforms include Habitat (for indoor navigation and manipulation), ManiSkill (for dexterous manipulation), MuJoCo (for physics-based motor control), AirSim (for aerial robotics), and MineDojo (for open-world exploration). Each platform emphasises different aspects of embodied intelligence, from low-level motor control to high-level strategic planning.

Dedicated benchmarks have emerged to evaluate specific capabilities. EmbodiedBench tests multi-modal understanding and task execution across diverse scenarios. MuEP evaluates multi-step embodied planning. ECBench focuses on embodied conversation abilities. BEHAVIOR-1K provides 1,000 household activities for comprehensive behavioural evaluation. These benchmarks are essential for measuring progress, but the survey cautions that benchmark performance does not always translate to real-world capability.

The sim-to-real gap remains the central evaluation challenge. Policies that achieve impressive performance in simulation frequently fail when transferred to physical robots due to discrepancies in physics simulation, sensor characteristics, and environmental complexity. Domain randomization — training across many randomized simulation parameters — remains the most common mitigation strategy, but the survey identifies world models trained on real-world data as a more fundamental solution, capable of learning the actual physics of target environments rather than approximating them through randomised simulation.

The Open X-Embodiment initiative, aggregating robotic experience data from over 20 institutions, represents an important step toward creating the large-scale, diverse datasets needed for robust generalisation. The survey emphasises that data scale and diversity — not just algorithmic sophistication — may be the binding constraint on embodied AI progress.

Embodied AI Applications: Service Robots, UAVs, and Industrial Automation

The survey identifies several application domains where embodied AI is moving from research to deployment. Service robotics — including household assistants, hospitality robots, and logistics systems — represents the largest near-term market opportunity. The integration of MLLMs enables these robots to understand natural language instructions from non-technical users, dramatically expanding the addressable use case space beyond pre-programmed tasks.

Rescue and inspection UAVs (unmanned aerial vehicles) represent another high-value application. Embodied AI enables autonomous navigation through damaged buildings, identification of victims, structural assessment, and coordination between multiple drones — tasks that are dangerous for human rescuers and require real-time perception and decision-making in unstructured environments. The world model paradigm is particularly valuable here, allowing UAVs to predict structural stability and plan safe navigation paths.

Industrial robotics is being transformed by embodied AI’s ability to handle variability. Traditional industrial robots require precisely controlled environments and pre-programmed motions. Embodied AI systems can adapt to variations in part placement, lighting conditions, and manufacturing tolerances, enabling flexible manufacturing and small-batch production. The economic implications are significant: according to the International Federation of Robotics, global robot installations continue to set records, driven increasingly by AI-enabled flexible automation. Our interactive analysis of Accenture’s Generative AI Enterprise Transformation explores how enterprises are deploying AI across operations.

Education, virtual environments, and space exploration round out the application landscape. Embodied AI tutors that can physically demonstrate concepts, virtual agents that interact naturally in immersive environments, and autonomous space robots that must operate with significant communication delays all benefit from the joint MLLM-WM architectures described in the survey.

Challenges and Future Directions for Embodied AI Research

Despite remarkable progress, the survey identifies several fundamental challenges that must be addressed before embodied AI achieves widespread deployment. The latency challenge is perhaps the most immediate: high-level MLLM reasoning requires hundreds of milliseconds to seconds, while robotic control loops demand sub-10ms response times. Bridging this temporal gap requires hierarchical architectures where fast reactive controllers handle immediate physical dynamics while slower deliberative systems handle strategic planning — but the synchronisation between these layers remains a hard problem.

Semantic-physical misalignment occurs when linguistically reasonable plans violate physical constraints. An MLLM might generate the plan “slide the object across the table to the target” without accounting for friction, mass, or surface properties that make the action infeasible. World models can catch some of these misalignments, but ensuring comprehensive physical grounding remains an open challenge, particularly for novel situations outside the training distribution.

Safety and explainability are critical for deployment in human environments. Embodied AI systems must be predictable, transparent in their decision-making, and robust against adversarial or unexpected conditions. The survey calls for advances in verifiable behaviour, interpretable planning representations, and fail-safe mechanisms that ensure safe operation even when the AI’s understanding of the situation is incomplete or incorrect.

Looking forward, the survey identifies five priority research directions: autonomous embodied AI with adaptive perception and real-time environmental awareness; specialised hardware and compiler optimisations for efficient edge deployment; swarm embodied AI with shared world models and social behaviour capabilities; explainability and trustworthiness frameworks for safety-critical applications; and lifelong learning with human-in-the-loop methods for continuous skill acquisition and preference alignment. These directions collectively define the roadmap for the next decade of embodied AI research and development.