Machine Learning Mental Health Forecasting: Traditional ML vs Deep Learning vs Foundation Models Compared

Why Mental Health Forecasting Matters More Than Detection

Machine learning mental health forecasting represents a fundamental shift in how artificial intelligence can support clinical care. Unlike mental health detection—which reactively identifies existing conditions from current behavioral signals—forecasting proactively predicts future mental health states before symptoms become apparent. This distinction is not merely academic; it carries profound implications for clinical intervention strategies and patient outcomes.

A landmark comparative study published in January 2026 by researchers from Nanyang Technological University and other institutions systematically benchmarked 14 models spanning traditional machine learning, deep learning, and large language model paradigms for this exact task. The study analyzed smartphone passive sensing data from 215 college students over five years, creating the most comprehensive evaluation framework for mental health forecasting to date.

The clinical motivation centers on enabling Just-in-Time Adaptive Interventions (JITAI)—personalized, real-time support delivered at the precise moment a patient’s mental health begins to decline. With depression and anxiety affecting over 280 million people globally according to the World Health Organization, the ability to predict deterioration days in advance could revolutionize preventive mental healthcare. Traditional clinical approaches rely on scheduled assessments and self-reported symptoms, creating dangerous gaps where conditions worsen undetected. Smartphone-based forecasting promises continuous, passive monitoring that requires no active participation from patients, making it particularly valuable for populations who avoid seeking help.

Understanding how different AI paradigms perform on this challenge is essential for researchers, clinicians, and technology companies building the next generation of AI-powered healthcare tools. The findings reveal surprising strengths and limitations across each approach, challenging common assumptions about which models work best for time-series behavioral prediction.

The CES Dataset: Five Years of Smartphone Behavioral Data

At the foundation of this comparative study lies the College Experience Sensing (CES) dataset, released by Dartmouth College in October 2024. This dataset represents one of the most extensive longitudinal smartphone sensing collections ever assembled for mental health research, spanning a five-year period from 2017 to 2022 that encompasses pre-pandemic, pandemic, and post-pandemic behavioral patterns.

The dataset includes 24,778 data samples collected from 215 college students, with individual participation ranging from 61 to 432 samples per student (averaging 206 samples each). Data collection occurred through two complementary channels that together paint a rich picture of daily behavioral patterns and their relationship to mental health outcomes.

The first channel captured passive sensing data—continuous behavioral traces recorded automatically through smartphone sensors on an hourly basis. This included mobility patterns tracked through GPS and accelerometer data, physical activity levels distinguishing between running, walking, and stationary states, sleep duration and quality metrics, phone usage patterns including screen time and app interactions, and social interaction indicators. From an original set of 172 raw features, the researchers distilled 35 meaningful behavioral features per day through careful feature selection, further aggregated into five high-level behavioral dimensions: leisure time, personal time, phone usage, sleep quality, and social engagement.

The second channel consisted of weekly Ecological Momentary Assessment (EMA) surveys, where participants completed the Patient Health Questionnaire-4 (PHQ-4)—a validated clinical instrument producing scores from 0 to 12. These scores were mapped to four severity levels: Normal (scores 0–3, representing 62% of all samples), Mild (scores 4–6, comprising 26%), Moderate (scores 7–9, at 7%), and Severe (scores 10–12, accounting for just 4% of samples). This severe class imbalance—where the most critical patients are also the rarest in the data—poses one of the study’s central methodological challenges.

The forecasting task itself was carefully constructed: given two weeks of behavioral features, models must predict the mental health severity level at the end of the second week using only the first week’s data. This creates a genuine seven-day forecasting window, testing whether patterns in smartphone behavior can reliably signal mental health changes before they fully manifest. Researchers used a chronological, user-level data split—70% training, 10% validation, 20% testing—ensuring that models were evaluated on truly future data from each participant.

Traditional Machine Learning Models for Mental Health Prediction

The study evaluated six traditional machine learning approaches that represent the foundational toolkit for classification tasks: Logistic Regression, Support Vector Machines (SVM), Decision Trees, Random Forests, XGBoost, and LightGBM. These models process aggregated statistical features rather than raw temporal sequences, requiring careful preprocessing to transform two-week behavioral time series into fixed-dimensional input vectors.

Two input strategies were tested for traditional ML models. Statistical Aggregation averaged behavioral features across the two-week observation window, producing a single feature vector per sample. Sequential Flattening concatenated daily features chronologically, preserving some temporal ordering but dramatically increasing dimensionality. A key finding was that Statistical Aggregation consistently outperformed Sequential Flattening across all traditional ML models, suggesting that for these approaches, summary statistics capture the relevant behavioral patterns more effectively than raw temporal structure.

Among traditional ML models, Logistic Regression achieved the highest overall Macro-F1 score of 0.5003, demonstrating that a relatively simple linear model with appropriate regularization can be surprisingly competitive for this task. SVM followed closely with a Macro-F1 of 0.4914, excelling particularly on the Normal class (F1 = 0.7881) but struggling with the rare Moderate class (F1 = 0.1827). LightGBM reached 0.4534, while XGBoost achieved 0.4437—both ensemble methods performing below the simpler Logistic Regression, likely due to overfitting on the imbalanced training distribution.

The most dramatic performance gap appeared in the Moderate and Severe classes. Decision Trees, despite their interpretability advantages, collapsed to near-random performance on minority classes (Moderate F1 = 0.0211), yielding the lowest overall Macro-F1 of 0.2738. Random Forests partially recovered through ensemble averaging but still achieved only 0.4053 Macro-F1. These results underscore that traditional ML models, while computationally efficient and interpretable, lack the capacity to model the subtle temporal dynamics that distinguish between adjacent severity levels in behavioral data.

Feature importance analysis from XGBoost models revealed that sleep duration emerged as the single most predictive behavioral feature (median importance 0.1715), followed by running activity (0.1581) and on-foot physical activity. Dorm duration showed the most consistent importance across participants, suggesting that time spent in one’s living space is a reliable behavioral indicator of mental health state. These insights align with established clinical literature linking sleep disruption to depression and anxiety, validating the passive sensing approach.

Deep Learning Architectures and Temporal Pattern Recognition

Deep learning models represented the strongest performing category in the study, with four architectures evaluated: Multi-Layer Perceptron (MLP, based on the I-HOPE framework), Temporal Convolutional Networks (TCN), LSTM with Attention mechanisms, and Transformers. Unlike traditional ML approaches, these models can process raw temporal sequences directly, learning complex nonlinear relationships between behavioral patterns and future mental health states.

The Transformer architecture achieved the study’s best overall performance with a Macro-F1 score of 0.5808—a substantial margin above all other approaches. This superiority was particularly pronounced on the most challenging prediction targets: the Transformer reached F1 scores of 0.3868 for Moderate cases and 0.6411 for Severe cases, both representing the highest values achieved by any model. The attention mechanism’s ability to weight different time steps and behavioral features dynamically proved especially valuable for capturing the complex, multi-scale temporal patterns that precede mental health deterioration.

TCN models, which use causal dilated convolutions to capture local temporal dependencies, achieved the second-best deep learning performance with Macro-F1 of 0.5456. TCN’s unique strength lay in its ability to process high-dimensional temporal data—it was the only model that performed best under the (35-Dimension, Daily) configuration, leveraging fine-grained daily behavioral patterns that other architectures found overwhelming. This suggests that TCN’s hierarchical convolutional structure is particularly well-suited for extracting patterns from dense, high-frequency behavioral time series.

The MLP model (I-HOPE baseline), despite processing aggregated features without sequential order, reached Macro-F1 of 0.5439 and actually achieved the highest accuracy among all models at 0.6662. However, this apparent accuracy advantage was largely driven by strong Normal class performance (F1 = 0.7896), masking weaker minority class detection. This discrepancy between accuracy and Macro-F1 highlights why balanced metrics are essential when evaluating mental health classifiers—high accuracy can be misleading when the majority class dominates predictions.

LSTM with Attention achieved Macro-F1 of 0.5147, showing strong performance on the Mild class (F1 = 0.5380, the highest of any model) but relatively weaker results on Moderate and Severe categories. The attention mechanism helped LSTM focus on the most diagnostically relevant time steps, but the recurrent architecture’s sequential processing appears less effective than the Transformer’s parallel attention for this particular forecasting task. The choice of loss function also significantly impacted performance: Focal Loss worked best for high-capacity models like the Transformer (yielding the 0.5808 peak), while Weighted Cross-Entropy better suited the MLP, and vanilla Cross-Entropy was optimal for LSTM stability.

Foundation Models and LLMs in Clinical Forecasting

Perhaps the study’s most surprising findings concern large language models, which have recently demonstrated remarkable capabilities across diverse tasks. Four LLMs were evaluated: three open-source Qwen3 variants (4B, 8B, and 14B parameters) and the commercial GPT-4.1 model. The results challenge prevailing assumptions about LLM superiority and scaling behavior.

All four LLMs converged to nearly identical Macro-F1 scores: Qwen3-4B at 0.4419, Qwen3-8B at 0.4415, Qwen3-14B at 0.4414, and GPT-4.1 at 0.4411. This remarkable uniformity—with less than 0.001 difference across a 3.5x parameter range—demonstrates that scaling laws do not hold for this structured temporal prediction task. More parameters did not translate to better behavioral pattern recognition, suggesting that the bottleneck lies not in model capacity but in the fundamental challenge of converting continuous numerical time series into language-compatible representations.

The study tested seven distinct LLM adaptation strategies across three categories. Zero-shot prompting, where models received no examples, produced the weakest results and revealed systematic biases: Qwen3 models tended to classify nearly all samples as Normal, while GPT-4.1 showed a strong bias toward Mild predictions. In-Context Learning (ICL) approaches—particularly similarity-based and recency-based few-shot learning—consistently achieved the best LLM performance, with Macro-F1 scores above 0.44. This suggests that providing relevant behavioral examples helps ground LLMs in the domain-specific patterns needed for accurate forecasting.

Parameter-Efficient Fine-Tuning (PEFT) methods including LoRA and Prompt Tuning fell between zero-shot and ICL in effectiveness. LoRA consistently outperformed Prompt Tuning by adapting model weights more directly to the mental health forecasting task. Among knowledge injection strategies, Pattern-based approaches (where GPT-4.1 summarized behavioral patterns from training data) outperformed Statistical-based methods, and individual-level statistics proved more informative than population-level averages—reinforcing the importance of personalization in mental health applications.

A critical insight is that LLMs consistently performed best with the most compact data representation: 5-Dimension, Weekly features. This makes intuitive sense—LLMs process data as text tokens, and more compact representations reduce the cognitive load of parsing lengthy numerical tables. The researchers concluded that while LLMs show promise as “behavioral reasoning engines” capable of generating interpretable clinical narratives, they are not yet competitive as standalone numerical predictors for structured time-series forecasting tasks. Their strength lies in contextual understanding and explanation generation rather than raw predictive power.

Head-to-Head Performance Comparison Across All Models

The comprehensive benchmarking across 14 models reveals a clear performance hierarchy for machine learning mental health forecasting, with important nuances that should guide model selection for different clinical scenarios.

Several patterns emerge from this comparison. Deep learning models dominate the top four positions, with the Transformer maintaining a clear lead. The performance gap between the best deep learning model (0.5808) and the best traditional ML model (0.5003) is 0.0805—clinically meaningful when translated to patient outcomes. Perhaps most strikingly, all LLMs perform below even the best traditional ML approaches, despite their orders-of-magnitude greater computational cost.

The Severe class detection results are particularly important from a clinical perspective. The Transformer and MLP models both achieve 0.6411 F1 on Severe cases, meaning they correctly identify roughly two-thirds of patients at highest risk. By contrast, LLMs detect Severe cases with only 0.41 F1, missing nearly 60% of the most critical patients. For clinical deployment where identifying at-risk individuals is the primary objective, this performance gap could have life-or-death implications.

An important consideration is the relationship between model complexity and practical deployment. Traditional ML models require minimal computational resources and offer inherent interpretability—a Logistic Regression model can directly explain which behavioral features drove each prediction. Deep learning models require GPU resources but achieve meaningfully better performance. LLMs demand the most compute while delivering the weakest numerical predictions, though they uniquely offer the ability to generate human-readable clinical narratives explaining their reasoning. The optimal choice depends on whether the deployment context prioritizes raw accuracy, interpretability, or explanatory capabilities for different AI model selection scenarios.

The Personalization Effect on Machine Learning Mental Health Forecasting

One of the study’s most impactful findings concerns the dramatic effect of personalization on forecasting accuracy. By incorporating user-specific information into model training, researchers observed performance improvements that far exceeded those achieved by switching between model architectures—suggesting that knowing who you’re predicting for matters more than how you predict.

For deep learning models, personalization was implemented through learnable user embeddings—dense vector representations that the model learns to associate with each individual’s unique behavioral patterns. The results were striking: MLP improved by 0.3635 Macro-F1, TCN by 0.2899, LSTM+Attention by 0.2894, and Transformer by 0.2943. To put this in perspective, the largest improvement from personalization (MLP’s +0.3635) is more than four times larger than the performance gap between the best and worst deep learning architectures in user-agnostic mode.

The impact on Severe class detection was even more dramatic. Personalized models achieved Severe class Precision improvements of +0.9213 (MLP), +0.8802 (Transformer), and +0.9451 (Random Forest). These near-perfect precision gains mean that when a personalized model flags a patient as Severe, it is almost always correct—a critical property for clinical systems where false alarms can erode trust and waste limited intervention resources.

Traditional ML models showed more moderate but still meaningful personalization gains, ranging from +0.1144 (Decision Tree) to +0.2504 (Logistic Regression). For these models, personalization was implemented through one-hot user ID vectors appended to the feature space, a simpler mechanism than learnable embeddings but still effective at allowing the model to calibrate predictions per individual.

The LLM personalization results tell a cautionary tale. Researchers attempted to personalize LLM predictions by including user IDs in text prompts—a natural extension of the language-based paradigm. However, this approach proved ineffective or actively harmful: Qwen3-14B’s performance decreased by 0.0377 Macro-F1 with personalization. This failure highlights a fundamental limitation of current LLM architectures for personalized prediction—text-based user identifiers cannot capture the nuanced, continuous behavioral baselines that learnable embeddings encode. Future research may need to develop LLM-compatible personalization mechanisms, such as user-specific prompt templates derived from individual behavioral histories, to bridge this gap.

Feature Engineering and Data Granularity Strategies

The study systematically evaluated four feature representation strategies combining two dimensionality levels (5-Dimension aggregated vs. 35-Dimension detailed) with two temporal granularities (Daily vs. Weekly). The results reveal that more data is not always better—and that optimal feature engineering depends critically on the model architecture being used.

Traditional ML models and the MLP consistently performed best with the more compact 5-Dimension representation, which aggregates 35 raw behavioral features into five high-level categories: leisure, personal time, phone usage, sleep, and social engagement. This finding suggests that for models lacking built-in mechanisms to handle high-dimensional input, pre-aggregation serves as a form of beneficial regularization, reducing noise while preserving the most diagnostically informative behavioral signals.

TCN showed a unique preference for the most detailed (35-Dimension, Daily) configuration. Its hierarchical convolutional architecture, with causal dilated convolutions at multiple scales, can effectively extract local temporal patterns from dense behavioral streams without being overwhelmed by dimensionality. This makes TCN particularly suitable for deployment scenarios where preserving fine-grained behavioral detail is important—for example, detecting rapid behavioral changes that might signal acute mental health crises.

LSTM and Transformer models improved with reduced feature dimensionality or shorter temporal windows, suggesting that the attention and recurrent mechanisms perform better when they can focus on the most salient patterns without distraction from redundant features. The Transformer’s optimal configuration used Focal Loss with reduced dimensionality, achieving the study’s peak Macro-F1 of 0.5808.

LLMs consistently performed best with the most compact representation: 5-Dimension, Weekly features. This aligns with the inherent constraints of language-based processing—LLMs must convert numerical arrays into text tokens, and more compact inputs produce shorter, more parseable prompt sequences. The difference was meaningful: switching from 35-Dimension Daily to 5-Dimension Weekly improved LLM Macro-F1 by several percentage points, while requiring far fewer tokens (and thus lower inference cost) per prediction.

These findings carry practical implications for system design. A deployment pipeline might use TCN for continuous fine-grained monitoring with full 35-feature daily inputs, while running a parallel Transformer model on aggregated 5-Dimension weekly data for higher-accuracy weekly forecasts. LLMs could serve as an interpretive layer, generating clinical narratives from the compact weekly summaries that clinicians find easier to review and act upon. This multi-model architecture leverages each paradigm’s strengths while mitigating individual weaknesses.

Handling Class Imbalance in Mental Health Datasets

The severe class distribution imbalance in the CES dataset—62% Normal, 26% Mild, 7% Moderate, and 4% Severe—mirrors real-world mental health prevalence patterns and poses a fundamental challenge for all machine learning approaches. The study’s analysis of intra-class and inter-class similarity provides insights into why this imbalance is so problematic and which mitigation strategies work best.

Intra-class similarity analysis revealed that the Severe class has the highest behavioral coherence (similarity score 0.0446), meaning that severely affected individuals share more distinctive behavioral patterns than those in other severity levels. Paradoxically, despite this coherence, the Severe class is hardest to detect due to its extreme rarity in training data. Even more challenging, inter-class similarity between Mild and Severe is unexpectedly high (0.0405), meaning some Mild and Severe patients exhibit similar behavioral patterns—making the boundary between “manageable” and “critical” difficult for models to learn.

Three loss function strategies were compared across deep learning models. Vanilla Cross-Entropy, which treats all classes equally, provided the most stable training for LSTM+Attention (Macro-F1 = 0.5147) but systematically under-predicted minority classes. Weighted Cross-Entropy, which increases the loss penalty for misclassifying rare classes proportionally to their inverse frequency, worked best for the MLP (Macro-F1 = 0.5465). Focal Loss—which dynamically reduces the loss contribution from well-classified examples and focuses training on hard cases—proved most effective for high-capacity models: Transformer (0.5808) and TCN (0.5456).

The relationship between model capacity and optimal loss function is intuitive: Focal Loss requires sufficient model capacity to exploit the additional focus on hard examples without memorizing noise. For simpler models like MLP and LSTM, the gentler rebalancing of Weighted Cross-Entropy provides enough minority class attention without destabilizing training. For traditional ML models, the researchers applied class weighting through each algorithm’s native mechanism (e.g., class_weight parameter in scikit-learn), achieving partial but insufficient mitigation of the imbalance problem.

For LLMs, class imbalance manifests differently. Zero-shot LLMs displayed strong class biases—Qwen3 overwhelmingly predicted Normal while GPT-4.1 favored Mild—reflecting the prior distributions these models absorbed during pre-training. Few-shot ICL approaches partially corrected these biases by exposing models to balanced example sets, but the fundamental limitation remained: LLMs cannot adjust internal loss functions or sampling strategies the way supervised models can. The PEFT approaches (LoRA, Prompt Tuning) offered some improvement by fine-tuning on the imbalanced dataset, but without explicit class-balancing mechanisms, they inherited similar biases. These findings suggest that future LLM-based mental health systems will need creative approaches to class imbalance, perhaps incorporating chain-of-thought reasoning about severity level base rates or structured output calibration techniques.

Future of Machine Learning Mental Health Forecasting and Interventions

This comprehensive benchmarking study illuminates both the current capabilities and remaining challenges for machine learning mental health forecasting, pointing toward several promising research and deployment directions that could transform mental healthcare delivery.

The most immediate opportunity lies in hybrid architectures that combine the numerical precision of deep learning with the explanatory power of LLMs. A Transformer model could generate accurate severity predictions while a paired LLM interprets the behavioral patterns driving each forecast, producing clinical narratives like “Sleep duration decreased 23% this week while phone usage after midnight increased by 45 minutes—patterns historically associated with depressive episodes for this individual.” Such hybrid systems would address clinicians’ well-documented reluctance to trust black-box predictions, potentially accelerating clinical adoption of explainable AI in healthcare settings.

The personalization findings suggest that federated learning approaches could unlock even greater performance gains while preserving patient privacy. Rather than centralizing sensitive behavioral data, individual models could be fine-tuned on each user’s device, sharing only aggregated model updates with a central server. This approach would naturally encode deep personalization while complying with healthcare data regulations like HIPAA in the United States and GDPR in Europe.

Multi-modal integration represents another frontier. The current study relies exclusively on smartphone sensor data, but modern wearable ecosystems capture rich physiological signals—heart rate variability, skin conductance, sleep staging from accelerometry—that could provide complementary information about mental health states. Combining passive sensing with physiological data could improve detection of the challenging Moderate class, where behavioral changes may be subtle but physiological stress responses are more pronounced.

The early forecasting capability—achieving near-peak accuracy with just one week of data—opens possibilities for real-time intervention systems that continuously update predictions as new behavioral data streams in. Such systems could trigger graduated responses: an initial alert to a clinician after three days of deteriorating patterns, followed by direct outreach to the patient if predictions continue to worsen at day five, and emergency escalation protocols if the seven-day forecast indicates severe risk. The technical infrastructure for such systems exists today, but the clinical validation, ethical frameworks, and regulatory approvals needed for deployment remain works in progress.

Finally, the study highlights the need for larger, more diverse datasets. The CES dataset, while valuable, represents a specific population—college students at a single American institution. Mental health manifests differently across age groups, cultures, socioeconomic contexts, and clinical populations. Extending this research to diverse settings, including clinical populations with diagnosed conditions and individuals in low-resource environments where smartphone-based monitoring may be the only accessible mental health tool, will be crucial for ensuring that AI-powered forecasting benefits all communities equitably.

As the field continues to evolve, the combination of rigorous benchmarking studies like this one, advances in model architectures, and growing availability of longitudinal behavioral data positions machine learning mental health forecasting as one of the most impactful applications of artificial intelligence in healthcare.