SampleMix: How Sample-Wise LLM Pre-Training Data Mixing Transforms Model Performance

What Is SampleMix and Why LLM Pre-Training Data Mixing Matters

LLM pre-training data mixing determines how training corpora from multiple sources are combined before feeding into a large language model. The composition of this training data profoundly shapes everything a model learns — from factual knowledge and reasoning ability to linguistic fluency and domain expertise. A groundbreaking 2025 research paper from Peking University and Meituan Group introduces SampleMix, a fundamentally new approach that evaluates every individual document rather than treating entire data domains as monolithic blocks.

The scale of the challenge is enormous. Modern pre-training datasets like SlimPajama contain over 500 billion tokens drawn from sources including CommonCrawl, C4, Wikipedia, ArXiv, Books, and StackExchange. Deciding how much weight to give each source has traditionally been a coarse-grained decision made at the domain level. SampleMix demonstrates that this top-down approach leaves substantial performance on the table, and that a sample-wise bottom-up method achieves superior results while requiring nearly half the training compute.

This research arrives at a critical moment for the AI industry. As organizations invest billions in training foundation models, even marginal improvements in data efficiency translate into millions of dollars in compute savings. Understanding how financial stability frameworks increasingly depend on AI systems makes the efficiency of their training pipelines a matter of institutional concern.

The Problem with Domain-Wise Data Mixing Methods

Traditional LLM pre-training data mixing follows a two-step process. First, researchers determine what proportion of data should come from each domain — for example, 40% CommonCrawl, 25% C4, 15% Wikipedia, and so on. Second, they uniformly sample documents within each domain until the quota is filled. Methods like DoReMi, DoGE, DML, and BiMIX-OPT all follow this top-down paradigm, differing mainly in how they calculate those domain-level proportions.

The SampleMix researchers identify two critical flaws in this approach. The first is that domain boundaries based on data source do not correspond to semantic boundaries. Their analysis reveals staggering overlap: 99.9% of ArXiv clusters contain C4 samples, 99.6% contain CommonCrawl samples, and 76.6% contain Wikipedia samples. Consider a research article about Einstein’s contributions to physics — it could appear in ArXiv, Wikipedia, CommonCrawl, and Books simultaneously. Treating these domains as independent entities ignores their massive semantic redundancy.

The second flaw is uniform intra-domain sampling. Within each domain, all documents receive equal probability of selection regardless of their individual quality or uniqueness. A meticulously written Wikipedia article about quantum mechanics receives the same weight as a stub article with three sentences and broken formatting. Similarly, a redundant CommonCrawl page duplicating information found in thousands of other pages gets the same treatment as a uniquely informative analysis.

These limitations compound each other. Domain-wise methods cannot simultaneously optimize for what matters most: selecting the highest-quality, most diverse documents from across the entire corpus regardless of their source labels. As research on data-centric AI has consistently shown, data quality and curation often matter more than model architecture improvements.

How SampleMix Works: A Bottom-Up Data Mixing Approach

SampleMix fundamentally inverts the traditional paradigm. Instead of starting with domain proportions and working down to individual documents, it starts with individual documents and lets optimal domain proportions emerge naturally. The method proceeds through four stages: quality evaluation, diversity evaluation, sampling weight calculation, and training dataset construction.

In the quality evaluation stage, every document in the candidate pool receives a quality score between 0 and 10 based on seven carefully defined dimensions. In the diversity evaluation stage, documents are embedded into a high-dimensional vector space, clustered, and scored based on how much unique information they contribute. These two scores are then combined using a weighted formula to produce a single sampling probability for each document.

The key innovation is global cross-domain sampling. Rather than filling domain quotas sequentially, SampleMix performs a single global sampling pass across the entire corpus. High-quality, highly diverse documents from any source receive proportionally higher sampling weights. This means the final training set’s domain composition is an emergent property of document-level selection — not a pre-determined constraint.

This bottom-up approach offers three advantages. First, it naturally handles inter-domain overlap by evaluating documents on their own merits rather than their source labels. Second, it adapts automatically to different token budgets without requiring re-computation of domain weights. Third, it produces training sets where every included document earned its place through measurable quality and diversity contributions.

Evaluating Data Quality Across Seven Dimensions

SampleMix’s quality evaluation framework goes far beyond simple heuristics like document length or perplexity scores. The researchers define seven complementary dimensions that together capture what makes a training document valuable for language model learning. Each dimension receives a binary or ternary score, summing to a maximum quality score of 10.

The first four dimensions each contribute 0 or 1 point: clarity of expression and accuracy, completeness and coherence, structure and style, and content accuracy and credibility. The remaining three dimensions use a 0-to-2 scale for greater granularity: significance, knowledge richness, and logicality and analytical depth. This scoring system weights cognitive complexity and informational density more heavily than surface-level formatting.

To train the quality evaluator at scale, the researchers used GPT-4o to label 420,000 documents from the SlimPajama dataset, reserving 10,000 for testing. The evaluator backbone uses the gte-en-mlm-base model with an ordinal regression head rather than standard text classification. This design choice recognizes that quality scores have inherent ordering — a score of 6 is more similar to 7 than to 2.

The ordinal regression approach consistently outperforms text classification: mean squared error of 1.57 versus 1.95, mean absolute error of 0.72 versus 0.77, and close accuracy (within ±1 of true score) of 83.37% versus 82.24%. These improvements matter because even small errors in quality estimation propagate through billions of sampling decisions during corpus construction.

Perhaps most revealing are the quality distribution findings. CommonCrawl achieves an average quality score of 5.65, substantially higher than C4’s 4.20 — contradicting the common assumption that C4’s additional filtering makes it universally superior. Wikipedia contains surprising amounts of low-quality content including stub articles, parsing errors, and incomplete entries. ArXiv and Books exhibit the highest average quality as expected, but their distributions still show significant variance.

Measuring Data Diversity Through Clustering Analysis

While quality ensures individual documents are well-written and informative, diversity ensures the training set covers the broadest possible range of knowledge domains and writing styles. SampleMix quantifies diversity through a three-stage clustering pipeline that measures how much unique information each document contributes to the corpus.

The process begins with embedding generation using SimCSE-BERT, producing 768-dimensional vector representations of each document. These embeddings capture semantic meaning rather than surface-level features, allowing the system to identify topical similarity across different writing styles and source domains. The researchers then apply K-Means clustering with the number of clusters set to the square root of the sample count, using Facebook’s FAISS library for computational efficiency.

Each cluster’s diversity contribution is measured through two complementary metrics. Cluster compactness calculates the average distance between member documents and their cluster centroid — tighter clusters indicate more redundant, overlapping content. Cluster separation measures the distance between a cluster’s centroid and all other centroids — more isolated clusters represent more unique topical niches. A document’s final diversity score combines both: d(x) = compactness × separation.

The diversity distributions reveal important patterns across domains. C4, CommonCrawl, and Books show the highest diversity, containing documents spanning the widest range of topics and styles. StackExchange shows the lowest diversity, reflecting its concentrated focus on technical question-answer content. Within individual domains, diversity varies dramatically — C4’s distribution approximates a normal curve while other domains show distinct peaks and tails. These variations explain why uniform sampling within domains is fundamentally suboptimal for LLM pre-training data mixing.

LLM Pre-Training Data Mixing Benchmark Results

The researchers evaluated SampleMix against six established baselines using a 1-billion-parameter LLaMA architecture trained from scratch on 100 billion tokens from SlimPajama. The evaluation suite includes eight downstream tasks tested with 5-shot prompting via the LM-eval Harness, plus perplexity measurements on the Pile and xP3 evaluation sets.

SampleMix achieves an average accuracy of 47.77% across all eight benchmarks, decisively outperforming every baseline. The next best method, DoReMi, reaches 46.40%, followed by CE at 46.18% and Vanilla (proportional) sampling at 46.13%. The gap widens on specific tasks: SampleMix leads on ARC-Challenge by 1.28 points (29.86% versus 28.58%), on RTE by 2.17 points (53.79% versus 51.62%), and on ARC-Easy by 1.24 points (48.73% versus 47.49%).

Perplexity results tell an equally compelling story. On the Pile evaluation set, SampleMix achieves a perplexity of 25.63, compared to 26.20 for CE and 26.45 for DoReMi. On the multilingual xP3 set, SampleMix reaches 46.38 versus DoReMi’s 47.08. Lower perplexity indicates the model assigns higher probability to natural language — a fundamental measure of language modeling capability that underlies all downstream performance.

What makes these results particularly significant is their consistency. SampleMix leads on five of eight individual tasks and never falls below third place on any single benchmark. By contrast, baseline methods show volatile performance — DML leads on WiC but collapses on LAMBADA; DoReMi excels on WinoGrande but underperforms on ARC-Challenge. This consistency suggests SampleMix produces more balanced training sets that develop robust general capabilities rather than accidental specialization. Organizations tracking how AI capabilities intersect with digital infrastructure frameworks should note how training methodology directly impacts model reliability.

Training Efficiency: 1.9x Faster Convergence

Beyond final performance, SampleMix demonstrates remarkable training efficiency advantages. Convergence analysis shows that SampleMix achieves baseline-equivalent accuracy using 1.4x to 2.1x fewer training steps, depending on the baseline being compared. On average across all six baselines, SampleMix reaches comparable performance in approximately 100,000 training steps — a 1.9x speedup.

This efficiency gain has profound practical implications. Training a 1-billion-parameter model on 100 billion tokens requires significant GPU cluster time. A 1.9x reduction in required training steps translates almost linearly into compute cost savings, since each training step involves a fixed-cost forward and backward pass through the full model. For organizations training at the frontier with models ten or a hundred times larger, these savings multiply into millions of dollars.

The efficiency improvement also has environmental significance. The energy consumption of large-scale AI training has become a growing concern, with individual training runs consuming megawatt-hours of electricity. Achieving the same performance with roughly half the compute directly reduces the carbon footprint of foundation model development. As financial stability reports increasingly examine technology sector resource consumption, training efficiency methods like SampleMix represent meaningful progress.

The convergence curves reveal another important property: SampleMix not only converges faster but also to a higher final performance level. This means the method is not simply front-loading easy gains that plateau early — it is genuinely finding a better training data distribution that yields both faster learning and superior final capabilities.

Balancing Quality and Diversity in LLM Pre-Training Data Mixing

SampleMix combines quality score q(x) and diversity score d(x) using a weighted formula: sampling weight p(x) = α·d(x) + (1−α)·q(x), where alpha controls the balance between the two signals. The researchers conduct a systematic sweep of alpha from 0.0 (quality only) to 1.0 (diversity only) in increments of 0.2.

The results produce a clear inverted-U curve peaking at alpha equals 0.8. Pure quality-based selection (alpha=0.0) yields the worst average accuracy at just 45.53% — even below several baseline methods. Performance improves steadily as diversity weight increases: 46.72% at alpha=0.2, 47.20% at 0.4, 47.73% at 0.6, and the optimal 47.77% at 0.8. Pure diversity-based selection (alpha=1.0) achieves 47.58%, strong but slightly below the optimum.

This finding carries a counterintuitive message: for pre-filtered datasets where baseline quality is already reasonable, diversity matters substantially more than quality for pre-training performance. The researchers hypothesize that when a corpus has already undergone basic quality filtering (as SlimPajama has), further quality-based selection primarily removes documents at the margins while diversity-based selection ensures the model encounters the broadest possible range of linguistic patterns, factual knowledge, and reasoning structures.

The practical recommendation is context-dependent. For datasets with minimal prior filtering and high noise levels, a lower alpha value (more quality weight) would likely be appropriate. For well-curated corpora, the diversity-dominant weighting of alpha=0.8 appears optimal. This nuance highlights why a one-size-fits-all approach to LLM pre-training data mixing fails — the optimal strategy depends on the characteristics of the available data.

Scaling SampleMix to 8B Parameter Models

A critical question for any data mixing strategy is whether gains observed at smaller scales transfer to larger models. The SampleMix researchers address this directly by training 8-billion-parameter models — eight times the size of their primary experimental setup — using the same data mixing configurations.

At the 8B scale, SampleMix achieves an average accuracy of 54.86%, compared to 53.58% for DoReMi, 53.17% for Vanilla sampling, and 53.15% for CE. The absolute gap of 1.28 percentage points between SampleMix and the best baseline is consistent with the 1.37-point gap observed at 1B scale, demonstrating that the method’s advantages persist rather than diminishing with scale.

The 8B experiments also reveal interesting scaling dynamics. All methods improve substantially from 1B to 8B (gains of 7-8 percentage points), but SampleMix maintains its relative advantage. This suggests that the benefits of sample-wise data selection are orthogonal to — rather than redundant with — the benefits of increased model capacity. A larger model can learn more from any given training set, but a better-curated training set provides more to learn from regardless of model size.

These scaling results have significant implications for frontier model development. If SampleMix’s advantages hold at even larger scales — 70B, 405B, or beyond — the cumulative training efficiency gains would represent enormous cost savings. The researchers note that their method’s computational overhead for quality and diversity scoring is a one-time cost that amortizes over the entire training process, making it increasingly cost-effective as model size and training duration grow. Research into global electricity demand from AI infrastructure underscores why such efficiency innovations matter at industry scale.

Implications for Future LLM Pre-Training Data Mixing

SampleMix represents more than an incremental improvement — it challenges a fundamental assumption that has guided pre-training data curation since the earliest large language models. The shift from domain-wise to sample-wise thinking opens several important research directions that could reshape how the AI industry approaches data preparation.

First, the quality evaluation framework demonstrates that automated document scoring at scale is both feasible and impactful. The 420,000-document GPT-4o labeling dataset and ordinal regression evaluator could serve as foundations for increasingly sophisticated quality assessment tools. As evaluation models improve, the seven-dimension framework could expand to capture additional aspects of training value such as factual recency, reasoning complexity, or cultural representation.

Second, the diversity measurement approach via clustering offers a principled alternative to ad hoc deduplication methods. Rather than simply removing near-duplicate documents, SampleMix quantifies the marginal diversity contribution of every document and uses this signal to prioritize unique content. This approach naturally handles the spectrum from exact duplicates to topically related but distinct documents — a nuance that binary deduplication misses entirely.

Third, the method’s ability to adapt automatically to different token budgets without re-tuning is particularly valuable in practice. Organizations frequently adjust training budgets as compute availability changes, new data becomes available, or project timelines shift. A data mixing strategy that gracefully handles these changes without manual re-optimization reduces operational friction and accelerates experimental iteration.

The broader lesson from SampleMix aligns with a growing consensus in the AI research community: the era of “just add more data” is giving way to an era of data intelligence. As foundational datasets approach the limits of available high-quality internet text, the ability to extract maximum training value from existing data becomes a decisive competitive advantage. Methods that treat individual documents as first-class citizens in the curation process — rather than anonymous members of domain-level categories — point toward a more efficient and effective future for institutional knowledge processing and AI development alike.