When Emad Mostaque’s team trained StableLM, he recalled, too much Reddit data spoiled the result. The model became “a bit stupid and nasty,” and its improvement with more training fell off the expected trajectory. His explanation was simple: “You are what you eat.”
That experience helped frame a discussion on Moonshots with Peter Diamandis about an experiment by Dwarkesh Patel and Jerry Han. In a controlled comparison of datasets and model recipes spanning 2019–2025, they reported a 12-fold improvement in compute efficiency from better data, versus 3.7-fold from better architectures and training recipes, at their largest tested compute budget.
For the panel, that opened a business question: if choosing what a model learns matters so much, could a company’s private records give it an advantage that competitors cannot easily copy? BloombergGPT became the counterexample to the most optimistic answer.
What the experiment measured
Pretraining is the broad learning stage in which a language model absorbs patterns from large amounts of text. The dataset supplies that text; the architecture is the model’s internal design, and the training recipe governs how it learns.
Patel and Han’s experiment mixed and matched representative historical model recipes with public datasets. They trained at five computing budgets, from 10^17 to 10^19 FLOPs—floating-point operations, a measure of computational work.
The runs used the same system for splitting text into tokens, the chunks a model processes, and the same maximum input length. The authors varied model size and the amount of training text, and repeated runs with different random initializations. They evaluated capability using OLMES, aggregating ten mostly multiple-choice benchmarks.
Compute efficiency here means the computation needed to reach a given test score. A 12-fold gain means reaching comparable performance with roughly one-twelfth as much computation; it does not mean a model becomes twelve times more capable. At the largest budget, the estimated gain from dataset improvements was more than three times the gain from architectures and recipes.
Adding separate model-recipe and dataset effects explained 88% of the variation in scores. In this setting, the improvements largely stacked. That figure is not a finding that data and architecture explain 88% of all AI progress.
The authors also preserve a role for architecture beyond efficiency at a fixed budget: better designs can enable larger, more stable training runs. Their experiments were far smaller than those used for leading frontier models, and that small scale may overstate curation’s importance at the frontier. Limited tuning of training settings and extrapolation add uncertainty. Expanding training with synthetic, AI-generated data was not tested.
An information diet, not just a bigger pile
The panelist addressed as Alex said the result fit an argument he had made in an earlier essay, “Datasets Over Algorithms”: assembling the right learning material can unlock progress that clever algorithms alone do not.
He pointed to the Hutter Prize, a competition to compress Wikipedia. Recent improvements there, he said, appeared to come from reordering articles before compression. He connected that to curriculum learning—choosing the order in which material is presented—and to human education: the same information can be more useful when it arrives in a helpful sequence.
Mostaque’s StableLM story supplied the less tidy version. Collecting a large dataset was not enough; the balance of its ingredients affected both performance and behavior. He described labs as continually working to improve those ingredients. His account concerned an unhelpful training mixture, rather than a claim that Reddit data is always harmful.
BloombergGPT and the shelf life of an advantage
The panelist addressed as Dave saw an opening for entrepreneurs with specialized records. A model trained or adapted for a narrow purpose, he argued, could outperform general-purpose models on that particular use case. The panel recalled an entrepreneur with 30 years of engineering data from his firm: material that might be valuable if cleaned and made easy to use.
Alex agreed that internal business data could have value outside the organization, but questioned how long the advantage would last. He offered BloombergGPT, Bloomberg’s finance-focused language model, as his cautionary tale.
In his telling, Bloomberg’s extensive financial data initially helped its specialist model. Then a new generation of leading general-purpose models overtook it on relevant financial benchmarks. The advantage lasted “about five minutes, maybe a few months,” he said—an informal description of a short window, not a measured duration.
His argument was not that Bloomberg’s records had become worthless. It was that exclusive access to data did not necessarily preserve a model’s performance lead. Competitors could improve through broader training, better methods and other information sources, without acquiring the identical archive.
From owning records to maintaining a learning process
Another panel anecdote put a much larger price on proprietary data. An unnamed company reportedly separated its data assets into a subsidiary, cleaned them up and monetized them, then brought in an accounting firm to value the business. The speakers put the parent’s worth at roughly $8–10 billion and the data subsidiary’s valuation at $32 billion. They did not identify the company or explain the valuation method.
The discussion connected that potential value to proprietary learning loops: using information from a business’s work to improve its AI, then feeding what it learns from subsequent work back into further improvements. That is an ongoing operation, not simply possession of an archive.
The panel’s proposed response to a shrinking lead was continual reinvention. For the firm with 30 years of engineering records, that would mean treating the archive as a starting point: cleaning and organizing useful examples, adapting a model to a specific task, and continuing to improve it as new work produces new information. BloombergGPT’s reported experience leaves a concrete test for that effort: whether the specialist still performs better when the next general-purpose model arrives.