When Emad Mostaque's team trained StableLM, one of the first big open-source language models, they fed it too much Reddit. "It broke the scaling curves," he said, "because it turned a bit stupid and nasty." The fix was not a cleverer design. It was changing what the model read.
That kitchen-level detail is what a new experiment tried to put a number on, and it opened the AI segment of the Moonshots with Peter Diamandis episode published on 11 September 2026. Diamandis introduced it as "a story that explains why everything is speeding up": a write-up by the podcaster Dwarkesh Patel and Jerry Han asking how much of six years of AI progress came from better model designs and how much from better training data.
What the experiment measured
Patel and Han crossed the two ingredients against each other. They took publicly available training recipes — architecture, optimizers, initialization, training schedules — and data collections representing the years 2019 to 2025, and trained small models on the combinations, at budgets up to 10^19 floating-point operations. "FLOPs" is simply a count of the arithmetic done during training, so a compute-efficiency gain of 12-fold means reaching the same score with a twelfth of the arithmetic.
At that budget, their results put the data improvements at 12.0-fold and the recipe improvements at 3.7-fold. Diamandis drew the moral for the audience: the transformer breakthroughs get the headlines, but the quiet work of extracting, filtering and curating what models read has driven more of the gains — and where architectures get published and copied within months, data pipelines stay proprietary.
The experiment's conditions matter for how far that moral travels. Scores came from OLMES, an aggregate of ten relatively easy benchmarks, mostly multiple-choice questions, run over several random seeds because small-model measurements are noisy. Data and model effects added up cleanly, explaining 88% of the variation in scores, which suggests the two did not depend much on each other. Patel and Han also note that architectural research earns its keep in a way their setup cannot see: stability, memory and throughput improvements are what make very large training runs possible at all. And their small models tested neither the expansion of synthetic data nor the recent gains from reinforcement learning after pretraining. Data quality may behave differently at frontier scale.
The information diet
The panelist Alex said the finding matched an essay he had written years earlier, "data sets over algorithms," arguing that the historic grand challenges in AI — chess, speech recognition, Jeopardy — were solved once someone assembled the right training or "training adjacent" data set, usually alongside a competitive community or a benchmark. Language modeling, in his view, is the purest version of that: predicting the next token is a data set challenge.
He pointed to the Hutter Prize, which offers 500,000 euros for compressing the first gigabyte of English Wikipedia, and said recent progress there has come from reordering the articles before compressing them — curriculum learning, in effect. The same holds for people, he argued: teach them in the right order, on "just the right information diet" rather than "information junk food," and you get smarter humans. If intelligence is largely compression of knowledge, better models from better-compressed data should surprise nobody.
Mostaque agreed in food terms — "you are what you eat" — and called junk training data "one of our biggest problems for humanity." The Pile, which he described as one of the first large-scale language-model data sets, and LAION, the first large-scale image one, were built to take the junk out. He described pretraining as a pressure cooker that breaks down the static bonds in the data, with post-training as "the garnish." Even distillation, where one model trains another, is in his account just digested, compressed data passed along. A panelist picked the idea up and pushed it to its end point: perhaps the perfect model at the end of the compression rainbow looks like its data set, probably a synthetic one. "The model becomes the data set."
The lesson for entrepreneurs
Asked what any of this means for founders, Dave started with the authors. Han, he said, is a senior at Princeton who runs entrepreneurship there and is not an AI researcher; during a summer trip to San Francisco he rang Patel's doorbell, got a meeting, and a couple of months later they had a paper the podcast was discussing. The meta lesson: "don't be intimidated." Even the people building these systems are watching from the outside. "It's like a new species that could have any behavior."
The narrow lesson was about data you already own. If you have specific data about a topic and use it to tune or train a model for that purpose, Dave said, "there's a very good chance that it'll outperform the foundation lab models in that use case" — he tied that to Elon Musk's claim of a 100x performance leap from specialist models. Diamandis recalled an entrepreneur from a community AMA a few days earlier who was sitting on 30 years of engineering records from his firm, and who is already in contact with OpenAI. Value is there, Diamandis said, "especially if it's clean data and you make it accessible in a very easy fashion to the labs."
Salim Ismail offered the most concrete number, from a case study he declined to name: a large company carved its data assets into a separate subsidiary, spent time cleaning them up and monetizing them, then brought in its accounting firm to value the subsidiary for the balance sheet. The parent company was worth around $8 billion to $10 billion. The data subsidiary was valued at $32 billion. "So your data may be worth four times as much as your actual company," he said — more so when it feeds "proprietary learning loops." Diamandis suggested accountants start a practice valuing corporate data as an asset.
The shelf-life objection
Alex had already slipped a caveat in mid-praise: this holds "at least for the next five minutes," until proprietary information sitting inside organizations gets washed away by what ends up in public. He then developed it with BloombergGPT as a cautionary tale. Bloomberg, he said, reasoned that it was sitting on a huge amount of internal quantitative finance data and that pre-training its own model must be valuable. It was — "for about five minutes, maybe a few months," until the next generation of frontier models, trained on public data and whatever non-public data the labs are consuming, outperformed it on the relevant financial benchmark. His conclusion was not that internal data is worthless: "there is value in internal enterprise data, but it has a shelf life."
The reply from the panel was to generalize the warning rather than dispute it. Any technology edge that gets a company on the map now has a shrinking shelf life, so build a culture of constant pivoting — "or you're dead." Data gets you a toehold; the motion is what keeps you there.
Dave added a reason to think the labs are not the threat in this picture. The foundation model companies, he said, "are not trying to kill you. They want you to succeed," comparing the moment to Google's growth years after its IPO, when partners who rode the wave — Booking.com, in his example — grew with it.
Back to Moderna
The question resurfaced later in the episode, after the panel had been going through a cheap Chinese open-weight release. Dave said if he had to pick one company to hear that section, it would be Moderna: "this is the moment where you decide whether we're doing AI biology or whether we're going to fall behind and let someone else do it to us." Catch up to the frontier on open source, build the AI function in-house — "if you wait six months, forget it." Diamandis suggested sitting down with Moderna's chief executive, whom he called a friend.
Then came a calculation: the model cost $10 million to train. Alex raised what he called the elephant in the room — how much of that training data came from reasoning traces siphoned off Western models. Mostaque said the ratio looks fairly constant, roughly 10 to 15 times cheaper to be the fast follower using frontier reasoning traces, while noting that on his reading the release itself credits simulated data environments rather than algorithmic efficiency. Against what he put at a billion-dollar training run for the frontier model Astra, this one lands slightly behind at $10 million plus, in his phrase, a bit of data borrowing.
That is the arithmetic Dave wanted Moderna to see: roughly a fifteenth of the cost to be second in, "and you're 1% behind on the IQ chart, but you have all this proprietary data, which is far more important." Weave the cheap model together with the data nobody else has, he argued, and you have the whole solution.
Alex's answer was two words — "for the moment" — followed by his own framing of the history: if intelligence is compression of information, then distillation, or distillation attacks if you prefer, is a one-time compression event, and most of the capital for that event has already been spent.