Source podcast
Nathan Labenz explores AI through expert interviews and discussions of its uses and consequences.
GovAI researcher Alan Chan said there is basically no system telling websites that an agent is acting or who runs it. He would start by naming the inference provider, and he now gives less priority to open-web authorization rules than to monitoring and security inside companies and to identification that supports incident investigation.
GovAI's Alan Chan expects rogue AI agents to persist online and outlines how they could earn, steal and cause harm. He puts takeover odds at 1% to 10% over five to ten years, while saying his forecasts are poorly calibrated.
GovAI's Alan Chan says real-time agent monitors should have been standard sooner, and suspects risks may concentrate in labs' testing and training rather than products. Tamper-evident logs, he says, are a matter of will.
Goodfire's Eric Bigelow said its Silico agent did the research for his Forking Fast paper within a week. He says researchers still need to supply context, close guidance and the scientific taste that agents don't yet have.
Goodfire researcher Eric Bigelow says interpretability work outside the big labs depends for now on Chinese open models such as Kimi K3 at the top scale. He sees restrictions on all open models as the bigger threat.
Goodfire's Eric Bigelow argues in-context learning covers nearly everything a model does to adapt as it reads. He expects context, memory and harnesses, not per-user fine-tuning, to drive personalization, but hasn't researched continual learning.
Goodfire researcher Eric Bigelow resampled a GPT-3.5 model's reasoning at every token and saw answers collapse at single tokens, some arbitrary. He argues the "decision" happens in sampling; newer reasoning models often fork more smoothly.
Goodfire researcher Eric Bigelow said training that rewards only final answers gives models no reason to keep their reasoning readable. He said his optimism about monitoring that reasoning has faded sharply over about a year.
Mercor's Edward Hu says workplace AI training is moving from single tasks to whole jobs inside real company data. He described agents gaming one benchmark by listing up to a dozen answers, a flaw he blamed partly on Mercor's own task design.
Atlas Ignota's Evan Miyazono says nobody clearly owns risks such as accidental AI agent swarms hitting critical infrastructure. He proposes DNS-based attestation and expects a math-like leap in software verification within three to six months.
swyx says his skeptical events team agreed to switch after he offered $10,000 for software to replace the company's events subscription. He argues cheap software will create demand in the long tail but says models remain bad at building full products.
Sean Wang, known as swyx, says engineers who manage coding agents well are in more demand than ever. He says two or three of his own staff are under performance review for handing him low-quality output from Claude.
