AI scientists are agent systems that run parts of the research cycle on their own: reading literature, proposing hypotheses, writing and running experiments, and drafting papers. This page explains how they work, the mathematics underneath, and where they still break.
Research as a sequential decision problem: choose experiments \(e_t\) that buy the most information about hypotheses \(h_t\) per unit cost.
Most systems decompose science into six stages and wire LLM agents, tools and evaluators around them. Select a stage to see what the agent does there, the governing idea, and which systems emphasise it.
Each AI scientist is an engineering answer to a classical problem. These are the objects you should be able to write down in a qualifying exam.
The agent's "world model" is a posterior over hypotheses. Kosmos makes this explicit with a structured world model shared between its literature and data-analysis agents.
Optimal experimental design: run the experiment whose outcome you are most uncertain about, in the way that matters. Robotic labs approximate this with Bayesian optimisation.
UCT balances exploiting promising experiment branches against exploring new ones. AI Scientist-v2 replaces linear templates with a best-first agentic tree search over experiment nodes.
FunSearch and AlphaEvolve let an LLM mutate programs and keep those an automatic evaluator scores higher. The evaluator, not the LLM, guarantees correctness.
Google's AI co-scientist ranks hypotheses through pairwise simulated debates. Ratings converge to a quality ordering even when individual judgements are noisy (try it below).
When an agent tests thousands of hypotheses with prior plausibility \(\pi\), most "significant" results can be false. Automation raises throughput, not the prior.
If each step of a pipeline succeeds independently with probability \(p\), an \(n\)-step project succeeds with probability \(p^{n}\). A verifier that catches a fraction \(r\) of errors and allows one retry raises per-step success to \(p_{\text{eff}} = p + (1-p)\,r\,p\).
A deliberately simple independence model. Real errors correlate, which can make things better (shared fixes) or worse (systematic bias). Kosmos runs of ~200 agent rollouts show why checkpoints and traceable claims matter.
An AI scientist can screen far more hypotheses than a lab. Set the prior plausibility of its ideas, the statistical power of its tests and the threshold \(\alpha\), and see how many reported findings are real.
Eight hypotheses have hidden true quality (amber tick). A noisy judge compares random pairs; the winner follows a logistic model of the quality gap. Watch Elo ratings sort the ideas, the core loop of a generate–debate–evolve system.
Filter by what the system automates. Each entry links to its primary source.
A working ladder for reading claims critically. Click a level to see where today's systems sit. Most published systems operate at levels 2–3 with humans choosing the problem and validating the result.
Ideas that are incremental recombinations, or rediscoveries of published work the agent failed to retrieve. Automated novelty checks rely on search recall.
References that do not exist, or that do not support the claim attached to them. Every claim should trace to a source or a code cell.
The AI Scientist was observed editing its own code to extend a time limit. Agents optimise the metric they are given, including loopholes in it.
Wrong normalisation, leakage between train and test, or a plotting bug that flips a conclusion. Code runs; the science is wrong.
Fast iteration over analyses inflates false positives (section 04). Preregistration and held-out confirmation sets still apply.
Cheap papers shift cost onto reviewers. Venues increasingly require AI-use disclosure; check your target venue's policy before submitting.
Your progress is saved in this browser only.