The Explore-Exploit Tradeoff in Automated Science

Bayesian optimization starts with an objective, a finite evaluation budget, and an acquisition rule for choosing the next sample. Exploitation samples where the predicted value is highest; exploration samples where posterior uncertainty is greatest. Upper confidence bound, expected improvement, and Thompson sampling usually combine these pressures with different weighting schemes.

The formulation is mathematically tidy and performs well in simulation. It assumes, however, that every evaluation has the same price and that the budget is fixed. Wet-lab work violates both premises, so the policy must change with them.

Why Regret Minimization Is the Wrong Objective

For a multi-armed bandit, cumulative regret is the total difference between obtained reward and the reward available under perfect knowledge across all rounds. Minimizing it means identifying strong options quickly and exploiting them.

Wet-lab search has a different endpoint. We want conditions that meet a target criterion, then we want to stop. Reward is not accumulated indefinitely across cycles; experiments continue until confidence is sufficient to commit and advance the project. This is satisficing with a terminal condition, rather than cumulative reward optimization over a fixed horizon.

A regret-minimizing policy therefore exploits too heavily for many scientific searches. It keeps testing nearby variants of the current leader when resolving uncertainty elsewhere could be more valuable. That experiment may not improve the best observed result, but it can eliminate broad regions and let the team stop sooner.

The Cost Heterogeneity Problem

Textbook bandits assign the same price to every pull. Wet-lab experiments do not. A plate-reader viability assay may require $40 in consumables and an hour of technician time. An HPLC run with preparation costs $150 and three hours. Synthesizing and characterizing a new alloy composition may cost $1,200 and consume two days of fabrication before measurement.

Equal-cost treatment is poorly calibrated here. If two experiments offer comparable expected information gain, one priced at $50 and the other at $800, the cheaper option should win by a clear margin. Standard UCB has no cost term because textbook bandits provide no price difference to model.

We use an acquisition score that divides expected information gain by estimated experiment cost. During project setup, researchers provide an approximate price for each experiment type. The model is coarse but useful: when costs span an order of magnitude, it materially changes the ranking. That situation is common when discovery work pairs inexpensive screens with costly confirmation experiments.

How the Loop's Policy Diverges from RL Defaults

The three familiar Bayesian optimization choices are Upper Confidence Bound, Expected Improvement, and Thompson Sampling. Each was developed for i.i.d. observation noise and uniform evaluation cost. Under those conditions, all three have provable regret bounds and strong empirical results.

Our primary acquisition function is expected information gain, followed by cost normalization. For a proposed experiment, it is the expected decrease in entropy of the posterior over the hypothesis tree after a result arrives. This is not expected improvement over the current leader. It is the experiment's expected value in bits of uncertainty resolved.

The distinction is most consequential mid-project, after a promising region appears but much of the space remains untested. Expected improvement concentrates measurements around that region. Expected information gain may instead select a distant region with high uncertainty if its result could eliminate a large portion of the search space. For a satisficing goal with early stopping and sufficient confidence, that policy reaches the stopping condition sooner.

The Noise Structure Is Different

Standard bandits treat observation noise as independent and identically distributed across experiments. Wet-lab measurements satisfy neither condition.

Measurements are linked: runs from one batch and instrument share systematic error from both. If an instrument drifted 2 percent between Monday and Wednesday, results from that interval shift in the same direction. Noise is correlated within batches and over time, rather than i.i.d.

The distribution also changes. Instrument behavior drifts over months, reagent lots differ by supplier, and laboratory conditions vary seasonally. Identical experiments in January and June can disagree because the measurement environment changed, not because the underlying system did.

We represent batch correlation with a correction term in the covariance update and down-weight older observations over time during posterior computation. These approximations are imperfectly calibrated. Consequently, expected information gain is less certain than in textbook conditions, and the policy explores somewhat more aggressively than a well-calibrated model. We accept that bias: extra exploration costs experiments, but seldom produces a wrong answer.

Where the Policy Fails

Correlation-based search struggles most with a sharp, isolated optimum in a large parameter space. If no sampled region offers a gradient path toward the optimum, covariance-guided exploration will not approach it naturally.

This limitation is not unique to our policy. It follows from assuming correlation in the parameter space. When that assumption is wrong, automated search misses the optimum, and the researcher's knowledge of why the region matters becomes essential.

We observed this in two early-access projects. In each, researchers pinned experiments manually to force sampling in a region the model had neglected. The loop located the optimum within two cycles of the pin. Without that intervention, we think the budget would have run out first.

That is why the policy accepts researcher overrides. The explore-exploit acquisition function covers the usual case, while physical intuition unavailable to the model needs a route into the loop. Reordering the ranked list is therefore an intentional input channel, not a workaround. In both projects, a manual pin preceded the optimum by two cycles.