Parallel Experiment Queues: Scheduling for Higher Information Gain

Most queues run experiments serially, but parallel runs collect evidence sooner. Our first queue was one ordered list: the loop ranked experiments, the researcher sent each to a free instrument, and results returned in arbitrary order. This worked with one instrument type. It broke when a lab used three instrument classes with very different cycle times, because the queue could not account for them.

Scheduling research across several instruments is difficult for structural reasons, not mainly computational ones. Instruments differ in throughput, failure behavior, reagent lead times, and dependence on one another's outputs. A naive queue assumes experiments are interchangeable. They are not.

Multiple Instrument Reality

A mid-size discovery lab may use four instrument classes in one week: plate readers for high-throughput assays in 96 or 384-well formats, with cycle times measured in hours; HPLC for purity and yield measurements, with lower throughput and 15 to 60 minute runs per sample; an NMR instrument shared across projects, scheduled by appointment, handling one sample at a time for 30 minutes to several hours per sample; and computational simulation on a cluster, with runtime varying according to the CPU queue.

These four classes have different operating timescales and availability rules. Without that context, the loop can produce a technically valid priority list that no researcher can use. The highest-ranked experiment may need a reagent requiring three days of preparation, or the HPLC may be booked through Thursday.

Our Priority Model

In Discovery Loop's queue model, each candidate has four scheduling fields in addition to its information-gain score: instrument class, estimated duration, reagent lead time, meaning days until materials are ready, and dependency flags showing whether another queued experiment must finish first.

The scheduler makes two passes. First, it checks feasibility and marks candidates blocked when reagent lead times or dependencies remain unresolved. Next, it ranks the feasible set with a composite score balancing information gain and estimated lead time. A high-information experiment that can start now outranks one with slightly greater information gain but a three-day reagent preparation window.

This seems obvious now, but it took us three iterations. Initially, we treated lead time as descriptive metadata instead of a scheduling constraint. Researchers accepted the priority order, then spent 15 minutes rearranging it around practical limits the system could not see. That manual edit showed us the queue model was incomplete.

Failure Modes by Instrument

Instrument classes fail differently, so the queue must represent those differences. Plate reader failures are uncommon, but a failure means requeuing the entire well plate rather than one well. HPLC drift accumulates across long sequences, producing baseline artifacts and requiring recalibration after roughly every 20 to 30 samples, depending on the matrix. A run in slot 28 of a 30-run sequence may therefore receive a recalibration flag. NMR shimming can drift as room temperature changes, especially in older magnets.

For every instrument class, we keep a failure-mode table with failure type, whether transient, systematic, or catastrophic, mean time between failures for that category, and the recovery action: immediate retry, requeue after cooldown, or human review. Transient faults cause automatic requeue. Systematic faults prompt recalibration before the next batch. Catastrophic faults block all queued work for that instrument and raise an alert.

Retrying automatically has a trap: if the cause remains, we waste time and reagents repeating a bad experiment. Before an immediate retry, our logic checks whether the failure signature matches a known transient or resembles a pattern preceding systematic failure. Most transient faults pass through this heuristic quickly. For ambiguous signatures, we use a short cooldown instead of requeuing immediately.

Cross-Cycle Dependencies

In a closed loop, the hardest scheduling problems involve dependencies crossing cycle boundaries. For some workflows, experiment A in cycle 3 determines which version of experiment B enters cycle 4. If they use different instrument classes with different turnaround times, that dependency imposes a wait and can hold up the whole cycle.

We store these relationships in a directed graph beside the queue. Each experiment node lists upstream requirements, the results needed before scheduling, and downstream consumers waiting for its output. When a result arrives, the system resolves blocked consumers and moves them into the feasible pool for the next scheduling pass.

Usually, a single cycle's graph has no more than two or three dependency levels. A deeper chain generally indicates that the experimental design needs restructuring, not that the scheduler needs extra machinery. When we find five or more levels, we show the researcher a design note: the workflow has a long critical path that will limit cycle time regardless of scheduling efficiency.

Parallel Batch Planning

For labs combining plate assays with slower analytical instruments, we often split scheduling into two tracks. The fast track handles high-throughput work, such as plate-reader assays and computational screening. It runs continuously and returns results in short cycles of 4 to 8 hours. The slow track handles confirmatory HPLC and NMR work, selecting leading candidates from the fast track for detailed characterization on a cycle of 1 to 3 days.

This arrangement preserves a fast information cycle for inexpensive experiments while reserving slow instruments for the strongest candidates. A single queue that treats every experiment equally can starve slow instruments because fast work keeps arriving, or suppress fast work by assigning too much weight to slow confirmatory measurements.

Limits of the Current Design

Our current model handles some scheduling cases poorly. Shared instrument booking is the clearest example: a core facility controls the time externally, rather than the loop. When availability comes from an outside scheduling system, we cannot match queue priority to real slots. We can only provide a preferred order for the researcher to map onto available time.

We also have limited visibility into reagent shortages before they block the queue. Lead time is currently a static field, so the model does not monitor stock or warn that a high-priority experiment's reagent is running low. That data lives in the lab inventory system. Inventory integration is on our roadmap, but it needs a deeper connection than most labs have established with us so far.

The queue is effective at choosing among feasible experiments. It is less effective at managing the conditions that create feasibility. That is the next gap we are addressing.