Building a Research Loop: Architecture Notes from Early Development

At month four, two architectures had already failed in production. Our first used synchronous request-response calls to instruments and deadlocked when a device exceeded its expected return time. Polling fixed that case, but with three concurrent loops the poll queue became the limiting resource. The third design, now in use, made us define experiment state precisely.

These notes cover the decisions we kept and the reasons behind them. They are for builders orchestrating physical experiments, where each unit has its own timing, failure behavior, and result schema.

Why We Rebuilt Twice

Both early failures came from the same abstraction. We represented an experiment with a small enum, pending, running, completed, or failed, then modeled the loop as a linear state machine. That fails when only part of an experiment succeeds, such as selected wells in a plate reader run, or when analysis finds a calibration artifact and a completed result must be reconsidered.

A linear machine can handle those cases only through exceptions such as "completed-but-invalid" and "failed-awaiting-rerun," each requiring additional transition rules. Once we had six special states and 14 transitions, the abstraction was plainly wrong. We changed the model from state transitions to an event log.

Event Sourcing for Experiment State

For experiments, event sourcing means retaining the ordered events rather than only the latest state. We reconstruct the present by replaying that history.

Each experiment in our system owns an immutable event stream. It can contain ExperimentQueued with initial parameters, InstrumentAcquired identifying the device, RunStarted with its timestamp and calibration hash, ResultSegmentReceived for streamed partial output, RunCompleted, AnalysisStarted, AnalysisCompleted with the result payload, ResultFlagged after a failed quality check, and ResultInvalidated with its reason and downstream consequences.

We derive current state by replaying the stream. That lets us inspect an experiment exactly when its run began, before analysis, and identify when a quality flag was added and which rule added it. This is essential when debugging loop behavior or reviewing why the system proposed a particular next experiment, since the proposal uses the result state available at that moment.

Event sourcing increased the initial implementation work, but removed a whole class of debugging gaps. Previously, overwritten state often made "why is this experiment in this state" impossible to answer. With the complete history retained, the answer is recoverable.

Async Instrument Adapters

Integrated instruments expose very different interfaces. Newer systems may provide REST APIs. Older systems often use RS-232 through vendor software with proprietary queries. Others write output into a watched directory, expose a status endpoint for polling, send completion webhooks, or require a person to move a USB drive and confirm completion in the UI.

Our first adapters forced every device into synchronous request-response: call sendExperiment and receive a result. That held in the lab with one modern instrument and a proper async API, then broke at the first legacy integration.

Now each connector follows a shared interface while retaining its native mechanism: synchronous calls, polling, events, or file watching. The adapter translates that mechanism into a standard event stream consumed by the experiment log. To the loop, every instrument emits events; protocol differences stay inside the adapter layer.

Adapter development remains the slowest part of adding an instrument. Our quickest took half a day, for a plate reader with a clean REST API. The slowest required three weeks, for a mass spectrometer with a proprietary file format, an undocumented CSV column layout, and vendor software that had to stay open for export. Once complete, the adapter contains that complexity and the rest of the system remains unaware of it.

Hypothesis Versioning

The hypothesis layer stabilized last. Initially, one mutable object was updated whenever new results arrived, replacing its prior contents. That made proposal provenance impossible when we needed to explain an experiment, because the motivating hypothesis no longer existed in its original form.

We moved to append-only versions. Each version records a version number, the result set behind it, including experiment IDs and result hashes, the reasoning state, including surrogate-model parameters and acquisition-function settings, and the experiments it proposed. New results and a hypothesis update create another version.

This provides the provenance needed for reproducibility. For any queued experiment, we can recover the exact hypothesis version that motivated it and the evidence available at proposal time. That record supports debugging and lets a researcher examine the loop's reasoning instead of simply accepting or rejecting its output.

A Loop Cycle in the Data Model

In the data model, one cycle moves a batch of experiment IDs from "queued" to "running," emitting events as work begins. Results arrive, analysis processes each segment, and completion emits more events. The hypothesis layer ingests those results and creates a new version, emitting hypothesis-updated. The acquisition function then evaluates that version and emits proposals-generated with ranked candidates for the next batch.

We define cycle time as the interval from completion of the last experiment in one proposal batch to startup of the first experiment in the next. Analysis, hypothesis updating, and proposal generation occupy that interval and should be minimized. Excluding instrument runtime, our median is around three minutes for a standard batch of 8 to 12 experiments. Analysis usually dominates, especially when parsing and normalizing complex instrument output.

Stage-level latency tracking shows where each cycle spends time. Analysis is currently slowest; historically, proposal generation suffered when acquisition optimization covered large candidate sets. For batches above 200 candidates, we adopted approximate optimization, using a greedy DPP variant. In benchmarks, proposal latency fell by roughly 70% with negligible change in proposal quality.

What We Would Do Differently

Starting over, we would use event sourcing from day one instead of adding it after two failed designs. The initial investment is a few extra days of data modeling. Avoiding it would have saved two months of debugging consistency defects that stayed hidden until they produced incorrect proposals.

We would introduce hypothesis versioning sooner as well. A mutable hypothesis feels simpler, and remains simpler until an audited production run requires the reasoning state at proposal time and that state cannot be reconstructed.

We would keep the adapter pattern. Instrument protocols differ, and this boundary absorbs that variation in the right place. A good adapter has a real one-time cost. Without a clean separation between instrument communication and loop logic, as in our first design, each added instrument increases coupling and makes later changes more difficult.

These notes reflect month four, and we expect them to change. The event log has remained reliable under load, while adapters have worked across more instrument types than anticipated. Hypothesis versioning is newer, so its limits are still unknown. Our next likely redesign will focus on those limits.