Parsing Lab Instrument Output at Arrival

CSV with one stable schema failed immediately. In three months, we added adapters for eleven file formats across six instrument classes and rebuilt the parser twice.

Every vendor defines a proprietary instrument format. That is the central infrastructure barrier to interoperable lab data, not a small inconvenience. It was our first major engineering problem after the hypothesis engine functioned.

Why Standards Failed

Agilent, Waters, Shimadzu, Thermo Fisher, BioTek, and PerkinElmer developed acquisition software internally over decades. None had much reason to align output schemas with competitors. The result: several formats per instrument class, plus version changes within vendors across software releases.

For HPLC, Agilent ChemStation writes .asc files with a custom header block, Waters writes tab-delimited .arw files with unusual metadata, and Shimadzu LabSolutions writes binary .lcd files that need a separate ASCII export before text parsing. BioTek plate readers write .xlsx files with data beginning on row 8 after instrument metadata. A PerkinElmer Victor writes CSV with timestamps in column one and a different well-position layout. The schemas share no field naming convention.

Two Parsing Layers

We split parsing into two layers. The outer adapter handles an instrument class and vendor, reading file structure, finding metadata, extracting rows, and returning a normalized object. The inner normalizer maps vendor fields onto our common schema.

Our schema is deliberately small. Its eight required fields are result identifier, run timestamp, instrument class, instrument identifier, sample identifier, signal value, signal unit, and parameter block. The block stores run conditions as key-value data, varying by instrument. HPLC entries include solvent gradient, column identifier, flow rate, and column temperature. Plate reader entries include wavelength, gain setting, and well layout reference.

The schema does not depend on the measurement type. Fluorescence plate reader and mass spectrometer results enter the hypothesis engine identically. It requires the numerical result and the conditions that generated it, not an interpretation of measurement physics.

Spectroscopy Adapters

UV-Vis exports are simple in structure but not automatically comparable. Most contain wavelength and absorbance columns, while range and step size differ by instrument and method. A different range from the previous run can make a naive comparison report signal changes that are only measurement artifacts.

Before schema insertion, our UV-Vis adapter interpolates each spectrum onto a canonical wavelength grid. Resolution decreases somewhat, but spectra from different instruments and methods become comparable, enabling valid hypothesis-engine updates across hardware. Changing spectrometers mid-project should not erase cycle history because the grid changed.

Fluorescence requires paired excitation and emission conditions. On one sample, 485 nm excitation with 528 nm emission is a different measurement from 485 nm excitation with 535 nm emission. We store both values in the parameter block and separate wavelength pairs into distinct series unless the researcher links them.

Plate Reader Normalization

Plate readers generate the most data among our current users. Biochemical assays commonly return fixed-wavelength absorbance, fluorescence intensity, or luminescence counts, one value per well in a 96- or 384-well plate.

Well layout is the parsing problem. Labs place samples, controls, and blanks according to local conventions. Some reserve the first two columns for replicated controls, some use the outer ring, and some keep a separate layout file for each run. Without the map, the parser cannot separate sample signal from background blank.

We keep a layout reference outside the instrument export. Researchers define standard plate maps once; at parse time, the parser loads the map and labels wells before normalization. This adds setup work, but omitting it creates silently incorrect results, a worse outcome than the extra configuration.

Why Parse Time Matters

Parse time means processing results as they arrive instead of in a nightly batch. Each new result triggers an incremental posterior update and a revised experiment ranking. If parsing waits for hours, the morning ranking may exclude a run finished the previous night.

This matters for short overnight assays. A run completed at 11 PM can revise the experiment queue by 11:01 PM. The next morning, the researcher has a ranked run ready to confirm. Parsing at 8 AM adds eight hours to the cycle without a technical need.

Our current latency is under 90 seconds for every supported format when files are below 100 MB. Larger files, mainly from imaging instruments, enter an asynchronous queue. The hypothesis engine updates after parsing finishes, rather than at upload.

What Stays Manual

Some instrument outputs remain outside automatic parsing, especially raw mass spectrometry data. Thermo .raw, Bruker .d, and Waters .raw are complex binary formats. Useful extraction requires spectral deconvolution and database matching. We accept processed MS exports only: CSV peak lists containing mass, intensity, and optionally retention time.

We chose this boundary because automated MS results vary with deconvolution parameters. We do not want hidden preprocessing decisions to alter findings. The researcher processing the MS data owns those choices. Discovery Loop reads that processed output, not the raw acquisition file.

This principle applies throughout our integration stack. We read existing instrument output without changing the measurement chain or its meaning. We add value at parse time; upstream signal-processing choices remain with the researcher and instrument software.