Reading HPLC Data: How We Built the Chromatography Integration

How can a lab normalize HPLC results across instruments from different vendors? We built this integration for pharmaceutical development, process chemistry, and quality control, where HPLC is common but exported data is difficult to parse consistently.

An HPLC run yields a chromatogram: detector response over time, with peaks at compound-specific retention times. Peak integration tables, baseline annotations, method settings, and system suitability results add further structure. Vendors arrange each layer differently. Plate reader exports are usually text, but some HPLC systems provide binary files that only proprietary software can read.

We spent roughly three months building the HPLC pipeline before its outputs were dependable. This note explains the design and the decisions that proved less straightforward than expected.

The Vendor Landscape

Our early-access users most often bring us Agilent, Waters, and Shimadzu data, roughly in that order.

Agilent ChemStation and OpenLAB CDS support several export types. We most commonly parse the .asc report: plain text with a fixed header followed by peak rows. Its header includes method name, instrument identifier, sample name, and injection settings. Rows record retention time, area, height, width, and, when calibration data exists, compound name.

Waters custom reports use .arw files: tab-delimited text with a proprietary header block. The same field may appear as "Ret. Time" in one template and "RT" in another, so the parser must inspect surrounding fields. Our Waters alias table currently contains 23 retention-time variants across the versions we have seen.

Shimadzu LabSolutions is harder to ingest. Its native .lcd output is binary and undocumented. We therefore rely on the built-in ASCII export, a verbose but structured text representation, rather than decoding .lcd ourselves. Shimadzu users must perform that export before Discovery Loop receives the data, an unresolved integration friction point.

Stage One: Format Parsing

The first stage identifies the source and selects an adapter. Because .asc can represent exports from multiple vendors, we inspect the first 200 bytes and compare header signatures before routing the file.

Every adapter emits one internal model: a RunRecord containing instrument identifier, run timestamp, sample identifier, method name, a list of PeakRecord objects, and the complete unparsed source in raw. Each PeakRecord stores retention time, area, height, width, optional compound name, and a manual-integration flag.

The manual flag is important. Operators often redraw boundaries when automated integration splits a peak or misses a shoulder. That judgment can improve one result while adding person-dependent variation across the series. We pass the flag to the hypothesis engine, which can weight those observations differently under researcher-defined settings.

Stage Two: Baseline Correction Accounting

HPLC baselines commonly slope or drift. Column bleed, gradient-solvent UV absorbance, lamp aging, and the column's equilibration state can all contribute. Integrating against that moving baseline biases area relative to integration against a flat one.

Instrument software usually corrects the baseline before writing the peak table, but algorithms differ by vendor and version. Agilent defaults to a modified Rubberband method, while Waters software uses Asymmetric Least Squares with configurable smoothing. For broad, asymmetric peaks, the two approaches can differ in area by 5 to 15 percent.

We do not apply a second correction to exported peak tables. In most workflows the raw detector trace is unavailable, and silently reprocessing an already processed result would obscure provenance. Instead, we compare method parameters and flag runs in one series that used materially different baseline settings. The flag lives on RunRecord and appears in the UI, where the researcher checks area comparability before the data reaches the hypothesis engine.

Stage Three: Retention Time Normalization

Retention drift is the hardest aggregation issue. Column aging, solvent-lot variation, and temperature changes gradually move compound peaks. A maintained system may drift under 0.1 minutes over 50 injections, while a high-throughput setup can exceed 0.5 minutes within a few weeks.

When Discovery Loop combines a run series, it must associate peaks with compounds. A compound at 8.2 minutes in run 1 and 8.7 minutes in run 30 can defeat an exact lookup, or cause run 30 to be assigned to another compound eluting near 8.7 minutes.

We normalize retention time against reference compounds. The researcher selects stable, well-resolved peaks present in every run and expected not to respond to the variable under study. From their observed and expected positions, we calculate a per-run linear or piecewise-linear correction. The same transform moves other peaks before compound matching.

This handles uniform column drift, not selectivity changes from serious degradation or major method revisions. Those cases need a more involved model. It also depends on identifying each reference in every run, so anchors must remain well separated from neighboring peaks throughout the series.

What Breaks the Pipeline

Co-elution is the usual failure mode. Nearly coincident compounds may be merged, inconsistently split, or represented only by the more abundant species. Such outputs cannot be safely aggregated or treated as meaningful evidence by the hypothesis engine.

We flag possible co-elution using unusual width-to-height ratios, pronounced tailing consistent with an unresolved shoulder, and inconsistent peak presence within a retention-time window across runs. These peaks bypass automatic hypothesis updates and go to researcher review. The researcher can confirm the assignment and remove the exclusion.

System suitability failure is another frequent issue. Regulated methods commonly require a minimum theoretical plate count, a maximum tailing factor, and adequate resolution for a reference standard. These checks identify runs whose instrument performance is outside specification. We read the suitability result from the sequence header and quarantine every result from a failed sequence. Quarantined data stays out of hypothesis updates until the researcher confirms the quarantine or overrides it with a documented rationale.

What We Are Still Working On

Shimadzu and several other vendors still leave us with a binary-format problem. Shimadzu .lcd files and Bruker raw MS data need vendor export steps before ingestion. We are discussing programmatic run-data API access with one vendor. Until that is available, the extra export remains a friction point.

Column lifecycle tracking is the other open problem. Across hundreds of injections, aging changes retention, broadens peaks, and lowers efficiency. A Discovery Loop project that runs for months is effectively working with a different column in month four than in month one. Our normalization captures systematic drift while the column is stable, but not progressive aging as degradation. A column-age model is on the roadmap. For now, we recommend starting a new loop project after a column change instead of carrying prior anchors across the replacement.