Erol

Posted on Aug 28

Reverse engineering the Agilent ChemStation .ch and .uv file formats

#reverseengineering #javascript #opensource #binary

Agilent ChemStation writes chromatography data into a .D directory of binary files. The vendor has never published the layout. Nothing in the files is encrypted and there is no licence check anywhere in the format, so they can be read offline, but you have to work the layout out first.

What follows is what is in them, from writing a reader and checking it against real files from six instruments.

File structure

A .D is a directory, not a file. The signal data is in files inside it:

.ch: one detector channel over time. A single column of numbers.
.uv: diode array data. A full spectrum at every time point.

Both start with a number identifying the layout. Read it first and branch on it. Do not branch on the extension, since the same extension covers layouts with almost nothing in common.

Type	Ext	Header	Body
30	`.ch`	`0x400`, one byte per char	deltas, big endian
130	`.ch`	`0x1800`, UTF-16	deltas, big endian
179	`.ch`	`0x1800`, UTF-16	float64, little endian
31	`.uv`	`0x200`, one byte per char	deltas, little endian
131	`.uv`	`0x1000`, UTF-16	deltas, or float64. Both occur.

An unrecognised type should be rejected by number, not routed to the nearest matching path. A wrong chromatogram looks plausible on inspection and will get used.

Delta encoding

Types 30, 130, 31 and 131 store changes rather than values. Each segment begins with a count, followed by that many signed 16-bit values, each added to a running total. A value of -0x8000 is an escape: the next four bytes are a new absolute total.

let acc = 0;
for (let i = 0; i < count; i++) {
  const d = view.getInt16(p, false); // big endian for .ch
  p += 2;
  if (d === -0x8000) {
    acc = view.getInt32(p, false);   // escape: new absolute total
    p += 4;
  } else {
    acc += d;
  }
  out.push(acc * scale);
}

Cost is 16 bits per point, and 32 only where the signal jumps. The file above uses the escape 22 times across 2,100 points.

Byte order

.ch stores these values big endian. .uv stores them little endian.

The wrong byte order does not throw. The segment walk loses alignment, reaches a byte pair that reads as the escape, and terminates:

The result is hard to distinguish from a run where acquisition stopped early.

Undocumented offsets

Four values are not in any published description. The rainbow project published the notes this work started from, and two of its values do not match files in the wild.

The .uv scaling factor is at 0xC0D, not 0x127C. The .uv header ends at 0x1000, so the documented address falls inside the data body:

This one can be ruled out without a sample file. The correct value sits at 0xC0D, immediately before the units string that the same document places correctly at 0xC15.

Legacy string layout. Types 30 and 31 are not a truncated modern header. Strings sit at unrelated offsets and use one byte per character: acquisition date at 0xB2, units at 0x244.

Legacy scaling factors. 0x284 for .ch type 30, 0x13E for .uv type 31.

OpenLab .uv segments. Label 70 with a plain little endian float64 body, rather than the documented label 67 with a delta encoded body. Both occur and both have to be handled.

Verification

A parser cannot be tested against the author's own reading of the format, since that reading is what is under test. The check has to be external.

Every decoded value is compared against rainbow's output across 11 files from six instruments, covering every supported type: 1,487,470 values. All match, at a maximum difference of zero rather than inside a tolerance.

A second test re-derives each reported offset independently, so a claim about where a field lives holds up without sharing a constant with the parser. Together these catch the usual failure in this kind of work, which is an offset that is correct only for the sample set in hand.

Known gaps

Type 181 is implemented from the layout it shares with 179 and has never been checked against a real file, because no public sample exists.

.ch retention times are reconstructed, not read. The header carries a first time, a last time and a point count, and times are an even ramp between them, which is all the format provides. A non-uniform sample rate would be reported wrongly by any reader. Every file examined so far is uniform.

.uv carries a timestamp per time point, so those are read rather than inferred.

DEV Community