Three services, one lot, and a spread of two and a half points

The headline finding is that within-laboratory repeatability was good in every case and between-laboratory reproducibility was not, and that the entire gap is explained by disclosed method choices rather than by anything hidden. Each service answered the question it was asked using the method it publishes. Each answer is defensible. The spread between them is nonetheless larger than the differences vendors compete on in their marketing, which means the number a buyer is comparing is not a property of the material.

What each service can be asked for

Purity by reversed-phase chromatography is available from all three assay services and is the commodity product of this market. Identity by intact mass is available from all three, with instrument class varying and rarely specified in the cheapest tier. Peptide content by elemental nitrogen determination is available from two, and is the analysis this department considers most under-purchased relative to its usefulness; the compendial route by quantitative amino-acid analysis is the older and more laborious alternative.¹

Water content, counter-ion determination and residual solvent are available on request from at least one service and are seldom ordered. Bacterial endotoxin testing sits outside all four as a routine offering and requires a laboratory equipped for it. Sterility testing is not offered by any of the four, takes fourteen days of incubation and cannot be compressed. Sequence confirmation by peptide mapping is available by arrangement from one service and has been ordered, on the Journal’s enquiries, a handful of times.

The pattern is that the market can buy what it asks for and asks almost exclusively for the cheapest test. The specification framework for a biotechnological article assumes a coordinated panel of tests whose members answer different questions.² What exists here is a single test purchased in isolation, which is not a failure of the services but a description of demand.

Custody begins at the reception desk

What a laboratory can record about a submitted sample begins at the moment of receipt, and the better reports in this market record a surprising amount: the date and time of arrival, the condition of the outer packaging, whether cold-chain materials were present and still cold, whether the vial seal was intact, whether the crimp showed evidence of having been lifted, and the appearance of the cake. Two of the four services record most of this as standard. The others record some of it.

These lines are worth more than they look. A vial arriving with a collapsed cake or a discoloured plug is telling a reader about storage and transit history that no purity figure captures, and a report that notes it has converted an unknown into a datum. Condition on arrival is also the only point in the entire chain at which anybody independent observes the material’s physical state, and it is therefore the only opportunity to catch a transit problem before it is analysed away.

What no laboratory can record is what happened before the package was posted. The submitter’s storage, the route from vendor to submitter, the number of times the vial changed hands, the ambient temperature of a fortnight in a sorting facility: all of it is prior to custody and none of it is knowable. The Journal accordingly cites third-party purity figures with the submission date and, where we know it, the submitter, because those two facts bound what the figure can support.

The submitter pays, and the submitter decides

Reduce the incentive problem to its mechanism and it is very simple. The party who pays for a test receives the report. The report is that party’s property. Nothing obliges them to publish it. Therefore the set of third-party results visible in this market is not the set of results obtained; it is the subset that somebody with a commercial interest chose to make visible.

The size of the resulting distortion is unknown and unknowable from published data, which is precisely the difficulty. If a vendor submits twelve batches over a year and publishes eight, the four unpublished results are invisible, and no amount of scrutiny applied to the eight will recover them. This is the mechanism the research literature calls publication bias, and it is well characterised: the distortion it produces is larger when the number of tests is small, when the cost of a test is high relative to the value of a favourable result, and when nobody records that a test was commissioned.

The accreditation standard for testing laboratories requires a laboratory to identify and manage risks to its impartiality arising from its commercial relationships, and to be able to demonstrate that it has done so.³ That obligation sits on the laboratory and it is the right place for it. What the standard cannot reach is the client’s filing cabinet, and the filing cabinet is where this problem lives.

When a vendor commissions testing of its own product

The clearest case of the conflict is also the most common. A vendor wishing to advertise independent verification commissions a test, selects the sample, pays the invoice, receives the report and publishes it. Each of those steps is ordinary commerce and none of them is improper. Taken together they mean the verification of a product has been arranged and funded by the party selling it, which is a conflict of interest in the plain sense of the term.

It is worth being exact about what the conflict does and does not imply. It does not imply that the laboratory’s measurement is wrong; there is no reason to think it is, and the laboratory has strong reputational reasons for accuracy. It does not imply that the vendor is dishonest. What it implies is that the result is not independent of the vendor’s interests in the way the word independent suggests, because the vendor controlled the sample, the timing and the disclosure.

The comparable arrangement in other industries — a manufacturer paying an accredited laboratory to test its own product — is normal and is managed by requiring that results be reported to a regulator or a purchaser with enforcement power, so that disclosure is not optional. Neither exists here. The Journal therefore labels vendor-commissioned results as such wherever it cites them, which is the only lever a publication has.

The blind duplicate exercise

Twelve vials, one lot, purchased at retail without disclosure of purpose. Two vials were sent to each of the three assay services under two different submitter names and addresses, so that each laboratory received two nominally unrelated submissions of the same material some three weeks apart. Six further vials were retained. Each service was asked for its standard purity determination at its standard price and turnaround, with no special instructions.

The design tests two distinct quantities that the trade conflates. Repeatability is the agreement between duplicate determinations within one laboratory; reproducibility is the agreement between laboratories. Interlaboratory studies in analytical chemistry consistently find the second to be substantially worse than the first, and the variance decomposition that separates them is standard methodology.⁴ The distinction between repeatability and intermediate precision is formalised in the validation guidance,⁵ and multi-site studies in adjacent fields have repeatedly found between-laboratory agreement on identical samples to be the harder problem.⁶

All three services were informed after the fact, before publication, and each was given the opportunity to comment on its own method as printed and on the comparison as a whole. All three responded. Two supplied additional method detail that has been incorporated. One disputed the framing of the comparison, and its objection is printed in the correspondence below. None of the three asked for its result to be withheld, which the Journal records because it did not have to be that way.

What came back

Within-laboratory repeatability was good. The two determinations from each service agreed to within 0.3 percentage points in every case, and to within 0.1 in one, which is about what a well-controlled chromatographic method should deliver on duplicate material and is a genuinely reassuring result.

Between-laboratory reproducibility was another matter. The three services returned figures spanning 2.1 percentage points on material from one lot. Every point of that spread is accounted for by disclosed method differences: gradient duration, integration threshold, the retention-time cut-off defining the solvent front, and whether an orthogonal second gradient was run and the lower figure reported. Rerun the raw data from the shallowest method with the fastest method’s integration threshold and the two figures converge to within 0.4 points, which is the strongest available demonstration that the disagreement is methodological rather than analytical.

Identity results agreed completely: all three found a single dominant species at the expected mass, and none reported evidence of an unrelated compound, which is the ordinary outcome of intact-mass confirmation on submitted material.⁷ The two services reporting peptide content returned 93% and 91% of label, a difference within the stated uncertainty of nitrogen determination. The material, in short, was what it claimed to be, and the disagreement was confined to the second significant figure of the number the market competes on.⁸

What this exercise cannot establish

Four limitations, stated because the alternative is letting readers over-read a small study. First, one lot of one compound from one supplier is not a sample from which the performance of these services in general can be inferred; it is an existence proof about method-driven spread. Second, three services is too few for any statistical treatment beyond the descriptive; published round-robin studies of peptide purity use seven or more participants for exactly that reason.⁹

Third, and most important, the exercise tested reproducibility, not accuracy. All three could be equally wrong: without a certified reference standard of known purity, there is no true value against which to score them, and the compendial approach to validating a purity procedure requires exactly such a reference to establish accuracy rather than mere agreement.¹⁰ What we measured is dispersion around an unknown centre, and the same constraint applies to any quantitation attempted without a matched standard.¹¹

Fourth, blind submission tests a laboratory’s ordinary process, which is the point, but it also means we bought the cheapest standard product from each service rather than the most thorough. A comparison of each service’s best available package would be a different and probably more flattering study, and it would tell a buyer less, because almost nobody buys the best available package.

The Journal will repeat the exercise annually with a different compound and, funding permitting, against a certified reference standard. The design is published so that others can run it.

Four changes, none requiring a regulator

First, publication of submissions rather than only of results: the date, the vendor as named by the submitter, and the compound, with the result confidential where the client requires it. This defeats most of the selection effect and costs nothing.

Second, submitter type on every report — vendor, buyer, publication or reseller — which the laboratory knows and which determines what the result can support. Third, lot-linked badges carrying a lot number, a date and a link, so that a verification claim expires with the lot it describes. Fourth, gradient and integration threshold printed on every purity report, which is the only way two figures from different services can be compared at all.

None of the four requires a regulator, new legislation, or any change in analytical practice. Three of them require a laboratory to print information it already holds; the fourth requires a vendor to accept that a badge should perish. The Journal has put all four to each of the services covered here. Responses have been mixed and mostly constructive, and are printed in our correspondence pages as they arrive. We will publish an annual note on which have been adopted, because the alternative is making the same request indefinitely without recording the answer.

Our own position, stated in full

Janoshik Analytical and PeptideMeter both advertise in The Compound Journal. Both relationships are disclosed by name on our funding page, together with every other sponsor. No advertiser sees editorial copy before publication, no advertiser has any role in commissioning or reviewing coverage, and the analytical-chemistry desk is contractually barred from consulting for any vendor, testing service or compounding pharmacy. This article was edited by the standards desk under the same rules as every other piece in the department.

The Journal also pays these services. We have submitted samples to three of the four on commercial terms, at list prices, and the blind duplicate exercise described above was funded from editorial budget. We are therefore simultaneously a customer of the institutions we are reporting on and a recipient of advertising revenue from two of them. Readers are entitled to weigh that, and the only useful response we can offer is to state it plainly and to publish objections.

Our position on the substance is unchanged by any of it. All four services are legitimate operations and we have no evidence of dishonesty by any of them. The problems this article describes are structural — who commissions testing, who decides what is published, and what a sample can support about a batch — and they would persist unchanged if every person working at all four organisations were beyond reproach. Correspondence to standards@compoundjournal.com.

What the Journal thinks these services are worth

A summary judgement, since a critical article of this length invites the inference that we think the sector is worthless. We do not. Independent testing in this market is the only mechanism by which a buyer can obtain information about material that is not supplied by the party selling it, and its existence is the difference between a market with some evidence in it and a market with none. Several of the reports these services produce are better documents than the manufacturer certificates they are checking, which is a low bar cleared with room to spare.

The three criticisms we would press are narrow. A sample is not a batch, and the trade cites samples as batches. The party paying for a test decides whether anybody sees it, and the visible corpus is therefore selected. And a badge on a listing has dropped every particular a reader would need. None of these is an analytical failing and none is a failing of the services in isolation; the second and third are properties of the market that surrounds them.

What we would tell a reader is this. A third-party report on a vial you selected and posted yourself is strong evidence about that vial. A third-party report published by the vendor is weaker evidence, of an amount you cannot determine. A badge is not evidence. And nothing in any of the three is a statement about whether anybody should administer the contents to anything.

The Journal’s summary of this sector is that it is better than the market deserves and weaker than the market believes. Three laboratories and an auditor, selling individual tests mostly by post, mostly to private individuals, produce documents that are frequently better than the manufacturer certificates they check. What they cannot produce, because no institution in this market can, is a result that the party being examined does not control the disclosure of.