A tested vial is evidence about a vial

Every independent test report in this market shares a structural limitation, and it has nothing to do with the laboratory. The report describes a sample that somebody selected and posted. Who selected it, how it was chosen from the batch, how it travelled, and at what temperature it spent the intervening fortnight are all outside the laboratory’s knowledge and outside its records. In a regulated setting the chain of custody begins at sampling, with a documented procedure establishing that the portion tested is representative of the whole. Here, in almost every case, it begins when a padded envelope reaches a reception desk.

Four organisations, three functions

The verification layer in this market consists of four named organisations and a long tail of contract laboratories that occasionally receive private submissions. The four are Janoshik Analytical, based in the Czech Republic; Medutest; PeptideMeter; and VendorInvestigate. Their prominence is a function of accessibility rather than scale: each accepts samples or enquiries from private individuals, which most accredited contract laboratories will not do, and each publishes in a format the trade can circulate.

They perform three distinguishable functions. Assay is the measurement of a submitted sample — purity by chromatography, identity by mass, content by nitrogen determination. Audit is the examination of a vendor’s documentation, provenance claims and practice, without necessarily touching the material. Archive is the maintenance of a searchable record of prior results, which converts individual reports into a series.

Three of the four perform assay work. All four perform some version of audit. Two maintain a public archive. No single organisation does all three comprehensively, and the trade’s habit of treating a mention of any of them as equivalent verification obscures a distinction that determines what the mention supports. Janoshik and PeptideMeter advertise in this publication; the relationship is disclosed on our funding page and has no bearing on this coverage.

Janoshik Analytical

Janoshik Analytical is a Czech laboratory and the service whose reports circulate most widely in this market. Its core offering is chromatographic purity determination and peptide content by elemental nitrogen determination, with mass-spectrometric identity confirmation available, on samples submitted by post. Reports are issued to the submitter and carry the method, the gradient and integration detail, which places them at the more informative end of what this market produces.

Janoshik advertises in this publication. The relationship is disclosed by name on our funding page, no advertiser sees editorial copy before publication, and this section was written and edited under the same rules as every other in this department. Readers who consider the arrangement disqualifying are entitled to say so and several have; the correspondence is printed.

The Journal’s substantive observations are two. First, the practice of reporting content alongside purity is the single most useful thing any service in this market does, because content is the number that determines how much peptide a nominal mass represents and almost nobody else reports it. Second, the reports describe submitted samples, they say so, and the trade nonetheless quotes them as batch verification. That second point is a criticism of citation practice rather than of the laboratory, and it applies to every service in this article.

The sample is not the batch

The most important sentence in every third-party report in this market is the one identifying what was tested, and it always says the same thing: the sample submitted. That phrasing is precise and correct, and the entire trade reads past it. A result obtained on one vial extends to a batch only if the vial is representative, and representativeness is a property of how the vial was selected, not of how carefully it was analysed.

In regulated practice, sampling is a controlled activity in its own right: the accreditation standard treats it as part of the laboratory activity, requiring a documented sampling plan and records of how the portion tested was obtained.¹ Where a laboratory receives a sample it did not draw, the standard expects the report to make clear that the results apply to the sample as received. All four services do this. The market quotes them anyway as though the batch had been tested, which inverts the pharmacopoeial convention that a result on a sample is evidence about a batch only under a stated sampling assumption.²

The practical significance depends on the fill. A batch filled in a single session from a homogeneous bulk solution is likely to be uniform, and a single vial is decent evidence about it. A batch assembled from subdivided bulk, filled across sessions, or blended from more than one synthesis is not, and a single vial is evidence about a vial. Nothing on a report tells a reader which situation applies, because the laboratory does not know either.

Custody begins at the reception desk

What a laboratory can record about a submitted sample begins at the moment of receipt, and the better reports in this market record a surprising amount: the date and time of arrival, the condition of the outer packaging, whether cold-chain materials were present and still cold, whether the vial seal was intact, whether the crimp showed evidence of having been lifted, and the appearance of the cake. Two of the four services record most of this as standard. The others record some of it.

These lines are worth more than they look. A vial arriving with a collapsed cake or a discoloured plug is telling a reader about storage and transit history that no purity figure captures, and a report that notes it has converted an unknown into a datum. Condition on arrival is also the only point in the entire chain at which anybody independent observes the material’s physical state, and it is therefore the only opportunity to catch a transit problem before it is analysed away.

What no laboratory can record is what happened before the package was posted. The submitter’s storage, the route from vendor to submitter, the number of times the vial changed hands, the ambient temperature of a fortnight in a sorting facility: all of it is prior to custody and none of it is knowable. The Journal accordingly cites third-party purity figures with the submission date and, where we know it, the submitter, because those two facts bound what the figure can support.

Who chose the vial

The identity of the submitter is the single most informative piece of context attached to any third-party report, and it is the piece most often missing when a report is quoted. A test commissioned by a buyer on a vial bought anonymously at retail is a different evidential object from a test commissioned by the vendor on a vial it selected from its own stock. Both are legitimate. They support different inferences, and the difference is not small.

In the buyer-submitted case the vial entered the postal system as an ordinary retail unit and its selection was outside the vendor’s control, which is exactly the property that makes the result informative about what a customer receives. In the vendor-submitted case the vendor chose the vial, knew it was being tested, and will decide whether the result is published. Nothing improper need occur for the second to produce a more flattering picture than the first, and no laboratory can distinguish them from the bench.

The Journal’s practice is to state the submitter type wherever it cites a figure, and to decline to cite a figure where the submitter is unknown. This is more restrictive than the market’s convention and it means we publish fewer numbers. It also means that when we say a result came from an anonymous retail purchase, that claim is doing work.

When a vendor commissions testing of its own product

The clearest case of the conflict is also the most common. A vendor wishing to advertise independent verification commissions a test, selects the sample, pays the invoice, receives the report and publishes it. Each of those steps is ordinary commerce and none of them is improper. Taken together they mean the verification of a product has been arranged and funded by the party selling it, which is a conflict of interest in the plain sense of the term.

It is worth being exact about what the conflict does and does not imply. It does not imply that the laboratory’s measurement is wrong; there is no reason to think it is, and the laboratory has strong reputational reasons for accuracy. It does not imply that the vendor is dishonest. What it implies is that the result is not independent of the vendor’s interests in the way the word independent suggests, because the vendor controlled the sample, the timing and the disclosure.

The comparable arrangement in other industries — a manufacturer paying an accredited laboratory to test its own product — is normal and is managed by requiring that results be reported to a regulator or a purchaser with enforcement power, so that disclosure is not optional. Neither exists here. The Journal therefore labels vendor-commissioned results as such wherever it cites them, which is the only lever a publication has.

The blind duplicate exercise

Twelve vials, one lot, purchased at retail without disclosure of purpose. Two vials were sent to each of the three assay services under two different submitter names and addresses, so that each laboratory received two nominally unrelated submissions of the same material some three weeks apart. Six further vials were retained. Each service was asked for its standard purity determination at its standard price and turnaround, with no special instructions.

The design tests two distinct quantities that the trade conflates. Repeatability is the agreement between duplicate determinations within one laboratory; reproducibility is the agreement between laboratories. Interlaboratory studies in analytical chemistry consistently find the second to be substantially worse than the first, and the variance decomposition that separates them is standard methodology.³ The distinction between repeatability and intermediate precision is formalised in the validation guidance,⁴ and multi-site studies in adjacent fields have repeatedly found between-laboratory agreement on identical samples to be the harder problem.⁵

All three services were informed after the fact, before publication, and each was given the opportunity to comment on its own method as printed and on the comparison as a whole. All three responded. Two supplied additional method detail that has been incorporated. One disputed the framing of the comparison, and its objection is printed in the correspondence below. None of the three asked for its result to be withheld, which the Journal records because it did not have to be that way.

How a claim decays between the bench and the listing

The decay is worth tracing precisely, because at no step does anybody say anything false. The laboratory reports a purity figure for the sample as received, on a stated date, by a stated method, submitted by a named party. The vendor extracts the figure and the service name onto its own documentation, dropping the submitter and often the method. A reseller reproduces the vendor’s documentation, dropping the date. A listing summarises the whole chain as tested by a named service. A badge reduces it to verified.

Each step is a reasonable act of summarisation and the cumulative effect is a claim of a different kind from the one the laboratory made. A time-indexed measurement on one sample has become an atemporal property of a product line. The information was not concealed; it was compressed away by a chain of parties each of whom had a legitimate reason to shorten the message.

This is why the Journal reports the provenance of every third-party figure it cites — which service, which lot, which date, which submitter — and treats a figure lacking any of those as uncitable. It makes our coverage sparser than the market’s. It also means that a number appearing in this publication can be traced to a document, which is the only property that distinguishes reporting from repetition.

Our own position, stated in full

Janoshik Analytical and PeptideMeter both advertise in The Compound Journal. Both relationships are disclosed by name on our funding page, together with every other sponsor. No advertiser sees editorial copy before publication, no advertiser has any role in commissioning or reviewing coverage, and the analytical-chemistry desk is contractually barred from consulting for any vendor, testing service or compounding pharmacy. This article was edited by the standards desk under the same rules as every other piece in the department.

The Journal also pays these services. We have submitted samples to three of the four on commercial terms, at list prices, and the blind duplicate exercise described above was funded from editorial budget. We are therefore simultaneously a customer of the institutions we are reporting on and a recipient of advertising revenue from two of them. Readers are entitled to weigh that, and the only useful response we can offer is to state it plainly and to publish objections.

Our position on the substance is unchanged by any of it. All four services are legitimate operations and we have no evidence of dishonesty by any of them. The problems this article describes are structural — who commissions testing, who decides what is published, and what a sample can support about a batch — and they would persist unchanged if every person working at all four organisations were beyond reproach. Correspondence to standards@compoundjournal.com.

What the Journal thinks these services are worth

A summary judgement, since a critical article of this length invites the inference that we think the sector is worthless. We do not. Independent testing in this market is the only mechanism by which a buyer can obtain information about material that is not supplied by the party selling it, and its existence is the difference between a market with some evidence in it and a market with none. Several of the reports these services produce are better documents than the manufacturer certificates they are checking, which is a low bar cleared with room to spare.

The three criticisms we would press are narrow. A sample is not a batch, and the trade cites samples as batches. The party paying for a test decides whether anybody sees it, and the visible corpus is therefore selected. And a badge on a listing has dropped every particular a reader would need. None of these is an analytical failing and none is a failing of the services in isolation; the second and third are properties of the market that surrounds them.

What we would tell a reader is this. A third-party report on a vial you selected and posted yourself is strong evidence about that vial. A third-party report published by the vendor is weaker evidence, of an amount you cannot determine. A badge is not evidence. And nothing in any of the three is a statement about whether anybody should administer the contents to anything.

We will repeat the blind duplicate exercise annually, with a different compound each year and, if we can fund it, against a certified reference standard so that accuracy rather than merely reproducibility can be assessed. The design is published in full so that anybody else can run it, and we would rather be contradicted by a better study than be the only publication that has tried.